Exploring Effective Stimulus Encoding via Vision System Modeling for Visual Prostheses
Authors: Chuanqing Wang
Di Wu
Chaoming Fang
Jie Yang
Mohamad Sawan
Venue: ICLR 2024 poster (2024)
Keywords: Visual prostheses
Spiking recurrent neural network
Dynamic vision sensor
Biological phosphene model
Primary area: applications to neuroscience & cognitive science

Abstract: Visual prostheses are potential devices to restore vision for blind people, which highly depends on the quality of stimulation patterns of the implanted electrode array. However, existing processing frameworks prioritize the generation of stimulation while disregarding the potential impact of restoration effects and fail to assess the quality of the generated stimulation properly. In this paper, we propose for the first time an end-to-end visual prosthesis framework (StimuSEE) that generates stimulation patterns with proper quality verification using V1 neuron spike patterns as supervision. StimuSEE consists of a retinal network to predict the stimulation pattern, a phosphene model, and a primary vision system network (PVS-net) to simulate the signal processing from the retina to the visual cortex and predict the firing rate of V1 neurons. Experimental results show that the predicted stimulation shares similar patterns to the original scenes, whose different stimulus amplitudes contribute to a similar firing rate with normal cells. Numerically, the predicted firing rate and the recorded response of normal neurons achieve a Pearson correlation coefficient of 0.78.

Published as a conference paper at ICLR 2024
EXPLORING EFFECTIVE STIMULUS ENCODING VIA VI- SION SYSTEM MODELING FOR VISUAL PROSTHESES
Chuanqing Wang'; ; Di Wu'; Chaoming Fang', Jie Yang'! Mohamad Sawan
1 CenBRAIN Neurotech, School of engineering, Westlake University, Hangzhou.
2 Westlake Institute for Optoelectronics, Fuyang, Hangzhou, 311421, China wingchuanging, wudi, yangjie, sawan @westalkesseeeeeeeeee edu.kk..eedddc.
ABSTRACT
Visual prostheses are potential devices to restore vision for blind people. which highly depends on the quality of stimulation patterns of the implanted electrode array. However, existing processing frameworks prioritize the generation of stim- ulation while disregarding the potential impact of restoration effects and fail to assess the quality the generated stimulation properly. In this paper, we propose for the first time an end-to-end visual prosthesis framework (StimuSEE) that generates stimulation patterns with proper quality verification using VI neuron spike patterns supervision. StimuSEE consists of a retinal network to predict the stimulation pattern, a phosphene model, and a primary vision system network (PVS-net) to simulate the signal processing from the retina the visual cortex and predict the firing rate of V1 neurons. Experimental results show that the predicted stimulation shares similar patterns to the original scenes, whose different stimulus amplitudes contribute to a similar firing rate with normal cells. Numerically, the predicted firing rate and the recorded response of normal neurons achieve Pearson correlation coefficient of 0.78.
INTRODUCTION
Visual prostheses are potential devices to restore vision for patients with severe vision disorders, such as age-related macular degeneration (AMD) and retinitis pigmentosa (RP) (Busskamp et al., 2010; Sehajic-Kapetanovii et al., 2022: Prévot al., 2020). These prostheses utilize image sensor to record external scenes and processing framework to predict stimulus for the electrode array (Soltan al.. 2018). The implanted electrode array stimulates ganglion cells with predicted stimulus to restore visual perception in the visual cortex (Berry et al., 2017; Sahel et al., 2021). In the whole workflow, the processing framework to obtain suitable stimulation patterns is the most critical factor in affecting visual prostheses' restoration effect, attracting many research endeavors (Turner et al., 2019). The performance of stimulation patterns predicted by the processing framework is highly related to the choice of the supervised signal and processing model (Tumee et al., 2019).
One strand of the processing framework to construct stimulation patterns utilized a region-based detection method to process the recorded external frame signals (Figure 1(a)). The region of interest of the input frame was converted to grayscale and directly mapped to the electrode location for stimulation without any supervised signal (Guo et al., 2018; Wang et al., 2023). Due to the unsupervised nature of these methods, the quality of the stimulation patterns can not be verified before deploying this framework in implanted devices for animal experiments.
Another strand of research adopted neural networks to predict stimulation patterns and a biological phosphene model simulate phosphene and generate decoded images (Figure 1(b)) (Bureu et al., 2022: Granley et al., 2022; Wang et al., 2022a). The phosphene represents visual perception without light entering the eyes. Then, the original image is used as a supervised signal to update the stimulation pattern, making the output decoded image similar to the original one. However, we argue that the visual perception in the visual cortex neural spikes but not images. Consequently, this processing
'These authors contribute equally to this work.
1 *oorrsponning author.

---

Published as a conference paper at ICLR 2024
(a) Region detection-based processing framework
Frame signal recording
region of interest
Grayscale
Mepping electrode arry
Directly stimulate ganglion cells
(b) Hybrid Neural Autoencoders
T
Puise with
Natural scene
H Frequency Stimulus
Amplltude
Bnggeness
Frame signal recording
Encoder: deep leamnin model
Decoder: phosphene model of retinal prosthesis
Predicted percept
Figure Illustration of existing processing frameworks. (a) The external scene is recorded with series of frame signals in region detection-based processing framework. Such a framework utilizes region-based detection detect region of interest and simplifies with grayscale and pixelation techniques to obtain stimulation patterns. These patterns are mapped into electrode array to stimulate ganglion cells directly for vision rehabilitation. (b) Hybrid neural autoencoders (usually deep neural networks) take frame signals input and predict suitable stimuli for retinal prosthesis. Another decoding model used to simulate phosphene and generate the decoding image. The original image is adopted supervision signal during autoencoder training, whose performance evaluated by comparing the similarity between the decoded image and the original image.
framework only finds suitable stimulation patterns that favor original image reconstruction rather than finding suitable patterns for better vision perception in the cortex. Thus, this stimulus with low biological similarity restricts the vision restoration effect of visual prostheses low-level (Montazeri et al., 2019).
To remedy the inadequacy of existing frameworks. the proposed framework (StimuSEE) generates stimulation patterns that trigger the same spike responses in the visual cortex normal visual perception when given the same scene information. As demonstrated in Fig. 2, StimuSEE combines retinal network, phosphene model, and primary visual system network (PVS-net). Retinal prostheses require the processing framework to be energy efficient for prolonged use. To ensure energy efficiency, the retina network adopts spiking recurrent neural network (SRNN) to predict stimulation patterns for electrode arrays. Compared to the traditional CNN models, SRNNs have a low spike firing rate during their calculation process without the need for floating point multiplication. The phosphene model and Pett are designed specifically to validate the effectiveness predicted stimulation patterns. The phosphene model biological simulation model accurately simulate phosphene in the retina after stimulation by the implanted prosthesis. PVS-net mimics the function from the retina the visual cortex and predicts the firing rate of primary visual cortex (VI) neurons. The predicted values of StimuSEE fit well with that of normal neurons, whose performance achieves 0.78, evaluated by the Pearson correlation coefficient. We hope that the resemblance shown between the predicted stimulation patterns and neural spikes with normal neurons could contribute to more practical visual prosthesis development.
In short, we summarize the main contributions of our work the following:
To the best of our knowledge, the first to propose end-to-end visual prosthesis framework (StimuSEE) that generates stimulation patterns with proper quality verification using VI neuron spike patterns.
We designed a phosphene model and PVS-net to simulate the visual signal process from the retina to the primary visual cortex validate the effectiveness of predicted stimulation patterns.
Beside quality verification of stimuli, StimuSEE is designed in an energy-efficient manner aiming to bring truly practical visual prostheses into reality.
2

---

Published as a conference paper at ICLR 2024
Frequanccy
Spkke trsins from DVS
Predicting with SRNN model
Stimulus ampiltude
Hyperparamder from human experieent
Simulated phosphene
Prodictig with PVS net
Firing rate of V1 neurons
for electroces
model trainabie
PVS-net trainable
Pusso width
Amplitude
1889
Read
Firing
Spike layer
Spiike layer
Froquency
Brighines Threshold
Size
Conv 1.09
MP LIF
326:11 x11
32611 x 11
Spherical Coordinate traneforatioon
Simulsled shosphere in the rotina
Recurrent
Conv
Convolution layer
Leaky intagrato and fire neuron
MP LIF
LIF nouron
Membrane potental
PVSinnt
systerm nelwork
Primary visual
The proposed optimization framework mainly comprises the retina network, phosphene model, and PVS-net. The retina network utilizes dynamic vision sensors (DVS) to encode natural scenes into sparse spike trains. Then, the SRNN model designed to predict the stimulus amplitude of the electrode array with input spikes. This model consists of two spike layers and recurrent block. The phosphene model integrating predicted stimulus amplitude and hyperparameters from human experiments simulate phosphene in the retina. This model fully considers the impact of spherical coordinate transformation, phosphene brightness, threshold, and size effects. The PVS-net can mimic the function of the visual system, especially lateral geniculate nucleus (LGN) and primary visual cortex, with the input of simulated phosphene. The model output is the firing rate of multiple VI cells compared with the recorded response of normal cells to evaluate stimulus performance.
Figure 2:
2 RELATED WORKS
Region-based detection. The initial processing framework for visual prostheses adopted region- based detection to find region of interest first and simplify obtain stimulation patterns (Li et al., 2018; Itti et al., 1998). A pixelated model was proposed to detect the interest region by selecting caattiiongrabbing local features, then pixelated this region stimulation pattern (Li al., 2005; Boyle et al., 2008). Later, a self-adaptive FCM clustering method was proposed to identify all clusters containing objects and choose the one closest the center as a region of interest (ROI). This ROI was transformed into grayscale image, and edge detection was used extract edges from background information (Wang al., 2016; Parikh et al., 2013). The final pattern was constructed with grayscale foreground region and background edge information. This method preserves more features for recognition compared with other region-basse detection methods. In addition, several studies focused on constructing stimulation patterns from dynamic scenes (Guo al., 2019; Zhou et al., 2003). combined processing strategy, including edge detection and zooming techniques, was proposed to segment moving objects from dynamic scenes (Guo al., 2018; Wang al., 2014; Barnich & Van Droogenbroeck, 2011).
Hybrid neural autoencoder with phosphene model. Apart from predicting stimulation patterns, several studies focused on constructing biological simulation model simulate phosphene and verify the quality of generate patterns (van der Grinten et al.. 2022). A stimulation vector mapping method proposed to convert stimulation patterns into light spots in the electrode location (van Steveninck et al., 2022). Then, a biologically plausible simulation model was designed to consider several biological features, including phosphene brightness and size (Granley & Beyeler, 2021 1). Recently, hybrid neural autoencoder was proposed predict the stimulation patterns with the CNN model and generate simulated phosphene with the above phosphene simulation model (Granley et al., 2022). This phosphene was decoded to original images to verify its performance. Another processing framework utilized a similar method to generate simulated phosphene but added a reinforcement learning agent to improve the performance in decoding original images (Burcu et al., 2022).
3

---

Published as a conference paper at ICLR 2024
3 STIMUSEE
In this study, we employ the neural firing patterns of V1 neurons to assess the efficacy of stimulation. Our fundamental premise is straightforward: If the neural spikes collected in VI neurons of visually impaired individuals following stimulation resemble the spikes recorded from VI neurons in sighted individuals, it suggests an effective stimulation for vision restoration. StimuSEE comprises retina network, a phosphene model, and a PVS-net (Figure 2). The retina network predicts the stimulation patterns that are applied to the stimulation array of the implanted prosthesis. The retina network adopts the SRNN model and takes input from a dynamic vision sensor to mimic the function of the human retina for high-quality stimulus while ensuring low power consumption. The phosphene model fully considers the coordinate system transition and phosphene brightness, size, and threshold effect to simulate phosphene at the retina accurately. Finally, we adopt PVS-net to mimic the function from the retina to the primary visual cortex (V1) and predict the firing rate of multiple V1 neurons (Lindsey et al., 2019). The predicted neurons' firing rate from the PeS-net is then compared with the recorded response of normal V1 neurons to evaluate the performance of the generated stimulation.
3.1 RETINA NETWORK
Inspired by the paske-based processing characteristic of the human retina, we proposed a retina network that includes a dynamic vision sensor (DVS) and SRNN model to predict stimulation patterns with spike signals (Figure 2). Recent studies demonstrated that spike-based perception and processing methods with DVS and SRNN models achieve better prediction performance with much less energy consumption than frame-basee sensor and CNN model (Wang et al., 2022a). The dynamic vision sensor could encode scenes into sparse spike trains. The SRNN model comprises two spike layers and one recurrent block, suitable for processing the input spatial-temporal signal.
We utilize a dynamic vision sensor and a synchronous recording system to encode scene information into sparse spike trains. This vision sensor only encodes the moving part of scenes into spike trains, which are of high temporal resolution and low data volume. Then, the recording system receives and stores these spike trains synchronously through TCP/IP protocol. Before constructing an event dataset for the proposed SRNN model, it is necessary to denoise the recorded spike trains to improve their quality (Ledelo-Bardallo al., 201 Finally, due to the large spatial resolution of the vision sensor, the input resolution of spike trains is shrunk to 120 x 120 for the SRNN model.
The proposed SRNN model is designed to predict stimulation amplitude with the input spike trains (Figure 2). This model utilizes spike layers and recurrent block to achieve spatial- feature extraction in low power consumption condition. Both spike layers and recurrent blocks use the convolution layer to extract spatial features. In addition, the temporal feature extraction ability depends on the dynamic characteristics of the Leaky Integrate and Fire (LIIF) and Membrane Potential LIF (MP LIF) neurons. For LIF neurons, the membrane potential update value is obtained by input spikes multiplied by corresponding weights (Figure 3(a)). Then, the new membrane potential must be compared with the threshold to determine whether to fire spike. The neuron will fire spike when membrane potential exceeds the threshold (Figure 3(b)). Thus, LIF neurons could memorize the temporal feature with updated membrane potential.
For MP_ LI_ neuron, the calculation process and membrane potential updated details of MP_LIP neuron are depicted in Figure 3(c, d). The dynamic characteristic of MP_LIF can be written as:
V/2 -1 -1- V- v 1
where the membrane time constant 7 controls the balance between remembering X, and forgetting Vi-1, which is similar to the function of the recurrent layer (Zhu al., 2022). Thus, it can be considered a simple version of the recurrent layer. The SRNN model with spatial-tempont feature extraction ability suitable for processing input dynamic spike signals and can predict the stimulation amplitude for retinal prostheses.
3.2 PHOSPHENE MODEL
The phosphene model is designed to simulate phosphene in the retina according to the stimulus of the electrode array, which includes stimulation amplitude, duration, and frequency (Figure. 2). The
4

---

Published as a conference paper at ICLR 2024
d
Membrane potential
Mambrane potential
Vtt
V12
V13
Vih
Σ
time
Leaky integrate and fire (LIF) nouron
Integrate leaky and fire (LIF)
Membrane potential LIF nouron
Membrame polential neuron
Figure 3: The dynamic features of LIF and MP_LIF neurons. (a, b) The updated membrane potential obtained by adding corresponding weight of input spikes. This potential added to existing membrane potential of LIF neuron, which is used to compared with threshold to decide whether to fire a spike. (c, d) The MP LIF neuron has same membrane potential update mechanism. However, each time step, outputs the real-valued membrane potential rather than spikes.
amplitude comes from the results of the above SRNN model, and the pulse width and frequency are hyperparameters summarized from human experiment (van der Grinten et al., 2022). Phosphene means visual perception without light entering the eyes. The feature of phosphene is recorded in patients with retinal implants. After the stimulation of each electrode, the patients will report the location, size, and brightness of the phosphene. These features suitable parameters for the phosphene model. This model first maps the stimulation parameters with the location of the electrode array. Then, it converts the location in one hemisphere to the plane coordinate system and calculates the phosphene brightness and size with the stimulus parameters. Finally, whether generate a phosphene is also decided by the threshold of retinal ganglion cells.
summarized a dataset obtain
Coordinate system transition The implanted electrode array is located at the surface of the retina hemisphere, which utilizes spherical coordinates to represent the location information. For calculating the specific location of phosphene, it is necessary to convert Spherical coordinates to Cartesian coordinates. We adopt reverse wedge-dipole visuotopic model to achieve this conversion (Polimeni et al., 2006).
ab(eab b
== A
where z and w are the locations of the visual field and one hemisphere. a scaling factor that scales the mapping to realistic proportions in retina distance, and and b are parameters that control the singularities of the dipole model.
Phosphene brightness, size, and threshold The phosphene brightness is affected by various factors, including the stimulation parameters, the distance to the stimulation electrode, and the distance the soma along the curved axon. To calculate the effective stimulation intensity, the remaining current amplitude is multiplied by the frequency and duration.
Lefs max(0, (Istim Iseak) Pw f),
(3)
where Leekk Pw. and f are leak current, pulse width, and frequency. The phosphene is determined by the effective stimulation intensity:
Foright =f0 Less
if 2 lels <1
where 17 is the threshold to control the intensity. The phosphene size Faise is only affecceed by the stimulation amplitude but not with duration or frequency (van der Grinten et al., 2022). Then, Fstreak modeling phosphene elongation is function of time (Granley & Beyeler, 2021). More details are summarized as follows:
Esize Lstim + b1,
Fstreak = -az!" bz,
where dj, by a2 by, À are hyperparameters to modify the function to fit the recorded users' response. The final phosphene brightness is contributed by three parts: efficient stimulation amplitude (Fbrighe), the distance to the electrode (radial decay rate, p), and the distance to the soma (axonal decay rate, A).
{-L de
d 25
Brigheness max Σ
Foright exp
aεA
2p²
eEB
5

---

Published as a conference paper at ICLR 2024
where A is the cells' axon trajectory, E is the set of electrodes, de is the path to the electrode, and d is the path length along the axon trajectory (Jansonius et al., 2009). In sum, the above phosphene model is designed to simulate phosphene in the retina after stimulated by retinal prosthesis, taking coordinate system transition and threshold function into consideration compared with other phosphene models, making it more biologically plausible (Granley et al., 2022).
Parameter selection The hyperparameters of the phosphene model utilized in this work are based on various experimental and modeling results from different components of corresponding research: In Eq. 2, we on data of an human experiment from (Horton & Hoyt, 1991). In Eq. 3, Eq. 4, Pw = 3.3ms,
0.75, b 120, k = 1.73, based on fitting method of (Polimeni et al., 2006)
= 20H,, threshold 0.3m.W/ 33mm are obtained from the setting and human experimental results with retinal prosthesis (Granley & Beyeler, 2021). We use Ieak = 23.9puA based on a data fitting from (van der Grinten et al., 2022). In Eq. 5, Eq. 6, @1 02. b, and b2. adopted from retinal phosphene model (Granley et al., 2022). In Eq. 7, da, des P. and y was obtained from a human retina model (Jansonius et al., 2009).
Surrogate method for phosphene model The phosphene model is a complex function during simulating phosphenes, affected by many parameters and experiment setups. However, in this proposed processing framework, reduced the entire model to function of the input stimulus and carefully determined the appropriate values for these parameters. In addition, the threshold function that decides whether to generate phosphene is not differentiable, hindering the backpropagation process. Thus, we adopt adjusted ReLU function f(x) In(1 c(*-))) replace threshold function to make it differentiable in the training, where r7 is the threshold in Eq. 4.
3.3 PRIMARY VISUAL SYSTEM NETWORK
The primary visual system network (PVS-net) adopted to predict the firing rate of VI neurons with the simulated phosphene. This network comes from a unified theory of early visual representations from the retina to cortex (Lindsey et al., 2019; Tanaka et al., 2019). This theory utilizes a retina-net and vision system network to explore visual information processing features and find biological similarities between the proposed network and visual processing features of human beings. Thus, we adopt PVS-net to replace the function of the lateral geniculate nucleus and primary visual cortex. We utilize an online trainable feature bank to capture semantic aware neural responses upon different visual stimulation. In particular, the feature bank contains 50 embeddings 150 with the same shape as the output x of the feature extraction module of the Pet- During the fitting process of the spike number of V1 neurons, we first replace with the closest embedding eg the feature bank:
R(z) 2 Ej, where l argmin, || - -eyll2-
(8)
where R(-) denotes the replacement operation. We then apply a read-out layer to predict the firing rate of VI neurons:
f(x) = ReLU FC(R(x)))
where (xx) FC, ReLU means the firing rate of V1 neurons, the fully connected layer, and the activation function (Cadena et al., 2019).
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Dataset. In this paper, we validate the proposed StimuSEE framework with the DVS-VI dataset (Wang et al., 2022b; Cadena al., 2019). The dataset comprises 7250 visual stimuli with corre- sponding VI responses. The original visual stimuli are selected from the Imagenet and transformed into sparse spike trains by a DVS sensor. We also designed a synchronous recording system to record these spike signals in high quality. More details about this recording system and the quality of the spike signal are summarized in the appendix. The V1 responses were recorded from macaque monkeys with 32-channel microelectrode array when displaying the above-selected images to these monkeys. Because this experiment was conducted for four sessions, each stimulus has four different V1 responses. The average value of these responses is used labels to train the proposed framework.

---

Published as a conference paper at ICLR 2024
Extemal Spike signals
Stimulus amplitude
Simulated phosphane
Extemal
Spike signals
Stmuuuus amplitude
Simulated phosphene
Figure 4: We randomly selected six external scenes from the test dataset to display the signal- processing process of StimuSEE. The external scene is converted to spike trains by DVS sensor, and then the spike train within one-time window is constructed to binary image to ensure its quality. The constructed binary image demonstrates the spike signals contain fruitful features, including edge and texture information. The predicted stimulation amplitude by the SRNN model also contains scene features, whose value contributes similar firing rate that of normal neurons. The simulated phosphenes by a biological phosphene model will be transferred to the visual cortex for perception.
70% of the spike trains from the DVS sensor and corresponding recorded VI neural responses were used for training. Another 10% was used to validate the model during training, and the remaining 20% were used to test the final performance.
The external scenes are recorded to sparse spike trains and then recon- structed into binary images to validate their quality. As is shown in Figure 4, the recorded spike signals contain fruitful features, including edge and texture information. Then, the predicted stimulus amplitude of the SRNN model has similar shape with constructed binary image of spike signals but has quite different intensity. During the animal experiment, the stimulus amplitude is an impor- tant factor affecting neurons' firing rate. In this framework, the stimulus amplitude is trained with supervised signals from the neural response of VI cells, which makes the stimulation patterns more effective for vision restoration. The simulated phosphenes are obtained by the proposed biological simulation model, which is the input signal for PVS-net to fit the neural response of V1 neurons.
Data flow of StimuSEF.
Loss function. In this study, adopt the Mean absolute emror (MAE) loss function to minimize the difference between the predicted firing rate of VI neurons against the ground truth. We also applied an L2 penalty to the Laplacian of the weights encourage spatial locality and impose regular changes in space.
L Smoothnese
E(w,+ L),
-1
4
-1
(10)
-1
Thus, the total loss summarized as follows:
Liotal = LMAE + LSmoothness
(11)
Training details. The phosphene model is biological model whose parameter is adjusted based on experimental data and kept fixed during training. The PVS-net is pre-trained with the CIFAR-10 dataset. The parameters of the PVS-net and the feature bank are updated with stopgradient operation. The StimuSEE trained to fit the responses of the 166 neurons with the spike trains from the DVS sensor input. The framework is implemented with Pytorch on NVIDIA RTX 3060. We use Adam optimizer with base learning rate of le 3, 31. B2 = 0.9, 0.999. and a weight decay of 0.01 to minimize the joint loss.

---

Published as a conference paper at ICLR 2024
Dite Prediction
Data
Number of spikes 4-
Number of takle Prediction
10 Neuron
10
20 Images
Figure 5: The performance of StimuSEE to fit the response of VI neurons in the test set. (a) In the spatial domain, 20 neurons were randomly selected from 166 neurons to show their performance. The data represented the recorded firing rate of several V1 neurons when displaying the image to a monkey. The prediction in this figure means the predicted spike number of the StimuSEE framework. The predicted spike number of V1 neurons in one scene can fit well with the recorded response of normal cells. (b) In the temporal domain, the front 20 image prediction of StimuSEE in one selected neuron is compared with that of a normal cell. The spike train in the red rectangle was generated according the corresponding firing rate of one neuron. This spike train's time distribution and spike number followed the Poisson distribution. In addition, this generated process is repeated ten times to avoid randomness, as shown in the yellow rectangle. The spike train of the StimuSEE framework adopts the same generated method. The comparison between the firing rate of recorded data and predicted values was depicted in the bottom column. The comparison in spike train and spike number proved the predicted results of StimuSEE could fit well with the recorded data in the temporal domain.
4.2 PERFORMANCE EVALUATION OF STIMUSEE
Fitting the response of primary visual cortex. The performance of stimulation patterns and simulated phosphenes are verified by fitting the spike response of 166 VI neurons. We first measure the performance by calculating the difference between predicted values and recorded spike number of 166 neurons all test images. The percentage of difference between [-0.5, 0.5) achieves 0.54 that represents good fitting performance in all test dataset. Then, we measure the performance from both the spatial and temporal domains. The performance in the spatial domain is measured by the similarity between the predicted firing rate of 166 neurons to one scene and the recorded values of V1 cells. The similarity between two values is measured by Pearson correlation coefficients (PCC). The percentage of PCC in all test scenes that is above 0.75 exceeds 80%. In addition, select 20 neurons' responses to one scene to demonstrate its performance, where it has high similarity between predicted value and recorded response, including cells with high and low spike firing rates (Figure 5(a)).
The temporal domain performance is measured by the similarity between the single neuron's 's response to various external scenes and corresponding predicted values. The proportion of Pearson correlation coefficients (PCC) between these two values that exceeds 0.75 is approximately 0.69. To better illustrate its similarity, the generated spike trains for 20 scene images are depicted in the first column, as shown in Figure 5(b). These spike trains were generated based on the recorded firing rate of randomly selected V1 neuron. The spike train in the red rectangle displays the spike number and time distribution in 660 ms, corresponding to 20 time steps. This generated process follows Poisson distribution, repeated ten times to avoid randomness. The spike train of the SRNN model adopts the same method. The comparison results show that the spike train of the SRNN model can fit well with the recorded response. Then, the bottom column compares the spikes between those values. The comparison results prove that the predicted values can fit well with the number of spikes of single cell for various scenes. In addition, the performance of fitting results achieves 0.78 in the test dataset evaluated by the Pearson correlation coefficient.
Comparison with State-of-the-art works. The proposed StimuSEE verifies the quality of stim- ulation patterns by comparing the predicted firing rate V1 neurons with that normal neurons. Other existing works either neglect the distortion effects, which are modeled by Phosphene model,
8

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of performance between proposed StimuSEE and other prediction models
Reference
(Yan et al., 2020)
(Cadena al., 2019)
(Wang al., 2022b)
This work
Processing
CNN
CNN
SNN
SRNN
model
Validate
BSM & PVS-net
N/A
N/A
N/A
component
Target cells
Ganglion cells
VI cells
VI cells
V1 cells
PCC
0.7
0.72
0.65
0.78
BSM Biological simulation model;
Table 2: Comparison of power consumption between SRNN and CNN model with similar architecture
First layer
Second layer
Recurrent layer
Total
Model
Firing rate
Power
Firing rate
Power
Power
SFR
Power
SRNN
%
11 JJ
9.97%
11.65 JJ
0.13 PJ
7.29%
22.78 JJ
CNN
N/A
59 JJ
N/A
59.6 jJ
N/A
N/A
118.6 JJ
or verify stimulation qualities by image reconstruction quality. We argue that these approaches lack feasibility for real-world retinal prostheses. Moreover, StimuSEE predicts the stimulation patterns with the SRNN model, whose spike-based processing characteristic makes it energy-efficienn efficient and more suitable for retinal prosthesis. See Table A2 for a detailed comparison of StinuSEE and existing works.
Although the primary contribution of this work is not to design a framework that boosts the per- formance of predicting VI cells firing rate, we still compare with some works that only focus on predicting the neural response of ganglion cells or VI cells for reference (Table 1). These frameworks achieve 0.65-0.72 with the evaluation method by Pearson correlation coefficient. However, in our framework, we utilize the SRNN model to obtain stimulation patterns, the phosphene model to simulate phosphene the retina, PVS-net to fit the neural response. still achieves 0.78 in the final fitting tasks, which proves that this framework can obtain more effective stimulation patterns and has a high potential improve the restoration effect for blind people.
Power Consumption Analysis of SRNN model. We evaluate the energy efficiency of the proposed SRNN model because only this model needs to be deployed into an implanted prosthesis for prediction. As shown in Table 2, the spike firing rates (SFR) of the first and second spike layers are 4.61 % and 9.97 %, respectively. The overall SFR is 7.29 %. This low firing rate characteristic can significantly decrease the amount of calculation and power consumption. We compare the power consumption of the SRNN model with a CNN model of similar architecture where the power consumption of multiplication and addition with 32-bit is 3.7 pJ and 0.9 pJ, respectively (Horowitz, 2014). From Table 2, we see that for the SRNN model, the energy of the first, second, and recurrent layers are 50.1 J.J, 749 J.J, and 51.9 J.J. The total energy of the SRNN model in one prediction is 851 pJ, around five times lower than the CNN model, which consumes 3.982 m.J in one prediction.
5 CONCLUSION
In this study, we propose for the first time an end-to-end stimulation patterns optimization framework (StimuSEE) for retinal prostheses. StimuSEE consists of a retina network, a phosphene model, and a PVS-net to predict stimulation patterns and validate its effectiveness. Besides, adopts VI neurons" spike response supervision during training, which provides a better optimization direction to obtain effective stimulation patterns for vision restoration. The predicted stimulation patterns generate firing rate similar to V1 cells, whose performance evaluated by Pearson correlation coefficient achieves 0.78. In sum, the proposed StimuSEE is promising processing framework to obtain effective stimulation patterns to empower visual prostheses for better vision restoration.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENTS
This work was supported by the ST12030-Major Project (Grant No. 2022ZD0208805) and Key Project of Westlake Institute for Optoelectronics (Grant No. 2023GD004).
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Olivier Barnich and Marc Van Droogenbroeck. Vibe: A universal background subtraction algorithm for video sequences. IEEE Transactions on Image Processing, 20(6):1709-172 2011. doi: 10. 10.11009112200022011111.
Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, 364(64439:caaa44336 2019.
Michael Beauchamp, Denise Oswalt, Ping Sun, Brett L Foster, John F Magnotti, Soroush Niketeghad, Nader Pouratian, William H Bosking, and Daniel Yoshor. Dynamic stimulation of visual cortex produces form vision in sighted and blind humans. Cell, 181(4):774-783, 2020.
Michael H Berry, Amy Holt, Joshua Levitz, Johannes Broichhagen, Benjamin M Gaub, Meike Visel, Cherise Stanley, Krishan Aghi, Yang Joon Kim, Kevin Cao, et al. Restoration of patterned vision with an engineered photoactivatable g protein-coupled receptor. Nature communications, 8(1): 1-12, 2017.
Justin Boyle, Anthony Maeder, and Wageeh W Boles. Region-offinterest processing for electronic visual prostheses. Journal of Electronic Imaging, 17(1):01300,, 2008.
Burcu, Bodo Rueckauer, Nasir Ahmad, Jaap de Ruyter van Steveninck, Umut Güçlü, and Marcel van Gerven. Optimization of neuroprosthetic vision via end-to-end deep reinforcement learning. bioRxiv, pp. 2022-02 02, 2022.
Volker Busskamp, Jens Duebel, David Balya, Mathias Fradot, Tim James Viney, Sandra Siegert, Anna Groner, Erik Cabuy, Valérie Forster, Mathias Seeliger, et al. Genetic reactivation of cone photoreceptors restores visual responses in retinitis pigmentosa. science, 329(5990):413-417, 2010.
Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLoS computational biology, 15(4):e1006897, 2019.
Jasmina Cchajic-Kapetanovii Mandeep S Singh, Eberhart Zrenner, and Robert E MacLaren. Bio- engineering strategies for restoring vision. Nature biomedical engineering, pp. 1-18, 2022.
Jacob Granley and Michael Beyeler. A computational model of phosphene appearance for epiretinal prostheses. In 202/ 43rd Annual International Conference of the IEEE Engineering Medicine & Biology Society (EMBC), pp. 4477-4481. IEEE, 2021.
Jacob Granley, Lucas Relic, and Michael Beyeler. Hybrid neural autoencoders for stimulus encoding in visual and other sensory neuroprostheses. In Advances in Neural Information Processing Systems, 2022.
Fei Guo, Yuan Yang, and Yong Gao. Optimization of visual information presentation for visual prosthesis. International journal of biomedical imaging. 2018, 2018
Fei Guo, Yuan Yang, Yang Xiao, Yong Gao, and Ningmei Yu. Recognition of moving object in high dynamic scene for visual prosthesis. IEICE TRANSACTIONS on Information and Systems, 102(7): 1321-1331, 2019.
Mark Horowitz. 1.1 computing's energy problem (and what we can do about it). 2014 IEEE International Solid-State Circuits Conference Digest of Technicul Papers (ISSCC), pp. 10-14. IEEE, 2014.
Jonathan C Horton and William F Hoyt. The representation of the visual field in human striate cortex: a revision of the classic holmes map. Archives of ophthalmology, 109(6):816-88)) 6-44, 1991.
Laurent Itti, Christof Koch, and Emsst Niebur. A model of saliency- based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence. 20(11): 1254-1259, 1998.
11

---

Published as a conference paper at ICLR 2024
Nomdo M Jansonius, Jukka Nevalainen, Bettina Selig, LM Zangwill, PA Sample, WM Budde, JB Jonas, Wolf Alexander Lagrèze, PJ Airaksinen, Reinhard Vonthein, et al. mathematical description of nerve fiber bundle trajectories and their variability in the human retina. Vision research, 49(11) (977211577221,3, 2009.
Susanne Klauke, Michael Goertz, Stefan Rein, Dirk Hochl, Uwe Thomas, Reinhard Eckhorn, Frank Bremmer, and Thomas Wachtler. Stimulation with a wireless intraocular epiretinal implant elicits visual percepts in blind humans. Investigative ophthalmology & visual science, 52(1 (4(1):449444) 2011.
Juan Antonio Barddll,,, Teresa Perrano-Gotarredona and Bernaéé Linares-Barranco Barranco. 3.6 jis latency asynchronous frame-free event-driven cisumic-visionsennee IEEE Journal of Solid-Star State Circuits, (466)):443-14555 2011.
Heng Li, Xiaofan Su, Jing Wang, Han Kan, Tingting Han, Yajie Zeng, and Xinyu Chai. Image processing strategies based on saliency segmentation for object recognition under simulated prosthetic vision. Artificial intelligence in medicine, 84:64-78, 2018.
Ruonan Li, Xudong Zhang, and Guangshu Hu. A computational pixelization model based on selective attention for artificial visual prosthesis. In International Conference on Natural Computation, pp. 654-662. Springer, 2005.
Jack Lindsey, Samuel Ocko, Surya Ganguli, and Stephane Deny. A unified theory of early visual representations from retina cortex through anatomically constrained deep cnns. arXiv preprint arXiv:1901.00445 2019.
Keith Mathieson, James Loudin, Georges Goetz, Philip Huie, Lele Wang, Theodore I Kamins, Ludwig Galambos, Richard Smith, James Harris, Alexander Sher, et al. Photovoltaic retinal prosthesis with high pixel density. Nature photonics, (6):
Leila Montazeri, Nizar El Zarif, Stuart Trenholm, and Mohamad Sawan. Optogenetic stimulation for restoring vision to patients suffering from retinal degenerative diseases: current strategies and future directions. IEEE transactions on biomedical circuits and systems, 13(6):1792-1880) 2019.
Neeti Parikh, Laurent Itti, Mark Humayun, and James Weiland. Performance of visually guided tasks using simulated prosthetic vision and saliency-bar based cues. Journal of neural engineering, 10(2): 026017, 2013.
Jonathan R Polimeni, Mukund Balasubramanian, and Eric L Schwartz. Multi-area visuotopic map complexes in macaque striate and extra-striate cortex. Vision research, 46(20):3336-3359, 2006.
Carlos R Ponce, Will Xiao, Peter F Schade, Till S Hartmann, Gabriel Kreiman, and Margaret Livingstone. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 17994)999--1009 2019.
Paul-Henri Prévot, Kevin Gehere, Fabrice Arcizet, Himanshu Akolkar, Mina A Khoei, Kevin Blaize, Omar Oubari, Pierre Daye, Marion Lanoë, Manon Valet, et al. Behavioural responses to a photo- voltaic subretinal prosthesis implanted in non-human primates. Nature Biomedical Engineering, (2):172-180, 2020.
José-Alain Sahel, Elise Boulanger-Scemama. Joseph N Martel, Simona Degli Esposti, Alexandre Delaux, Jean-Baptiste de Saint Aubert, Caroline de Montleau, et al. Partial recovery of visual function in a blind patient after optogenetic therapy. Nature Medicine, 27(7): 1223-1222--229 2021.
Scemama, Chloé Pagot, Angelo Arleo, Francesco Galluppi,
Sekhar, A Jalligampala, Zrenner, and DL Rathbun. Correspondence between visual and electrical input filters of and off mouse retinal ganglion cells. Journal of neural engineering, 14(4): 046017, 2017.
Ahmed Soltan, John Martin Barrett, Pleun Maaskant, Niall Armstrong, Walid Al-Atabany, Lionel Chaudet, Mark Neil, Evelyne Sernagor, and Patrick Degenaar. head mounted device stimulator for optogenetic retinal prosthesis. Journal of neural engineering, 1566)065002,, 2018.
12

---

Published as a conference paper at ICLR 2024
Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Advances in neural information processing systems, 32, 2019.
Maxwell H Turner, Luis Gonzalo Sanchez Giraldo, Odelia Schwartz, and Fred Rieke. Stimulus-and goal-oriented frameworks for understanding natural vision. Nature neuroscience, 22(1):15-24, 2019.
Maureen Louisa van der Grinten, Jaap de Ruyter van Steveninck, Antonio Lozano, Laura Pijnacker, Bodo Rueckauer, Pieter Roelfsema, Marcel van Gerven, Richard van Wezel, Umut Guclu, and Yagmur Gucluturk. Biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses. bioRxiv, pp. 2022-12, 12, 2022.
Jaap de Ruyter van Steveninck, Umut Güşlüü Richard van Wezel, and Marcel van Gerven. End-to-end optimization of prosthetic vision. Journal of Vision, 22(2):20-20, 2022.
A Vanhoestenberghe and N Donaldson. Corrosion of silicon integrated circuits and lifetime predic- tions in implantable electronic devices. Journal of neural engineering, 10(3):031002, 2013.
Chuanqing Wang, Chaoming Fang, Yong Zou, Jie Yang, and Mohamad Sawan. Spikesee: energy-efficient dynamic scenes processing framework for retinal prostheses. arXiv preprint arXiv:2209.07898, 2022a.
Chuanqing Wang, Jie Yang, and Mohamad Sawan. Neurosee: A neuromorphic energy efficient processing framework for visual prostheses. IEEE Journal of Biomedical and Health Informatics, 2022b.
Chuanqing Wang, Chaoming Fang, Yong Zou, Jie Yang, and Mohamad Sawan. Artificial intelligence techniques for retinal prostheses: comprehensive review and future direction. Journal of Neural Engineeringg 2023.
Jing Wang. Yanyu Lu, Liujun Gu, Chuanqing Zhou, and Xinyu Chai. Moving object recognition under simulated prosthetic vision using background-subtracee wuw.rouuddd-uuttttoioooooooooo based image processing strategies. Information Sciences, 177-512-524, 2014.
Jing Wang, Heng Li, Weizhen Fu, Yao Chen, Liming Li, Qing Lyu, Tingting Han, and Xinyu Chai. Image processing strategies based on visual saliency model for object recognition under simulated prosthetic vision. Artificial organs, 40(1):94-100 2016.
Qi Yan, Yajing Zheng, Shanshan Jia, Yichen Zhang, Zhaofei Yu, Feng Chen, Yonghong Tian, Tiejun Huang, and Jian K Liu. Revealing fine structures of the retinal receptive field by deep-learning networks. JEEE transactions on cybernetics, 2020.
Dengyong Zhou, Jason Weston, Arthur Gretton, Olivier Bousquet, and Bernhard Schölkopf. Ranking on data manifolds, Advances in neural information processing systems, 16, 2003.
Lin Zhu, Xiao Wang. Yi Chang, Jianing Li, Tiejun Huang, and Yonghong Tian. Event-based video reconstruction via potential-assisted spiking neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2554-36044 2022.
13


Reviews:
[{"review_id": "0nGlXbG0gX", "reviewer": "Reviewer_QbGv", "summary": "The authors explored a better way of encoding the image stimuli, using spiking recurrent network, phosphene generation model, building in many details of early visual system and used a V1 encoding model. Then they optimize the electric stimuli generation parameters. constrained by V1 recording data. The model reached competitive results for modeling V1 data.", "questions": "### Questions\n\n- Not quite clear to me what “*the front 20 times prediction of*” mean in Figure 5 (b) caption. What are the x-axes “Times” in this plot? is it seconds?\n- Given the four repeats in your V1 data, could you calculate the noise ceiling / self consistency of V1 response itself? then you can kind of compare your obtained pearson correlation to that “ceiling”.\n- How do you compute the energy efficiency of Spiking RNN? also by counting the multiplication and addition in the simulated neuron firing process? —— if that’s the case won’t that depend on how do you discretize time?\n- For more data to train the model, [1] seems to be quite relevant.\n    \n    [1] Chen, X., Wang, F., Fernandez, E., & Roelfsema, P. R. (2020). Shape perception via a high-channel-count neuroprosthesis in monkey visual cortex. *Science*, *370*(6521), 1191-1196.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "### \n\n- The authors provided adequate background literature for evaluating this work.\n- The authors built in numerous biological relevant details to make this stimulation system works, which is laudable.\n- The primate V1 prediction results seems quite competitive.\n- The energy efficiency is one highlight for this method!", "weaknesses": "### Weakness\n\n- The method and experimental procedure is still a bit vague to me, i.e. I’m not sure how data and gradient flow through this system and how this system trains. Correct me if I’m wrong, during training you use the DVS → spiking RNN → stimulation model → phospene model → PVS-net V1 model and then fit all parameters with the primate V1 data? Some parameters of the phosphene model are constrained by human patient data, is that also transferred to fit primate V1 data?\n- Eq. 8 and Eq. 9, the notations are somewhat confusing, are capital `Index` and `index` the same thing? What does $\\cdot$ and $[]$ mean in the Eq. 9?.  I’m also not sure about the dimension of any of the variables… are they scalar or vector integer or what? Please clarify these in the paragraph following it.\n- The biological data validation figure seems a bit weak.\n    - Usually you’d like to plot the scatter plot between predicted spikes and actual spikes for all images and summarize the correlation for each cell. Figure 5 a) looks nice but lack the population summary.\n    - If the data presented Figure 5 b is real biological spike data from times, it seems overly sparse. Can you clarify the image stimulation onset and offset time schedule of it? —— if the no spike period is just gray screen stimulation, then the temporal accuracy is just defined by the stimuli time course, which doesn’t mean much.\n    - Further the temporal precision between the data and the prediction in Figure 5b seems **overly high.** With 10 repetitions, i.e. it can predict all the cases where spikes are missing? Maybe I misunderstood what data means in this case, is it the firing rate data for 166 V1 neurons, with some Poisson spike generator attached? How could the model know the differences between these different runs?\n    - Is Figure 5 training set or test set?\n    - Also not sure about the bottom panel in Figure 5 b), the x axes `Times` means different images?\n- The work is technical solid, but the contribution and impact seems limited to the neuroengineering domain, not sure the technique will fit or benefit the neurips community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gqtOu8ABwZ", "reviewer": "Reviewer_Fc3Q", "summary": "This paper proposes an end-to-end framework called StimuSEE for visual prosthetics. The proposed frameworks makes use of v1 neuron responses as feedback signal to train a retinal network to generate stimulation pattern. Their results show that with this framework, they could general meaningful simulation patterns that could predict v1 neurons with substantial performance.", "questions": "The paper only briefly mentions that V1 neuron response in blind animals are comparable to those in sighted animals, it is unclear how this approach directly transfers to blind animals. Does this framework end up with different learned model based on coverage of V1 neurons in recording? Could you use two sighted animal to estimate how transferrable the model is from one animal to the other? Also since the frameworks requires many pre-determined parameters for other non-trainable models, could the authors provide some insight on how sensitive is the SRNN network training with respect to different parameter setting among the other models? \n\nCould you elaborate on the purpose of the parameter index in equations (8)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The approach proposed in the paper is novel to my knowledge. The training pipeline is simplistic and clever. \nThe paper itself is well written, and thorough about providing experimental details for future replication.", "weaknesses": "Though the framework demonstrate reliable results in predicting V1 neurons, because of the different performance metric (neuronal responses vs. reconstruction error), it is hard to directly compare this method against other existing methods. One potential solutions is adding a decoder layer on top of the trained StimuSEE framework and see whether it indeed also facilitate reconstruction as well.  \n\nSince the StimuSEE frameworks consists of multiple models, the paper perhaps should make it more clear which of the models are trained with the error signals from predicting V1 responses. For example, in Figure 2, it could be helpful to use a bounding box with a different color or the SRNN model or an arrow signifying the direction of training feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XU7Xvq6AyS", "reviewer": "Reviewer_NiJW", "summary": "This paper introduces an end-to-end framework (StimuSEE) designed for visual prostheses through the integration of a retinal network, a phosphene model, and a primary visual system network (PVS-net). Specifically, this method generates stimulation patterns by using the corresponding V1 neuron spike patterns as supervision. The performance of this framework is impressive (Pearson correlation coefficient = 0.78 between the predicted and groundtruth firing rate).", "questions": "**Questions**:\n- It's unclear to me what is the novelty of the proposed phosphene model compared to Granley et al. (2022).\n- \"The training details\" section requires further elaboration to enhance clarity for readers and to ensure the work is reproducible by other researchers, for example:\n    - How is PVS-net pre-trained? Was the same dataset utilized both for the PVS-net pretraining and the end-to-end training?\n    - What is the temporal resolution of the label (neural firing rate)? How many time points per image?\n- Since the key innovation of this paper is using the V1 neural prediction instead of image reconstruction as the objective, a more detailed discussion and presentation of results in Fig.5 would be beneficial to convey the claims:\n    - In Fig 5a, the authors only showcased 20 neurons. It would be informative if the authors could provide quantitive results on the model's fit across all 166 neurons. Is Fig. 5b showing the results from a single neuron? If so, is it the best-performing one?\n    - What would be the chance-level performance for both Fig. 5a and Fig. 5b?\n    - Is Fig. 5a showing spike counts from 20 neurons for a single time point or single image? If yes, why not show the overall statistics for all images in the test dataset?\n    - Fig. 5b shows the neural spike train over time. What is the temporal resolution here? Also, why comparing the spike train randomly generated 10 times from the firing rate, instead of directly comparing the time series of the predicted and groundtruth firing rate?\n- How did you get the PCC in Table 2? Is it the correlation between the predicted and groundtruth firing rate along time? then averaged across all 166 neurons?\n\nMinor point:\n- Some fonts in equations (8) and (9) are not standard. Typically, functions or operators such as min, max, and ReLU are not italicized.\n- It will be more reader-friendly to explain some abbreviations (e.g., DVS, LIF) in the caption of Fig. 2.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, I think this paper showcases a notable contribution to the field, especially with its V1 neural encoding approach for visual prostheses. The writing is clear and well-organized. The methods and results are solid. The work has novelty and can lead to potential clinical applications, such as restoring vision for the blind.", "weaknesses": "Some details in methods and results need to be further elaborated in the main text. \nI am willing to raise the score if the authors could provide clarification to my questions listed below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ELMdadB4U", "reviewer": "Reviewer_JqK3", "summary": "This research proposes a new end-to-end architecture that can be used to generate an appropriate electrical stimulation pattern of visual prostheses based on an input natural image through a network of SRNNs, which can then be passed through a network of CNNs to induce the desired V1 neuronal activity. If this information flow can be successfully implemented, the network can be used to generate a corresponding electrical stimulation pattern based on the image. This would have great applications in patients with retinal dysfunction.\n\nOverall, the idea is great and I see great potential for this approahc in developing useful visual prostheses.", "questions": "1. What is the adjusted ReLU function for surrogate gradient?\n2. What are the differences between LIF and MP_LIF ? \n3. How are the parameters of the phosphene model selected? It says \"fitting the experimetnal data recorded from patients with retinal protheses\". I didn't see any fitting procedure or decription of the data.\n4. Table 2, The performance of this work is better than CNN approach (Cadena et al, 2009). But can you show the number of parameters of each approach?\n5. Table 3 shows the energy cost of SRNN and CNN with a similar achitechture. The advange of SRNN is not surprising given the intrisically low energy cost of SRNN. However, the energy cost of SRNN should be compared to that of a CNN with similar performance (such as, Cadena, 2009) rather than similar achitechture.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. End-to-end prediction of V1responses using visual prothesis is rare.\n2. Uses an SRNN model with low energy cost is used.\n3. The design of the overall framework is elegant.", "weaknesses": "The overall writing is good, but some technical details are not clear", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)