Natural image synthesis for the retina with variational information bottleneck representation
Authors: Babak Rahmani
Demetri Psaltis
Christophe Moser
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neuroscience
neural coding
generative model
variational autoencoders
neural latent models
information bottleneck

Abstract: In the early visual system, high dimensional natural stimuli are encoded into the trains of neuronal spikes that transmit the information to the brain to produce perception. However, is all the visual scene information required to explain the neuronal responses? In this work, we search for answers to this question by developing a joint model of the natural visual input and neuronal responses using the Information Bottleneck (IB) framework that can represent features of the input data into a few latent variables that play a role in the prediction of the outputs. The correlations between data samples acquired from published experiments on ex-vivo retinas are accounted for in the model by a Gaussian Process (GP) prior. The proposed IB-GP model performs competitively to the state-of-the-art feedforward convolutional networks in predicting spike responses to natural stimuli. Finally, the IB-GP model is used in a closed-loop iterative process to obtain reduced-complexity inputs that elicit responses as elicited by the original stimuli. We found three properties of the retina's IB-GP model. First, the reconstructed stimuli from the latent variables show robustness in spike prediction across models. Second, surprisingly the dynamics of the high-dimensional stimuli and RGCs' responses are very well represented in the embeddings of the IB-GP model. Third, the minimum stimuli consist of different patterns: Gabor-type locally high-frequency filters, on- and off-center Gaussians, or a mixture of both. Overall, this work demonstrates that the IB-GP model provides a principled approach for joint learning of the stimuli and retina codes, capturing dynamics of the stimuli-RGCs in the latent space which could help better understand the computation of the early visual system.

Natural image synthesis for the retina with variational information bottleneck representation
Babak Rahmani
Demetri Psaltis
Christophe Moser EPFL
EPFL
rahmani. bt1 1991@mmil... com
demetri cometri.ppalriseefff.cc
Abstract
the early visual system, high dimensional natural stimuli are encoded into the trains of neuronal spikes that transmit the information to the brain to produce perception. However, all the visual scene information required to explain the neuronal responses? In this work, we search for answers to this question by developing a joint model of the natural visual input and neuronal responses using the Information Bottleneck (IB) framework that can represent features of the input data into few latent variables that play a role in the prediction of the outputs. The correlations between data samples acquired from published experiments ex-vivo retinas are accounted for the model by a Gaussian Process (GP) prior. The proposed IB-GP model performs competitively the state-of-the-an feedforward convolutional networks in predicting spike responses to natural stimuli. Finally, the IB-GP model used in closed-loop iterative process to obtain reduced- complexity inputs that elicit responses elicited by the original stimuli. We found three properties of the retina's 's IB-GP model. First, the reconstructed stimuli from the latent variables show robustness in spike prediction across models. Second, surprisingly the dynamics the high-dimensional stimuli and RGCs' responses are very well represented in the embeddings of the IB-GP model. Third, the minimum stimuli consist of different patterns: Gabor-type locally high-frequency filters, on- and off-center Gaussians, or mixture of both. Overall, this work demonstrates that the IB-GP model provides a principled approach for joint learning of the stimuli and retina codes, capturing dynamics of the stimuli- in the latent space which could help better understand the computation the early visual system.
Introduction
fundamental challenge in neuroprosthetics is finding the proper input to a sensory or motor system that yields desired functional output. This is achieved naturally by the underlying physiological circuit in an unimpaired system. For example, in the vision system, when an image formed onto the photoreceptors, the electrical neural activity is processed by several layers of neurons within the retina, and train of electrical neural spikes is sent to the visual cortex via the optic nerve producing perception of the visual scene. However, the input stimuli and still obtain the same functional behavior? In other words, what are the reduced spatiotemporal stimuli that elicit the same responses those of the digher-dimensional original inputs? These are important questions both for understanding the neural computations of the retina and possibly for designing visual prosthesis in which minimal information stimuli that optimally drive neurons may prove beneficial [36]] 18 22]
ask: is it possible reduce the complexity of
In this work, focus the natural vision system and search for answers to these questions. Our system consists of the spatiotemporal stimuli impinging on the photoreceptors in the retina up to the spiking responses emitted by the Retinal Ganglion Cells (RGCs). Linear-nonlinear (LN) [[] and
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Xac
Paci
Z E
A xx
ن
rell
منا ا
Putch 2
enc
dec
Datch
Z' KNER
Figure Overview of the IB-GP model of the retina responses. Batches of T high-dimensional stimuli are encoded into low-dimensional space and decoded back to the batches of N neuronal response trains of length T. The latent space consists of latent vectors. Each latent vector 2f is modeled by a GP prior to account for the sample correlations in the original space.
were proposed for encoding of the retinal neural codes. Although successful for artificial stimuli such white noise, they could not faithfully describe RGCs' responses to natural stimuli [13]. The convolutional neural network (CNN) model McIntosh et al.
generalized linear models (GLMs)
25 26
21 was the first model capable of such computation. This was followed by other efforts such as employing recurrent neural networks
[5] or variants CNNs (0). Although nigh-dimensional in nature, neural activity can be well explained by low-dimensional representations [12 14] 28). This opens up the possibility of using models that learn a latent representation data, such as VAEs [16] and ICAs [9]. Accordingly, latent dynamical methods that inspect sources of di-densiiiinnll structure in the neural response have recently gained growing interest
35
we introduce the Information Bottleneck Gaussian Process (IB-GP), a latent space model that can extract the principal features of joint stimuli-responses distribution that are sufficient for predicting the retina responses. The schematic of the IB-GP is depicted in Fig. This model a variational approximation to the original Information Bottleneck framework of Tishby et al. [3I]. The objective of IB is to obtain reduced representation of the input source that preserves maximum information about the output response. DVIB assumes the IB's latent codes are i.i.d. This is poor assumption for the retina dataset. The reason for this is two-fold. First, while observing a scene, images captured by the retina are naturally very similar. Second, during fixation, the eyes undergo dynamic movements that shift the gaze's center. Therefore, incorporating the temporal correlations among data samples should provide a better model. Accordingly, IB-GP uses Gaussian Process prior modeling the retina dataset.
Inspired by the Deep Variational Information Bottleneck (DVIB) 2
9
Our contributions to this work are:
A probabilistic model: We propose IB-GP, a new model for the RGC's spike train in response to the complex natural visual stimuli using the IB framework. The model integrates GP prior into the IB latent code that can learn the temporal dynamics of the data in a lower dimension space. This model shows competitive performance compared to the state-of-the-arr feedforward CNN models of the retina tested on real-world experimental data.
Latent dynamics analysis: We analyze the latent space of the proposed model and extract low- dimensional dynamics consistent with the high- -igh-iimnssioaal data dynamics. We show that the extracted dynamics are more predictive of the neural responses compared to dynamics extracted by a model with no temporal prior constraint.
Closed loop image synthesis: Utilizing the IB-GP model, which only allows principal features of the input stimuli to pass the bottleneck, the original input stimuli are pruned to contain the minimum information required for producing the RGC responses. The resulting complexity-redueed stimuli could be used for the next round of measurements in an iterative process.

---

In the following, we start by reviewing the related work in section 2 the general setting of the problem and our model are presented in section B] and the closed-loop stimuli optimization procedure is
outlined in section 4 Experiments and results are presented in sections 5 and [ Finally, the results are discussed section
2 Related work
In the RGC modeling literature, our work is the closest to McIntosh et al. (2 in which a CNN is used model the responses of RGCs
of input stimuli. Despite the simplicity of this approach, the feedforward CNNs for image classification or regression cannot be directly utilized to extract the principal features of the stimuli contributing the neural responses. On the other hand, latent neural models such as 17 (23 29 are primarily applied to cortical recordings to learn sources of neural variability in the motor cortex data. Unlike our model, which directly relates the stimuli to the neural responses, the stimuli in those lines of work are utilized as a guide star to disentangle the latent factors. Therefore, these methods do not directly apply to modeling the RGC response to input stimuli. In the closed-loop physical system input optimization literature, our work is related In particular. for neuroscience applications, authors in 4 control the activity of the individual neuronal sites in V4 by optimizing individual input stimulation, whereas in our work, thanks to the learned latent code of the entire population of the targeted neurons, all stimuli are optimized together. In Shah et al. RO, the RGCs' responses to electrical stimulation were optimized by first developing a model for the electrical stimuli and the spiking probabilities and then using the model for adaptive stimulation. The model is obtained by parameterizing the spike amplitude and electrical stimulation threshold by a few parameters and then maximizing an evidence lower bound on the spiking probabilities. Our work assumes no relevant relation between the input stimuli and the spiking probabilities. Instead, the relevant parts the input for predicting the output are discovered automatically.
to [27 30
3 Methods
Below, introduce the IB-GP method, which models the neural activities by learning the latent space underlying the stimuli and RGCs' responses. We also summarize additional methods, including a variant of our proposed method that we used for evaluating the IB- model.
3.1 IB-GP model
Notation and problem formulation The input stimuli dataset X E MTT consists of consecu- that are projected in the retina at tive spatial inputs of size M, i.c. X [[xM, M ..M.T.
constant rate to elicit count responses of the form Y E N XT where Y [UN, 1, JN.2, , /N,T] and N denotes the number RGCs. With the above assumptions, we formulate the problem as finding the latent variables Z E IR 4xT that have maximal mutual information 7(Z, Y) with targets Y. We note that is comprised of T consecutive data points, i.c. Z = [ZL,1, ZL,2, ... ZL.T] where T < T. Also Z is simultaneously constrained to have minimal mutual information with X. Therefore, the constraint optimization problem can be written as
LiB max TTB]
where
In (() (((,
< designates the parameters of the model, i.e. 0 and ó, (notation henceforth dropped for brevity) and 33 is variable to adjust the amount of reduced and preserved information in Z. To obtain these latent variables, we approximate their posterior distribution with parametric stochastic encoder Po(2 (Z Due to the apatio-tempont nature of the data, assumption of data independence along the time dimension is not valid. The same is true in the latent space. To account for this fact, we assume a posterior of the form:
L
Ppl (Z()
(, 1=1
(2)
'Code for reproducing the results provided in Appendix 9.

---

where z' denotes the I-th latent vector of size T. In Eq. 2 H R' and ΣS E TTT are the mean and covariance matrix of a multi-variate normal distribution that are functions the inputs. We choose the structure of the covariance of the multivariate normal distribution in Eq. to reflect the time correlations in the data. Hence, similar to previous work model by the product of bidiagonal matrices:
3 73], we construct the E in our
{ e 1o
THE [T,
E -1 viv, + where Vdrre
(3)
O.W.
We note that the simple GP kernel assumption makes the computation required for drawing samples from Ps linear in time [3] Expanding the IB objective in Eq. (details in the Appendix 1), have:
f dY dYd dz PYY 7.) log Z P(Y)
3 f - p(x p(x,
log (Z
TIB -
(4)
(Z))
Although all terms in the RHS of Eq. are fully defined, computing marginal distributions p(Z) and pl (Y) may be intractable. We use variational approximation for p(||Z) that is parameterized with 0. On the other hand, the prior on the latent variables, i.e. pZZ), is modeled using GPs. In particular, assume GP prior on Z defined as multivariate normal:
(Z))
N K) 1=1
(5)
where p is the variational approximation for the prior and K is the covariance function that models the temporal correlations in the latent space. In more details, the covariance between the T-th and the 7/-th samples is computed as K K(T, 7/) where K. is the kernel function. We used the Cauchy kernel:
(T (T- )
T(T,
1
where and 1 are the magnitude and temporal scales of the kernel function, respectively. Cauchy kernels have proved successful in modeling of the time-series systems with multi-scale dynamics as shown before [33. Substituting the variational approximations of the intractable marginals in Eq. and using the fact that the Kullback Leibler (KL) divergence is always positive, we obtain lower bound for the IB objective:
Tin T Σ (ep. go -
where r(t) is to represent time dependence up to time See Appendix I for more details.
3.2 Other models considered
Maximum likelihood feedforward CNN is used as comparison baseline. We use the same CNN network that was shown in the previous work to obtain oftte-of-the-arr results on a larger version of the Natural dataset (2T) The predicted outputs the network are the averaged maximum likelihood estimation of the retina responses given the input stimuli. Artificial noise is injected into the model during optimization to account for the variability in retinal spiking.
IB-Disjoint optimizes the objective from Eq. 7 Jusing the same training procedure as the original method. The difference lies in that the IB-Disjoint assumes latent samples are temporally independent. Accordingly, an isometric Gaussian distribution is employed as the prior in Eq. 7 This model is akin to the standard Variational Auto-encoder [6), or vanilla DVIB 2 which samples in the data and latent space are assumed to be independent (no latent GP).
4 Closed loop stimulation
In this section, we take advantage the model developed in the previous sections devise an algorithm that uses the prior recorded data to optimize the stimulation by iteration in subsequent

---

measurements in a closed loop. Specifically, we assumee the latent variables of the learned model, henceforth referred as the forward have captured the principal rules governing the underlying biological system. We intend to find transformed version of the original input stimuli that yields a set of latent variables that produce the most correlated responses with the original target responses in the subsequent round of measurements. Of course, the original stimuli themselves are one set of solutions. However, this is not a useful transformation. Instead, we are interested in obtaining the best transformation subject to constraint on their complexity. Hence, we define a parametric function f& : 12A4X + R- xT' that maps the complex original stimuli X onto the transformed stimuli X". Passing X" to the forward model, the parameter of the mapping functions, i... &, are optimized so that the forward model's 's output responses Y' are the most correlated with the original responses Y. Denoting the original stimuli and responses as X = X and yorig = Y, the objective function reads as:
min DI Di Y*) + a D2 ongg x*) + Dx')
where:
X X- ((("B)))) Z' ~ -p(('"X"), Y 'o("
(8)
Therein, Ps and go are the encoder and decoder of the forward model trained on the prior data. D1 is a measure to ensure the closeness of the original target outputs and the responses of the forward model to X". D2 is a similarity measure between the original and synthesized stimuli X and X" which encourages the image synthesizer to focus on finding the essential features in the stimuli rather than synthesizing entirely new solutions. Finally, D3 is used to constrain the synthesized stimuli X* to be smooth. Contributions of D2 and D3 to the total loss in Eq. 8 are tuned by hyperparameter a. Two major mechanisms help to prevent fg from being simply an identity mapping. One, during the training of the fe. the encoder's 's latent variables Z are masked by zeroing out all the latent variables except for the most informative one. This way, X is no longer the optimal solution that minimizes the first term in the objective function. Two, fe is implemented using an Autoencoder architecture with limited latent capacity. Finally, the optimized stimuli for subsequent rounds of measurements can be obtained by the following procedure: (1) map the original stimuli to the optimized stimuli using mapping fe. (2) encode the new input X to the posterior Z", (3) decode the latent variables, (4) construct the metric D and backpropagate the error to fit fe. (5) once converged, use the mapping function to obtain the optimized stimuli X (6) send new inputs X to the true biological system (or its model) and observe the new true system outputs, (7) to repeat the procedure, use the true system's new inputs and outputs to fine-tune the forward model and (8) repeat.
5 Experiments
5.1 Dataset
We focus on applying our model to the time-series dataset containing the 2D train of input stimuli (images) entering the vision circuit of an example subject and the corresponding elicited count responses of a group of neurons. Specifically, we use two real-world experimental datasets. The first one, Natural dataset, contains natural scene images as input stimuli and spike trains from nine RGCs is a public dataset which contains images with disc-like shapes having Brownian movements and the spike trains elicited in 91 Rat's RGCs [8].
in Salamander (21) 2 The second dataset, nrownian-moveneee
5.2 Forward modeling
Training specifications We used the models outlined in section to fit the neural activities and input stimuli. The Natural dataset is from the Natural images from the birthplace of the human [32] This dataset consists of Natural images in the wild that do not belong any particular category. The train set contains 3950 unique images (cach unique image is jittered 100 times) of which 20% was randomly chosen for validation. The test set contains 50 averaged repeated trials of novel stimuli
'The Natural dataset, used in this work. is a subset of the original dataset of [21] that was provided by the authors therein. This dataset contained responses of only nine RGCs.
5

---

where each unique image is jittered 100 times. One batch of the test data consists of 10 unique images. We note that the output responses were first binned at 10 ms and then smoothed using a Gaussian filter with standard deviation equal to the bin size. The Brownian-movent contains 129600 pairs of input images and output spikes binned at 12.5 ms in the training set, of which 20% was randomly chosen for validation. The test set contains 1800 novel stimuli. We varied the number of neurons in the Brownian dataset for which we tested our model. We report the results for nine neurons in the main paper and 15, 21, 27, 36 and 51 neurons in Appendix 5.
dataset
Models and evaluation metrics To account for the discrete nature of the output responses (count time series), we employ a Poisson regression model for both IB-GP and IB-Disjoint decoder as well as the feedforward CNN loss term, i.e. qo (v: [Z) = Po(ye (f(()) where fo is the parametric network. We sweep the number of latent factors and the prior pressure constraint 3 for both IB-GP and IB-Disjoint. For the Natural and Brownian datasets, the number of latents is swept between I to 15 (step 1) and to 50 (step 5) variables, respectively. Except for the latent space the IB-GP, which requires more variables to model its covariance matrix. IB-GP and IB- -i-Djjoont are implemented using the same architectures. All hyperparameters in the training of the two models are fixed to be the same. We perform three-fold fitting of the models for the spike prediction results. We report the models' Poisson negative log- liglliicls on the ground truth data. Morcover, we use the Pearson correlation 66] between the ground truth data and the networks' prediction as another metric for evaluating the performance of the models. The architectures of the networks and the training setup are explained in more details in Appendix 2.
Traversal analysis We perform traversal analysis on the latent space of IB-GP and IB- D--jjoiont models to study the amount of information picked by each latent variable. To this end, the encoder (Z|X) is fed with batch of input stimuli initialize the latents. In the IB-GP model, the 1-th latent is sampled from (00 mK) where m is a multiplier that is traversed in the range [-3,3] 3] while keeping all the other latents their inferred values. The latent variables in the IB- (B-Disjoint model are sampled similarly but from the normal univariate distribution N(0, ml). The latents are then passed to the decoder that reconstructs the output responses. The output responses of the traversed model then compared against the non-traversed responses.
5.3 Closed-loop image synthesis
We use both the IB-GP and BB-Disooint with a latent size of 4. The prior constraint parameter / is set to 0.05. These hyperparameters are chosen as trade-off between the latent space compression and RGC's neural activity fitting accuracy. The image synthesizer Se is implemented using an Autoencoder architecture (33) with latent size of 15. The rationale for this choice of architecture is explained in Appendix 2. To allow for multi-phase iterative optimization, we use neural network as the proxy for the true biological system trained on the Natural dataset. This model is trained once and kept fixed. We use the feedforward CNN network of McIntosh et al. [2 as the true model in order to rule out the possibility of obtaining correlated responses due the similarity of the IB- IB-GP model and the true model. We optimize the stimuli in three-phase closed the Poisson regression loss as the D2 measure in the objective function[8] [8] The similarity measure D2 between the synthesized and original stimuli is computed using a Mean Squared Error (MSE). Total Variation (TV) metric [10 used as the D3 measure. We analyze the contributions of all measures to the quality of the synthesized stimuli by sweeping the hyperparameter a.
experiment. We used
6 Results
6.1
Forward modeling
RGC neural activity fitting Performances of the IB-GP, IB-Disjoint, and Feedforward CNN in fittings to neural activities versus the number of latents for both Natural and Brownian datasets are plotted in Figure[] Fi shows that IB-GP outperforms IB- IB-Disjoint almost in all cases. For a fixed 3 value, the performance of the IB-Disjoint usually closer to that of the IB-GP with the increase in the number of latents. This indicates that IB-GP learns neural activities with fewer number of latents. Similar is the case with 3 value decreasing to the extent where IB-Disjoint's performance reaches that of the IB IB.GP. For less constrained models, IB- BB-Disjoint surpasses IB-GP

---

Negative Log Likelihood
Peasso Correlatio
Peeffrraadd CNN
0.1
3a 200, IB-CP 32 200 DiyDeei
3: 0190 IB-C2"
5 (100) [B- IB-Deeyimt
3 0133, IB-Ce
3 0103. IB- DB-Digyin
5: (1070 IB-C2
5: 01000 (B- (B-Deeymm
Number of latents
Number of latents
Negalive Log Likeltheod
sG
Pearson Corrlation
Pesdforward
-00000
3=003 BB-Bieytire
0.364
0.342
3-0010 0.000
0.000
0.54
0:00 (B-CP
0.000 (B-Ibeyxtrr
0.08
0
00
50
20
Number of latents
Nomber latents
Figure 2: Overview of the RGCs' neural activity prediction accuracy for the IB-GP, IB-Disjoint, and Feedforward CNN trained the Natural and Brownian dataset. (a) Poisson negative log-likelihood of the predicted neural responses and (b) Pearson correlation of the predicted and true spike responses as function of the number of latents for different values of the prior pressure constraint 3 for the Natural dataset. (c, d) the same as (a,b) but for the Brownian dataset.
(refer to Appendix 4). Interestingly, IB-GP achieves good neural prediction performance for the Natural dataset even with two latents. Figure 2 shows that for the Brownian dataset and larger Gs. IB-GP keeps its performance gap with IB- IB-Disjoint for a large number of latents. This is due to the strong correlation between the Brownian stimuli dataset (refer to Appendix to observe some examples of stimuli), Both IB-GP and IB-Disjoint outperform the Feedforward CNN when sufficient number of latents are used (~ latents and B < 0.033) for Natural and (~ 10 latents and
B < 0.01) for the Brownian dataset.
Comparison of the IB-GP and IB-Disjoint learned dynamics Figure 3 we analyze the learned dynamics of the IB-GP and IB IB-Disjoint trained under the same conditions on the Natural dataset (results for the Brownian dataset are in Appendix 5). For 3 = 0.05 and 15 latents, performances of the IB IB-GP and IB-Disjoint in prediction of RGCs' responses ~ 0.43, respectively). Nevertheless, the amount of information picked by latents in each model varies significantly. To show this, perform a traversal analysis of the latents and report the variability in the output responses for each traversed latent in Figure 3 aa.d). The KL divergence for each latent is also plotted. As observed in the figure, IB-GP lumps most of the information required to explain the neural responses in one factor. In comparison, this information is spread over five factors in IB-Disjoint. Analyzing the dynamics of the latents in each model can provide further insight. Figure 3lb.e) visualizes the inferred dynamics of the most informative latent in each model after reducing the dimension 2 with T-SNE. As observed in the figure, the dynamics of the most informative latent in the IB IB-GP have 50 clusters. Strikingly, these 50 clusters correspond to the 50 unique stimuli in the dataset. The test set consists of 5 batches in which each batch has 10 unique images (each jittered 100 times). Each batch is colored differently in the figure. Auto-Correlation (AC) plots of the latents corroborate these findings. As evident from 3 AC of the most informative latent in the IB-GP closely follows the statistics of the true and predicted responses of the model (averaged over all neurons). While the AC of the IB-Disjoint's is predicted responses is similar to that of the true responses, the AC of its latents is very different. To demonstrate the efficacy of the IB-GP model in fitting the RGCs' responses, neural activity predictions of both models are shown in Figure 3 (g.h). Refer to Appendix 6 for a more detailed interpretation of IB-GP's G latent dynamics.
similar (correlation 0.45 vs.
*e should emphasize that the hyperparameters of all three networks were not optimized to achieve the best results. Proposing a retina model that outperforms all the other methods is not the main focus of the work: for that requires much further experimentation.
7

---

h
Parrss comelation
(noemallzee))
TSNE dimension
Autoooralaion
0.1
33
12
-30
1tth
TSNE dimencion
Timme lug (5)
d
Lidend BIGC
Parson dediit
8.42-
M
KL. (ncemalizedi
TSNE dimeesioon
Autoooendatten
TSNE dimmension
Time (5)
0.63
0.9
5.58
0.81
0.89
0.58
0.25
0.25
0.00
2.00
Rate (5phes221
00.02
40.01
0.25
0.25
0.00
5.A
0.50
0.50
0.25
0.25
6.0
0.5
7.0
7.0
0.0
0.0
8..
9.0
6.0
6.5
7.0
7.0
10.0
0.0
0.5
9.0
Time (4)
Timm
Figure Analysis of the IB-GP (first row) and IB-Disjoint's (second row) learned dynamics evaluated on the Natural images in the test dataset. Third row plots examples of the RGCs' neural responses obtained by the IB-GP (left) and IB-Disjoint (right). (a,d) The average prediction accuracy of the IB-GP and IB- IB-Disjoint, measured by the Pearson correlation, is shown as a flat line. The variability in the accuracy of the model for traversed latent factors depicted as blue shades. Averaged normalized KI. divergence of each latent is also plotted. (b.e) The temporal information of the most informative latent variable from (a.d) are visualized after reducing the dimensions to 2 with T-SNE. Each color denotes one batch of the test set. (c.f) AC of the most informative latent is compared against those of the true temporal activity and the model's s predicted responses averaged over the number of neurons. (g.h) Neural activity prediction of the models for some example cells. Numbers show the correlation of the predicted responses with the true activities.
6.2 Closed loop image synthesis
T the training results of the closed-loop experiment after the first round of measurements. Both the IB-GP and IB-Disjoint are used the forward model in the training of the image synthesizer. We consider two scenarios when tuning the hyperparameters in the objective function of the image synthesizer. Scenario one: the image synthesizer is trained only to reduce the complexity of the original stimuli independently from the RGCs' responses (dropping the Dj term in the objective function of the image synthesizer). In this scenario, we also consider the case where the TV smoothness constraint is dropped. These are important cases because they answer this valid question: can simple aivensionnaity-reduttiiii
technique be used for synthesizing images that elicit responses correlated with the original ones? Scenario Two: the image synthesizer is trained in conjunction with RGCs' responses (a f 0). Figure 4 a. b) plot the Pearson correlation and negative log-likelihood of the RGCs' responses predicted by the true model in response to the stimuli obtained by the image synthesizer network for the two scenarios. The TV loss of the obtained synthesized stimuli in each case is plotted in Figure 4.. As can be observed in the Figure 4d, dimensionality reduction of the stimuli

---

Negatie Log Likelihnod
0.12
Join
Pearson Correlation
log Loss]
Dijolattiinig with contiit
0.30
Dujoint tniniig cormonn
0.11
01
85-10"3
0.5
Original stimuli
Disjoint
Disjoint train with
Joint
No TV
IB Disjoint a=0.1
IB-GP
Receptive Fields
0.8
0.4
0.2
Figure 4: Summary of the closed-loop image synthesis experiment. The image synthesizer is trained either with the forward model (joint training) or without the forward model in the loop (disjoint training). We used both the IB-GP and IB- IB-Disjoint as the forward model in the former case. In the latter case, the synthesizer is trained only to reconstruct the high- high-dimensional stimuli without incorporating any information from RGCs. The disjoint training either subject constraint on the reconstructed images or with constraint.
smoothness
(a) Poisson negative log-likelihood and (b) Pearson correlation for the RGC neural activity predictions. (c) TV loss for the synthesized flat black line. Joint
stimuli. Performances of the original stimuli in the true model are depicted training with IB-GP with a = 0.1 as the forward model achieves the best performance among all cases. (d) Examples of the synthesized images in all scenarios. IB-GP synthesized images contain particular patterns that are within the RFs of cells. The RFs of the neurons are depicted scale.
without neural information results non-smooee smoothed images (column 2) or smoothed images (column 3) but poor RGC response accuracy (rightmost results in Fig. 4a-c). On the contrary, incorporating RGC neural information obtains synthesized stimuli with excellent smoothness and significant RGC response accuracy (Fig. 4a-c and columns 5,6 in Ha)).. These results demonstrate the robustness of the proposed algorithm in finding synthesized images that provide accurate responses. Strikingly, when IB-GP is used as the forward model in the closed-loop experiment, complex curvature shapes appear the synthesized images. Expectedly, these shapes appear where the neurons' receptive field is located. The receptive fields (RFs) of the neurons, obtained by Spike-triggered Average (STA) analysis (T. are also shown in the figure. Finally, we continue the optimization the image synthesizer for the IB-GP model with hyperparameter 0.1 for another two rounds. The Pearson

---

correlation of the synthesized stimuli in all three rounds reads as: 0.353, 0.366, and 0.361. Details of the iterative optimization and evolution of the synthesized images in each phase are explained in Appendix 6. The last row in the Figure 4 shows a failure mode of the image synthesizer algorithm. We tested the image synthesizer (trained on the Natural dataset) on the white noise data, As seen from the figure, IB-Disjoint provides superior synthesized image than other methods. This is in line with our observations that the IB-GP model fails to fit neural data elicited by the white noise. More results are available in Appendix 8.
7 Discussion
This work presents the IB-GP model, latent space variational model that learns low-dimensional representation of the Natural stimuli and RGCs' responses. We observed that the IB formulation of the neural activity fitting allows learning of low-dimensional representation of the stimuli and RGC responses which is more predictive than the feedforward models. We demonstrated that incorporating temporal prior the latent space of the IB model factorizes the sources of neural variability into fewer latents. The Latent analysis of the model revealed the superior performance of the IB-GP to IB- Disjoint with temporal prior the latents. We observed that the dynamics of the high-dimensional stimuli and RGCs' responses were very well represented the embeddings of the IB-GP model.
The model was then used in closed-loop experiment to synthesize stimuli that elicit neuronal responses as those elicited by the original complex stimuli. Previously published work in the literature also looked into closed-loop image synthesis, mainly to find optimal stimuli that maximally activate neurons in the retina (24), or more recently in V1 and V4 regions in the mouse [34] 4). The goal is to obtain spatial features that do not necessarily resemble the original stimuli but still elicit the same responses. This allows us to correctly use the resources available, for example, by placing the stimulation energy at suitable locations and with the correct spatiotemporal features. We observed various types of optimal stimuli for the Natural dataset, such as Gabor-like filters, on- and off-center Gaussians, mixture of both.
Given the promising results demonstrating the usefulness of the model, the IB-GP model has some limitations too. First, it is unclear what type of temporal prior is sufficient for the best factorization of latents. Although the Cauchy kernel has proved efficient in modeling multi-scale dynamics [K], it might be that other temporal kernels, combination of several kernels, or even spatio -temporal kernels [9] could improve disentangling of the latents. Second, we observed the failure mode of the IB-GP model in the fitting of the white noise stimuli due to the lack of temporal correlation. Third, computing the covariances of the IB-GP's prior is computationally intensive, especially for the long temporal datasets. This might also be possible negative societal impact of our work, resulting in large carbon footprint.
Acknowledgments and Disclosure of Funding
The authors would like to thank all the anonymous reviewers for their constructive feedback helping us improve the manuscript, and especially for bringing the initial overfitting in Figure 2 to attention, which allowed us to resolve it and improve the results. Authors declare no competing financial interests funding.
References
[1] Ahn, B. Rueckauer, Y. Yoo, and Y. S. Goo. New features of receptive fields in mouse retina through spike-triggered covariance. Experimental Newrobiology, 29(1):38, 2020.
[2] A. A. Alemi, I. Fischer, V. Dillon, and K. Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 16/2000400 2016.
[3] R. Bamler and Mandt. Structured black box variational inference for latent time series models. arXiv preprint arXiv: 170.00077 2017.
[4] Bashivan, K. Kar, and J. J. DiCarlo. Neural population control via deep image synthesis. Science, 364(6439), 2019.
10

---

[5] E. Batty, J. Merel, N. Brackbill, A. Heitman, A. Sher. A. Litke, E. Chichilnisky, and L. Paninski. Multilayer recurrent network models of primate retinal ganglion cell responses. 2016.
[6] Benesty, Chen, Y. Huang, and Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1-4. Springer, 2009.
[7] D. M. Blei and J. D. Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pages 113-120, 2006.
[8] V. Botella-Soler, S. Deny, G. Martius, O. Marre, and G. Tkačik. Nonlinear decoding of complex movie from the mammalian retina. PLoS computational biology, 14(5)ee1006057 2018.
[9] P. Casale, A. V. Dalca, Saglietti, J. Listgarten, and N. Fusi. Gaussian process prior variational autoencoders. arXiv preprint 1rXiv:/810.117:80 2018.
[10] A. Chambolle, V. Caselles, D. Cremers, M. Novaga, and T. Pock. An introduction to total variation for image analysis. Theoretical foundttions and numerical methods sparse recovery, 9(263-340):221 2010.
[11) E. Chichilnisky. A simple white noise analysis of neuronal light responses. Network: computa- fion neural systems, 12(2):199, 2001.
[12] G. Elsayed and Cunningham. Structure in neural population recordings: an expected byproduct of simpler phenomena? Nature neuroscience, 2009):1110-131,, 2017.
[13] V. Fortuin, D. Baranchuk, G. Rätsch, and S. Mandt. Gp-vae: Deep probabilistic time series imputation. In International conference on artificial intelligence and statistics, pages 1651-1661 PMLR, 2020.
[14] J. A. Gallego, M. G. Perich, L. E. Miller, and S. A. Solla. Neural manifolds for the control of movement. Neuron, 94(5):978-9844 2017.
[15] A. Heitman, Brackbill, M. Greschner, Sher, A. Litke, and Chichilnisky. Testing pseudo-linear models of responses to natural scenes in primate retina. biorxiv, 045336, 2016.
[16] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
[17] C. Hurwitz, A. Srivastava, K. Xu, J. Jude, M. Perich, L. Miller, and M. Hennig. Targeted neural dynamical modeling. Advances in Neural Information Processing Systems, 34, 2021.
[18) I.. H. Jepson, Hottowy, A. Weiner, W. Dabrowski, M. Litke, and E. Chichilnisky. High-fidelity reproduction of spatiotemporal visual signals for retinal prosthesis. Neuron, 83(1): 87-92, 2014.
[19] I. Khemakhem, D. Kingma, R. Monti, and Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[20] N. Maheswaranathan, L. McIntosh, H. Tanaka, S. Grant, D. B. Kastner, J. B. Melander, A. Nayebi, L. Brezovec, J. Wang, S. Ganguli, et al. The dynamic neural code of the retina for natural scenes. BioRxiv, page 340943, 2019.
[21] L. McIntosh, N. Maheswaranathan, models of the retinal response to natural scenes. Advances in neural information processing systems, 29: 19:133991777, 2016.
Nayebi, S. Ganguli, and S. Baceus. Deep learning
[22] S. Nirenberg and C. Pandarinath. Retinal prosthetic strategy with the capacity to restore normal vision. Proceedings of the National Academy of Sciences, 109037):15112-1507, 2012.
[23] C. Pandarinath, D. J. O'Sha, J. Collins, R. Jozefowicz, S. D. Stavisky, J. C. Kao, E. M. Trautmann, M. T. Kaufman, I. Ryu, I., R. Hochberg, et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10):805-815 2018.
11

---

[24] L. Paninski, J. Pillow, and J. Lewi. Statistical models for neural encoding, decoding, and optimal stimulus design. Progress in brain research, 165:493-500 -07, 2007.
[25] W. Pillow, L. Paninski, V. J. Uzzell, E. P. Simoncelli, and E. Chichilnisky. Prediction and decoding of retinal ganglion cell responses with probabilistic spiking model. Journal of Neuroscience, 25(47): 1003-) 1003-11013 2005.
[26] J. W. Pillow, J. Shlens, L. Paninski, A. Sher, A. M. Litke, E. Chichilnisky, and E. P. Simoncelli. Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature, 454(7207):995-999 2008.
[27] B. Rahmani, D. Loterie, E. Kakkava, N. Borhani, U. Tegin, D. Psaltis, and C. Moser. Actor neural networks for the robust control of partially measured nonlinear systems showcased for image propagation through diffuse media. Nature Machine Intelligence, 2(7))403-410, 2020.
[28] T. Sadtler, K. M. Quick, M. D. Golub, S. M. Chase, S. Ryu, E. C. Kabar- B. M. Yu, and A. P. Batista. Neural constraints on learning. Nature, 512(7515):43-4466 2014.
[29] O. G. Sani, H. Abbaspourazad, Y. T. Wong, B. Pesaran, and M. M. Shanechi. Modeling behaviorally relevant neural dynamics enabled by preferential subspace identification. Nature Neuroscience, 140,),,4 2021.
[30] N. Shah, S. Madugula, P. Hottowy, A. Sher, A. Litke, L. Paninski, and E. Chichilnisky, Efficient characterization of electrically evoked responses for neural interfaces. Advances in Neural Information Processing Systems, 32: 14444-14458, 2019.
[31] N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
[32] G. Tkačik, P. Garrigan, C. Ratliff, G. Milčinski, J. M. Klein, L. H. Scyfarth, P. Sterling, D. H. Brainard, and V. Balasubramanian, Natural images from the birthplace of the human eyc. PLoS one, 6(6)), 2011.
[33] M. Tschannen, O. Bachem, and M. Lucic. Recent advances in autoencoder-basee representation learning. arXiv preprint arXiv:1812.05000 2018.
[34] E. Walker, F. H. Sinz, E. Cobos, T. Muhammad, E. Froudarakis, G. Fahey, A. S. Ecker, J. Reimer, X. Pitkow, and A. S. Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22(12):0000-266; 2019.
[35] D. Zhou and X.-X. Wei. Learning identifiable and interpretable latent models of high- dimensional neural activity using pi-vae, Advances in Neural Information Processing Systems, 0472334-2247, 2020.
[36] E. Zrenner, K. U. Bartz-Schmidt, H. Benav, D. Besch, A. Bruckmann, V.-P. Gabel, F. Gekeler, U. Greppmaier, A. Harscher, S. Kibbel, et al. Subretinal electronic chips allow blind patients read letters and combine them words. Proceedings of the Royal Society B: Biological Sciences, 178(1711);149--199) 2011.
Checklist
1. For all authors:
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See section 6 and
(b) Did you describe the limitations of your work? [Yes] See section 7
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
12

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes] See section 3 and Appendix
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL.? [Yes] See the footnote
in section[] and Appendix 9
(b) Did you specify all the training details (e-g., data splits, hyperparameters, how they were chosen)? [Yes] See sectior and Appendix 2
and
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See section[6]
(d) Did you include the total amount of compute and the type of resources used (c.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See the footnote in section and Appendix 9
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? (Yes] The dataset asset are cited in section 5
(b) Did you mention the license of the assets? [Yes] One dataset is public and the other is proprietary. See Dataset subsection in section 5
(c) Did you include any new assets either in the supplemental material or as a URL? [No] a sample of data will be added in the supplementary material for verification purposes. (d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [Yes]] One dataset is public. The other provided by the authors of the cited paper. See Dataset subsection in section 5
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "nBlZ2Oin6A", "reviewer": "Reviewer_6crN", "summary": "In this manuscript the authors develop a new variant of the variational bottleneck architecture with temporal dependencies in the latent space and apply it to the prediction of retinal ganglion cell responses. The temporal dependencies are introduced by using a Gaussian process prior, i.e. by adding a prior covariance across time to the latent space variables. Their architecture is quite successful at predicting retinal responses beating a previous network without a bottleneck. Finally, the authors optimize images to be smooth and produce the same latent representation as original input stimuli, which results in much simpler stimuli that are expected to produce similar responses in the neurons.", "questions": "1) Could you please clarify why the image generation is helpful?\n2) Is there any chance to use a larger dataset for this?", "limitations": "---", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The introduction of temporal dependencies into the latent is an interesting extension of this architecture, the implementation seems sensible and the predictive performance of the approach is good. However, the authors predict only a very small number of neurons and I am a bit lost to why the image generation process is helpful. Thus, this paper seems like a sensible step forward, but not like the solution to the visual prosthetics problem as it is framed by the authors.\n\nConcrete points:\n1) The authors fit only 9 neurons in the main paper and up to 36 neurons in the supplement. This is extremely small. Large retinal recording datasets record from hundreds of ganglion cells electrophysiologically (https://doi.org/10.1162/neco_a_01395 for example use more than a 1000 for decoding) or using calcium imaging (https://www.nature.com/articles/nature16468 for example). Especially when stimuli are generated that presumably aim to activate all or at least many retinal ganglion cells in an area, this seems like a substantial drawback.\n2) Similarly, the stimuli used here are in generated from only 10 different images. This restriction may contribute to the success of the small networks here and casts a little doubt on whether the described techniques would generalize to new images.\n3) The authors generate smooth images that are meant to generate similar responses as the original images. This strikes me as odd, because smoothness appears largely unrelated to the constraints stimulation by a prosthetic device would face. Wouldn’t it make more sense to build in such constraints rather than generating images?\n\n\nSmall points:\n- Line 116: The bi-diagonal precision matrices here are simple, but not low rank.\n- Equation (7) should really be a lower bound not an approximation.\n- Quite a few of the figures have extremely small annotations, especially when printed. Please, invest some time to make them more readable, remove unnecessary whitespace and boxes, etc.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kwSCSBfyTYo", "reviewer": "Reviewer_daMk", "summary": "The paper reports the results of analysis and simulation using the information bottleneck method to find a space of reduced stimuli that produce the same population neural response as complex natural stimuli.  \n", "questions": "Figure 3 shows IB-GP and IB-Disjoint, but which is which in the figure?  also what do the blue vs. grey traces correspond to in panels g,h?\n", "limitations": "see above", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- a principled approach using IB method\n\nWeaknesses:\n- motivation is not clear, difficult to ascertain the punchline of what has been achieved here and how it helps us to understand the retina or build a better prosthesis.\n\nThe work appears to be motivated by the desire to build more effective neural prostheses for vision.  Current technology can only stimulate a fraction of neurons in a region of retina, resulting in un-natural percepts.  The authors then turn to the question of whether you can derive a reduced stimulus space that yields the same neural responses as complex natural images.  This seems like a non-sequitur.  If you want to fix the neural prostheses problem, wouldn't you want to find a modified stimulation protocol that yields more natural percepts?  This would presumably require identifying precisely which neurons you are stimulating and somehow adjusting the stimulation so that the cortex can fill in the desired percept from the fraction of neurons you are stimulating.  Instead the authors turn to the question of deriving a simplified stimulus space.  Although a valid scientific question, I don't see how it addresses the neural prostheses problem.  Also, by the end of the paper, I don't see how it helps us to understand retinal mechanisms in general.  If you are only recording from a fraction of neurons then of course there will be a null space (not necessarily linear) in the stimulus space.  Even if you had access to an entire population, there would still be a null space because the retinal transform is certainly lossy.  One could hypothesize how lossy based on our current understanding of retinal mechanisms and compare this to results of stimulating with derived stimuli as shown here, but that's not something this paper does.  So I'm left scratching my head.  Overall it seems like a very nice theoretical and principled approach, but the scientific question and what we learn from this is missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gCXNmbXbwTp", "reviewer": "Reviewer_SLVr", "summary": "The authors propose to use the Information Bottleneck method to compress the visual input while capturing a low-dimensional dynamic of the Retinal Ganglion Cell (RGC) spike trains.\nMany variations of their model and one other CNN-based model are compared. Performances reach similar levels but one of their model is performing best both in term of Pearson Correlation and Negative Log-Likelihood. \nThe authors also propose to generate images which would produce a similar RGCs activity while reducing their \"complexity\".", "questions": "- image complexity is not quantified (figure 4),\n- I do not understand why the latent variable Z must have high mutual information with Y and low mutual information with X ? Can you give some intuition about how it achieve its goal ?\n- In other works, it is sometimes assumed that mutual information between X and Y is maximized. How does it relate to information bottleneck method ? \n- Its seems the latent Z has mainly learned stimulus labels, I don't understand why it is presented as an inferred dynamic ? (Figure 3 be)\n- What is the interest of the closed-loop image synthesis ? Do you obtain a \"population RF\" ?\n- In figure 4, we can only see the RF location not the actual filter (or maybe it's too small). If the obtained image is some sort of population RF, is linearly predictable from the individual RF obtained by STA ?\n\n\nRemark :\n- Eq (7), if it's a lower bound do not write I_{IB} \\approx  but I_{IB} >=.\n\n\nMinor remarks :\n- l 106 (and where applying) : do not start a sentence with a mathematical variable,\n- use \\cite{smith2020blabla,tommy2021bliblou} to get [1,2] instead of [1][2],\n- the text in all figures is too small,\n- ", "limitations": "What is the energy consumption of the training ? How many GPUs ? For how long ?\nDoes the gain in performance compared to non-CNN/DNN models justified in comparison to the training cost (duration, consumption, CO2) ?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n- the model is sufficiently described\n- good performances\n- model comparison\n\nWeaknesses :\n- comparison to a single model from the literature (are there non-CNN/DNN based method out there ?)\n- the learned dynamics are not sufficiently shown, I would expect to see something like a trajectory as function of time\n- the closed-loop image synthesis is not sufficiently motivated. There is a lack of intuition behind the resulting images.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A91-Sq_u08V", "reviewer": "Reviewer_G6SJ", "summary": "The paper proposes a VAE model for generating the ganglion cell responses for visual images. Their motivation is that this could be used for visual prosthetic devices which have a limited number of electrodes so can only handle a sample of the input so it is better to send most important information. They attempt to compress the input images to the lowest possible information required for the response to be identical.", "questions": "- I'm not sure I'm understanding this correctly but shouldn't number of latent variable be lower than the number of outputs for this to be a bottleneck model? The latent dimension is swept from 1 to 50 with only 9 neurons\n- The IB-GP model appears from figure 3 to learn the order the stimuli were sent in rather than the content of the image\n- Line 272 \"(equivalent to setting α = ∞ in the 273 objective function of the image synthesizer)\". Consider dropping this as it is not mathematically sound. One cannot minimize a function with infinity except turning the factor to negative.\n- Can you plot the synthesized images only around the field of view of each neuron? The given synthesized images don't seem to be informational at all to the given original stimuli", "limitations": "Given that this was tested on each dataset separately and with training and test sets being from the same distribution, it is highly likely that this model is overfitting to just compress a choice out of 10. The only thing that the model seems to be recognizing well would be that the jittered images are parts of the same category.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The model appears to outperform the FF CNN model baseline at a larger number of latents.\n\nWeaknesses:\nMajor:\n- I am very skeptical of this model. It appears that the model with temporal information included has learned the sequence of stimuli rather than the actual contents and stored that in one latent variable.\n- The minimal information image does not show any resemblance to the original image and it seems that what the model learned is how to create a category of images that maximize the latent distance between categories but does not care about the actual content of the image.\n- The above two points can be tested if you add out of distribution categories in your test set.\n- The number of latent variables in high performance conditions is larger than the number of RGCs so it is not a bottleneck model\n\nMinor:\nLine 78 \"the CNN cannot be directly utilized to extract the principal features of the stimuli\" this statement is inaccurate, it should be CNN for image classification or something like that, your model is a CNN", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept