Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper reports the results of analysis and simulation using the information bottleneck method to find a space of reduced stimuli that produce the same population neural response as complex natural stimuli.  

strengths_and_weaknesses: Strengths
- a principled approach using IB method

Weaknesses:
- motivation is not clear, difficult to ascertain the punchline of what has been achieved here and how it helps us to understand the retina or build a better prosthesis.

The work appears to be motivated by the desire to build more effective neural prostheses for vision.  Current technology can only stimulate a fraction of neurons in a region of retina, resulting in un-natural percepts.  The authors then turn to the question of whether you can derive a reduced stimulus space that yields the same neural responses as complex natural images.  This seems like a non-sequitur.  If you want to fix the neural prostheses problem, wouldn't you want to find a modified stimulation protocol that yields more natural percepts?  This would presumably require identifying precisely which neurons you are stimulating and somehow adjusting the stimulation so that the cortex can fill in the desired percept from the fraction of neurons you are stimulating.  Instead the authors turn to the question of deriving a simplified stimulus space.  Although a valid scientific question, I don't see how it addresses the neural prostheses problem.  Also, by the end of the paper, I don't see how it helps us to understand retinal mechanisms in general.  If you are only recording from a fraction of neurons then of course there will be a null space (not necessarily linear) in the stimulus space.  Even if you had access to an entire population, there would still be a null space because the retinal transform is certainly lossy.  One could hypothesize how lossy based on our current understanding of retinal mechanisms and compare this to results of stimulating with derived stimuli as shown here, but that's not something this paper does.  So I'm left scratching my head.  Overall it seems like a very nice theoretical and principled approach, but the scientific question and what we learn from this is missing.

questions: Figure 3 shows IB-GP and IB-Disjoint, but which is which in the figure?  also what do the blue vs. grey traces correspond to in panels g,h?

limitations: see above
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose to use the Information Bottleneck method to compress the visual input while capturing a low-dimensional dynamic of the Retinal Ganglion Cell (RGC) spike trains.
Many variations of their model and one other CNN-based model are compared. Performances reach similar levels but one of their model is performing best both in term of Pearson Correlation and Negative Log-Likelihood. 
The authors also propose to generate images which would produce a similar RGCs activity while reducing their "complexity".
strengths_and_weaknesses: Strengths :
- the model is sufficiently described
- good performances
- model comparison

Weaknesses :
- comparison to a single model from the literature (are there non-CNN/DNN based method out there ?)
- the learned dynamics are not sufficiently shown, I would expect to see something like a trajectory as function of time
- the closed-loop image synthesis is not sufficiently motivated. There is a lack of intuition behind the resulting images.
questions: - image complexity is not quantified (figure 4),
- I do not understand why the latent variable Z must have high mutual information with Y and low mutual information with X ? Can you give some intuition about how it achieve its goal ?
- In other works, it is sometimes assumed that mutual information between X and Y is maximized. How does it relate to information bottleneck method ? 
- Its seems the latent Z has mainly learned stimulus labels, I don't understand why it is presented as an inferred dynamic ? (Figure 3 be)
- What is the interest of the closed-loop image synthesis ? Do you obtain a "population RF" ?
- In figure 4, we can only see the RF location not the actual filter (or maybe it's too small). If the obtained image is some sort of population RF, is linearly predictable from the individual RF obtained by STA ?


Remark :
- Eq (7), if it's a lower bound do not write I_{IB} \approx  but I_{IB} >=.


Minor remarks :
- l 106 (and where applying) : do not start a sentence with a mathematical variable,
- use \cite{smith2020blabla,tommy2021bliblou} to get [1,2] instead of [1][2],
- the text in all figures is too small,
- 
limitations: What is the energy consumption of the training ? How many GPUs ? For how long ?
Does the gain in performance compared to non-CNN/DNN models justified in comparison to the training cost (duration, consumption, CO2) ?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a VAE model for generating the ganglion cell responses for visual images. Their motivation is that this could be used for visual prosthetic devices which have a limited number of electrodes so can only handle a sample of the input so it is better to send most important information. They attempt to compress the input images to the lowest possible information required for the response to be identical.
strengths_and_weaknesses: Strengths:
- The model appears to outperform the FF CNN model baseline at a larger number of latents.

Weaknesses:
Major:
- I am very skeptical of this model. It appears that the model with temporal information included has learned the sequence of stimuli rather than the actual contents and stored that in one latent variable.
- The minimal information image does not show any resemblance to the original image and it seems that what the model learned is how to create a category of images that maximize the latent distance between categories but does not care about the actual content of the image.
- The above two points can be tested if you add out of distribution categories in your test set.
- The number of latent variables in high performance conditions is larger than the number of RGCs so it is not a bottleneck model

Minor:
Line 78 "the CNN cannot be directly utilized to extract the principal features of the stimuli" this statement is inaccurate, it should be CNN for image classification or something like that, your model is a CNN
questions: - I'm not sure I'm understanding this correctly but shouldn't number of latent variable be lower than the number of outputs for this to be a bottleneck model? The latent dimension is swept from 1 to 50 with only 9 neurons
- The IB-GP model appears from figure 3 to learn the order the stimuli were sent in rather than the content of the image
- Line 272 "(equivalent to setting α = ∞ in the 273 objective function of the image synthesizer)". Consider dropping this as it is not mathematically sound. One cannot minimize a function with infinity except turning the factor to negative.
- Can you plot the synthesized images only around the field of view of each neuron? The given synthesized images don't seem to be informational at all to the given original stimuli
limitations: Given that this was tested on each dataset separately and with training and test sets being from the same distribution, it is highly likely that this model is overfitting to just compress a choice out of 10. The only thing that the model seems to be recognizing well would be that the jittered images are parts of the same category.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this manuscript the authors develop a new variant of the variational bottleneck architecture with temporal dependencies in the latent space and apply it to the prediction of retinal ganglion cell responses. The temporal dependencies are introduced by using a Gaussian process prior, i.e. by adding a prior covariance across time to the latent space variables. Their architecture is quite successful at predicting retinal responses beating a previous network without a bottleneck. Finally, the authors optimize images to be smooth and produce the same latent representation as original input stimuli, which results in much simpler stimuli that are expected to produce similar responses in the neurons.
strengths_and_weaknesses: The introduction of temporal dependencies into the latent is an interesting extension of this architecture, the implementation seems sensible and the predictive performance of the approach is good. However, the authors predict only a very small number of neurons and I am a bit lost to why the image generation process is helpful. Thus, this paper seems like a sensible step forward, but not like the solution to the visual prosthetics problem as it is framed by the authors.

Concrete points:
1) The authors fit only 9 neurons in the main paper and up to 36 neurons in the supplement. This is extremely small. Large retinal recording datasets record from hundreds of ganglion cells electrophysiologically (https://doi.org/10.1162/neco_a_01395 for example use more than a 1000 for decoding) or using calcium imaging (https://www.nature.com/articles/nature16468 for example). Especially when stimuli are generated that presumably aim to activate all or at least many retinal ganglion cells in an area, this seems like a substantial drawback.
2) Similarly, the stimuli used here are in generated from only 10 different images. This restriction may contribute to the success of the small networks here and casts a little doubt on whether the described techniques would generalize to new images.
3) The authors generate smooth images that are meant to generate similar responses as the original images. This strikes me as odd, because smoothness appears largely unrelated to the constraints stimulation by a prosthetic device would face. Wouldn’t it make more sense to build in such constraints rather than generating images?


Small points:
- Line 116: The bi-diagonal precision matrices here are simple, but not low rank.
- Equation (7) should really be a lower bound not an approximation.
- Quite a few of the figures have extremely small annotations, especially when printed. Please, invest some time to make them more readable, remove unnecessary whitespace and boxes, etc.

questions: 1) Could you please clarify why the image generation is helpful?
2) Is there any chance to use a larger dataset for this?
limitations: ---
review_title: null

Meta-review: metareview: In this paper, the authors describe a new method for estimating the stimulus-response characteristics of biological visual neurons from the retina. The authors employ the Information Bottleneck method to compress the visual representation and compare their results to other models including CNN-based architectures. The authors find that their model is most performant in terms of Pearson correlation and log-likelihood on real neural spike trains recorded in response to natural imagery and Brownian movement. Lastly, the authors use the model to reconstruct stimuli using the learned latent representation. 

The reviewers applauded the principled approach to the estimation and analysis of neural receptive fields, the clarity of presentation of the method and comparisons across models and the scale of the data. In the responses, the authors were able to showcase new results indicating that the method could scale to a larger number of neurons. One reviewer did take issue with the introduction relying too heavily on the discussion of neural prosthetics to motivate this work as opposed to a discussion of just the neural coding problem. I agree with the reviewer in this sentiment and would like to see the introduction of the paper updated accordingly to better emphasize that this work is focused on the issue of modeling the stimulus-response relationship of a neural population. 

Given the strong support of the reviewers, this paper will be conditionally accepted provided two updates are performed by the authors: (1) update to the introduction section and (2) update to Figure 4D to fit within the margin.

confidence: Certain
Decision: Accept