Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper implements a photoreceptor model in Keras and uses it as a front end in training a shallow CNN model to predict retinal responses to white noise stimuli. The paper reports that the photoreceptor model allows the model to generalize better to new lighting levels. This observation is demonstrated using monkey and rat RGC data. The light levels in the rat experiments used a more dramatic range of variation (4 orders of magnitude) that required a more complex cross-validation procedure due to largely non-overlapping rod and cone activation across levels. 
strength_and_weaknesses: Strengths

I found the paper clear and easy to follow. The results are simple and straightforward.

The paper provides decent evidence that the photoreceptor model improves generalization to new lighting levels, which is an important form of variation.

Exploiting insights from biology to improve machine learning is rare and potentially exciting contribution.

Weaknesses

Applications to vision and ML are preliminary and left to the appendix (Appendix C), which seems likely to limit the impact of the work to the ICLR community. The photoreceptor model is rather complex which I worry will limit its popularity and impact. The authors toy with creating a simpler, convolutional layer based on their model, but again this part of the paper is preliminary and left to the appendix. 

The generalization effect in monkey RGCs is only substantial for the case when training on high light levels and testing on low light levels. No measures of statistical significance or confidence are reported.

It would be interesting to know whether the model would have difficulty generalizing to a variety of lighting conditions when trained using naturalistic variation in retinal stimulation. It would also be useful to quantify model performance using naturalistic stimulation since one of the benefits of having a DNN model is that it can potentially predict neural responses to real-world stimuli, as opposed to just white noise.

If one does use a training set with a wide range of lighting levels is there any benefit of the photoreceptor model? For example, does the model require less training time to reach a given level of accuracy? For ML applications, it’s not clear whether the benefit of the photoreceptor model would be worth the added complexity, particularly if it’s relatively easy to create robustness by applying a bit more diversity to the training regime. 

clarity,_quality,_novelty_and_reproducibility: Combining a biophysical photoreceptor frontend with a standard DNN architecture is novel to my knowledge.

The writing is clear and easy to follow. 

As noted above, I do not believe the paper reports any statistics or measures of confidence…

The cross-validation approach used to evaluate generalization to extreme lighting conditions is convoluted but seems sound.

The authors do not report any data on whether training the photoreceptor model is important or not. How did the authors choose which parameters to optimize? The photoreceptor model has a small number of parameters compared with the DNN layers, and this mismatch might make it hard to jointly train using first-order optimization algorithms such as ADAM. 

It is not clear how the L1 and L2 penalties were selected (grid search?). Were the hyper-parameters chosen using the same test data used to compute explained variance? If so, this would introduce an upward bias in prediction accuracy. It would be preferable to select hyper-parameters using validation and measure performance in a separate test data set. 

The authors split the trials into 2 when estimating the noise variance. However, I believe they use all the data when estimating prediction accuracy, which produces a mismatch. This mismatch is unlikely to matter since the authors report that the responses were highly reliable, but it would be better to use the same amount of data when evaluating predictions and noise variance or to correct for the extra data in the predictions.

Am I correct that the data reported is from a single rat and a single monkey?

I do not have the expertise to evaluate whether the retinal model is properly implemented. 

summary_of_the_review: The combination of a photoreceptor frontend with a DNN to predict RGC responses is novel and potentially interesting. I am overall convinced that this addition helps with generalization to novel lighting levels, at least when the differences are extreme. It is unclear to me whether this insight will in practice be useful to the ML community. There might be simple training strategies that would avoid this issue, and the photoreceptor model is complex and seems unlikely to become a mainstay of computer vision models. 

Response after rebuttal:

I apologize for my slow response. The authors have addressed all of my comments, which I appreciate. I think this paper provides a solid contribution to the computational neuroscience literature and seems well-suited to the Neuroscience and Cognitive Science section. I still find the generalization results in monkey RGC a little underwhelming, and my best guess is that the impact of the paper will be modest. Nonetheless, I have increased my score from 5 to 6 based on the authors' argument that their paper is appropriate for the noted section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors showed that putting a photoreceptor front-end with adaptive dynamic gain control allows a deep neural network to predict more reliably the responses of retinal ganglion cells in different lighting conditions than the STOA CNN retinal model. 
strength_and_weaknesses: Introducing an adaptive gain control mechanism as a front end to CNN is a sensible and good idea.  The fact that such an adaptive mechanism should improve a CNN's prediction of neural response in different lighting conditions is not surprising. Nevertheless,  demonstrating empirically it actually does the job is still meaningful and an accomplishment. Making the adaptive mechanism itself partially learnable is also novel. 
clarity,_quality,_novelty_and_reproducibility: Clear enough.
summary_of_the_review: While the technical and conceptual contribution in ML and representational learning are relatively limited, this paper is acceptable for demonstrating or proving  that an adaptive gain control mechanism as a front-end of CNN can improve the prediction of RGC's responses in different lighting condition. Even though it might be proving the obvious, it is still important and might have significant technical implication for visual prothesis and vision system research in the future.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors indicate that modern computer vision systems tend to fail under dynamic lighting conditions. This is because computer vision algorithms do not explicitly make use of adaptation mechanisms similar to the ones used in retinal ganglion cells. Because current deep learning models of the retina have no in-built notion of light level , they are unable to accurately predict RGC responses under lighting conditions that they were not trained on. The authors use a model of RGC and test it on monkey and rat retinal data. The authors argue that their algorithm can be used with computer vision algorithms to adapt to dynamic environments.
strength_and_weaknesses: Strengths:

S1: the authors identify and important problem in computer vision algorithms and make some biologically motivated suggestions on how this problem could be solved.

S2: the paper is extremely well written and is up to ICLR standards

Weaknesses:

W1: With respect to making computer vision algorithms more robust, why do they feel that such an adaptation approach is necessary? Why not simply give HDR data (high dynamic range data) as input to algorithms (ie data that is represented with more than 8 bits per channel). Wouldn't this be sufficient to deal with saturation issues in high intensity situations, and low contrast in low light situations?

W2: page 1 typo: "We focus here on retina". Missing "the"

W3: prior related work discussed in the paper does not go before 2015-2016. I suggest an expansion on prior work detailing the effects of sensor bias in experimental methods.

W4: section 3.3. The authors do electrophysiology experiments from ex vivo macaque monkey and rat retina. It is not often that you see such experiments in machine learning/computer vision conferences. Machine learning conferences like ICLR are more focused on algorithms that improve real-world performance. The way the paper is written it seems to be focused more towards a computational neuroscience audience. While not necessarily bad, in this case this seems to be to the detriment of demonstrating whether this algorithm actually leads to improvements in accuracy, performance, speed, power of computer vision systems. The main focus of the authors should have been to show that their algorithm leads to some quantifiable improvements compared to other algorithms that are solving a certain real-world problem. The authors seem to try to touch upon this in the appendix, but not in a lot of detail. This is the paper's rather major weakness in my opinion. The appendix algorithm should be in the main paper and its evaluation on a real world problem expanded upon.

W5: page 5: I suggest the authors introduce parasol and midget cells and how their behavior differs, since the typical ICLR audience will not be familiar.

W6: Eq 1: Should it be T or T-1? Is it sample of population variance?

W7: Eq 1: clarify how Var[y] was obtained.

W8: page 5: how were the sets A, B obtained? Are there confidence intervals on these estimators? Is eq 2 an unbiased estimator? Clarify if this estimator is used in Cadena 2017. It is unclear whether the average is across time. Clarify over what the expected value is taken.

W9: will source code and the data be provided to make this reproducible?

W10: if you connect a live camera as input, how well would the algorithm adapt to changes in the sensor gain and shutter speed?

W11: page 6: 180 frames are used. At 30fps this is 6 seconds. This means it takes 6 seconds for adaptation to happen? Could this be a drawback in computer vision algorithms?
clarity,_quality,_novelty_and_reproducibility: The paper is well written. It is an interesting, though somewhat incomplete effort at biologically plausible computer vision. It is not clear whether data and source code will be provided to make this reproducible.
summary_of_the_review: Overall, while this is a well written paper, I feel the authors need to do a better job adapting it towards the typical ICLR audience. Currently it is addressed more towards a computational neuroscience audience. As indicated on page 6, I do not feel comfortable accepting to ICLR a paper that has only been tested with white noise input. The effort in appendix C to compare it with actual algorithmic performance is incomplete and needs more work
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work Introduces a new photoreceptor-inspired CNN layer that mimics adaptation to light level of biological cells in the retina.  While the direct focus of the work is on improving biological modeling, a secondary motivation is to develop approaches that will improve performance of computer vision algorithms. The novel layer is a biophysically-inspired adaptation model that is implemented as a differentiable layer in Keras.  This method outperforms CNN baselines on settings with variable lighting conditions.

Two out of three reviewers initially leaned negative, but one of the reviewers adjusted their score in light of author responses.  Reviewer b3Ty initially gave a score of 5 and found the paper clear and technically sound in terms of the computational neuroscience contribution.  Author responses prompted this reviewer to raise their score to a 6, but they remained underwhelmed by the results and didn't feel it would be impactful.  Reviewer TU1Q found the core innovation sensible and the empirical validation sufficient, but noted the technical and conceptual contributions are limited from an ML perspective.  Reviewer Rcws initially read the paper primarily from the perspective of a bio-inspired solution to a computer vision problem, rather than as a computational neuroscience effort in its own right.  In response to this, the authors made a reasonable request for the AC to clarify the suitability of their submission.  I replied both to the authors and to the reviewer, indicating my agreement with the authors that the topic matter is suitable for an ICLR paper.  

However, in the authors' "Response to the main concern raised by all the reviewers" as well as in a message to the AC, the authors oversimplify the more nuanced concerns of the reviewers, incorrectly asserting that all reviewers are improperly evaluating the paper based on topic. For example, both reviewers Rcws and b3Ty completely reasonably raised a concern about the evaluation only being performed on white noise stimuli.  While the authors replied that white noise data is standard in neuroscience, I don't really find this acceptable given that the authors had considerable flexibility and collected their own data for this paper (i.e. "monkey data were collected by us as part of this study"). Another paper that was itself accepted to ICLR back in 2017 entitled "Multilayer Recurrent Network Models of Primate Retinal Ganglion Cell Responses" also looks at modeling primate retinal cells and involves presentation of natural images to the retina, further indicating the authors' claims that white noise is sufficient isn't entirely consistent with what has been previously studied. That paper focuses on RNNs to capture adaptation phenomena, which also points to the possibility that RNNs of some sort might be an alternative to the biophysically inspired adaptation layer (a baseline not considered in the present work).  

In addition, there remains a bit of a confusingly split motivation to the paper as it was written.  Two reviewers pointed out in their reviews some version of confusion as to whether the approach introduced here for modeling biology is a good idea for computer vision.  Reviewer b3Ty noted that applications are only covered in the appendix and wondered why not train models across multiple light levels.  Reviewer Rcws pointed out that for computer vision applications, high-dynamic range data could be a more practical choice for dealing with light variation.  The authors responded in multiple places that this work is focused on predicting neural data as the application.  While, as noted previously, I agree that neural data prediction is a completely valid problem space, I believe the authors invited this confusion by making claims that their method may be useful for computer vision applications.  For example, the abstract states, "Such processing capabilities are desirable in many settings, including computer vision systems that operate in dynamic lighting environments like in self-driving cars, and in algorithms that translate visual inputs into neural signals for use in vision-restoring prosthetics."  It isn't clear to me that self-driving cars and retinal prosthetics are really facing the same challenge.  And the discussion + appendix further muddle the message by making claims about the CV-ulitity of the approach with only weak support.  Indeed the author "Response to the main concern raised by all the reviewers" doubles down on the claims of utility for CV applications while at the same time claiming the paper is being unfairly evaluated.
justification_for_why_not_higher_score: I think the core biophysically-inspired layer is a reasonable contribution.  I think there are some legitimate concerns about this contribution in its own right raised by the reviews as well as questions about the impact of this innovation even within computational neuroscience.  On top of this, the authors don't seem to appreciate that their paper actually does make claims (even if they are secondary) about the utility of their innovation for CV settings that multiple reviewers legitimately question in their reviews.
Decision: Reject