A new photoreceptor-inspired CNN layer enables deep learning models of retina to generalize across lighting conditions
Authors: Saad Idrees
Greg D Field
Frederick Rieke
Joel Zylberberg
Venue: Submitted to ICLR 2023 (2023)
Keywords: retina model
photoreceptor model
bio-inspired artificial vision
retina predictor
dynamic environments

Abstract: As we move our eyes, and as lighting changes in our environment, the light intensity reaching our retinas changes dramatically and on multiple timescales. Despite these changing conditions, our retinas effortlessly extract visual information that allows downstream brain areas to make sense of the visual world. Such processing capabilities are desirable in many settings, including computer vision systems that operate in dynamic lighting environments like in self-driving cars, and in algorithms that translate visual inputs into neural signals for use in vision-restoring prosthetics. To mimic retinal processing, we first require models that can predict retinal ganglion cell (RGC) responses reliably. While existing state-of-the-art deep learning models can accurately predict RGC responses to visual scenes under steady-state lighting conditions, these models fail under dynamic lighting conditions. This is because changes in lighting markedly alter RGC responses: adaptation mechanisms dynamically tune RGC receptive fields on multiple timescales. Because current deep learning models of the retina have no in-built notion of light level or these adaptive mechanisms, they are unable to accurately predict RGC responses under lighting conditions that they were not trained on. We present here a new deep learning model of the retina that can predict RGC responses to visual scenes at different light levels without requiring training at each light level. Our model combines a fully trainable biophysical front end capturing the fast and slow adaptation mechanisms in the photoreceptors with convolutional neural networks (CNNs) capturing downstream retinal processing. We tested our model’s generalization performance across light levels using monkey and rat retinal data. Whereas conventional CNN models without the photoreceptor layer failed to predict RGC responses when the lighting conditions changed, our model with the photoreceptor layer as a front end fared much better in this challenge. Overall, our work demonstrates a new hybrid approach that equips deep learning models with biological vision mechanisms enabling them to adapt to dynamic environments.

Under review as a conference paper at ICLR 2023
A NEW PHOTORECEPTOR-INSPIRRD CNN LAYER EN- ABLES DEEP LEARNING MODELS OF RETINA TO GEN- ERALIZE ACROSS LIGHTING CONDITIONS
Anonymous authors Paper under double-blind review
ABSTRACT
As we move our eyes, and as lighting changes in our environment, the light inten- sity reaching our retinas changes dramatically and on multiple timescales. Despite these changing conditions, our retinas effortlessly extract visual information that allows downstream brain areas to make sense of the visual world. Such process- ing capabilities are desirable in many settings, including computer vision systems that operate in dynamic lighting environments like in self-driving cars, and in al- gorithms that translate visual inputs into neural signals for use in vision-restoring prosthetics. To mimic retinal processing, we first require models that can pre- dict retinal ganglion cell (RGC) responses reliably. While existing state-of state-of-thhe-rt -hrt deep learning models can accurately predict RGC responses to visual scenes under steady-state lighting conditions, these models fail under dynamic lighting condi- tions. This is because changes in lighting markedly alter RGC responses: adapta- tion mechanisms dynamically tune RGC receptive fields on multiple timescales. Because current deep learning models of the retina have no in-built notion of light level or these adaptive mechanisms, they are unable to accurately predict RGC responses under lighting conditions that they were not trained on. We present here new deep learning model of the retina that can predict RGC responses to visual scenes at different light levels without requiring training at each light level. Our model combines fully trainable biophysical front end capturing the fast and slow adaptation mechanisms in the photoreceptors with convolutional neural networks (CNNs) capturing downstream retinal processing. We tested our model's general- ization performance across light levels using monkey and rat retinal data. Whereas conventional CNN models without the photoreceptor layer failed to predict RGC responses when the lighting conditions changed, our model with the photoreceptor layer as front end fared much better in this challenge. Overall, our work demon- strates a new hybrid approach that equips deep learning models with biological vision mechanisms enabling them to adapt to dynamic environments.
INTRODUCTION
A key problem in visual neuroscience is to generate models that accurately predict how neu- rons will respond to visual stimuli. Along with their role in basic neuroscience, these models have applications in prosthetic devices and can form the basis for bio-inspired computer vision systems that aim to mimic the impressively robust functions of the human visual system. Use of machine learning models for such applications neuroscience has become increasingly ubiquitous given their strong performance in computer vision applications like object recognition (Chollet] 2017] Simonyan & Zisserman 2015) Krizhevsky et al. (017). For example using convolutional neural
networks (CNNs) to predict responses of neurons in visual cortex (Kindel et al.. 2019 Cadena et al.) 2017) and retina (McIntosh et al., 2016 Tanaka et al. 2019 Yan et al. 2022 Goldin et al.) 2022) to visual stimuli. We focus here the retina. Under carefully controlled experimental conditions, and with constant lighting conditions, CNN models can predict responses of retinal ganglion cells (RGCs, the "output" cells of the retina, whose axons form the optic nerve) visual stimuli with high accuracy (McIntosh et al. 2016). In natural vision, however, lighting conditions are highly dynamic: the amount of light falling on the retina change by several orders of magnitude mul-

---

Under review as a conference paper at ICLR 2023
tiple timescales. For example, light imput can change locally region of the retina by an order of magnitude in less than a second following rapid eye movements such as saccades; global light levels may fluctuate by order of magnitude every few seconds on cloudy day, as clouds pass between the observer and the sun. These changes in light level substantially alter RGC responses to visual
stimuli (Tikidj-Haamburaall 2013), and pose the us-of-yet-unansweer responses under these more challenging conditions.
al., 2015 |udd al. 22200: Idrees et al.] 2020 2022 Farrow al.,
question of whether CNNs can accurately predict RGC
To answer this question, we first tested the ability of conventional CNN models (i.e., the Deep Retina model of (Meintosh et al. 2016)) to predict RGC responses in the presence of changing light levels. These CNNs could accurately predict RGC responses when tested under the same lighting conditions under which they were trained but were unable to accurately predict responses under different lighting conditions.
To improve the performance of machine learning (ML) models in predicting RGC responses under varied lighting conditions, we created a new type of convolutional neural network layer front- end for CNNs. This fully trainable input layer mimics the transformation of light into electrical signals at the retina's photoreceptors, including the adaptation mechanisms that modulate photore- ceptor response sensitivity and kinetics and by doing so enable the retina to operate under a wide and dynamic input range. This in-built adaptation allows the model to adapt to changing light levels locally, at each pixel location in the input, the same way our retinas do. CNNs with this new input layer could generalize test lighting conditions very different from the lighting conditions under which they were trained.
We anticipate that our new photoreceptor-inssired CNN layer could have several significant impacts in various fields of vision science. First, our model can be used directly in vision-restoring prosthet- ies by enabling the translator algorithms to function under diverse lighting conditions, and thereby improve the operating range of these prosthetics. Second, our model can be used by visual neurosci- entists to investigate dynamic retinal computations. Using the model as front-end for algorithms that predict the responses of neurons in visual cortex, will enable those cortical models to operate under more naturalistic and more varied lighting conditions. Finally, because their local (pixel-by- pixel) adaptation to changing light levels can filter out image changes due to changing lighting and shadow, our photoreceeporrinspieee CNNs could have substantial application for computer vision systems operating in outdoor environments. There, lighting changes and shadows are known to confuse existing object detection and recognition algorithms (Yarron & Tynan 2016 Levin 2018] Janai et al., 2020; Gomez-Ojeda et al. 20155 Kolaman et al. 2019) they do not naturally separate image changes due to local lighting variations from image changes due to changing object content. Here, photoreceptor-inspiee
CNN models can discount disruptive events in video stream, such as sudden large changes in the amount of light reflected off' a tracked object due to shadows or changes in intensity of the light source.
2
RELATED WORK
2.1 DEEP LEARNING MODELS OF THE VISUAL SYSTEM
Deep learning approaches have become increasingly popular in modeling the visual system as they can describe neural responses to visual scene markedly more accurately than dinear-nonlings (LN) models. The current state-of-the-art retina predictor, called Deep Retina (McIntosh et al., 2016). is based on a 2-layer CNN. This model takes as input movie from which spatio-temporal features are extracted by the CNN layers, and outputs the spike rate of retinal ganglion cells. This model, when trained to predict tiger salamander retinal ganglion cell responses to spatiotemporal white noise stimuli, outperformed other models such as LN and the generalized linear models (GLMs). Such CNN based models of the retina have been used for describing responses to natural stimuli (MeIn- tosh et 2016) Tanaka et al.[2019 (019)) and for explaining the underlying neural computations that lead to RGC responses (Tanaka et al. 2019) Yan et al., 2022 Goldin et al., 2022). CNN based models can also better capture the activity of other visual areas such as the primary visual cortex (V1), than standard LN models (Kindel et al. 2019, Cadena et al. 2017). While these models have been shown to reliably predict responses to visual stimuli having similar image statistics as the training set, it is unclear whether they will generalize to stimuli with different image statistics, such as different ambient luminance levels.
2

---

Under review as a conference paper at ICLR 2023
2.2 HOw THE VISUAL SYSTEM FUNCTIONS DYNAMIC AMBIENT LIGHT CONDITIONS
The light falling on our retina spans over twelve orders of magnitude and can change at different timescales: local changes retina can occur within a few hundred milliseconds as we make eye movements such as saccades or by shadows cast on a tracked object whereas slower global changes can occur cloudy day. While extracting features under such dynamic conditions is still chal- lenge for artificial vision systems (Janai et al.) 2020 Gomez-Ojeda et al. 2015 Kolaman et al.)
2019), our retina effortlessly extracts reliable information for the downstream areas make sense of our visual world (Hikbuj- et al.
2013 Idrees et al. 2020: 2022 Goldin et al. 2022
Schreyer & Gollisch Hosoya et al., 2005; Kastner & Baccus 2011). This is achieved part by signal processing done by photoreceptors (Angueyra et al., 2021; Yu et al. 2022 Clark et al.) 2013), the very first cells in the visual processing hierarchy. They adapt strongly match their neu- ral sensitivity to the ambient light conditions. This adaptation itself is highly dynamic, and occurs at multiple timescales. As a result, the photoreceptor dynamics affects both the gain and the kinetics with which light inputs are converted into electrical signals in the retina. This adaptation is also highly local: each photoreceptor can adapt independently. Biophysical models of phototransduction in photoreceptors that translate light input into electrical signals are well established (Lamb & Jr. 1992 Rieke Baylor 1998| Invergo al. 2014), and faithfully capture a range of responses. A more recent model by Angueyra et al. (2021) incorporates adaptive changes in the photoreceptor output in conditions where light falling on the retina changes by large magnitudes rapidly. We im- plemented this signalling cascade
fully-trainable neural network layer, and used it as the first layer to deep CNNs to predict RGC responses to visual stimuli.
METHODS
3.1 PHOTORECEPTOR LAYER
Our new photoreceptor CNN layer builds upon biophysical model of the phototransduction cascade by Angueyra et al. (021) that in-
Stimulus 31,000
Model photoreceptor output
corporates the various feedforward and feed- back molecular processes that convert photons into electrical signals, and therefore faithfully captures the photoreceptors adaptation mecha- nisms (Fig. 1).
20,000
10 PA
10 000
R "receptor's s'
0.15
4,000
2,000
We implemented this biophysical model as fully-trainable neural network layer in Keras (Chollet et al., (2015) in Python. This layer con- verts tire-narying light intensity signal in units of receptor activations per photoreceptor per second (R (R*receptor 's 11 into photocurrents (pA) by solving the differential equations that underlie the phototransduction cascade (bio- physical model from Anguuyra al. T02T)> produced in Appendix A). The photoreceptor layer has twelve parameters corresponding to the twelve interaction parameters of the bio- physical model, all of which can (if needed) be trained through backpropagation using the Keras and TensorFlow package in Python. We fixed 8 parameters to their values from photore-
1,080
Figure 1: Dynamic gain adjustment in the pho- toreceptor model. Left column: step stimuli at two light levels: 10,000 R*receptor (topp) and 1,000 R Reeeeppor (bottomm Right col- umn: model photoreceptor output to the stimuli on left. The difference in response to two small intensity steps at the lower light level (red arrow in bottom traces) was similar to two large inten- sity steps higher light level (red arrow in top traces), showing that the model photoreceptor dy- namically adjusts its sensitivity (Angueyra et al., 2021).
(top)
ceptor data fits. These parameters relate to molecular processes such as the concentration of cyclic guanosine monophosphate (cGMP) in darkness and other factors governing cGMP conversion into current, and are similar for both rod and cone photoreceptors. The remaining parameters, set to be trainable, include the rate and the gain factors of the molecular processes such as the opsin activa- tion, the phosphodiesterase activation and decay, and the rate of calcium removal. These trainable parameters differ across photoreceptor types. In the current version of our model, the photoreceptor
3

---

Under review as a conference paper at ICLR 2023
parameters shared by all the input pixels: Le., the conversion of each pixel into photocurrents only depends on that pixel's 's previous values and not on the values of the other pixels.
3.2 MODEL ARCHITECTURE
We used our new photoreceptor layer at the input stage of a deep CNN (Fig. 2 The general ar- chitecture of our deep CNN was similar to Deep Retina (Mclntosh al. 2016). which we describe briefly here. In Deep Retina, the model input was a movie which passed through a convolutional layer operating in both space and time, followed by another convolutional layer that only operated in the spatial dimension. The output of the final dense layer was the predicted spike rate of RGCs. Our model differs from Deep Retina in two key ways: 1) the addition of the photoreceptor layer at the input stage; and 2) the use of batch normalization layers. We performed some experiments on models in which the photoreceptor layer was removed, and some in which the normalization layers were removed. These latter experiments provide the closest comparisons to Deep Retina.
Similar to Deep Retina, our model takes as input a movie (180 frames per training example where each frame corresponds to 8 ms) and outputs an instantaneous spike rate for each RGC at the end of that movie segment. To obtain the time series of RGC responses to longer movie stimuli, we feed into the model many 180-frame video samples taken from that longer movie, that correspond to I-frame shifts. Lc., the model receives as inputs frames 1-180, 2-181, 3-182, etc., and outputs RGC responses at the times of movie frames 180, 181, 182, etc.
The photoreceptor layer of our model, converts each pixel the input movie (where each pixel is in R R*receptor"'s" 's into photocurrents (pA), based on the history of the pixel's value. The first 60 time points are then discarded to account for edge effects. The output of this layer is a movie that is 120 frames long, and the same spatial dimensions as the input visual stimuli. This photocurrents movie is then normalized by its mean and variance using a Layer Normalization layer (LayerNorm). The resulting movie is then passed through a conventional 3-layer CNN, where the first layer is a 3D convolutional layer operating in both the spatial and temporal dimensions. The output of the 3D convolutional layer is a 2D image which normalized using Batch Normalization (BatchNorm) and then passed through a rectifying (ReLU) nonlinearity. To down sample the spatial dimensions, we applied 2D max pool operation that took the maximum value over 2x2 patches of the previous layer's output. The subsequent 2D CNN layers are followed by final, fully connected layer that outputs the predicted spike rate for each RGC in the dataset. We refer to this model as the Chotoreccepor-CNN model, or PR-CNN model for short.
Model parameters were optimized using Adam (Kingma & Ba 2015), where the loss function was given by the negative log- likeliood under Poisson spike generation. Photoreceptor parameters were initialized to their known values from fits to experimental data. This initialization helped stabilize the joint training of the photoreceptor model with the downstream layers. The network layers were regularized with a L2 weight penalty at each layer, including the photoreceptor layer, to prevent loss of information and be more robust outliers. In addition, Lı penalty was applied to the output of the dense layer because the neural activity itself is relatively sparse and L1 penalties are known to induce sparsity. The number of channels each CNN layer and the filter sizes were optimized by a grid search for each model type. These details therefore differed for the models with and without the photoreceptor layer or normalization layers. Learning rates were set to 0.001.
3.3 RETINA ELECTROPHYSIOLOGG
To develop and test our models of the retina, we used data from ex vivo macaque monkey and rat retina electrophysiology experiments. The rat data was previously published by Ruda et al. (2020). whereas the monkey data were collected by us as part of this study. The basic experimental approach was the same in both sets of experiments: the retina of an euthanized animal was extracted and placed with the RGC-side down on a multi-electrode array (MEA).
We recorded monkey RGC activity to the same stimuli at three different mean light levels, each differing by log unit: 0.3 R ""receptor"' -I. Rereceptor -1 -s- and 30 R*receptor''s light levels fall under the scotopic regime, where mostly rod photoreceptors contribute to vision. The stimulus was a binary white noise checkerboard movie with 39 pixels x 30 pixels, where each pixel edge corresponded approximately 140 jmm on the retina surface. The movies across
These
4

---

Under review as a conference paper at ICLR 2023
the three light levels only differed in their mean pixel values which were 0.3 R "eceeptor"'s 3 R*receptor 's 's'' and 30 R Rrrecepoor 's In this work, we used a subset of 37 recorded RGCs that were classified high quality units after spike sorting. This subset contained both the midget RGCs (that process fine details given their small receptive fields) and parasol RGCs (sensitive to motion given their large receptive fields) of ON and OFF types.
The rat experiments Ruda et al. 2020) were performed at two light levels differing by 4 log units: I Rereeeptor -s (scotopic light level where mostly rod photoreceptors contribute vision) and 10,000 R R*reeepoor (photopic " to vision). The white noise checkerboard movie in these experiments had 10 pixels 11 pixels, with each pixel edge corresponding to approximately 252 ym on the retina. In this work, we used data from two rat experiments: subset of 61 RGCs from Retina A and a subset of 58 RGCs from Retina B that were classified as high quality units after spike sorting. This subset contained OFF brisk sustained and OFF brisk transient RGC subtypes.
light level where cone photoreceptors predominantly contribute
3.4 MODEL TRAINING AND EVALUATION
Models were trained using 40 minutes of neural recordings and associated stimulus movies for least 50 epochs with early stopping based on validation dataset separate from the test dataset used for measuring the model's performance.
Trained models were then evaluated using the held out test dataset that contained multiple repeats of 5-10s of binary white noise checkerboard movie not seen during the training, along with the associated neural recordings. We quantified the model performance with the fraction of explainable variance in each RGC's response that was explained by the model (FEV). This quantity (Eq. [ was calculated as the ratio between the variance accounted for by the model and the explainable variance (denominator in Eq. D Such metrics to quantify how well model predicts neural data have been used in previous studies like Cadena et al. (2017). We calculate FEV as
FEV- = 1- - + Ei= (v/ )1 o2 noise
(1)
Var(y 2 noise
where,
2 noise [(, v(a)
(2)
yd and 3 are the observed spike rate of RGC calculated as average across set of repeats A and set of repeats B respectively. The sets A and were obtained by randomly splitting the total number of repeats into two. Út represents the predicted spike rate by the model at time bin t. The explainable variance (denominator in Eq. [ is the variance of each RGC attributable to the stimulus, computed by subtracting an estimate of the observed noise from the variance across time (Eq. 3 in the actual RGC's responses, calculated as
T 2 1=1
1
Varly
T (
(3)
where ÿ is the the observed spike rate A averaged across time. In all neural data sets we con- sidered, the number of trials was large and hence the estimated noise variance was quite low. As result, our FEV values are quite similar what is obtained using the usual fraction explained vari- ance calculation, which does not correct for unexplainable noise. By definition, FEV can be negative the prediction error is larger than the variance in the actual responses. We report each model's performance across all RGCs as the median FEV across the set of RGCs. For ease in interpretation, we present FEV as a percentage throughout our results.
4 RESULTS
4.1 EXISTING DEEP LEARNING MODELS OF THE RETINA DO NOT GENERALIZE WELL ACROSS LIGHT LEVELS
We first tested whether existing deep learning models of the retina could generalize across light levels. To do this, used the Deep Retina architecture (MeIntosh al. 2016) (i.e., the architecture
5

---

Under review as a conference paper at ICLR 2023
RGC H1 M
RRC #2 MMn
30
180
120
18
ROC AN
16
39
channels
pixels
White noise movie ftames
180 frumes (-1.5 s) 120 frames
200
Layer shape
Convolutional BatchNorm ReL.U
Photorceeptor layer LayerNorm
Max pooling
Dense Softplus
Figure 2: PR-CNN model architecture. The photoreceptor layer at the input takes 180 frames of movic as a single sample and converts each pixel (green color) into photocurrents. The photocurrents then passed to the first convolution layer that has 8 channels where each kernel (orange color) operates over spatial dimensions of 9 pixel x 9 pixel and temporal depth 120 frames. The result normalized through Batch Normalization layer and then passes through Rectified Linear Unit (ReLU). The output is then max pooled to reduce the spatial dimension and then passes through another two convolution layers. The model output layer dense layer that has N units based on the number of RGCs in the dataset. It converts previous layer outputs into RGC spiking output (blue traces). By traversing through the input movie 180 frames at time, an entire time series of RGC responses is achieved.
in Fig. with the photoreceptor layer and normalization layers removed), and re-optimized the numbers of convolutional layers, filters in each layer, and the filter sizes with a grid search. This model had three CNN layers followed by dense layer with 8, 16 and 18 channels in each of the CNN layers, respectively, and filter sizes of 9x9, 7x7 and 5x5 respectively.
We trained this model predict monkey RGC responses using the training dataset that contained binary white noise movies at 30 **receptor 1 and 3 R in the monkey data set. We then used the test data receeptor 1s--1 the two higher light levels including the testing light level (0.3 R "receptoor Layer Normalization layer at the imput normalized each pixel in input movie sample by the mean and variance across all pixels in the movie. The model could reliably predict responses to stimuli 3 R Prreceptor-s--1100
evaluate its performance at all light levels not used during the training (Fig. (p) A
the two training light levels, 30 recceptor
3p. columns 1-2),
with FEV (medion + 95%c.i.) of 84% + 11% and 78% + 3% respectively (Fig. 41). However, this model performed poorly at the testing light level (0.3 ##ecceptor -'s-1 (Fig. 3p, column 3)
explaining only 16% + 15% of the variance in the population data (Fig. 44)
To overcome internal covariate shift and stabilize the network, we included batch normalization lay- prior to each activation function in the model. This marginally increased the overall performance of the model at each light level (Fig. 4p). The generalization performance at the test light level was however still quite low with FEV of 24% + 15% (Fig. 4p).
4.2 PHOTORECEPTOR FRONT END ENABLES DEEP LEARNING MODELS OF RETINA TO GENERALIZE ACROSS LIGIIT LEVELS
To test whether our new PR-CNN model was better at generalizing across lighting conditions, we repeated the same task as above, i.e., trained the model end-to-end including the photoreceptor layer to predict monkey RGC responses to the binary white noise movie at the two higher light levels, 30 R receeptor - R*receptor -s- Similar to the model without the photoreceptor layer, this model could also reliably predict responses to test stimuli at the two training light levels (Fig. 4 columns 1-2 and Fig. 44). Importantly, this model was able to explain 54% + 11% of the variance in responses at light level lower than those at which it was trained (Fig, 4 column 3 and Fig. 4c). 4b) This a notable improvement over the model without the photoreceptor layer (but with the normalization layers included), which only explained 24% + 15% of the variance (Fig. 4p).
Next, we repeated this analysis with all different combinations of training and testing light levels: each time, trained the model data from two light levels, and then evaluated the model test

---

Under review as a conference paper at ICLR 2023
Stimulus
30 R receeporr 's
3 R "receptor" 's
0.3 R"receptor
Time
CNN model (similar Deep Retina)
RGC predicted response at respective light levels
RGC real response
spike is (normalized)
mm m
FEV 94%
FEV 78%
in Manlar
FEV -2%
0.5
Photerceeptor-CNN CNN model
Wom Momer Mumlan
FEV = 84%
FEV= = 88%
FEV= 55%=58%
Figure 3: Example monkey retinal ganglion cell (RGC). a. White noise movie at three differ- ent light levels (columns). Models were trained on data 30 R*receptor "s 1) and 3 RR*recepoor - ''s (columm 2). b. Actual spike rate of an example RGC (gray) and those predicted by CNN model similar to Deep Retina (colored) each light level in (columns). FEV values above each trace quantifies the performance of the model for this RGC at the corresponding light levels. Same as but for our Photorecceporr-NN. model.
data from different light level (Fig. 4t.. For all combinations of training and testing light levels, the PR-CNN model performed better at generalizing to new light levels than the conventional CNN model without the photoreceptor layer (but with normalization layers included). The difference was smallest for the case where the testing light level (3 R*teceptor -ss-) was intermediate between the training light levels (30 #*rccppoor -- 0.3 R*receptor 1s-1) (Fig. 4d, column 3). This suggests that while standard CNN based models can generalize to conditions that can be interpolated from training conditions, they are not as well equipped to generalize conditions that have to be extrapolated. Our new PR-CNN model leverages the biophysics of photoreceptor adaptation to do much better at this challenging extrapolation task.
4.3 GENERALIZATION ACROSS EXTREMELY DIFFERENT LIGHT LEVELS
Having observed that our PR-CNN model can generalize well between light levels that differ by 1-2 orders of magnitude, we next wondered whether it could also generalize well across more ex- treme variations in lighting. To answer this question, trained our model to predict responses at relatively bright (photopic) light level (10,000 "#"rceptor--s- predominantly contribute to vision and evaluated the model's generalization performance at much dimmer (scotopic) light level (1 R recceptor--s- where only rod photoreceptors are active.
where cone photoreceptors
For comparison, we first performed this experiment with the conventional CNN model including normalization layers (as in Fig. 4b). That model failed (FEV of -52% + 9%) to predict responses recepptor
level after having been trained at the 10,000 R R#receptor level (Fig. cl.c). Our proposed PR-CNN model, however, did notably well (Fig. 5p), achieving FEV of 54% 8% on this task (Fig. 5. For this experiment, we trained PR-CNN model at then replaced that model's photoreceptor parameters (which corre- spond cone cells at this light level) with those corresponding rod cells, as explained below and in Fig. B.I in Appendix B
the R$
10,000 R rs
First, we trained our model end-to-end to predict rat RGC responses (from Retina A) to stimuli at the bright (photopic) light level (10,000 R*receptor "sreceptor-ss-)) This led to estimate of photoreceptor and CNN parameters for the cone pathway. We then estimated parameters reflecting rod photoreceptors by re-training the model at the dim (scotopic) light level (1 R Receeepoor 'ss ss')) while keeping the CNN weights and biases fixed to their already-leart values. Thus, the retraining altered only the photoreceptor parameters based the data from Retina A.

---

Under review as a conference paper at ICLR 2023
CNN (similar Deep Retina)
b. CNN with Batch Norm
PhhtoorecptorrCC... Performance all test light levels
CNN with norm layers Photorecetor-CNN
FEV
Median 84%
78%
16%
84%
84%
24%
82%
80%
54%
24%55%%
38%
47%
53%
55%
205% et.
115
5%
15%
14%
3%
15%
8%
75
11%
15% 11%
9%
9%
6%
8%
100
0,009
0
0.7
60
40
FEV
0
-20
N 57
-40
RGCS
Light
30R*
3R
0.3R*
30R* 3R
0.3R*
30R* 3R* 0.3R*
Ttaining Testing 0.3R
30R 3R
level
Trainin
Tessingg
Figure 4: Model performance comparisons. Performance on the test data set for (a) a CNN model that had similar architecture Deep Retina, (b) CNN model of but with normalization layers included, and (c) our proposed photoreceptor-CNN model. a-c. Each model was evaluated at three light levels (labelled below each box plot): 30 R*receptor -s- 3R R*receptor "oseepttrr-s--2 --s--at which the models were trained at and 0.3R*receptor during the training. The box plot at each light level shows the distribution of FEVs across 37 monkey RGCs. Numbers at the top of each box plot are the median FEVs + 95%c.i.. d. Performance of the CNN model with normalization layers (blue color; same model as b), and the photoreceptor- CNN model (green color; same model as in с) at all combinations of training and test light levels. For each column, the legend below the box plot panel shows the two light levels the models were trained at (black outline) and the third light level at which it was tested. The box plots show the distribution of FEVs at this testing light level. Testing light levels were 0.3 R*receptor" (srluuu (columm
(testing condition) which the models did not see
1), R receptor s (column 2), and 30 R*receptor s (columm 3). p-values were calculated by performing two-sample t-test on the FEV distributions from the CNN and PR-CNN model at each testing light level.
We then trained the PR-CNN model to predict RGC responses of another rat's retina (Retina B) to stimuli the bright light level. Here, we fixed the photoreceptor parameters the values learned from Retina A at the bright light level. We then exchanged those photoreceptor parameters with the ones learned from Retina A at the low light level. In this manner, we took the model where the CNN weights were learned on model of Retina B at the bright light level, and simply exchanged the cone photoreceptor parameters for rod parameters. Notably, in this procedure, none of the data from Retina B at the dim light level was used in training the model.
In retrospect, models without inherent knowledge light levels, like the conventional CNN, are likely to fail at this generalization task as response properties of RGCs substantially different at bright (photopic) and dim (scotopic) light levels. For example, rod mediated RGC responses at dim conditions slower than mediated responses at brighter light levels (Ruda al. 2020 Baylor & Fettiplace 1977). This could explain the temporal lag between the predicted and actual response dim light level (Fig. 5a): the model trained bright light levels (10,000 R recepptor --1) learnt faster kinetics of the cone pathway. While some of the differences in RGC responses may arise due faster response kinetics of the cones themselves (Cao et al., 2007 [aylor & Hodgkin 1973 Ingram et al., 20166) SSchnewweis & Schnapf, 1995). the relative contribution of photoreceptors and downstream retinal circuits are not well understood. Our PR-CNN model could be one way to dissect such underlying contributions.
DISCUSSION
Here, we introduced new class of machine learning models for neuroscience applications. The input layer to the model is a novel convolutional layer that incorporates biophysical photoreceptor model to leverage photoreceptor adaptation mechanisms and enable generalization across light lev- els. We found that this model strongly outperforms other CNN architectures, generalizing across light levels, making it potentially desirable tool for many applications, including visual prosthetics.
8

---

Under review as a conference paper at ICLR 2023
a. CNN model (including normalization layers)
c. Model performance FEV
FEV -1552%
RGC real response RRGC predicted response
Median -52%
54%
Als we WWM.
295%
9%
8%
sasiiii (nomalizzal)
100
aapikos is
M
Testing light level: 1 R receeptor'ss 's
FEV (%)
500 mi
b. Photoreceptor-coo CNN model
RGC real response RGC predicted response
-50-
55
FEV= 80%
-100
RGCs
m
CNN
Photoreceptor-
(with norm layers)
CNN
Figure 5: Generalization across extreme light levels. a-h. Rat Retina B example RGC responses at scotopic light level (1 (Prrecppoor-s--) Actual responses shown in gray. Predicted responses generated by the CNN model (trained at photopic level) without the photoreceptor layer (a, blue) and with photoreceptor layer(b, green). c. Box plots show the distribution of FEVs across RGCs (N = 55 RGCs, Retina B) for the CNN model with normalization layers (blue) and for the CNN model with the photoreceptor layer added, when the models were trained at photopic light level and tested at scotopic light level.
The photoreceptor model while complex, has parameters that map directly onto the biology, pre- senting the scientific community with opportunity to investigate visual processing of dynamic stimuli. For example, how much of the adaptation in the retina can be accounted for by photore- ceptors alone? And how much does the sensitivity of RGCs different stimulus features vary with lighting conditions? Moreover, models of retina that can leverage the power of deep learning to model multiple ganglion cells simultaneously, together with biologically interpretable components, could be used to dissect the relative contributions by different components. From a wider neuro- science perspective, our approach demonstrates the power of embedding neural dynamics in deep learning models of the brain: biophysical layers can match the sensitivity to prevailing input condi- tions that change dynamically whereas the downstream layers of the model can progressively extract relevant features from those dynamically adapting input stages.
The wider scientific community might not be interested in biological investigations, but could still benefit from the photoreceptor's light adaptation mechanisms to overcome challenges in computer vision. To this end we have developed an adaptive convolutional layer based upon simplified phenomenological photoreceptor model (Appendix C.T.) This model could potentially enable bet- ter computer vision systems for applications where lighting is highly dynamic, such as the object detection and recognition systems autonomous vehicles. Here, pixel-wise local luminance adap- tation (from using our ciaptive-convolutionn
layer as an input layer) could help filter out disrup- tive lighting changes induced by changing shadows, that would otherwise cause errors in object recognition and/or detection (Janai et al.) 2020; Gomez-Ojeda et al.] 20155] Kolaman et al.) 2019). A proof-of-concept that a pootoreceptoo-inspire convolution layer, Adaptive-Cony may be better than conventional CNNs at this task, is presented in Appendix [.T
Future work includes overcoming some of the limitations of this work. For example, the manual switch from cone to rod photoreceptors that is required for model trained at cone light levels to generalize to rod light levels (Fig. 5) In addition, the photoreceptor parameters learnt may also reflect some of the adaptive mechanisms in downstream retinal circuits. While this is not a concern when developing models that perform better at different light levels, it may limit some biological inferences. One way to overcome this limitation would be by adding layers downstream of the photoreceptor layer that incorporate other known biological sources of adaptation the retina.
REPRODUCIBILLITY STATEMENT
The photoreceptor-CNN model and Adaptive-Con Conv model can be downloaded from [Redacted for Double- Blind Review]. All retina clectrophysiology data used in this study can be made available upon reasonable request.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
All clectrophysiology experiments were performed accordance with the guidelines of Institutional Animal Care and Use Committee at [Redacted for Double-Bling Review].
REFERENCES
Juan M Angueyra, Jacob Baudin, Gregory W Schwartz, and Fred Rieke. Multiple time scales of adaptation allow cones to encode the inputs created by visual exploration of natural scenes. bioRxiv, pp. 2021.02.13.431100 2021.
D. A. Baylor and R. Fettiplace. Transmission from photoreceptors ganglion cells in turtle retina. The Journal of Physiology, 271(2):991-224, oct 1977.
D. A. Baylor and A. L. Hodgkin. Detection and resolution of visual stimuli by turtle photoreceptors. The Journal of Physiology, 2:4(1):163 oct 1973.
Santiago Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas $ Tollia,, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque V1 responses to natural images Author summary. Plos Computational Biology, pp. 1-28, 2017.
Dingeai Cao, Andrew J. Zele, and Joel Pokorny. Linking impulse response functions to reaction time: Rod and cone reaction time data and computational model. Vision Research, 47(8):106) 1074, apr 2007.
François Chollet. Xception: Deep learning with depthwise separable convolutions. Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-Jamuary: 1800-1807, 2017.
Francois Chollet et al. Keras, 2015. URL httpt/pps//ggttuuu..m/////cellll
Damon A. Clark, Raphael Benichou, Markus Meister, and Rava Azeredo da Silveira. Dynamical Adaptation in Photoreceptors. PLoS Computational Biology, 9(11), 2013.
Karl Farrow, Miguel Teixeira, Tamas Szikra, Tim J. Viney, Kamill Balint, Keisuke Yonehara, and Botond Roska. Ambient illumination toggles a neuronal circuit switch in the retina and visual perception at cone threshold. Neuron, 78(2):325-333 apr 2013.
Matías Goldin, Baptiste Lefebvre, Samuele Virgili, Mathieu Kim Pham Van Cang, Alexander Ecker, Thierry Mora, Ulisse Ferrari, and Olivier Marre. Context-dependent selectivity natural images in the retina. Nature Communications, 13(1):5556, 2022.
Ruben Gomez-Ojeda, Manuel Lopez-Antequera, Nicolai Petkov, and Javier Conzalez-Jimenae Training Convolutional Neural Network for Appearance-Inven Invariant Place Recognition, pp. 1- 9, 2015.
Toshihiko Hosoya, Stephen a Baccus, and Markus Meister. Dynamic predictive coding by the retina. Nature, 436((747):71 jul 2005.
Saad Idrees, Matthias P. Baumann, Felix Franke, Thomas A Münch, and Ziad M Hafed. Perceptual saccadic suppression starts in the retina. Nature Communications, 11(1):1977, dec 2020.
Saad Idrees, Matthias phillipp Baumann, Maria M. Korympidou, Timm Schubert, Alexandra Kling, Katrin Franke, Ziad M. Hafed, Felix Franke, and Thomas A. Münch. Suppression without inhi- bition: how retinal computation contributes to saccadic suppression. Communications Biology, 5 (1):692, dec 2022.
Norianne T. Ingram, Alapakkam P. Sampath, and Gordon L. Fain. Why are rods more sensitive than cones? Journal of Physiology, 594(19):515-55466 2016.
Brandon M. Invergo, Daniele Dell' Orco, Ludovica Montanucei, Karl Wilhelm Koch, and Jaume Bertrampetit. A comprehensive model of the phototransduction cascade in mouse rod cells. Molec- ular BioSystems, 10(6): 1481- 2014. ISSN 17422051. doi: 10. 30.55344.
10

---

Under review as a conference paper at ICLR 2023
Joel Janai, Fatma Güney, Aseem Behl, and Andreas Geiger. Computer Vision for Autonomous Ve- hicles: Problems, Datasets and State the Art. Foundations and Trends@ in Computer Graphics and Vision, 12(1-3):1-300 2020.
David B Kastner and Stephen A Baccus. Coordinated dynamic encoding in the retina using opposing forms of plasticity. Nature neuroscience, I 14(10):317-22,, oct 2011.
William F. Kindel, Elijah D. Christensen, and Joel Zyllberberg. Using deep learning to probe the neural code for images in primary visual cortex. Journal of Vision, 19(4):29, apr 2019. ISSN 1534-7362.
Diederik Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. 3rd Interna- lional Conference on Learning Representations, ICLR 2015 Conference Truck Proceedings, pp- 1-15, 2015.
Amir Kolaman, Dan Malowany, Rami Hagege, and Hugo Guterman. Light invariant video imag- ing for improved performance of convolution neural networks. IEEE Transactions on Circuits and Systems for Video Technology,
2019.
Alex Krizhevsky, lya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convo- lutional neural networks. Commuicaations of the ACM, 60(6):84 -00 may 2017.
D Lamb and E N Pugh Jr. A quantitative account of the activation steps involved in phototrans- duction in amphibian photoreceptors. journal of phy, pp. 719-758, 1992.
Sam
Levin.
Uber crash shows 'catastrophic failure' of self-driving technology,
2018.
//wp:///wwwww. URL http:::/ww..hegarrdddaa... an httpa///w.aaoe/eeooooooo////////
w..-ddiiing--ooe car -be deat oht.
Lane T. McIntosh, Niru Maheswaranathan. Aran Nayebi, Surya Ganguli, and Stephen A. Baccus. Deep learning models of the retinal response to natural scenes. Advances in Neural Information 2016.
Processing Systems, pp.
F Ricke and D A Baylor. Origin of Reproducibility in the Responses of Retinal Rods to. Biophysical Journal, 75(4):1836-1857, 1998. ISSN 0006-3495.
Kiersten Ruda, Joel Zylberberg, and Greg D. Field. Ignoring correlated activity causes a failure of retinal population codes. Nature Communications, 11(1):1-15, 2020.
David M. Schneeweis and Julie L. Schnapf. Photovoltage of rods and cones in the macaque retina. Science, 263 1056, 1995.
Helene Marianne Schreyer and Tim Gollisch. Nonlinear spatial integration in retinal bipolar cells shapes the encoding of artificial and natural stimuli. Neuron, 199210)):622110660 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, pp. 1-14, 2015.
Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen A. Baccus, and Surya Ganguli. From deep learning mechanistic understanding in neuroscience: the structure of retinal prediction. (11 2019.
Alexandra Tikidii-Hambuyyan, Katja Reinhard, Hartwig Seitter, Anahit Hovhannisyan, Christo- pher a Procyk, Annette E Allen, Martin Schenk, Robert Lucas, and Thomas a Münch. Retinal output changes qualitatively with every change in ambient illuminance. Nature Neuroscience, 18: 66-74, 2015.
Danny Yadron and Dan Tynan. Tesla driver dies in first fatal crash while using autopilot mode, 2016. URL wtw.-:/wea..a..ee---ooooooo-oaaa tesla-autopi wwt.:pppppoteeeeeeeeee
/www http://aaaa.aaaacooc//ocoooooolo
Qi Yan, Yajing Zheng, Shanshan Jia, Yichen Zhang, Zhaofei Yu, Feng Chen, Yonghong Tian, Tiejun Huang, and Jian K. Liu. Revealing Fine Structures of the Retinal Receptive Field by Deep- Learning Networks. IEEE transactions cybernetics, 52(1):39-50, jan 2022.

---

Under review as a conference paper at ICLR 2023
Zhou Yu, Maxwell H. Tumner, Jacob Baudin, and Fred Rieke. Adaptation in cone photoreceptors contributes to an unexpected insensitivity of primate On parasol retinal ganglion cells to spatial structure in natural images. eLife, 11:1-28, 2022.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX: PHOTORECEPTOR BIOPHYSICAL MODEL
The biophysical model of the photoreceptor that we used is described in Angueyra et al. (2021) Below reproduce the model facilitate the readers.
Input to the model is the stimulus intensity as a function of time (in units of R* 1s- given by (l)m(() in Eq. The output of the model is the photoreceptor outer segment current function of time, (t) given in Eq. 11
dR (1) yStim(!)) RR)
(4)
dt
dP(()
dt = R² cP(t) +
(5)
dG(t)
dt - P(tG)))))
(6)
dCall) qT(()) 3Ca(l)
dt
Ca(0) Cdarr
(8)
Smaz
S(I)
(9)
Ca(t)
1 1
Kac
dCastow (t) Batono (Castow (1) Ca(())
(10)
dt
T(1) k(a
(11)
In our Keras photoreceptor layer, the parameters 7, Φ. 77, 9, B, Cdark, KGc, Bstows m, KCa. and can all be set to trainable. this study however, we only allowed a, 3, 6 and 1j to be trained. Remaining parameters were fixed to their values (Angueyra et al. 2021 from photoreceptor data fits.
13

---

Under review as a conference paper at ICLR 2023
B APPENDIX: TRAINING PROCEDURE FOR GENERALIZING ACROSS EXTREME LIGHT LEVELS
Rat Retina
10,000 R receptoor''
Predicted RGC
M
response
Step
Photorecepter (cone)
CNN
CNN
CNN
DENSE
Time
R Rereceptor 's
Fixed to weights learnt above
Step
Photoreceptor (rod)
CNN
CNN
CNN
DENSE
mmw
Rat Retina B
10,000 Rereeeptrr's'
Photoreceptor (cone)
CNN
CNN
CNN
DENSE
Step
1aamt from Retina A
Fixed weights learnt above
Rereceptor's'
Step
Photoreceptor (rod)
CNN
CNN
CNN
DENSE
WLAM
Figure B.1: Schematic for training across extreme light levels. Step 1: PR-CNN model was trained end-to-end to predict Rat Retina 'A' RGC responses at photopic light level (10,000 R "rreceptor"s"" 1)_ This led to estimate for cone photoreceptor parameters and model for the inner retina circuit (the CNN layers). Step 2: The model was re-trained at scotopic light level (1 R*receptor''s
set to non-trainable and fixed to weights learnt in Step 1. In this case, the photoreceptor model learnt parameters reflecting rods. Step 3: The model was trained predict Rat Retina 'B' responses photopic light level. Photoreceptor layer to non non-trainable with parameters fixed to cone parameters learnt in Step 1. Step 4 (testing step): Model was tested to predict Rat Retina 'B' responses scotopic light level. Here, we used the rod photoreceptor parameters learnt in Step 2 and rest of the model representing inner retina pathways of Retina B learnt Step
but the CNN layers
set
C APPENDIX: ADAPTIVE-CONY
C.I PROOF-OF-CONCEPT DEMONSTRATION OF APPLICATION IN COMPUTER VISION
Given that the photoreceptor adaptive mechanisms enable the retina (and the PR-CNN model dis- cussed in the main paper) work under wide range of steady-state and dynamic light intensities, we hypothesized that those same adaptive mechanisms could also be useful for computer vision tasks in dynamic lighting conditions. Specifically, we consider conditions with potential disruptive events such as shifting cloud cover or shadows due to moving tree branches, that can rapidly and dramatically change the lighting part or all of visual scene. These events can shift the scale of the pixel values in an image, either on some pixels or on all of them, depending on the spatial scale of the event.
If the spatial scale is large, and the whole image exhibits the change in lighting level, simple nor- malization (i.e., layer norm) could remove the change. For localized changes, however, there remove the lighting change. Moreover, these local changes in pixel intensities can be easily confused for changes in the edges in the scene, hence
no obvious strategy through which normalization
confounding object detection and identification (Gomez-Ojeda al. 2015 Kolaman et al. 2019
Janai et al. 2020).
To address this challenge, have developed a novel convolution layer, the Adaptive- which contains a simplified functional model of the photoreceptors (see Appendix. [C2 for technical de- tails). As proof-of-concept that such photoreceptor-inspinn models could be better solving
14

---

Under review as a conference paper at ICLR 2023
computer vision tasks, considered a toy example where the input to the model is object's intensity represented by a single pixel in space. The pixel value (intensity) varies in time around a fixed mean value (object intensity in Fig. p. column 1). At different time points, brief disruptive events of different durations and intensities (spanning to 5 log units) (source intensity in Fig. C. column 2), change the amount of light reflected off the object( Fig. PT.pp, column 3). We attempted to train neural networks with, or without, our new Adaptive-Cont layer, to extract the original in- tensity of the object (red line
C.I.a column 3), thereby removing the effects of the disruptive light change (blue line in Fig. C. column 3)
We trained model with the Adaptive-Com layer to perform this task (Fig. C. p, left panel). The model takes as input a movie with 200 frames. The first layer is an Adaptive- claptive-Conv layer with 40 channels followed by dense layers each containing 80 units. These hyperparameterss were optimized using a grid search. The width of the temporal kernel in the Adaptive-Cony layer was set to 200. This model could reliably extract the original intensity of the object, discounting the disruptive events (Fig. C.Ip. right panel). In contrast, conventional ID CNN model where the standard convolution layer was used (Fig. C.T. left panel), failed to extract the original intensity following the disruptive events (Fig. C.I.. right panel), despite having 4 times as many parameters as the model with the Adaptive-Com Conv. (For this model as well we optimized the hyperparameters using grid search which resulted in 100 channels for the first convolution layer, followed by dense layers each with 200 units.) Our ongoing work will test this Adaptive-Cony approach on more complex vision tasks and see if the same observation holds, namely that the photoreceptor model discounts the sudden large changes in intensity, thereby removing the impacts of those intensity changes on object detection and/or categorization.
C.2 ADAPTIVE-CONY LAYER
The Adaptive-Con Coop layer is trainable layer that constructed using Keras (Chollet et al. 2015) package in Python and builds upon a modified version of phenomenological photoreceptor model deseribed in (Clark et al. 2013). In this layer, the input signal at each pixel, xx(t) is convolved in time, f, with two temporal kernels, as in
v(1)= f Ky(1 (()a(('))
(12)
x(1) = S K2(t - ())((')dd
(13)
where xt) is the pixel intensity at time t, Ky and K₂ are the two temporal, of the form
thv
(t(
RT +I 0(t)
(14)
g
Ke(1)-
TT e(1
(15)
K2(1) = (1(0)==((0++(1--))
n ng. 1. 0(1)
(16)
Here, Ty, Te: Tu, specifies the time scale and Ny. nhe ny specifies the rise behaviour of the kernels. Parameter y (constrained
< 7 S 1) weights the relative importance of the two kernels with different temporal profiles making up K_. and @ is the Heavyside function. All these parameters are trainable within the Adaptive-Conv layer.
The two convolved input signals, y(t) and =t) are then weighted and combined divisively to form the Adaptive-Coom Cony layer's output, through the following operation
a
o(t) = + Bz(l) v(t),
(17)
15

---

Under review as a conference paper at ICLR 2023
where a, and r are trainable weights. In total, the Adaptive-Conv layer has 10 parameters, all of which are fully trainable through backpropagation.
Similar to the biophysical model used in the main paper, key feature of this phenomenological model that the gain of the input is dynamically controlled by x+p2(() over the time scale of g.
based on prevailing inputs
Stimulus
Object
Source
Object x Source
Light inunsay (au.)
(M Mh Myy J
Object
10
10:
10:
10
107
10
10
1000
400
600 1000
1200
200
400
500
1000 4400
Frames
Frames
Model with Adaptive-oonvllttion
Layer shape Adaptive-Cony ReLU
Dense BatchNorm + ReLU
Actual object intensity
Dense Sottplus
Model extracted object intensity
(wr) Ansnoo
I pixel
X
pixel frames
channels
units
200
6oo 1000 200 1600
Frames
Model with conventional convolution
Depthwise convolution
BatchNorm
pixel
x
Intersiyy
pixel 200 frames
100 channels
200 units
200
100
COo
s0O
1000
1200 1:00 1600
Frames
Figure C.I: Adaptive-Cone layer. a. Snippet of the toy example stimuli where intensity reflected off an object varies in time around mean value (red trace in column 1) multiplied by a signal having large intensity variations (log scale; purple trace in column 2) to introduce disruptions in the intensity reflected off the object (blue trace column 3). b. Left panel: a model with the Adaptive- Conv layer trained to extract the original intensity of the object from the combined signal (blue in a). Right panel: original intensity of the object that the model is trained to extract (red solid line) and the intensity extracted by the model (dashed black line). Same in b but with model where a conventional convolution layer replaced our Adaptive-Coo Coan layer. Cyan arrows the right panel indicate examples where the model with conventional convolution layer failed to extract the object intensity from the combined signal.
16


Reviews:
[{"review_id": "m0MagZkzKX", "reviewer": "Reviewer_Rcws", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors indicate that modern computer vision systems tend to fail under dynamic lighting conditions. This is because computer vision algorithms do not explicitly make use of adaptation mechanisms similar to the ones used in retinal ganglion cells. Because current deep learning models of the retina have no in-built notion of light level , they are unable to accurately predict RGC responses under lighting conditions that they were not trained on. The authors use a model of RGC and test it on monkey and rat retinal data. The authors argue that their algorithm can be used with computer vision algorithms to adapt to dynamic environments.", "summary_of_the_review": "Overall, while this is a well written paper, I feel the authors need to do a better job adapting it towards the typical ICLR audience. Currently it is addressed more towards a computational neuroscience audience. As indicated on page 6, I do not feel comfortable accepting to ICLR a paper that has only been tested with white noise input. The effort in appendix C to compare it with actual algorithmic performance is incomplete and needs more work", "strength_and_weaknesses": "Strengths:\n\nS1: the authors identify and important problem in computer vision algorithms and make some biologically motivated suggestions on how this problem could be solved.\n\nS2: the paper is extremely well written and is up to ICLR standards\n\nWeaknesses:\n\nW1: With respect to making computer vision algorithms more robust, why do they feel that such an adaptation approach is necessary? Why not simply give HDR data (high dynamic range data) as input to algorithms (ie data that is represented with more than 8 bits per channel). Wouldn't this be sufficient to deal with saturation issues in high intensity situations, and low contrast in low light situations?\n\nW2: page 1 typo: \"We focus here on retina\". Missing \"the\"\n\nW3: prior related work discussed in the paper does not go before 2015-2016. I suggest an expansion on prior work detailing the effects of sensor bias in experimental methods.\n\nW4: section 3.3. The authors do electrophysiology experiments from ex vivo macaque monkey and rat retina. It is not often that you see such experiments in machine learning/computer vision conferences. Machine learning conferences like ICLR are more focused on algorithms that improve real-world performance. The way the paper is written it seems to be focused more towards a computational neuroscience audience. While not necessarily bad, in this case this seems to be to the detriment of demonstrating whether this algorithm actually leads to improvements in accuracy, performance, speed, power of computer vision systems. The main focus of the authors should have been to show that their algorithm leads to some quantifiable improvements compared to other algorithms that are solving a certain real-world problem. The authors seem to try to touch upon this in the appendix, but not in a lot of detail. This is the paper's rather major weakness in my opinion. The appendix algorithm should be in the main paper and its evaluation on a real world problem expanded upon.\n\nW5: page 5: I suggest the authors introduce parasol and midget cells and how their behavior differs, since the typical ICLR audience will not be familiar.\n\nW6: Eq 1: Should it be T or T-1? Is it sample of population variance?\n\nW7: Eq 1: clarify how Var[y] was obtained.\n\nW8: page 5: how were the sets A, B obtained? Are there confidence intervals on these estimators? Is eq 2 an unbiased estimator? Clarify if this estimator is used in Cadena 2017. It is unclear whether the average is across time. Clarify over what the expected value is taken.\n\nW9: will source code and the data be provided to make this reproducible?\n\nW10: if you connect a live camera as input, how well would the algorithm adapt to changes in the sensor gain and shutter speed?\n\nW11: page 6: 180 frames are used. At 30fps this is 6 seconds. This means it takes 6 seconds for adaptation to happen? Could this be a drawback in computer vision algorithms?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XYLFOYciWf", "reviewer": "Reviewer_TU1Q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors showed that putting a photoreceptor front-end with adaptive dynamic gain control allows a deep neural network to predict more reliably the responses of retinal ganglion cells in different lighting conditions than the STOA CNN retinal model. ", "summary_of_the_review": "While the technical and conceptual contribution in ML and representational learning are relatively limited, this paper is acceptable for demonstrating or proving  that an adaptive gain control mechanism as a front-end of CNN can improve the prediction of RGC's responses in different lighting condition. Even though it might be proving the obvious, it is still important and might have significant technical implication for visual prothesis and vision system research in the future. ", "strength_and_weaknesses": "Introducing an adaptive gain control mechanism as a front end to CNN is a sensible and good idea.  The fact that such an adaptive mechanism should improve a CNN's prediction of neural response in different lighting conditions is not surprising. Nevertheless,  demonstrating empirically it actually does the job is still meaningful and an accomplishment. Making the adaptive mechanism itself partially learnable is also novel. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Hp0mJsbt0G", "reviewer": "Reviewer_b3Ty", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper implements a photoreceptor model in Keras and uses it as a front end in training a shallow CNN model to predict retinal responses to white noise stimuli. The paper reports that the photoreceptor model allows the model to generalize better to new lighting levels. This observation is demonstrated using monkey and rat RGC data. The light levels in the rat experiments used a more dramatic range of variation (4 orders of magnitude) that required a more complex cross-validation procedure due to largely non-overlapping rod and cone activation across levels. ", "summary_of_the_review": "The combination of a photoreceptor frontend with a DNN to predict RGC responses is novel and potentially interesting. I am overall convinced that this addition helps with generalization to novel lighting levels, at least when the differences are extreme. It is unclear to me whether this insight will in practice be useful to the ML community. There might be simple training strategies that would avoid this issue, and the photoreceptor model is complex and seems unlikely to become a mainstay of computer vision models. \n\nResponse after rebuttal:\n\nI apologize for my slow response. The authors have addressed all of my comments, which I appreciate. I think this paper provides a solid contribution to the computational neuroscience literature and seems well-suited to the Neuroscience and Cognitive Science section. I still find the generalization results in monkey RGC a little underwhelming, and my best guess is that the impact of the paper will be modest. Nonetheless, I have increased my score from 5 to 6 based on the authors' argument that their paper is appropriate for the noted section. ", "strength_and_weaknesses": "Strengths\n\nI found the paper clear and easy to follow. The results are simple and straightforward.\n\nThe paper provides decent evidence that the photoreceptor model improves generalization to new lighting levels, which is an important form of variation.\n\nExploiting insights from biology to improve machine learning is rare and potentially exciting contribution.\n\nWeaknesses\n\nApplications to vision and ML are preliminary and left to the appendix (Appendix C), which seems likely to limit the impact of the work to the ICLR community. The photoreceptor model is rather complex which I worry will limit its popularity and impact. The authors toy with creating a simpler, convolutional layer based on their model, but again this part of the paper is preliminary and left to the appendix. \n\nThe generalization effect in monkey RGCs is only substantial for the case when training on high light levels and testing on low light levels. No measures of statistical significance or confidence are reported.\n\nIt would be interesting to know whether the model would have difficulty generalizing to a variety of lighting conditions when trained using naturalistic variation in retinal stimulation. It would also be useful to quantify model performance using naturalistic stimulation since one of the benefits of having a DNN model is that it can potentially predict neural responses to real-world stimuli, as opposed to just white noise.\n\nIf one does use a training set with a wide range of lighting levels is there any benefit of the photoreceptor model? For example, does the model require less training time to reach a given level of accuracy? For ML applications, it’s not clear whether the benefit of the photoreceptor model would be worth the added complexity, particularly if it’s relatively easy to create robustness by applying a bit more diversity to the training regime. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject