Title: Multilayer Recurrent Network Models of Primate Retinal Ganglion Cell Responses
Authors: Eleanor Batty
Josh Merel
Nora Brackbill
Alexander Heitman
Alexander Sher
Alan Litke
E.J. Chichilnisky
Liam Paninski
Venue: ICLR 2017
Keywords: Deep learning
Applications

Abstract: Developing accurate predictive models of sensory neurons is vital to understanding sensory processing and brain computations. The current standard approach to modeling neurons is to start with simple models and to incrementally add interpretable features. An alternative approach is to start with a more complex model that captures responses accurately, and then probe the fitted model structure to understand the neural computations. Here, we show that a multitask recurrent neural network (RNN) framework provides the flexibility necessary to model complex computations of neurons that cannot be captured by previous methods. Specifically, multilayer recurrent neural networks that share features across neurons outperform generalized linear models (GLMs) in predicting the spiking responses of parasol ganglion cells in the primate retina to natural images. The networks achieve good predictive performance given surprisingly small amounts of experimental training data. Additionally, we present a novel GLM-RNN hybrid model with separate spatial and temporal processing components which provides insights into the aspects of retinal processing better captured by the recurrent neural networks.


Review discussion:
I do not understand the experimental setup. In 3.1 on page 2 you talk about two separate experiments but you don't say what they are.
Then you talk about static images and movies. Which did you use? And why are training data interleaved with test data, and why do you need repetitions of the test data? What are the light and dark bars represent in figure 2A?
Also, note that you over the 9 page limit with references.
Thanks for the questions.
1) We perform the same experimental procedure in two retinas. We fit models to these two experiments separately due to animal to animal variability in cell properties, such as receptive field size and firing rate.
2) The movies consisted of a series of images presented for one second each. Within each second, the image is spatially jittered to replicate small eye movements. At the end, the image is replaced with a new image.
3) Test and training movies are interleaved because sometimes neural responses systematically drift over time. Interleaving means that the testing data spans the same period of time as the fitting data, and therefore is experiencing the same range of conditions.  This is a standard protocol in sensory neuroscience.
4) We have repetitions of the test movie so that we predict the average firing rate over multiple trials. On each trial, we assume that spikes are drawn from the average firing rate using a Poisson distribution, so averaging over repetitions produces a less noisy estimate of the true firing rate.
5) As shown in the legend, the light and dark colors indicate the results from the two different experiments throughout the manuscript.
We have uploaded a revision clarifying these details of the experimental set-up in Section 3.1
Sec. 2 -- was the decision to train separate models for on and off cells made from first principles, or did you try both together in preliminary experiments, and find worse performance with a combined model?
One of the results of your analysis is that the RNN is better at matching spike timing. You only run the models with a relatively coarse time binning though. Have you explored running the RNN at a time scale which is on the order of spike time precision when we can precisely characterize the inputs to a neuron (eg, 1 ms or less)? It might be that at shorter time scales, the improved ability of the RNN to match spike timing causes it to more dramatically outperform traditional techniques.
Similarly, though I recognize it may be out of scope for this paper, are you planning to give the RNN access to recorded spike trains as well as the stimulus? Or possibly to the LFP signal? Spiking is often better aligned to recent spikes or ongoing oscillations than to stimuli, so this might buy you further gains over more traditional approaches.
1) The decision was made from first principles. ON and OFF cells receive inputs from separate populations of earlier retinal cells and have fairly significant differences in processing properties.
2) We haven’t thoroughly explored running the RNN at a smaller time scale although this is a planned experiment.
3) Future work will incorporate correlations into the model, although doing so using coupling filters from spike trains of neighboring neurons has not previously had a significant impact on predicting trial-averaged firing rates (Pillow et al 2008, Heitman et al 2016).
What was the motivation for using the fraction of explainable variance as the metric to assess the performance of the models? A reference to a recent paper is given without actually providing an explanation for why you would want to use this. Why not other metrics such as log-likelihood?
This paper fits models to spike trains of retinal ganglion cells that are driven by natural images. I think the title should thus include the word “activity” at the end for otherwise it is actually formally incorrect.
Anyhow, this paper proposes more specifically a recurrent network for this time series prediction and compares it to what seems to be the previous approach of a generalized linear model.
Overall the stated paradigm is that when one can predict the spikes well then one can look into the model and learn how nature does it.
In general the paper sounds plausible, though I am not convinced that I learned a lot. The results in figure 2 show that the RNN model can predict the spikes a bit better.
So this is nice.
But now what? You have shown that a more complicated model can produce better fits to the data, though there are of course still some variations to the real data. Your initial outline was that a better predictive model helps you to better understand the neural processing in the retina.
So tell us what you learned. I am not a specialist of the retina, but I know that there are several layers and recurrencies in the retina, so I am not so surprised that the new model is better than the GLM.
It seems that more complicated recurrent models such as LSTM do not improve the performance according to a statement in the paper.
However, comparisons on this level are also difficult as a more complex models needs more data.
Hence, I would actually expect that more layers and even a more detailed model of the retina could eventually improve the prediction even further.
I was also a bit puzzled that all the neurons in the network share all the same parameters (weights).
While the results show that these simplified models can capture a lot of the spike train characteristics, couldn’t a model with free parameters eventually outperform this one (with correspondingly more training data)?
Thanks for taking the time to review our manuscript. We will change the title as suggested. We think, as other reviewers have mentioned, that this work is exciting because it introduces the idea that deep neural networks can be used in a neural modeling context with limited experimental data.
In particular, the framework we introduced for sharing information across neurons allows us to fit richer models given less data, and we believe this will be a powerful approach for modeling data from a number of other brain areas - most of which are much less studied / understood than the retina.
Yes, a model with free parameters or more complicated structure could eventually outperform the shared model with enough training data but there are real biological and experimental limitations to the amount of data one can collect in these experiments and more broadly in most neuroscience experiments.
We also believe we have taken important first steps to understanding the model improvement through model comparisons. Whether the gap between linear-nonlinear model performance and optimal performance could be explained by a combination of  long-range spatial interactions and spatial and temporal nonlinearities might seem simple, but it is an important question in the field that has been difficult to address. We showed that the former is not true (at least up to the level of RNN performance) and that both spatial nonlinearities and nonlinear temporal processing are important.
These results can guide future research and experiments and further demonstrate the utility of our approach. This is a first step: in future work, we plan to further interrogate the RNN models to understand more specifically what information they are capturing. As noted above, we also plan to apply this framework to non-retinal neurons where there are larger gaps between what current models capture and the optimal predictions. In these systems, there is more room for improvement and more basic questions can be answered using our approach.
This is a clearly written paper with a nice, if straightforward, result: RNNs can be good predictive models of neuron firing rates in the retina.
On the one hand, the primary scientific contribution seems to just be to confirm that this approach works. On this particular stimulus locked task the gains from using the RNN seemed relatively modest, and it hasn't yet taught us anything new about the biology.
On the other hand, this (along with the concurrent work of McIntosh et al.) is introducing neural network modeling to a field that isn't currently using it, and where it should prove very effective.
I think it would be very interesting to see the results of applying a framework like this one with LFP and other neurons as input and on a shorter discretization time scale.
I suspect followup work building on this proof of concept will be increasingly exciting.
Minor comments:
Sec 3.2:
I didn't understand the role of the 0.833 ms bins.
Use "epoch" throughout, rather than alternating between "epoch" and "pass through data".
Fig. 4 would be better with the x-axis on a log scale.
Thanks for the review!
While it is possible that there is non-spiking, low-frequency information similar to the cortical LFP, this is not a measurement typically done in the retina, and due to technical reasons these signals are not recorded on our multielectrode array.  As mentioned previously, we plan to incorporate correlations in future work, although doing so using coupling filters from spike trains of neighbouring neurons has not previously had a significant impact on predicting trial-averaged firing rates (Pillow2008, Heitman2016).The 0.833 ms bins were used because a post spike history filter operates on a small time scale so we divided the 8.33 ms bins corresponding to the frame rate by 10. We need small bins for it to capture effects like the refractory period.
This paper explores the ability of nonlinear recurrent neural networks to account for neural response properties that have otherwise eluded the ability of other models.  A multilayer rnn is trained to imitate the stimulus-response mapping measured from actual retinal ganglion cells in response to a sequence of natural images.  The rnn performs significantly better, especially in accounting for transient responses, than conventional LN/GLM models.
This work is an important step in understanding the nonlinear response properties of visual neurons.  Recent results have shown that the responses of even retinal ganglion cells in response to natural movies are difficult to explain in terms of standard receptive field models.
So this presents an important challenge to the field.  If we even had *a* model that works, it would be a starting point.
So this work should be seen in that light.  The challenge now of course is to tease apart what the rnn is doing.  Perhaps it could now be pruned and simplified to see what parts are critical to performance.  It would have been nice to see such an analysis.
Nevertheless this result is a good first start and I think important for people to know about.
I am a bit confused about what is being called a "movie."  My understanding is that it is essentially a sequence of unrelated images shown for 1 sec. each.
But then it is stated that the "frame rate" is 1/8.33 ms.  I think this must refer to the refresh rate of the monitor, right?
I would guess that the deviations from the LN model are even stronger when you show actual dynamic natural scenes - i.e., real movies.  Here I would expect the rnn to have an even more profound effect, and potentially be much more informative.
Thanks for the review! The movie shown is a sequence of unrelated images, but within each 1 second segment, the image is jittered on the screen to simulate  the small eye movements that occur in a fixating animal.
Therefore, there is constant movement. The frame rate of this movie is 120Hz, which is also the refresh rate of the monitor.  This represents a common realistic setting where macaques gaze at part of a relatively static scene with fixational eye movements, and then saccade to another part of the scene (represented by the new image). The main difference between these images and natural movies is that there are no objects moving in the environment, only small translations of a sequence of static scenes. An example stimulus can be found at https://youtu.be/sG_18Uz_6OE (this link will be added to the manuscript).
We have posted a new revision of the manuscript in which we changed the subset of cells for which we show results to better align with a previous study. The results remain unchanged. One of our example cells in Figure 3 was no longer in the criteria-passing subset so we are showing responses from a different OFF example cell.
Additionally, we added a supplementary figure showing comparisons using a normalized log-likelihood metric, posted a link to a video of the stimulus, added “Responses” to the title, and made minor cosmetic changes to the figures.
The new revision reports individual LN/GLM metrics using parameters obtained from minimizing over individual validation error instead of over the validation error of all neurons. Again, results don't change but an outlier was removed.

Meta-review:
role: PC
data: comment: This work is an important step in developing the tools for understanding the nonlinear response properties of visual neurons. The methods are sound and the results are meaningful. Reviewer 3 gave a much lower score than the other two reviewers because Rev 3 does not appreciate the improvement of prediction performance as an advance in itself. For understanding of the visual algorithms in the brain, however, prediction performance is the most critical success criterion. The paper provides convincing evidence that the approach is promising and likely to facilitate further advances towards achieving this long-term goal.
 
 I am confident enough to defend acceptance of this paper for a poster.

Decision: Accept (Poster)