Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper explores the ability of nonlinear recurrent neural networks to account for neural response properties that have otherwise eluded the ability of other models.  A multilayer rnn is trained to imitate the stimulus-response mapping measured from actual retinal ganglion cells in response to a sequence of natural images.  The rnn performs significantly better, especially in accounting for transient responses, than conventional LN/GLM models.

This work is an important step in understanding the nonlinear response properties of visual neurons.  Recent results have shown that the responses of even retinal ganglion cells in response to natural movies are difficult to explain in terms of standard receptive field models.  So this presents an important challenge to the field.  If we even had *a* model that works, it would be a starting point.  So this work should be seen in that light.  The challenge now of course is to tease apart what the rnn is doing.  Perhaps it could now be pruned and simplified to see what parts are critical to performance.  It would have been nice to see such an analysis.   Nevertheless this result is a good first start and I think important for people to know about.

I am a bit confused about what is being called a "movie."  My understanding is that it is essentially a sequence of unrelated images shown for 1 sec. each.  But then it is stated that the "frame rate" is 1/8.33 ms.  I think this must refer to the refresh rate of the monitor, right?   

I would guess that the deviations from the LN model are even stronger when you show actual dynamic natural scenes - i.e., real movies.  Here I would expect the rnn to have an even more profound effect, and potentially be much more informative.

review_title: Review of "Multilayer Recurrent Network Models of Primate Retinal Ganglion Cells"

final_score: rating: null
review_content: What was the motivation for using the fraction of explainable variance as the metric to assess the performance of the models? A reference to a recent paper is given without actually providing an explanation for why you would want to use this. Why not other metrics such as log-likelihood?

review_title: Performance metric

final_score: rating: null
review_content: I do not understand the experimental setup. In 3.1 on page 2 you talk about two separate experiments but you don't say what they are. Then you talk about static images and movies. Which did you use? And why are training data interleaved with test data, and why do you need repetitions of the test data? What are the light and dark bars represent in figure 2A? 
Also, note that you over the 9 page limit with references. 
review_title: Experimental setup

final_score: rating: 7: Good paper, accept
review_content: This is a clearly written paper with a nice, if straightforward, result: RNNs can be good predictive models of neuron firing rates in the retina.

On the one hand, the primary scientific contribution seems to just be to confirm that this approach works. On this particular stimulus locked task the gains from using the RNN seemed relatively modest, and it hasn't yet taught us anything new about the biology.

On the other hand, this (along with the concurrent work of McIntosh et al.) is introducing neural network modeling to a field that isn't currently using it, and where it should prove very effective.

I think it would be very interesting to see the results of applying a framework like this one with LFP and other neurons as input and on a shorter discretization time scale.

I suspect followup work building on this proof of concept will be increasingly exciting.

Minor comments:
Sec 3.2:
I didn't understand the role of the 0.833 ms bins.
Use "epoch" throughout, rather than alternating between "epoch" and "pass through data".

Fig. 4 would be better with the x-axis on a log scale.
review_title: a clearly written paper with nice, if straightforward, results

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper fits models to spike trains of retinal ganglion cells that are driven by natural images. I think the title should thus include the word “activity” at the end for otherwise it is actually formally incorrect.

Anyhow, this paper proposes more specifically a recurrent network for this time series prediction and compares it to what seems to be the previous approach of a generalized linear model. Overall the stated paradigm is that when one can predict the spikes well then one can look into the model and learn how nature does it. 

In general the paper sounds plausible, though I am not convinced that I learned a lot. The results in figure 2 show that the RNN model can predict the spikes a bit better. So this is nice. But now what? You have shown that a more complicated model can produce better fits to the data, though there are of course still some variations to the real data. Your initial outline was that a better predictive model helps you to better understand the neural processing in the retina. So tell us what you learned. I am not a specialist of the retina, but I know that there are several layers and recurrencies in the retina, so I am not so surprised that the new model is better than the GLM. 

It seems that more complicated recurrent models such as LSTM do not improve the performance according to a statement in the paper. However, comparisons on this level are also difficult as a more complex models needs more data. Hence, I would actually expect that more layers and even a more detailed model of the retina could eventually improve the prediction even further. 
I was also a bit puzzled that all the neurons in the network share all the same parameters (weights). While the results show that these simplified models can capture a lot of the spike train characteristics, couldn’t a model with free parameters eventually outperform this one (with correspondingly more training data)?

review_title: What do we learn from the model?

final_score: rating: null
review_content: Sec. 2 -- was the decision to train separate models for on and off cells made from first principles, or did you try both together in preliminary experiments, and find worse performance with a combined model?

One of the results of your analysis is that the RNN is better at matching spike timing. You only run the models with a relatively coarse time binning though. Have you explored running the RNN at a time scale which is on the order of spike time precision when we can precisely characterize the inputs to a neuron (eg, 1 ms or less)? It might be that at shorter time scales, the improved ability of the RNN to match spike timing causes it to more dramatically outperform traditional techniques.

Similarly, though I recognize it may be out of scope for this paper, are you planning to give the RNN access to recorded spike trains as well as the stimulus? Or possibly to the LFP signal? Spiking is often better aligned to recent spikes or ongoing oscillations than to stimuli, so this might buy you further gains over more traditional approaches.
review_title: preliminary questions

Meta-review: This work is an important step in developing the tools for understanding the nonlinear response properties of visual neurons. The methods are sound and the results are meaningful. Reviewer 3 gave a much lower score than the other two reviewers because Rev 3 does not appreciate the improvement of prediction performance as an advance in itself. For understanding of the visual algorithms in the brain, however, prediction performance is the most critical success criterion. The paper provides convincing evidence that the approach is promising and likely to facilitate further advances towards achieving this long-term goal.
 
 I am confident enough to defend acceptance of this paper for a poster.
Decision: Accept (Poster)