Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors adopted a task-agnostic U-net to model the retinal encoding process and characterize the representation of "time in the natural scene" in a low-dimensional compressed latent space. The U-net takes PSTHs of the salamander retinal population as input and the reconstructed movie frames as output, and learns with error propagation. They found that the retina has a generalizable encoding for time in the natural scene, which learned from one movie can be used to represent time in another movie. They also showed that static textures and velocity features of a natural movie are synergistic. 
strengths_and_weaknesses: The paper is clearly organized and the logic flow is good. Discovering the low-dimensional feature representation in retinal neuronal firing (the time feature) with the combination of a ML based method is interesting. 
questions: Q1: since neural responses are delayed (when an input stimulus is presented), how to align the input (neural responses) and the output (movie frames) to make a proper input-output pairs? Is it necessary to shift few miniseconds to find a good alignment? 

Q2: beside the background and motino seperation showed in Fig2, is there any other disentangled feature found in the decoder layer? Is this feature disentanglement due to the utilization of the variational sampling layers? Any other tricks to find these disentangled feature?

Q3: does the decoding performance reported in Fig3 means to decode the temporal order of N movie frames? 
limitations: Yes, the authors addressed the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In their paper "Learning low-dimensional generalizable natural features from retina using a U-net", the authors modify a U-net with a ResNet encoder and variational sampling layers for skip connections to identify salamander retinal population response space-time features. They do this by reconstructing movie frames from neural responses. Moreover, they analyze the latent representations and find that static features and dynamic features are encoded synergistically.
strengths_and_weaknesses: The modifications of U-net in terms of the ResNet encoder and variational sampling layers for skip connections is an interesting innovation that might have broader applications and could be of interest to the wider machine learning community. However, the authors do not discuss other potential applications of their methodology and instead just mention that the architecture can learn many compressed latent representations simultaneously.

To the best of my knowledge, the finding that static features and dynamic features are encoded synergistically in the retina is novel. Given that this finding helps to elucidate how the retina processes visual stimuli, this is a significant contribution. Therefore, the paper should be of interest to the wider neuroscience community.

The paper is written in a clear and compact way except for a couple of minor points, which I mention in Questions. Figures are informative and are well integrated into the text.

I could not find any conceptual or technical mistakes. The authors clearly state their assumptions and analyze their data rigorously.

EDIT: The authors addressed my minor concerns. I increased my rating to 7.
questions: Do the authors see other potential applications of their U-net modifications outside of neuroscience?

Figure 2 caption: Bottom row appears twice. The authors should properly refer to all four rows.

Figure 3: The authors trained on the fish movie and tested on the other two movies. Would the results be similar if training on the leaf or water movie and testing on the other two movies?

Line 248: Network misspelled.

Eq. 2: Please add brackets for clarity. Moreover, there is no explanation that this approximation comes from an independent Gaussian assumption. Please clarify the text.

Line 270: Subscript "joint" broken.
limitations: The authors do not discuss any negative societal impacts of their work. In my opinion, this is adequate in this case.

There is an independence assumption, which is very briefly discussed in the Discussion. The authors also discuss the lack of proper predictive constraints and the particular type of synergy that they looked at. Overall, limitations have been appropriately discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
In this manuscript the authors describe the use of a deep-learning pipeline to understand the nature of encoding in the salamander retina. Namely, they collect PSTHs from neurons in response to short natural movies, then the PSTHs are converted to images and used as the input to a U-Net, which is trained to reconstruct the movie. Curiously, the latent representation can be used to accurately decode time in the movie, even for completely novel movies. This suggests that the model learns a low-dimensional representation of elapsed time. Lastly, the authors use information theoretic analysis to show that encoding of time requires jointly encoding of static and dynamic features.

strengths_and_weaknesses: Strengths

1)	I found the pipeline presented by the authors to be fairly novel and well-motivated. I am not aware of the use of Gramian angular fields and autoencoders to understand population coding. This has the potential to be useful in other fields of neuroscience more generally.

2)	Their result – that the salamander retina may have a movie-invariant code for time – is genuinely interesting.

3)	For the most part the paper was well-written.

Weaknesses

1)	Lack of controls and comparisons with other methods. I do not know how likely this is, but one could imagine that time could be encoded by fairly trivial aspects of the data – e.g. a change in the aggregate statistics of spiking. Perhaps neurons fire more to the onset of the movie and slowly decay to 0 by the end. Since simple controls aren’t presented, it is hard to know if this sort of triviality explains their results. 

    1.	First, the authors could simply shuffle the identity of neurons in the test phase. This would account for any trivial gross changes in spiking statistics accounting for decoding accuracy. 

    1.	Second, a much simpler model should be presented as a baseline. For instance, train a linear regression model to predict the time in the movie from the PSTHs (or a dimensionally reduced version of them), in addition to summary statistics computed across neurons (e.g. the mean and variance). How does the accuracy of this baseline model compare to the U-Net? The outcome of this comparison would provide a much better motivation for the authors’ pipeline. While it is novel, it is important to know how much extra information it provides.

    1.	Third, show more raw results. Visualize how the latent space evolves over time for each of the three movies along with the PSTHs. It would be nice to have a better intuition for how the model is transforming the spikes, and what features in the spike trains are relevant for encoding time. Perhaps there are no obvious features one can spot by eye, which provides even better motivation for the authors’ method.


Minor

1)	The authors mention domain-conversion as an application of U-Nets. This could be my own misunderstanding, but I am not sure what is meant here since one of the examples given is super-resolution imaging.

2)	Figure 2 caption has bottom row twice. I also don’t know how the authors computed this receptive field. More detail would be nice here.


questions: 1. Did the authors attempt to use simpler models to decode time?
1. Can you predict time in a held-out movie using visual features from another? Might be a remote possibility, but I want to know if there are any "simple" visual features that predict time. If you take frames from the fish movie and compute the pairwise distance to all frames in another movie, I assume there is no correlation with time?
1. Did the authors try simple autoencoder architectures with a single latent layer rather than a U-Net? I am curious what benefit is conferred by the multi-resolution architecture, especially given that they are using a GAF representation of spike trains.
limitations: I found the discussion of limitations adequate. No concerns with respect to societal impact.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: In this manuscript the authors try to apply a U-net architecture to predict future movie frames from salamander retinal cell responses to the previous frames. The main finding is a signal for the time since the start of the movie in the latent representation.

strengths_and_weaknesses: 
Unfortunately, I don’t think any of the methods or results in this paper are useable. The plan to train a large DNN architecture on the reconstruction of a few seconds of movie is doomed from the start. A U-Net architecture can probably reconstruct the whole movie clip from a single number for the frame as input without any retinal data. This also means that finding a signal for which frame to construct is not unexpected.
Moreover, I fail to understand what the underlying aim of training this U-Net was in the first place. A deep neural network trained to predict future video frames is a technique for unsupervised learning, i.e. explicitly to learn complex features that are not easily available in the original inputs and filter out quickly changing information. I don’t see how this would be helpful for understanding the encoding of retinal cells, even if it worked.

Besides these fundamental problems, there are some odd choices in the model design and sometimes it was not entirely clear what the authors did exactly:
1) The input is a stack of “Gramian angular field” images constructed from the PSTHs for individual neurons. This transformation has no spatial correspondence to the image. Thus, the whole U-net architecture makes little to no sense with this input.
2) Similarly the weights of a resnet pretrained on images are not sensible for this kind of input.
3) The authors add noise to the architecture at various points without much justification except a hope for more interpretable representations. 
4) At various places, I lost track which analysis were based on the U-Net and which ones were based on the additional VGG.
5) The training section in the supplementary material suggests that different samples were generated by dropping 2 cells randomly, but there is no explanation how the network could handle input with permuted cell identities sensibly. 
questions: Unfortunately, I don't think my concerns could be removed with anything the authors could do within the confines of a rebuttal phase.

In general, find a sufficiently large dataset, find an input format for the retinal responses that has some spatial correspondence and keeps which neuron (type) responded where, and try to include some interpretability of the output to find out what can be decoded.
limitations: ---
review_title: null

Meta-review: metareview: The authors analyze the latent representation of visual features from natural movies in the salamander retina using a U-net. They train an encoder to learn a compressed latent representation from a large population of salamander retinal ganglion cells responding to natural movies, while a decoder samples from this compressed latent space to generate the appropriate movie frame. They characterize its representation of “time in the natural scene” in the latent space of their model.

Overall, the reviewers expressed a lot of interest in the topic and valued the novel application to salamander retinal data. There were some questions about the significance of the finding, and through the rebuttal period, the authors provided a number of experiments to compare against other variants of their baselines (and ablations), with some reviewers increasing their scores in favor of acceptance. 

At the same time, there was concern that the U-Net architecture could potentially reconstruct the movie without any retinal data. Thus, it was not entirely clear whether the model was truly leveraging the retinal data to obtain meaningful outputs. Unfortunately, this concern was not fully addressed in the revision, leaving the reviewers overall with mixed views but the majority in favor of acceptance.

confidence: Less certain
Decision: Accept