Venue: NeurIPS 2022 Workshop NeurReps

Reviews:

final_score: rating: 7: Technically solid paper, with moderate-to-high impact, and good-to-excellent evaluation and reproducibility.
review_content: summary: This abstract shows that pre-training a classifier with a biologically plausible Hebbian learning rule on "retinal wave"-like patterns improves the separability of the network’s internal representations. Using a sophisticated geometrical framework to analyze manifolds, they show that the manifolds' reduced dimension as opposed to their radius is the primary driver of the increase in capacity of the network.



strengths_and_weaknesses: Significance: The theoretical findings are interesting for our understanding of the role of retinal waves in neuroscience. For machine learning, the practical implications of retinal-wave learning is not demonstrated in this abstract, because of the simplicity of the task (MNIST) and architecture (one hidden layer, no backprop) studied.

Originality: I haven't seen such a precise geometrical analysis on the benefits of retinal-wave pre-training for manifold separability.

Quality/ Clarity: The abstract is clearly written and the claims well supported by the evidence.

questions: - It is an interesting finding that the manifolds' reduced dimension as opposed to their radius is the primary driver of the increase in capacity of the network, but how to interpret this finding? Could it be for example that, thanks to their localized receptive fields, neurons pool from nearby correlated pixels, effectively reducing the dimensionality of the manifold?


limitations: Limitations are adequately addressed.
recommended_decision: 3: Accept
review_title: Interesting study showing how retinal waves combined with a hebbian learning rule can be leveraged to improve manifold separability.

final_score: rating: 5: Technically solid paper with some limitations e.g., limited analysis or evaluation.
review_content: summary: This work analyzes the usage of retinal waves as a pre-training signal for machine learning classifiers. This is achieved by comparing different initializations of the model: one where the retina waves are used as pre-trained features and a control one where they are randomly permuted before being used. The evaluation is performed both via a task-driven metric (MNIST classification) and a geometry-driven one (manifold capacity). The results suggest that retinal waves are indeed informative features to be used in image classification tasks.
strengths_and_weaknesses: My expertise is not in biology, but concerning that part, I did not have problems following the work. Therefore, I consider this a strength of the paper and thank the authors for keeping it more than accessible for researchers with different backgrounds. Not only is it well-motivated in the introduction, but it also is fascinating to see machine learning being used as a proxy to explore hypothesis biology-related: in this case, whether retina waves retain enough information to be used as features to discern between samples in a simple image classification task.

My main concerns are in the experimental setup (clearly, excluding the data collection, which is well detailed in the appendix). I had some issues getting a clear picture of the model structure both in the first part of section 2 (Results) and in section 4 (Method). I think that portion could benefit from some restructuring; maybe a figure could be of help. However, the main issue was not understanding the number and nature of each layer (which is clearly illustrated) but the model as a whole and its design choices. Especially in the experimental setting, many choices are left unjustified, which is, in my opinion, weakening the final discussion and conclusions. For example:

- Why choose first to embed the retina features via Hebbian learning instead of using them directly as input? That could add noise to the procedure instead of using the raw data.
- Why is the second hidden layer (the one mapping from 400 to 100 dimensions) not trained? It has been shown in the literature that simple dense layers randomly initialized can still reach good classification performances, but why the need to add one here instead of just removing it if it can't be trained?
- How are the simulated samples obtained? I’m curious about this because I was expecting the real and simulated versions to have a similar distribution. However, in Figure 3, they are shown as being quite dissimilar, especially when comparing the two scrambled versions.

To be clear: if I understood correctly, the experimental setting is fairly solid. The problem is that I found it a bit convoluted to comprehend completely, mainly because there are some non-standard design choices left unmotivated, which negatively impacts the soundness of the results.
questions: Have the authors tried to apply the SVM directly on the input features instead of adding the two hidden layers before it? I’m asking this because I expect the same gap between the clean and the shuffled versions, but lower scores overall.
limitations: I did not find any additional limitations to include here. 

In any case, I’m more than open to changing my ratings and recommended decision if the main concerns/weaknesses are addressed.
recommended_decision: 2: Borderline
review_title: Fascinating research question, thorough evaluation, concerns in the soundness of the experimental setting.

final_score: rating: 5: Technically solid paper with some limitations e.g., limited analysis or evaluation.
review_content: summary: This work studies the computational role of retinal waves (i.e. correlated activity patterns in the developing retina). The authors trained —with a Hebbian learning rule— the first layer of a three-layer neural network on simulated / recorded input retina wave patterns. They then trained the last layer on MNIST and measure its classification accuracy along with other object manifold geometry metrics. They found that training the first layer on retinal waves  leads to higher classification accuracy than the random weights control.
strengths_and_weaknesses: Although the work is certainly going in an interesting direction, I find that the current analysis is conceptually unsound to solve the central question.

If the authors want to test wether unsupervised pretraining on retinal waves facilitates object classification in higher areas of the ventral stream, why not consider the effect of such pretraining after visual experience on natural images is made available? In the end, what is of most ethological relevance to the animal is to discriminate objects in the world during actual visual experience. If the end result (e.g. structured RFs) can be similarly achieved with unsupervised rules applied to natural image stimuli once it is available, what’s the benefit of these retina patterns?

Strengths:

- The role of wave patterns in the developing retina is of great interest in vision neuroscience
- The authors use real recordings of retinal waves to support their claims. These recordings are of great value!
- The authors also make comparisons with models of simulated retina waves.

Weaknesses

- The accuracy on MNIST was too low for a 3 layer network (Fig 1). A single linear SVM trained on pixels achieves 92% accuracy already ([https://dmkothari.github.io/Machine-Learning-Projects/SVM_with_MNIST.html](https://dmkothari.github.io/Machine-Learning-Projects/SVM_with_MNIST.html)) This weakens the validity of the claims.
- Object separability is measured on a single dataset with little resemblance to natural images (MNIST) that is far from real visual experience.
- There was little motivation (biological or otherwise) for only pretraining the first layer on retinal waves (and not deeper ones). There was also no correspondence between network layers and areas in the brain that the model was attempting to match.
- There were no comparisons between initializing the network with pretrained weights before and after visual experience.
- The claims depend strongly on network choices (e.g. width and depth)
- Link to code disservice double blind review.
questions: - Does initializing the network with the weights resulting from pre-training on retinal waves (compared to random) lead to better object separability (or any other benefit) after training the whole network on visual inputs with either supervised or unsupervised objectives? Aren’t natural images enough to induce retinal spatiotemporal receptive fields (e.g. ON-OFF center surround RFs) from scratch?
- What is the effect of dimensionality (width of the network) when comparing random vs pretrained?
- What is the effect of depth when comparing random vs. pretrained? Presumably classes will be more separable if you add nonlinearities (even when the first layer has random weights).
limitations: The authors could also discuss the limitations on running this analysis on a simple binarized set of stimuli (MNSIT). 
recommended_decision: 2: Borderline
review_title: Conceptually lacking approach to understand the computational role of retinal waves in the developing brain.

Decision: Accept (Poster)