Venue: NeurIPS 2022 Workshop NeurReps

Reviews:

final_score: rating: 3: A paper with technical flaws, weak evaluation, or inadequate reproducibility.
review_content: summary: The paper is proposing to use deep recurrent autoencoders to neural science. 
strengths_and_weaknesses: Weakness:
- Unfortunately, the presentation of the current version of the paper is not friendly for readers that are not familiar with the field.
questions: - In the previous work, can the authors formulate the problem, i.e. the goal, the challenge, the meaning of the variables? The current version seems hard to interpret mathematically.
- How should we interpret figure 3? Specifically, what does the height of the histograms mean?
- Why is the propose framework in section 3 superior to the previous works?
limitations: See the weakness and strengths section.
recommended_decision: 1: Reject
review_title: A bit hard to interpret

final_score: rating: 8: Technically strong paper with novel ideas and high-to-excellent impact, and excellent evaluation and reproducibility.
review_content: summary: The paper proposes a neural network and training loss for learning a dictionary for natural images. It presents evidence showing that the proposed network learns a more biologically accurate diversity of image filters. They additionally suggest an extension to apply the network to a supervised learning task.
strengths_and_weaknesses: **Originality:**
Overall the work presents an original network and loss combination. The underlying architecture presented in figure 4 is quite reminiscent to that of Rolfe and LeCun, 2003, although their use of the LLD and sparsity losses are different, and in my opinion more compelling, than the standard L2 + L1 approach used in Rolfe and LeCun. I also believe the loss is unique in the sparse coding literature.

**Quality:**
While I would be curious to see several additional experiments (noted below), the methods exposition and experimentation provided are sufficient to support this as a promising direction of research.

**Clarity:**
The paper is well written and the ideas are sufficiently motivated and clearly presented. In a longer format I would appreciate a more pedagogical exposition of the LLD loss and how it compares to the sparse coding loss, which would make the paper more accessible to the broader sparse coding community.

**Significance:**
The work is a promising start for what could be a significant contribution to the field.

questions: **Recommendations for future work:**

(1) I recommend that the authors look into [1], which demonstrates that the diversity of filters learned by sparse coding can be considerably increased by increasing overcompleteness. My personal investigations have suggested that this difference in overcompleteness is the reason why Rehn and Sommer learned more diverse RFs, not the alternative loss. Exploring the effect of overcomlpeteness in your own studies would be very compelling, especially given that the advancements in compute capabilities allow for more extensive experiments.

(2)  The distinction between the architecture of the sparse coding network proposed by Rozell et al. and an unrolled autoencoder such as what was presented by Rolfe and LeCun is minimal. I would not suggest leaning on this as a source of novelty in future iterations of the work.

(3) Olshausen & colleagues often note that the reconstruction loss is a poor approximation to the objective of the visual system. It’s encouraging to see more sparse coding work moving away from l2 reconstruction, although I would suggest that you provide a more nuanced summary of the current thinking in the field. For example, as far as I know no one has suggested that explicit image reconstruction is a goal of biological vision systems.

(4) I would recommend that you use the LCA network (Rozell et al.) as your baseline comparison instead of SparseNet. It can be implemented easily in modern neural network libraries, and is more extendable in terms of available sparsity constraints.


(5) In addition to investigating the features learned, I would highly recommend that you explore how _inference_ is different between your proposed network and the SC comparison model. This can be done with typical sparse coding tasks, like denoising or inpainting. You could also explore convergence properties and sparsity vs representation quality.

**References:**
[1] Olshausen, Bruno A. "Highly overcomplete sparse coding." Human vision and electronic imaging XVIII. Vol. 8651. SPIE, 2013
limitations: If the authors intend to submit a full paper to a machine learning field then they should take time to compare their results to more standard (feed forward) neural network approaches. Convincing the ML field that sparse coding is good for semi-supervised learning tasks is often difficult, and bolstering the work with extensive ablation and comparison experiments will greatly improve the likelihood of acceptance.
recommended_decision: 3: Accept
review_title: Abstract review

final_score: rating: 5: Technically solid paper with some limitations e.g., limited analysis or evaluation.
review_content: summary: The authors extend the sparse coding framework of Olshausen and Field to use a different 'weighted' l1 regularization objective. They show that the resulting objective drives learning of basis functions which are more similar to V1 receptive fields in their degree of tuning.
strengths_and_weaknesses: Expanding theories of sparse coding to better capture the features of biological circuits is an area of much interest in neuroscience, and the authors provide interesting new ideas about what could be driving the emergence of V1 receptive fields.

A shortcoming of the abstract is that they seem to devote very little space to actually exploring their proposed solution, with the majority of the text and figures focusing on previous work and related methods. This also made it challenging to figure out what exactly their method did, with little description of the architecture (is it just a linear projection?).
questions: How good is the weighted l1 loss for reconstruction compared to the original sparse coding framework? Is there a price to pay in accuracy for the improved match to the Gabor-like V1 filters?

It is not super clear to me why 'algorithm unrolling' is used to learn 'A' and 'X' in Equation 1 -- is this more efficient than simple gradient descent on the objective in equation 1?

If I understand the method correctly, it is really a linear model (i.e. Yhat = AX is linear in X) and the 'recurrence' is only used for optimization. It therefore feels somewhat misleading to call it a 'deep recurrent autoencoder', and I would suggest instead focusing some of the discussion on the properties of different types of regularization rather than 'deep' vs 'shallow' models.

It is not clear to me what the utility is of discussing the iterative Laplacian scheme in equation (4) - is this used anywhere in the abstract?
limitations: It would be good to explore how their method compares to classical sparse coding on other metrics than just the distribution of spatial phases.
recommended_decision: 2: Borderline
review_title: More discussion of method?

Decision: Accept (Poster)