Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Presents mathematical analysis and computer simulation of model that maximizes mutual information between photoreceptors and the rgc outputs of the retina.  Shows that several different spatiotemporal filters are derived that mosaic the retina in different ways.
strengths_and_weaknesses: Strengths

A nice formulation and modeling approach to understand retinal function.  Impressive technical work.

Weakness

Not sure what the main take away is.  The goal appears to be to understand the neural encoding in the retina, but after that the analysis and results, there is no attempt to tie these back to neurobiological mechanisms.  It seems one could, but the paper just ends with the statement, "our results are in strong agreement with observed retinal data," which leaves you hanging.

Specific issues:

The difference of Gaussians model in eq. 8: it mentions that the center position of each kernel is different for each neuron, but is this also learned?  not mentioned.

Section 3:  linear model in the continuum limit - this is very unclear.  what is being continuized?  space?  The integral is over frequency space - not following what's going on.  principal vectors a_1, a_2 and reciprocal vectors b_1, b_2 - what are these?  

Section 4.1: " power spectral density can be well approximated by a product of spatial and temporal power-law densities" -  Dong & Atick is cited, but curiously the claim the exact opposite, it is not separable.

Figure 4, panel A shows striking clustering in temporal spectral centroids - they are all stacked neatly in tight columns, no scatter.  is this what emerges from the learned filters, or is somehow the quantization imposed?  

The mosaics are interesting to look at, but not clear what to take away from this.  

Overall this seems like a very promising direction, I want to like this paper, but I find it a bit confusing and lacking a clear message. 

questions: see above
limitations: see above;  no societal impact issues.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a model for retinal mosaic organization, derived
by application of the efficient coding principle to natural movies. In
particular, the model shows that the total number of retinal ganglion
cells is a key parameter that controls the emergence of distinct cell
type. The model is also used to extract predictions about the relative
phase of distinct retinal mosaics as a function of input noise.

strengths_and_weaknesses: ### Strengths
The paper tackles a relevant problem, that is modeling the
emergence of distinct retinal mosaics from an efficient coding
perspective, by building on a solid foundation (reference [10]).

### Weaknesses
The paper is very hard to read, mostly because (1) its
tight coupling with reference [10], and (2) how much of it is
relegated to the Appendix. With respect to (1), I am not sure I
understand fully what parts of the model, and which results, are fully
novel here and which are minor refinements or variants of the results
in [10]. Regarding (2), the Appendix includes not only derivations for
most of the equations shown in the main text, or technical details for
the realization of some of the plots, but also - to my understanding -
the basic logic of some of the results (I'll give more details on this
below, under "questions").

Additionally, the paper claims "strong agreement with observed data" but contains no data and no quantitative assessment of such claim.

questions: 1. Is it possible to clarify better how the differences between this
   work and [10] (listed in the "related work" paragraph) affect the
   main conclusions of the paper? Or is the main point that the
   results here are qualitatively similar to those in [10], despite
   the differences listed under "related work"? (I am referring mostly
   to the material in Section 3). For instance, the first sentence of
   the Discussion reads: *"Here, we have shown that efficient coding of
   natural movies results in the formation of multiple receptive field
   mosaics with complementary filtering properties [10]."* Is this
   suggesting that the main result of the paper is something for which
   ref [10] should be credited?
2. Lines 50-53 read: *"In the case of linear encoding, we show
   analytically how a tradeoff between information gains from
   increasing mosaic density and informationg gains from adding new,
   specialized cell types leads to the addition of new mosaics
   capturing higher temporal frequency information."* Can you make sure
   that the results backing up this statement are easier to find in
   the text? If I am not mistaken, the passage should be lines
   182--189. That part of the text hinges on the fact that *"as
   detailed above, adding filters covering non-overlapping spectral
   windows allows mosaics to specialize to different regions of
   spatial and temporal frequency"*. The "above" here probably refers
   to the previous paragraph, that ends with the following: *"In [10],
   the authors considered a form of this with spatial bandwidth
   partitioning (v(k)\*v'(k) = 0), but as we show below, this special
   case encounters a limit beyond the first two mosaics, when the
   spatial passband spectra of new filters substantially
   overlap. Rather, the more general solution is that mosaic filters
   are band-limited in both space and time, arranging themselves to
   tile minimally overlapping regions with highest power in the (k,ω)
   plane (Appendix A)."* This passage, in turn, contains a reference to
   ref [10], one to "below", and one to the Appendix. Forgetting about
   [10] and assuming that "below" here refers to Section 4 (which is
   not valid for the original point we were trying to find support
   for, as we're looking for an analytical solution, as claimed in the
   Introduction), we therefore follow this chain of references to the
   Appendix. The relevant section seems to be section A.4, "Effects of
   channel capacity and new mosaic formation", but it would be great
   to have this spelled out more clearly, and perhaps the result in
   the Appendix also connected back to the claim in the main text in a
   more explicit way.
3. The Discussion states (lines 278--279) that the results are "in
   strong agreement with observed retinal data". I found this
   statement surprising, given the absence of quantitative comparisons
   with empirical data in the paper. Is it possible to clarify what is
   meant by this, and to substantiate the statement with direct
   comparisons?
limitations: The limitations of the work are acknowledged very briefly in the
discussion (*"despite the strong assumptions of our model - linear
filtering, separable filters, firing rates instead of spikes -"*), but
not unsuitably so for a modelling work of this type.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a unified perspective on the appearance of different ganglion cell types (in terms of receptive field size, temporal properties and polarity) in the framework of efficient coding. They deploy a previously developed model and efficient coding framework to spatio-temporal movies. They show, first analytically in the case of a simplified linear model, later experimentally in the non-linear case, how optimal spatial and temporal filters change as the number of neurons (‘channel capacity’) increases. Additionally they investigate the resulting mosaics of similar types for (anti-) alignment and show that input as well as output noise has an important but opposed influence on the alignment of ON/OFF receptive fields. 
strengths_and_weaknesses: ### Strengths

The authors embed their work excellently into the previous literature and work clearly out the parallels and their novel contributions. The theoretical as well as the experimental approaches are well described and cover different interesting arrangements and detailed analyses. All experiments are carried out carefully and are of very high quality, as well as the figures. The authors push the understanding of retinal layout further and try to unify previous approaches. 

However, there are some minor weaknesses: 

### Weaknesses
1. There are some open questions (see below).
2. A simple schema for the model would help to understand the setup. 
3. From equation (2) it does not become clear what the differences to [7] are, and how the formula is derived (as I could not find the exact formulation in [7]). 
4. Some parts of section 3 (and especially Suppl. A) are quite technical and it is sometimes difficult to follow. The authors could try to strengthen a ‘red thread’ and give also an intuition where possible.
5. Sections 3 and 4 seem to be rather detached from each other. While formulas are derived thoroughly, their interpretation and link to section 4 could be improved. 
6. Their final conclusion that previous cell types change down in temporal frequency (l. 274) is not well supported by the presented experiments. While it is more obvious for the spatial frequency, an additional analysis for the temporal domain could help to support this argument. 


### Minor comments:
- l. 256: shouldn’t it reference to eq (8)?
questions: 1. Fig. 2 a: How do the ‘experimental dots’ relate to the presented isoclines?
2. Could the authors comment on how their argumentation in line l. 186 (and also later) is depend on the assumption of time-space separability? While a detailed analysis is beyond the scope of the manuscript, the specialization to different regions of spatial and temporal frequencies is certainly influenced by this assumption. 
3. l. 196: The data distribution was approximated by a multivariate Gaussian. Is this an appropriate approximations given the known power laws for natural scene statistics? How would a different distribution change the results?
limitations: The authors did not comment on limitations or negative societal impact. While latter remains quite abstract, the limitations could be indeed discussed in more detail.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors study how retinal mosaics emerge from an efficient coding framework. They extend the framework to spatial-temporal kernels and video inputs, which is a crucial extensions to understand how natural environmental statistics influence coding in neural systems.  Also, the approach is highly innovative as it allows to derive multiple optimized mosaics.
strengths_and_weaknesses: Strengths:
* Extend previous models to spatiotemporal signals 
* Study relationship between natural video statistics and optimal neural codes
* Interesting approach to derive multiple mosaics with interesting results

Weaknesses:
* Sometimes a bit jargony/dense - the authors should check how understandable each section is and make sure they can be followed by non-theoreticians
* Limited by focus on efficient coding objective - the authors could discucss whether reconstruction of the visual environment is really the ultimate goal of the visual system
questions: * Lines 124: The Bravais lattice is not sufficiently explained for non-physicists. The section up to Line 129 could be better explained. What are principal and reciprocal vectors? 

* Fig 2A: The colored points should be defined in the legend.

* Fig. 5 is very dense and hard to understand - maybe more intuitive labels would help?



limitations: The authors should discuss the limitation of the efficient coding objective - is that really all that the organism cares about?
review_title: null

Meta-review: metareview: This paper received 1 accept, 2 strong accepts and 1 reject. All reviewers agree that the proposed model is elegant and that the technical work is impressive (even the negative reviewer). The main criticism of the negative reviewer is that  the main take away is not clear. The authors submitted a revised version of the manuscript. Sadly, the reviewer did not read the rebuttal and/or engage in a discussion post-rebuttal. The AC considers that the main criticism of this reviewer was addressed. In light of this, the AC recommends the paper to be accepted.
confidence: Certain
Decision: Accept