Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a translation-equivariant neural model where the neural response to a stationary stimulus is equivalent to convolving with a Gaussian; however, through recurrence, the response is maintained in the absence of a stimulus.  Introduces the use of Lie algebra for deriving the translation generator.



strengths_and_weaknesses: Strengths:
- the framing in terms of group theory, equivariant representations, and the exposition is fairly clear. 

Weaknesses:
- it is unclear to me what parts of the paper are mostly for exposition (drawing on previous work of Kechen Zhang), and what parts are novel contributions. Most of the paper consists of theoretical derivations, and it’s often not made clear where results were proved in other papers.
- given the focus on the 1D case I think the title is a bit too general.

For example:
- Section 3.2 shows that the fixed points of continuous attractors are equivariant representations, but I’d be surprised if this result was not already known or demonstrated
- Section 4.1 uses the derivative as the translation operator, as does Kechen Zhang’s (1996) paper which is cited here but not really discussed. It seems that the real contribution is the connection between this derivative to group actions, but I wish this were more clearly laid out.

And some thoughts on the circuit model:
- The discussion of biological (im)plausibility seems weak. There’s not really a problem of multiplicative modulation — synaptic weight multiplied by variable input would do fine (as the authors do in eq 19b). And the other problem they mention — Dale’s law — is easily circumvented by having excitatory and inhibitory neurons, again as the authors do.
- The more critical biological concerns, I think, are (1) biological noise (spike timing, synaptic noise, etc.) and (2) that the Drosophila E-PG cells number about 40, whereas the attractors considered are in the infinite neuron limit. I think some acknowledgement of these points/experiments proving the analysis holds up under these constraints would strengthen the paper.
- in particular there’s a fair amount of literature with CANs and how they deal with noise, drift, limited precision (e.g. Burak & Fiete, 2009) — and it’s surprising to see very little of this discussed here.

Given that velocity-controlled inputs are already the inputs to many CANs for path integration, and examples beyond integrators aren’t discussed here, it seems the main contribution is an interpretation of/twist on existing models, rather than a new model per se. For this reason, I would have appreciated more suggestions on what to do with this Lie group understanding: whether on novel neural circuits, new neural mechanisms that implement the transport operator, or even learning of group of structure.


questions: I am curious about challenges the authors faced - i.e., what did not work?

limitations: general limitations, see above.
societal impact: n/a
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors provide an exciting extension to the classic bump circuit model [20]. The work provides a connection between Lie group translations and continuous differential recurrent circuits (or continuous attractor networks, CANs) to demonstrate that CANs can effectively encode a continuous variable shift, such as position. They derive the circuit equations, implement numerical simulations that match the theoretical assumptions, and suggest modifications to more appropriately accommodate known biological constraints.
strengths_and_weaknesses: **Originality:** Of course the concept of using a recurrent circuit to encode an environment variable, such as position, is not new. Additionally, the application of Lie algebra to achieve equivariance in neural circuits is not new. However, as the authors point out and as far as I know, the connection that they provide between these two concepts is both novel and exciting. I am fairly confident that the additional clarity and rigor that the present study provides over previous work in the space is an original contribution.

**Quality:** I believe the mathematics are technically sound, and the supporting experiments demonstrate the applicability of the theory. I have some minor comments below about including more discussion on weaknesses in applying the theory.

**Clarity:** I include minor comments below to improve clarity, but overall the exposition is clearly communicated.

**Significance:** I believe the submitted work is a significant theoretical contribution. It draws on several ideas from iconic work in neural coding [20], and the application of Lie algebra lends itself to several future extensions.

questions: 1) Implications of the gaussian tuning curve assumption (lines 87-88) – A Gaussian profile is certainly a fair assumption; it is used in a majority of neural sensitivity research. But there’s also good evidence to suggest that at least visually selective neurons (probably others as well) are better fit by leptokurtic tuning curves, which demonstrates an increased selectivity for preferred stimuli [1]. I think this could be covered in a future publication (i.e. is not necessary to implement in the rebuttal phase), but I am curious if you have thoughts on how one might apply the derivation in section 2 of the appendix to a generalized normal distribution? Additionally, does neuron selectivity have an impact when you are considering an infinite set?

2) notation clarifications – I initially ran into some confusion with the notion that I think could have been avoided with a more gentle introduction at the top of section 2. 

2.a) There are a lot of conceptually overlapping variables that correspond to the stimulus & its perturbations (s, s’), neuron preferences & their perturbations (x, x’), the addition of an index when referring to individual neurons (x_j), and resultants from perturbing a stimulus (e.g. ‘a’ in T(a)*s = s+a). I understand the need for all of them, but an early exposition on what they are to orient the reader would have helped me.

2.b) I also had confusion coming from switching between a discrete (countable) notation for the neurons and a continuous notation based on the preferred stimulus. As some examples of the former, consider the pictorial description in Fig 2.a; the use of the term “index” in the horizontal axis label in fig 2f; and the use of the subscript j in equation 6. An example of the latter is found in the use of an integral over alternate stimuli in the denominator of equation 8b, and the explanation of an “excitatory neuron preferring stimulus s=x” on lines 112-113. This is explained briefly on lines 121-126, although there you walk back the infinite neuron description and describe it as “a large population of neurons.” I am not suggesting that any of this is wrong, per say, but more recommending that you spend a moment earlier (for example near line 86 or at the top of section 2) to explain what is going on here and when/why you will be flexible with notation.

3) What are the empirical implications of the infinite neuron assumption? I would appreciate a discussion at the end of the paper on what it means to implement such a system with a finite set of neurons. I may have missed this in the paper, but what were the implementation details for the experiments shown in figures 3 and 4? How does the number of neurons (i.e. resolution of the discretization of the stimulus space) influence the analysis?

4) Line 22 “decode the instant orientation” did you mean instantaneous?

5) Typo on line 35 of the supplement – “clear to se”

[1] Ringach, D. L., Shapley, R. M., & Hawken, M. J. (2002). Orientation selectivity in macaque v1: Diversity and laminar dependence. Journal of Neuroscience, 22(13), 5639–5651.

limitations: The authors note limitations in sections 5 and 6. I note some questions that touch on limitations of the presented work (e.g. the mismatch between theory & experimentation).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Many artificial neural networks should obey symmetry relationships.  This paper derives a translation operator for a ring attractor network inspired by recent findings from the drosophila head direction system.   The resulting system is designed to be equivariant to translation and is at least roughly neurobiologially plausible.

strengths_and_weaknesses: Strengths:
This paper strikes a nice balance between abstractness and specificity.  The math is initiated at a very high level, but is then implemented for a very specific circuit.

The model makes a number of predictions that could be tested with existing technologies.

Weaknesses:
My major concern about this paper is that perhaps something similar was already worked out, perhaps for the grid cell system but I'm unaware of that work.

questions: 
35: ...explicitly represent...
77-8: ...the above derivation...
92: Not convinced it's strictly necessary that the units have identical tuning curves.  Agree that it's convenient.
115: Divisive normalization...
138: ...disrupted by noise.
171: ...utilize the ...
213: Don't know that much about drosophila specifically, but in general, multiplicative modulation of weights is not challenging in neurobiological systems.  Keywords are ``shunting inhibition'' and ``cortical gain control''.  

Would be good to note that there are in fact speed cells in the grid system, e.g., Sargolini et al., (2006).  However their firing rate profiles are relatively complex (see for instance, Dannenberg, et al., 2019, J Neuro; 2020, eLife).

Authors might be interested to compare these results to Tanaka & Nelson (2019, PRE).   In thinking about 3-D transformations, Shepard (1994, Psychonomic Bulletin & Review) may be helpful.


limitations: 
The paper is far too sanguine about extending this approach to place cells/grid cells.  This is likely to be much more complicated than one might expect from this paper.  Place cells do not monotonically map the 2-D environment.  Locations nearer to reward locations and boundaries are overrepresented.  In addition, the grid system is much more complicated than a 2-D surface.  2-D position is represented in conjunction with head direction and speed.  Moreover, there appears to be a continuous spectrum of time constants in the speed system (Dannenberg, et al., 2019).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a recurrent neural circuit model with a continuous attractor state that models the heading direction system in the Drosophila and includes a circuit mechanism for translating the heading direction at arbitrary speeds. Moreover, their model is closely related to the Drosophila heading direction circuit.
strengths_and_weaknesses: Originality: As someone who is not an expert in the field of continuous attractor networks, it is difficult for me to assess the originality of this work in part because it was not clear what exactly were the authors' contributions. My impression is that the main contributions were stated in sections 4 and 5, but even within those sections it is not clear which parts are novel (e.g., equation 19 appears to be known). While they include numerous references and the second paragraph of the conclusion discusses on the novelty of this work, it would still be useful (for me) if the authors further delineate what is their contribution versus what has already been established in the literature.

Quality: As far as I can tell, the paper is technically sound.

Clarity: I did not find the presentation particularly clear, but this could be due to my lack of expertise in the area. I've listed clarifying questions below.

Significance: The paper address an interesting problem in neuroscience; however, it is difficult for me to asses exactly what the authors' contributions are versus what was already known. I readily acknowledge this may be my own lack of expertise, but I would like the authors to clarify this for me.
questions: - Line 37: The sentence "For example, ..." needs a citation.
- Line 96: Why is eq 6 the mean response of all neurons? Are you assuming infinitely many neurons? Is there a citation justifying this? Am I missing something obvious?
- Eq (7): What does $\overline u(s)\equiv\overline{u}(x-s)$ mean? This doesn't appear to be true based on the definition of $\overline{u}(x-s)$.
- Sec 3.1: Is there no external input to the system? Is it easy to include? How would that affect the system?
- Line 131: What is the definition of "free parameter"?
- Eq (13): Can I interpret this effectively as a restatement of the fact that $\hat T(a)=\exp(a\hat p)$?
- To what extent are eqs (16) and (19) equivalent? The authors discuss the relationship, but its not clear to me what the precise relationship is.

Minor:
- Line 20: reflects -> reflect
- Line 43: whereas -> however?
- Eq (6): I find the notation $\cup_j$ confusing. It looks like you're taking the union over $j$.
- Eq (10): The definition of $\overline{u}$ is different here than in eq. (7)
limitations: - Eq (19ab): Can external stimulus inputs be included in this model? It seems like those would be reconciled with the speed inputs via some sort of Kalman filter. Is the speed $v$ assumed to be an external stimulus that feeds into all of the speed neurons? Presumably the speed is computed in part from the CAN responses? I'm not suggesting that this circuit should include these computations, rather I'm trying to understand to what degree this model is flexible enough to include these considerations.
review_title: null

Meta-review: metareview: The paper constructs recurrent neural circuits that represent stimuli equivariantly with respect to a given symmetry, taking the example of the 1D translation group. Most Reviewers were positively impressed by the general framing of the problem in terms of group theory and the elucidation of a connection between Continuous Attractor Networks and Lie groups.
The main weakness pointed out by the Reviewers was that the high-level exposition of the paper tended to conceal the distinction between novel original contributions and the previous literature. This concern was however resolved in the rebuttals in a way that seems to satisfy all Reviewers.
Further concerns about the biological plausibility of the proposed general construction and the potential difficulty of extending it beyond the restricted 1D case analyzed in the paper were also raised by several comments in the reviews, but also in this case the clarifications in the rebuttals seemed to convince Reviewers, who by and large expressed optimistic views about the significance of the work for its potential for future developments and applicability for modeling other neural systems.
confidence: Certain
Decision: Accept