Translation-equivariant Representation in Recurrent Networks with a Continuous Manifold of Attractors
Authors: Wenhao Zhang
Ying Nian Wu
Si Wu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural coding
Equivariant representation
Continuous attractor neural network
Lie group
Drosophila's heading system

Abstract: Equivariant representation is necessary for the brain and artificial perceptual systems to faithfully represent the stimulus under some (Lie) group transformations. However, it remains unknown how recurrent neural circuits in the brain represent the stimulus equivariantly, nor the neural representation of abstract group operators. The present study uses a one-dimensional (1D) translation group as an example to explore the general recurrent neural circuit mechanism of the equivariant stimulus representation. We found that a continuous attractor network (CAN), a canonical neural circuit model, self-consistently generates a continuous family of stationary population responses (attractors) that represents the stimulus equivariantly. Inspired by the Drosophila's compass circuit, we found that the 1D translation operators can be represented by extra speed neurons besides the CAN, where speed neurons' responses represent the moving speed (1D translation group parameter), and their feedback connections to the CAN represent the translation generator (Lie algebra). We demonstrated that the network responses are consistent with experimental data. Our model for the first time demonstrates how recurrent neural circuitry in the brain achieves equivariant stimulus representation.

Translation-quuiarriatt Representation in Recurrent Networks with a Continuous Manifold of Attractors
Wen-Hao Zhang 122- wenhao . www.gngusooottosstttm.. edu
Ying Nian Wu ywuêstat, ywudstat.ucla. ucla. edu
Si Wur
siwu@pku.edte edu. cn
Lyda Hill Department of Bioinformatics, UT Southwestern Medical Center.
"O'Donnel Brain Institute, UT Southwestern Medical Center.
3 Department of Statistics, UCLA.
Schooo of Psychology and Cognitive Sciences, IDG/McGovern Institute for Brain Research, Center of Quantitative Biology, Peking-Tsinghut Center for Life Sciences, Peking University.
Abstract
Equivariant representation is necessary for the brain and artificial perceptual sys- tems to faithfully represent the stimulus under some (Lic) group transformations. However. it remains unknown how recurrent neural circuits in the brain represent the stimulus equivariantly, nor the neural representation of abstract group operators. The present study uses dieeeiinnnional (ID) translation group explore the general recurrent neural circuit mechanism of the equivariant stimulus representation. We found that a continuous attractor network (CAN), canonical neural circuit model, self-consistently generates a continuous family of stationary population responses (attractors) that represents the stimulus equivariantly. Inspired by the Drosophila's compass circuit, we found that the ID translation operators can be represented by extra speed neurons besides the CAN, where speed neurons' responses represent the moving speed (ID translation group parameter), and their feedback connections the CAN represent the translation generator (Lie algebra). We demonstrated that the network responses are consistent with experimental data. Our model for the first time demonstrates how recurrent neural circuitry the brain achieves equivariant stimulus representation.
example
Introduction
Neuronal responses in the brain internally represent the stimulus in the world. When the stimulus changes due to some transformations, the brain updates its internal representation accordingly to faithfully reflect the stimulus change. For example, VI neurons encode the visual orientation, and their population responses track the changing orientation which enables subsequent brain areas to decode the instantaneous orientation. Transformations acting continuous stimulus be described by the Lie group theory [[], and a neural representation reflecting the stimulus transformation is called group equivariant representation 42.4. Mathematically, supposing Lie group G acting continuous stimulus s, the evoked neural population response ū(s) (a continuous function of s, and denotes the mean response) is equivariant with the action of group G (group homomorphism) if,
f ū(s) TTT 8). 4g G, Vs.
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A Stimulus
Equivariant map
B
Mean resporse
sie x)
Meen responis
B(x-a)
Fiet
Neural
(8))
((s)
response
Stimulus feature
index
Figure 1: (A) An equivariant map between the stimulus and the neural representation. (B) Tuning curves (mean responses) of a population of homogeneous neurons in response one-dimensional stimulus feature s, where each neuron j has its unique value of preferred stimulus 2j. The tunings of neurons have the same profile and only differ in the preferred stimulus zy. (C) one-dimensional translation operator T(a) shifts the population response ū(r - s) by amount a.
Tg and Tg are the induced actions (operators) with group element g E G which respectively act on neuronal responses and the stimulus, and denotes the action of group operators. Intuitively, the equivariant neural representation (Eq. [ implies the neuronal responses evoked from a transformed stimulus (c.g., rotating the orientation of Gabor images), i... a(T, s) (Fig. DA blue path) would be the same as transforming the neuronal responses evoked by the original stimulus along the stimulus manifold, i.e., Fg ū(s) (Fig. LA green path).
The equivariant representation is indispensable for the brain to represent the stimulus under transfor- mations. An equivariant neural circuit not only required to represent the stimulus equivariantly (0(s). Eq. l.. but it also needs to explicitly represents the Lie group operators (f,, Eq. Explicitly representing group operators enables the brain to actively synthesize a transformed stimulus and predict the changing stimulus under transformations [5]. For example, when tracking a moving object, the brain not only represents the object's 's instantancous position 8, but also represents its moving speed (the translation group parameter) in order to predict the stimulus position in the next time step. Yet it is open question as to how recurrent neural circuits in the brain equivariantly represent the stimulus, and how abstract group operators are represented in the brain.
Neuroscientists utilized the Lie group to investigate the perception and representation in the brain (e.g., 14) ), however, nearly none of previous studies linked the abstract group theory to concrete biological recurrent circuit dynamics. Meanwhile, although equivariant neural networks have been extensively studied recently in machine learning (ML) research to capture transformations stimuli (input data) defined graphs (e.g.. (15) or defined smooth manifolds (e.g.. 3 [ 16, 17)). nearly all these studies focused on developing feedforward weights which act as group convolution kernels in order to yield equivariant outputs under group transformations. There has not been much ML research studying equivariant recurrent networks, nor explicit network representations of group operators.
To provide theoretical insight into this fundamental question, we used 1D translation group acting on ID stimulus feature as an example to investigate the general recurrent circuit mechanism of equivariant representation. We found the equivariant stimulus representation can be naturally realized in continuous attractor networks (CANs) which self-consistently generate continuous family of stationary population responses (attractors) to represent the stimulus equivariantly. Moreover, the translation group operators are represented by separate neuronal populations whose responses represent the translation speed (translation group parameter). The feedback connections from speed neurons to CANs are shifted on the stimulus manifold, which encode the translation generator (Lic algebra). We compare the proposed recurrent circuit model with Drosophila's 's internal compass circuit to show its biological plausibility. The present study is one the first studies linking the equivariant representation to biologically plausible neural circuit model with rigorous mathematical analysis. It will shed light on the neural representations in the brain.
2 Translation group and homogeneous neural population code
We start from a concrete Lie group and neural representation to investigate the general circuit mechanism of equivariant representation. As an example, we consider ID translation Lie group T acting on ID stimulus which can be regarded to be heading direction, orientation, moving direction, spatial location, etc. A group element ((a) T translates the stimulus 8 to 8 + a, i.e., T(a) s a, and therefore an equivariant neural representation ū(s) should satisfy (Fig. IP

---

ū (T(a) ss = iis + a) ((a) a(s),
(2)
where T(a) is the ID translation operator inducing the translation of neural representation (Fig.
Several properties of the T(a can be directly derived from the definition (Eq. [18]
7(0) = 1, (b) Ta b) T())((a)... = -
(3a)
(3b)
(3c)
First, translation operator with zero translation should leave the neural representation of stimulus to
be the same as no transformation (Eq. [aa. Second, the translation operator can be composed together and the composition is not dependent on the order of operators (Eq. (b),, which implies the translation
operator is commutative or abelian [8). Third, the inverse a translation operator corresponds to an
operator translating the opposite direction with the same distance (Fq.. 3cb. Since the translation is continuous, the amount of translation can be made infinitesimally small. Consider an infinitesimal translation da 0 on the stimulus 8, then the corresponding change of neural representation is,
T (ōa) m(s) ū(s + da) i(s) da),ū(s) (1 + )ad,))(())
where 0 = 0jas denotes the derivative over s, and first order Taylor expansion was used in above derivation, ý O2 is called the translation generator characterizing the tangential direction of translation in the group space, and forms a basis of the Lie algebra. In physics, ý is usually regarded as the momentum operator [8]. Using the infinitesimal translation, the differential form of a translation operator can be derived as (see details of Lie group in Supplementary Information (SI)),
(f(a) da
T(a + 5a) T(a)_
(66a) -
lim 8a-10
lim 60-+0
M(a) = P-)
(5)
da
da
By exponentiating the two sides in Eq. (). the translation operator can be solved as m(a) = exp(ap).
which is an exponential map of the translation generator p.
We next define a concrete neural representation based on experimental observations. The neural representation ū(s) {s) y=1 regarded as the mean responses of population of N neurons. Since the translation operator (Eq. 2 acts on continuous function of neural population response, i... ūs)). we therefore consider the limit of large neuronal population with infinite number of neurons (N - 20), and then ū(s) effectively converges to a continuous function (a vector of infinite dimension Hilbert space). Inspired by the bell- bell-shape stimulus tuning curves widely observed the cortex, the mean response of the j-th neuron is modeled as a Gaussian function of the stimulus (Fig. TB).
{s)]
(+()) ū(2 8) = expl - /
(6)
where xy denotes the preferred stimulus of neuron, indicating the neuronal response reaches its maximum when the stimulus matches its preferred value, i.c. Iz. Uj and σ, denote respectively the peak firing rate and the tuning width of the j-th neuron. A necessary condition to construct an equivariant neural population code is homogeneous neurons in the population, which means the tuning curves of all neurons have the same profile, i..., U, = U and σj = a for any j. and the preferred stimulus of all neurons, {2z is uniformly distributed on the stimulus space. In this way, the mean responses of all neurons in response to a stimulus s are (Fig. [[C
i(s) = a(x 8) U Uexp[-(e ()) -)2/ /A0").
which can be verified to satisfy the translation equivariant representation (Eq. 2
3 Towards an equivariant recurrent neural circuit model
We next investigate how the equivariant homogeneous neural code, i.c., 0(x in Eq. 7 can be self-consistently generated by a concrete recurrent neural circuit dynamics. Specifically, we explore the recurrent circuit mechanism generate a stable stimulus representation ū(æ - s) without translation (corresponding to a zero translation operator, 7(0).. Afterward, we investigate the neural representation of translation operators and how it induces the translation of stimulus representation.

---

A
E
6
180 In
synupse
Nauron Index
Eigevvector B Byale
pool
8
Filng raee
Elgenvec.
180
10
Timo (4)
D
index
Rec woigt W(,,e) W
40
Moan risporse
Eigenvelues
üi s)
10
-100
180
10
Tuning disparlty
Neuron index
Neuron index
Eigevvaue index
Figure 2: A continuous attractor network (CAN) celf-consistently maintains an equivariant stimulus representation. (A) The CAN structure. Excitatory (E) neurons are aligned based on their preferred stimulus H, and a pool of inhibitory neurons provides negative feedback to ensure the network stability. (B) The recurrent connections between E neurons are translation-invaria
with respect to the preferred stimulus x. (C-D) An example of spatiotemporal responses of all E neurons in the network model (C) and their temporal average (D). Note that the non-zero response is self-sustained by the network without external input. (E) The state space of CAN's population responses contains a one-dimensional continuous manifold, on which each location denotes population response ū(z m s) representing the value stimulus feature s. Top: a schematic diagram of the energy landscape the CAN. Bottom: three example population responses which correspond three states the continuous manifold. (F) Eigenanalysis of the perturbed CAN dynamics. Top: the eigenvector (blue) with the largest eigenvalue (A1 1) corresponds to infinitesimal change of population responses (orange) along the stimulus feature manifold. Bottom: the eigenvalues the perturbed CAN dynamics where the largest eigenvalue is 1, indicating neutral stability along the stimulus manifold.
3.1
The autonomous dynamics of continuous attractor networks
Since the recurrent neural circuit dynamics needs to generate non-zero mean responses ū(æ s) for any value (Eq. 6), the neural dynamics should be stable any s. That is, there is manifold of fixed points (attractors) in the state space of network dynamics. The continuous manifold of attractors emerge from the autonomous dynamics (no external input drive) of continuous attractor networks (CANs, Fig. [A]), which is a canonical network model explaining the processing of continuous stimuli in neural circuits I9 200), including reading-direction (21), orientation [22 spatial location (23
and so on. In the present study, we consider a CAN dynamics as follows 24 25
Ou(x, t)
-u((x,)) + W. x2) r(æl, t)dx',
(8a)
Ot
[u(2,, 1))
r(x,t) Giu(xx = 1 kp flu(x', d)
(8b)
where u(x. 1) and r(x, f) denote respectively the synaptic input and firing rate of the excitatory neuron preferring stimulus & = x at time f, and r the time constant of synaptic input. G(.) is an activation function that is modeled divisive normalization where [++ is nonnegative rectification and & determines the global inhibition strength. The divisive normalization is canonical operation widely observed in the cortex (26), and resulted from the parvalbumin (PV) inhibitory neurons (7]
(x, denotes the recurrent weight from the neuron preferring x' the neuron preferring æ, which is modeled as Gaussian function of the disparity between the preferred stimuli of the pre- and post-synaptic neurons (Fig. (23, i.e.,
W.(x, x') Wr(x - x') - ww ( exp
where is scalar and determines the peak recurrent weight, and a the width of connection in the stimulus space. Because we consider a large population of neurons that are uniformly distributed in the stimulus space, the neuronal interactions effectively become integral in Eq. Sa with denoting the neuronal density the stimulus space. Furthermore, the maasslaiooninvariii recurrent connections, which only depend x x', imply that the recurrent interactions effectively

---

conducting convolution on the stimulus space, i..., W, (x 1 the symbol denotes the convolution over æ.
-r(a,, t)dz'' (W, (2,, t), where
3.2 A continuous attractor network equivariantly represents the stimulus
It can be checked that the CAN dynamics (Eqs. 8a can generate a family of non-zero Gaussian-
profile stationary responses that are consistent with the proposed responses (Eqq. 6 25 28
ā(z - s) U exp (- s)²/40"], F(x s) = R Rexp [- - s)² 2002
(10)
as long as the recurrent weight is stronger than critical value which is solved as We 222(2T) 1/4 /ka/p (see SI. Sec. 2). It worth noting that s is a free parameter in Eq. (TO). indicating that the network has continuous family of fixed points, i.e., ü(x 8), which self- (Eqs.
consistently emerge from the recurrent neural dynamies without the influence from external inputs
Sa 8b.. The stationary population responses (Eq. [T) are consistent with the equivariant homogeneous neural population code (Eq. 7) and therefore they represent the stimulus equivariantly.
The stationary population responses in the recurrent network should be stable in order to maintain the equivariant stimulus representation robustly, otherwise, the population responses can be easily disrupted by noises. Therefore we check whether the continuous family of population responses (Eq. [0] are stable (attractors), by performing the stability analysis of the network dynamics. We perturbed the network state around the fixed points by adding a small perturbation 8u(2z, t), i.e., u(x,t) t) u(x s) + 6u(x, t), and then the dynamics of the perturbation derived as (28).
T T du( t) = -du(z, + / K( x' 1')ssu(( t)dz'.
(11)
The interaction kernel K(x,, r'1s) = fW - x - ==)) s)/Ok(z' - 8) s)da", whose analytical expression be seen SI. Sec. ōu(z, t), all of its cigenvalues, Ans and (unnormalized) eigenfunctions, fn. can be analytically calculated (listed with descending order) (28).
3 Treating ((x, as operator acting on the perturbation
\ 1,
(x(s)) x 0,ū(x - s),
(12a)
A
V We? 2/w,"
fa(2(s) x ū(r - s),
(12b)
An = 22-n (n 2 3),
(x)()) =....
(12c)
The explicit expression of eigenfunctions for
can be found at SI. Sec. We see eigenvalues
are larger than (Fig. (FF indicating that the continuous family of fixed points (ø(x - 8) in Eq. 10) stable, i.e., the network has continuous family of attractors, and each attractor is an equivariant representation of the stimulus s (Fig. 2
4 Representing translation operators in CANs
is noteworthy that the eigenfunction fi (xr]s) of the perturbed network dynamics (Eq. 2 top) is parallel to the direction of translation generator (Eq. 12a). i.e., fi (r|s) xx 0,ū(r - s) p u(x s).
thus fi (ar]s) corresponds to the translation of neural responses along the continuous attractor manifold representing the stimulus 8 (Eq. F top). Meanwhile, its corresponding eigenvalue A = 1 suggests the eigenfunction fi (xx s) is neutrally stable, which is resulted from the translation invariance of recurrent connections in the network (Eq. 99. Nevertheless, although the CAN dynamics maintains the translation operator resulted from the external perturbation du(x, t), it does not self-consistently support non-zero translation generators in that the stationary network states are static without any translation (Eq. 10 Fig. CC-D), i.e., corresponding (()). We next explore self-sustained network
mechanism to represent non-zero translation operators, i.c., F(a), (a * 0) (Eq. 2 in order to induce translations neuronal responses ū(x - s).
4.1 Incorporating translation operators in a CAN
To embed translation operators in the CAN dynamics, we derive a differential equation of Eq. 2 and
compare it with the CAN dynamics. We consider the translation has instantaneous speed v(I), which determines the translation amount in time window dt da u(t)dt. Taking the derivative of Eq.
5

---

A
D
Commutation
180 R())
0 ex 0-0 0-9 6-0 1-0-0-0--0000 00000
Wiaaal support non-zero
Neuron Index
response ii
Speed
(auuu))c translation manifold
Speed (network)
Filing
Recurrnt weight
Speed
Bump postion
180
Speed
Integrated stimulus
30
0
Theory Sim
-50
Tuning disparity
Time
5
(rl
10
-50
Speed (thoory)
Time (d)
5
10
The derivative recurrent connection, 0₂ W, induces translation of CAN's population responses along the stimulus feature manifold, and represents the ID translation operator. (A) Illustration of the recurrent connections the network model. (B) An example of spatiotemporal responses in the network (top) and the internally represented stimulus feature moves over time with constant speed. (C) The actual moving speed population responses along the stimulus manifold is consistent with theoretical prediction by using the magnitude of O,W, (D) Commutative path integration in the CAN.
Figure 3:
2 over time t, and utilizing the differential form of translation operators (Eq. 5) we derive that (to simplify notation we 0 without loss of generality),
dū(x a) (a) u(x) da (dT(a) m(xx) = up T(a) ū(x) = vD ū(x - a).
dt
da dt da
Replacing the variable by x, we reach the differential equation governing the translation of neuronal responses with speed (t), du(x) = x()p a(x))
(13)
dt
In comparison, the CAN dynamics in the stationary state is dū(x) /dt = 0 (Eq. Sa), i.c., the network state statie. To incorporate the translation operator with speed vtt) (Eq. 13). we utilize that the stationary CAN state satisfying ū(x 8) pW, F(x - 8), and substitute it to the right hand side of Eq.
(14) where we have used 0, -. (Eq. 7). and swapped the convolution and partial derivative. The above equation provides insight that the translation generator be converted into the derivative of recurrent connections in the network, i.e., W, Therefore, it motivates us to consider a modified
Vp ū(x) = wpW F(x)] -rp(a. F(x),
CAN dynamics whose recurrent connections have component (0, W, Fig. 3A) 21 29).
Ōu(I, 1) Øt
u(x, t) + (W, Tua, W,) * r(x,t).
(15)
Next we verify whether the modified CAN dynamics (Eq. T5) embeds translation operators, i.e., spontaneously moving neural response along the stimulus manifold with speed We substitute the mean population response (ū(x - s) and f(x s). Eq. 10 into the above network dynamics, repeatedly use the stationary response that satisfies a(x
pW, F(z - s) to cancel terms,
Dū(r= 8)
01 1- 5) pW, F(x 3) v W,) F(x - 8),
(16)
-00 1pW, F(x x)] vp ū(x s).
Comparing Eqs. (66 and [3),, we see the stationary state in the modified CAN spontaneously translates along the stimulus manifold in way consistent with the action of translation operators (Eq. 13). Therefore the recurrent connection component W, represents the translation generator, and its magnitude determines the moving speed v. Numerical simulations of the network model also confirm our theoretical derivations (Fig. 3CC,,, where the actual moving speed is well predicted by the theory using the magnitude of recurrent connection 0 W,.

---

When the network response deviates from the stereotypical Gaussian profile (Eq. 10), the recurrent connection component W, will remove distortions because the eigenvalues of all perturbations perpendicular to the continuous stimulus manifold are smaller than (Eq. [2a [2c Once the population response settles into a smooth Gaussian profile, the network response translates along the continuous manifold with the desired speed (see details in SI. Sec. 4
4.2 Path integration in the modified CAN dynamics
The brain accumulates the effects of series of translation operators, and the combined translation effects are not dependent on their action orders (Eq. 3b), i.c., translations are commutative. The translation accumulation that is independent of action orders corresponds to the path integration in the neuroscience research [23 29 330, where the animals' navigation system integrates their walking path to update their walking distance. The path integration can be achieved in the modified CAN dynamics (Eq. [5]. To illustrate this, we compute the translation distance in the modified CAN
dynamics (Eq. 16) by using the chain rule, 0ū(x-s) Bū(z-s) As Dā(2-s)
v(),, and substitute it 8x
back to Eq. (16). Finally, the translation distance is derived as
ds (t)dtt
(17)
where ds is the translation distance along the continuous stimulus manifold in the CAN in a time window lasting dl. As an example, we consider a constant moving speed en lasts for time duration t1, followed by another constant speed U2 lasting for duration t2, and then we swap the order of two speeds (Fig. 3D. top). The moving distance in the two cases can be calculated as the same,
[8| = I di + Sr flLtt v2dtt = -f² u2dl + Srz flatt vidt 11 + 2212,
(18)
which shows the translation in the modified CAN is commutative, i.c., the network achieves path integration. This is also confirmed by numerical simulations (Fig. 3D, bottom).
5 A biologically plausible equivariant recurrent circuit
Although in the modified CAN dynamics the recurrent connection component 0, W, (Eq. 15 mathematically represents the translation generator and translates network responses, the connection O,W, is not biologically plausible due to two reasons. First, the magnitude of the connection component W, needs to be multiplicatively modulated by the speed u (Bq. 15), but multiplicative modulation of synaptic weights is challenging in neural systems. Second, with Gaussian- proffile connections (Eq. 99). the derivative a, W, means that a neuron has both excitatory and inhibitory synapses, which disobeys Dale's law that a neuron can be either excitatory or inhibitory but cannot be both. Therefore, we propose that is consistent with the biological neural circuitry. Mathematically, we will replace the term of multiplicative synaptic weight modulation (Eq. [5] last term) by a biologically plausible input term that is originated from separate neuronal populations whose responses represent moving speed v.
neural circuit model to represent the translation operator
The new circuit model is inspired by the Drosophila's compass system which consists of three neuronal populations (Fig. 4B, e.g., B1 32)). There is a ring of heading- direction (stimulus s) neurons whose dynamics is governed by CAN (Fig. 4B, E-PG neurons in blue disk, B3].. Besides, there are two populations of speed neurons (Fig- 4B, left and right P-EN neurons) whose firing rates encode the speed (translation distance in a unit time). Moreover, the feedback projections from speed neurons to heading- -iriinnn-dretiionn neurons are either clockwise or anti-clockwise shifted and they induce translations of aeading-directiinn model has three populations of neurons (Fig. [A): the stimulus s neurons which is modeled as CAN as before (Eqs. 8a 8b), and two populations of speed neurons which are supposed to encode the positive and negative speed respectively. The network dynamics is governed by 31 34),
neurons' responses. Therefore. our new recurrent circuit
T au(x
u(xx 1) + pW, * r(x, 1) + P Wm I'm (x, 1),
(19a)
Øt
Ou+ (x, t) - u (x, 1) Was (), r= = ((go + v)u - (Ir, 1)]
(19b)
Øt
Eq. 19a) the CAN dynamies of stimulus neurons. u. (ax, 1) (r (x, 1)) are the synaptic inputs (firing rate) of speed neurons which represent positive and negative speeds respectively and are driven
7

---

A D2 neuron
Shitted connections: translation generator
M
Syressm Sarg
Firing (a u.u.)
100
Moving spred
(ransiation (mme)
10
s neuron (CAN) ...
neuron
-10
Aur
180
-40
40
Neuron Index
Drosophila's compass circuit
F
Joint tunings of speed neurons
G
Left P-EN [B]] Right P-EN (M.)
10
M. neuron
10
12.
neuron
10
Fiing rioe (8.1).
Firing rate (a.u.)
Speed (net)
100
Speed
20
E-PG
0.5
0.5
-10
(represent 8)
-180
Stimulus
180
-180
Stmulus
0
180
-10
Speed (theory)
Figure 4: The network model with disentangled representations stimulus 8 and speed e (amount of translation). (A-B) The proposed network structure (A) is similar to the Drosophila's internal compass (B). The three bumps shown the middle represent the recurrent inputs from corresponding color coded neural populations. (C) The connection strength from speed neurons to stimulus neurons are shifted by Ar which represents the translation generator. (D-E) The difference of firing rate between two populations of speed neurons (D) determines the moving speed of population activities of stimulus neurons (E). (F) The joint tuning curves over stimulus and speed of two example neurons. (G) The proposed network implements the path integration with consistent speed theory.
by the stimulus neuron r(rr preferring stimulus æ. Note that stimulus neurons receive feedback inputs from speed neurons (Eq. 19a last term), which replaces recurrent inputs from biologically implausible recurrent weight ,, W, (Eq. 15 last term). Consistent with the Drosophila's compass
31 32 34]. recurrent connections W (a, x') from the speed neuron x' to the stimulus neuron æ is
shifted by FAr towards the purported moving direction encoded by speed neurons (Fig. AC).
W=(z,z') - (oooo) - exp [- (- A
(20)
Moreover, speed neurons' firing rates (xx f) are multiplicatively modulated by the speed (Eq. 19b).
This operation is necessary to reproduce gain modulation of speed neurons' responses (speed invariant stimulus tuming width) in present rate-based model [11, and it achieves the same computation with multiplicative modulation of synaptic weights (Eq. [5] last term). 9u is a constant determining the baseline activity of speed neurons. To emphasize the main computational mechanism, we simplify the circuit model (Eqs. 19a 19b) compared with the Drosophila's circuit. First, we ignored the interactions between speed neurons. Second, assumed that a speed neuron is only driven by stimulus neuron with the same preferred stimulus, which exhibited by the scalar Wes (Eq. 19h).
5.1 Neural representation of translation operators
We investigate how the abstract translation operator (a) is represented in the circuit model inspired by the Drosophila's compass. Similar to the theoretical analysis performed in Eqs. (6). we focus on the latent network dynamics along the dominant continuous stimulus manifold by ignoring the fluctuations along other directions. Then Eq.
(1aa) can be simplified into (see more details in SL. 5)
① Øt
PW
ū(x 3)
V2TU (R4 - R-) p- u)r-s)].
(21)
where R., and R_ are the peak firing rates of positive and negative speed neurons, and U is the peak synaptic input received by stimulus neurons (Bq. 10). The shifted connections W (1( (x, x²) from speed
neurons to stimulus neurons introduce the trans translation generator, in that the magnitude of translation generator is proportional to the connection shift Ar (Eq. 16). The excitatory W (x, x²) pulls the network response towards shifted directions (implementing the excitatory part of the translation generator), while the inhibitory part the translation generator is achieved through the network effect of global inhibition (Eq. 8b). Moreover, the firing rate difference between two populations of speed neuron, R, R_, determines the translation speed (ID translation operator parameter). Overall, the fixed translation generator ò is stored in the shifted connections from speed to stimulus

---

neurons, and the time-varying translation speed is represented by the firing rate difference between two speed neuron populations which can change in a short time scale.
Meanwhile, the speed neurons' dynamics (Eq. 19b) suggests R4 - R [[g + + - (g. - v)] 20wus R. To translate network responses with the desired speed, the coefficient the right hand side of Eq. 21 should equal to the speed determining speed neurons' response (Eq. 19b), which requires,
2pw Wys RAr = ~U.
(22)
We performed numerical simulation of the circuit model (Eqs. 19a 19b). whose parameters are set to satisfy Eq. (22). We firstly verified our theoretical prediction, i.e., the moving speed is proportional to the firing rate difference between two populations of speed neurons, i.c., R4 R_ (Fig. 4D-E). Indeed, the actual moving speed of stimulus neurons' population responses is consistent with the required speed that determines the firing rate of speed neurons (Fig. 4p), which supports the validity of our theoretical analysis and indicates the accurate path integration in the proposed circuit model. The joint tuning curves of two example speed neurons with respect to the stimulus s and moving speed plotted in Fig. 4F, which are multiplicatively modulated by the speed, i.e. the speed invariant stimulus tuning width, and this is consistent with experimental observations [31]
6 Conclusion and Discussion
Equivariant representation is necessary for the brain and artificial perceptual systems to reliably represent the stimulus under transformations 24 6 37. The present study investigates how recurrent neural circuits implement equivariant stimulus representation and Lie group operator representation. Our theoretical analysis shows that the stimulus can be equivariantly represented in homogeneous neural population code that collectively emerges from a CAN dynamics. The Lie translation group operators are represented by extra neuronal populations whose activities represent the translation speed (translation group parameter) and whose feedback connections to the CAN store the translation generator. Our study for the first time links the Lie group equivariance to concrete, biologically plausible recurrent neural circuit. It gives us insight into understanding the neural coding principle and may inspire a new building block for equivariant representation in ML tasks.
The CAN is a canonical neural circuit model widely used in neuroscience 20
25
38
39
Recurrent circuit models similar the present model were proposed before (e.g.. 29
31
36 40), whereas no previous studies theoretically linked the network dynamics to the equivariant
stimulus representation, nor the Lie group operator representation. Besides, our theory also derives
the condition that network responses translate with the desired speed (Eq. (2). which is testable by analyzing experimental data.
To highlight the basic mechanism of equivariant representation in recurrent circuit, the proposed neural circuit model has some simplifications compared with neural circuits in the brain. First, we considered homogeneous neurons and ignored the randomness on recurrent connections and neural noises in real cortical circuits. Apparently, the neural noise and random connections will break the perfect structure of equivariant representation. It has been suggested that the CAN dynamics is resistant to input noise by removing the noise component that perpendicular the attractor manifold [24 41, and then the equivariant representation can be approximately held. Another possibility is that the neural heterogeneity and neural noise can help the brain implement sampling-batt based Bayesian inference (e.g.. 47). where the instant response of the CAN and the activity of speed neurons represent the samples of heading direction and rotation speed, respectively (e.g., [48 4 Second, we analyzed a limiting case of infinite number of neurons in recurrent circuit model, resulting in that the translation operator (Eq. 2 acts on a continuous function of neuronal population response. Although it brings huge benefit in theoretical analysis, the real neural circuit only has finite number of neurons, especially there are only dozens of E-PG neurons in Drosophila's 's heading direction circuit (Fig. A. BI 32)). ). In the case of finite number of neurons, the recurrent circuit model can still have equivariant representation, however, the profile of population response and the convolution kernel will slightly deviate from Gaussian and the concrete profile can be obtained by numerical optimization, as suggested by a recent work [40].
The proposed neural circuit model can be extended to capture more characteristics in real neural circuits. For example, our model doesn't 't have the sensory input order to emphasize an internal mechanism inducing the translation of neural representation. The sensory inputs that represent

---

instantancous stimulus direction and the instantaneous velocity into Eqs. (19a and 19b) respectively, similar with in earlier results BT 34 Moreover, the neural circuit model considers Gaussian profile of population responses, while neuronal tunings sometimes have long tail with high kurtosis (50) In principle, the recurrent connection profile can be varied generate different tuning curves. In particular, the high kurtosis profile may be considered scale- mixture Gaussians, i.c., Gaussians with different scales, and then it can be generated by recurrent connections that is also consist of mixture of many Gaussian connection profiles
the recurrent circuit consiseestt
To elucidate the concepts clearly, the present study only investigates the 1D translation group while the proposed framework can be generalized to more complicated group transformations. For example, we may develop recurrent circuit with a 2D translation group equivariant representation to explain the spatial location coding in place and grid cells (c.g., [30] ST 52 A recent study reported that 2D torus manifold (53) which the version of the ring that is being studied in the present study. Meanwhile the CAN based on 21) torus And therefore is possible to generalize our current theoretical framework to the 2D domain to partially explain the properties of grid cells in future work. Another possibility developing a non-commmtaaiiv) 3D rotation group equivariant recurrent circuits to explain the 3D rotation encoding in the brain. These studies form our future research.
the population activities of grid cells appear to reside
was also proposed in previous studies (e-g.. 30 48
Acknowledgments
Y. W. is supported by NSF DMS-2015577. S. W. is supported by Science and Technology Innovation 2030-Bran 1300-Brain Science and Brain-inspired Intelligence Project (No. 20212D00002 and Beijing Academy of Artificial Intelligence.
10

---

References
[1] Authony Zee. Group theory in a nutshell for physicists, volume 17. Princcton University Press, 2016.
[2] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 2018.
[3] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pages 2990-29999 PMLR, 2016.
[4] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning, pages 2747-2755 PMLR, 2018.
[5] Sophie Deneve, Jean-Rend René Duhamel, and Alexandre Pouget. Optimal sensorimotor integration in recurrent cortical networks: a neural implementation of kalman filters. Journal of neuroscience, 27(21):5744-5755 2007.
[6] William C Hoffman. The lie algebra of visual perception. Journal of mathematical Psychology. 3(1):65-98, 1966.
[7] Peter C Dodwell. The lie transformation group model of visual perception. Perception & Psychophysics, 34(1):1-16, 1983.
[8] Jack L Gallant, Jochen Braun, and David C Van Essen. Selectivity for polar, hyperbolic, and cartesian gratings in macaque visual cortex. Science, 259(5091):100-103; 1993.
[9] RPN Rao and DL Ruderman. Learning lie transformation groups for invariant visual perception. M. Kearns, SA Solla, & D. Cohn, Ad6ances in neural information processing systems, 11, 1999.
[10] Xu Miao and Rajesh PN Rao. Learning the lie groups of visual invariance. Neural computation, 19(10):265-2669 2007.
[11] Joel Z Leibo, Qianli Liao, Fabio Anselmi, Winrich A Freiwald, and Tomaso Poggio. View- tolerant face recognition and hebbian learning imply mymeer-yymmetric neural tuning to head orientation. Current Biology, 27(1):62 67, 2017.
[12] Andrea Tacchetti, Leyla Isik, and Tomaso A Poggio. Invariant recognition shapes neural representations of visual input. Annual review of vision science, 4:403-422, 2018
[13] Irina Higgins, Le Chang, Victoria Langston, Demis Hassabis, Christopher Summerfield, Doris Tsao, and Matthew Botvinick. Unsupervised deep learning identifies semantic disentangiement in single inferotemporal face patch neurons. Nature communications, 12(1):1-14, 2021.
[14] Yanis Bahroun, Dmitri Chklovskii, and Anirvan Sengupta. A similarity- preserving network trained on transformed images recapitulates salient features of the fly motion detection circuit. Advances in Neural Information Processing Systems, 32, 2019.
[15] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
[16] Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups. International Conference on Machine Learning, pages 1755-1763. PMLR, 2014.
[17] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint arXiv:1801. 10130 2018
[18] David J Griffiths and Darrell F Schroeter. Introduction to quantum mechanics. Cambridge university press, 2018
[19] Shun-ichi Amari. Dynamics of pattern formation lateral-inhibition type neural fields. Biolog- ical cybernetics, 27(2):77-87, 1977.

---

[20] Si Wu, KY Michael Wong, CC Alan Fung, Yuanyuan Mi, and Wenhao Zhang. Continuous attractor neural networks: candidate of a canonical model for neural information representation. F F1000Research, 5, 2016.
[21] Kechen Zhang. Representation of spatial orientation by the intrinsic dynamics of the head-
direction cell ensemble: a theory. The Journal of Neuroscience, 16(6):2112-2126, 1996.
[22] R Yishaishai, R Lev Bar-Or, and H Sompolinsky. Theory of orientation tuning in visual cortex. Proceedings of the National Academy of Sciences, 9449)33844,3848, 1995.
[23] Bruce L McNaughton, Francesco P Battaglia, Ole Jensen, Edvard I Moser, and May-Britt Moser. Path integration and the neural basis the' 'uc'cogniiive map'. Nature Reviews Neuroscience, 7(8):6633678, 2006.
[24] Sophie Deneve, Peter E Latham, and Alexandre Pouget. Reading population codes: neural implementation of ideal observers. Nature Neuroscience, 2(8):740-745, 1999.
[25] Si Wu, Kosuke Hamaguchi, and Shun-ichi Amari. Dynamics and computation of continuous attractors. Neural Computation, 20(4))994- 2008
[26] Matteo Carandini and David J Heeger. Normalization canonical neural computation. Nature Reviews Neuroscience, 13(1):51-62, 2012.
[27] Cristopher M Niell. Cell types, circuits, and receptive fields in the mouse visual cortex. Annual review of neuroscience, 38:413-431, 2015.
[28] C. C Alan Fung. Y. Michael Wong, and Si Wu. A moving bump in a continuous manifold: comprehensive study of the tracking dynamics of continuous attractor neural networks. Neural Computation, 22(3):752-799 2010.
[29] John B Issa and Kechen Zhang. Universal conditions for exact path integration in neural systems. Proceedings of the National Academy of Sciences, 109(17):6716 16 1--720,, 2012.
[30] Yoram Burak and lla R Fiete. Accurate path integration in continuous attractor network models of grid cells. PLoS computational biology, 1(2)xc, 2009.
[31] Daniel Turner-Evans, Stephanie Wegener, Herve Rouault, Romain Franconville, Tanya Wolff, Johannes D Seelig, Shaul Druckmann, and Vivek Jayaraman. Angular velocity integration in fly heading circuit. Elife, 6:c23496, 2017.
[32] Jonathan Green, Atsuko Adachi, Kunal K Shah, Jonathan D Hirokawa, Pablo S Magani, and Gaby Maimon. A neural circuit architecture for angular integration in drosophila. Nature, 546(7656):101-006, 2017.
[33] Sung Soo Kim, Hervé Rouault, Shaul Druckmann, and Vivek Jayaraman. Ring attractor dynamics in the drosophila central brain. Science, 356(6340):449-83. 2017.
[34] Anna Kutschireiter, Melanie A Basnak, Rachel I Wilson, and Jan Drugowitsch. A bayesian perspective on the ring attractor for heading-direction tracking in the drosophila central complex. bioRxiv, 2021.
[35] William Skaggs, James Knierim, Hemant Kudrimoti, and Bruce McNaughton. A model of the neural basis of the rat's sense of direction. Advances in neural information processing systems, 1994.
[36] Xiaohui Xie, Richard HR Hahnloser, and H Sebastian Seung. Double-ring network model of the head-direction system. Physical Review E, 66(4):041902, 2002.
[37] Mario Livio. Why symmetry matters. Nature, 49007741)):472 2012.
[38] H Sebastian Seung and Haim Sompolinsky. Simple models for reading neuronal population codes. Proceedings of the National Academy of Sciences, 90(22):10749-111 10753, 1993.
[39] James Knierim and Kechen Zhang. Attractor dynamics of spatially correlated neural activity in the limbic system. Annual review of neuroscience, 35:267-285, 2012.
12

---

[40] Marcella Noorman, Brad K Hulse, Vivek Jayaraman, Sandro Romani, and Ann M Hermundstad. Accurate angular integration with only handful of neurons. bioRxiv, 2022.
[41] Si Wu, Shun-ichi Amari, and Hiroyuki Nakahara. Population coding and decoding in a neural field: computational study, Neural Computation, 14(5):99--1026 2002.
[42] Patrik O Hoyer and Aapo Hyvärinen. Interpreting neural response variability monte carlo sampling of the posterior. Advances in neural information processing systems, pages 293-300, 2003.
[43] József Fiser, Pictro Berkes, Gergö Orbán, and Máté Lengyel. Statistically optimal perception and learning: from behavior to neural representations. Trends in cognitive sciences, (4(3):119-134 2010.
[44] Lars Buesing, Johannes Bill, Bernhard Nessler, and Wolfgang Maass. Neural dynamics as sampling: model for stochastic computation in recurrent networks of spiking neurons. PL.S computational biology, 7(11):e1002211 11, 2011.
[45] Ralf M Haefner, Pietro Berkes, and József Fiser. Perceptual decision decision-making as probabilistic inference by neural sampling. Neuron, 9063):649-660, 2016.
[46] Gergö Orbán, Pietro Berkes, József Fiser, and Máté Lengyel. Neural variability and sampling- based probabilistic representations in the visual cortex. Neuron, 92(2):530-543, 2016.
[47] Rodrigo Echeveste, Laurence Aitchison, Guillaume Hennequin, and Máté Lengyel. Cortical-like dynamies in recurrent circuits optimized for sampling- -ased probabilistic inference. Nature neuroscience, 23(9): 138 1 149, 2020.
[48] Wen-Hao Zhang. Tai Sing Lee, Brent Doiron, and Si Wu. Distributed sampling-hase based bayesian inference coupled neural circuits. bioRxiv, 2020.
[49] Wenhao Zhang, Si Wu, Kresimir Josic, and Brent Doiron. Sampling-base based bayesian inference in recurrent circuits of stochastic spiking neurons. bioRxiv, 2022.
[50] György Buzsáki and Kenji Mizuseki. The log-dynamic brain: how skewed distributions affect network operations. Nature Reviews Neuroscience, 15(4))264-278, 2014.
[51] Ben Sorscher, Gabriel Mel, Surya Ganguli, and Samuel Ocko. A unified theory for the origin of grid cells through the lens of pattern formation. Advances in neural information processing systems, 32, 2019.
[52] Ruiqi Gao, Jianwen Xie, Xue-Xin Wei, Song-Chun Zhu, and Ying Nian Wu. On path integration of grid cells: Group representation and isotropic scaling. Advances in Neural Information Processing Systems, 34, 2021.
[53] Richard J Gardner, Erik Hermansen, Marius Pachitariu, Yoram Burak, Nils A Baas, Benjamin A Dunn, May-Britt Moser, and Edvard Moser. Toroidal topology of population activity in grid cells. Nature, 602(7895):123- 128, 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sections and 6.
(c) Did you discuss any potential negative societal impacts of your work? [No] The present paper a theoretical study investigating the computation of neural eircuits in the brain, and thus it won't 't have negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We clearly state the assumptions in theoretical analysis, and simplifications of the model compared with real neural circuits in the brain across the manuscript.
(b) Did you include complete proofs all theoretical results? [Yes] The main text only lists
the idea of proof and theoretical results. Detailed proofs and mathematical derivations are presented the Supplementary Information.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URLJ? [Yes] The simulation details presented in Supplementary Information and the code also uploaded.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] They presented in simulation details in Supplementary Information.
4. If you are using existing assets (e.g., code, data, models) or curating/releusing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "wpRyVOiyDSi", "reviewer": "Reviewer_fomB", "summary": "The authors propose a recurrent neural circuit model with a continuous attractor state that models the heading direction system in the Drosophila and includes a circuit mechanism for translating the heading direction at arbitrary speeds. Moreover, their model is closely related to the Drosophila heading direction circuit.", "questions": "- Line 37: The sentence \"For example, ...\" needs a citation.\n- Line 96: Why is eq 6 the mean response of all neurons? Are you assuming infinitely many neurons? Is there a citation justifying this? Am I missing something obvious?\n- Eq (7): What does $\\overline u(s)\\equiv\\overline{u}(x-s)$ mean? This doesn't appear to be true based on the definition of $\\overline{u}(x-s)$.\n- Sec 3.1: Is there no external input to the system? Is it easy to include? How would that affect the system?\n- Line 131: What is the definition of \"free parameter\"?\n- Eq (13): Can I interpret this effectively as a restatement of the fact that $\\hat T(a)=\\exp(a\\hat p)$?\n- To what extent are eqs (16) and (19) equivalent? The authors discuss the relationship, but its not clear to me what the precise relationship is.\n\nMinor:\n- Line 20: reflects -> reflect\n- Line 43: whereas -> however?\n- Eq (6): I find the notation $\\cup_j$ confusing. It looks like you're taking the union over $j$.\n- Eq (10): The definition of $\\overline{u}$ is different here than in eq. (7)", "limitations": "- Eq (19ab): Can external stimulus inputs be included in this model? It seems like those would be reconciled with the speed inputs via some sort of Kalman filter. Is the speed $v$ assumed to be an external stimulus that feeds into all of the speed neurons? Presumably the speed is computed in part from the CAN responses? I'm not suggesting that this circuit should include these computations, rather I'm trying to understand to what degree this model is flexible enough to include these considerations.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: As someone who is not an expert in the field of continuous attractor networks, it is difficult for me to assess the originality of this work in part because it was not clear what exactly were the authors' contributions. My impression is that the main contributions were stated in sections 4 and 5, but even within those sections it is not clear which parts are novel (e.g., equation 19 appears to be known). While they include numerous references and the second paragraph of the conclusion discusses on the novelty of this work, it would still be useful (for me) if the authors further delineate what is their contribution versus what has already been established in the literature.\n\nQuality: As far as I can tell, the paper is technically sound.\n\nClarity: I did not find the presentation particularly clear, but this could be due to my lack of expertise in the area. I've listed clarifying questions below.\n\nSignificance: The paper address an interesting problem in neuroscience; however, it is difficult for me to asses exactly what the authors' contributions are versus what was already known. I readily acknowledge this may be my own lack of expertise, but I would like the authors to clarify this for me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BhssC8rjE_", "reviewer": "Reviewer_C6Qo", "summary": "Many artificial neural networks should obey symmetry relationships.  This paper derives a translation operator for a ring attractor network inspired by recent findings from the drosophila head direction system.   The resulting system is designed to be equivariant to translation and is at least roughly neurobiologially plausible.\n", "questions": "\n35: ...explicitly represent...\n77-8: ...the above derivation...\n92: Not convinced it's strictly necessary that the units have identical tuning curves.  Agree that it's convenient.\n115: Divisive normalization...\n138: ...disrupted by noise.\n171: ...utilize the ...\n213: Don't know that much about drosophila specifically, but in general, multiplicative modulation of weights is not challenging in neurobiological systems.  Keywords are ``shunting inhibition'' and ``cortical gain control''.  \n\nWould be good to note that there are in fact speed cells in the grid system, e.g., Sargolini et al., (2006).  However their firing rate profiles are relatively complex (see for instance, Dannenberg, et al., 2019, J Neuro; 2020, eLife).\n\nAuthors might be interested to compare these results to Tanaka & Nelson (2019, PRE).   In thinking about 3-D transformations, Shepard (1994, Psychonomic Bulletin & Review) may be helpful.\n\n", "limitations": "\nThe paper is far too sanguine about extending this approach to place cells/grid cells.  This is likely to be much more complicated than one might expect from this paper.  Place cells do not monotonically map the 2-D environment.  Locations nearer to reward locations and boundaries are overrepresented.  In addition, the grid system is much more complicated than a 2-D surface.  2-D position is represented in conjunction with head direction and speed.  Moreover, there appears to be a continuous spectrum of time constants in the speed system (Dannenberg, et al., 2019).\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper strikes a nice balance between abstractness and specificity.  The math is initiated at a very high level, but is then implemented for a very specific circuit.\n\nThe model makes a number of predictions that could be tested with existing technologies.\n\nWeaknesses:\nMy major concern about this paper is that perhaps something similar was already worked out, perhaps for the grid cell system but I'm unaware of that work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B_XkWhXjuzF", "reviewer": "Reviewer_QibX", "summary": "The authors provide an exciting extension to the classic bump circuit model [20]. The work provides a connection between Lie group translations and continuous differential recurrent circuits (or continuous attractor networks, CANs) to demonstrate that CANs can effectively encode a continuous variable shift, such as position. They derive the circuit equations, implement numerical simulations that match the theoretical assumptions, and suggest modifications to more appropriately accommodate known biological constraints.", "questions": "1) Implications of the gaussian tuning curve assumption (lines 87-88) – A Gaussian profile is certainly a fair assumption; it is used in a majority of neural sensitivity research. But there’s also good evidence to suggest that at least visually selective neurons (probably others as well) are better fit by leptokurtic tuning curves, which demonstrates an increased selectivity for preferred stimuli [1]. I think this could be covered in a future publication (i.e. is not necessary to implement in the rebuttal phase), but I am curious if you have thoughts on how one might apply the derivation in section 2 of the appendix to a generalized normal distribution? Additionally, does neuron selectivity have an impact when you are considering an infinite set?\n\n2) notation clarifications – I initially ran into some confusion with the notion that I think could have been avoided with a more gentle introduction at the top of section 2. \n\n2.a) There are a lot of conceptually overlapping variables that correspond to the stimulus & its perturbations (s, s’), neuron preferences & their perturbations (x, x’), the addition of an index when referring to individual neurons (x_j), and resultants from perturbing a stimulus (e.g. ‘a’ in T(a)*s = s+a). I understand the need for all of them, but an early exposition on what they are to orient the reader would have helped me.\n\n2.b) I also had confusion coming from switching between a discrete (countable) notation for the neurons and a continuous notation based on the preferred stimulus. As some examples of the former, consider the pictorial description in Fig 2.a; the use of the term “index” in the horizontal axis label in fig 2f; and the use of the subscript j in equation 6. An example of the latter is found in the use of an integral over alternate stimuli in the denominator of equation 8b, and the explanation of an “excitatory neuron preferring stimulus s=x” on lines 112-113. This is explained briefly on lines 121-126, although there you walk back the infinite neuron description and describe it as “a large population of neurons.” I am not suggesting that any of this is wrong, per say, but more recommending that you spend a moment earlier (for example near line 86 or at the top of section 2) to explain what is going on here and when/why you will be flexible with notation.\n\n3) What are the empirical implications of the infinite neuron assumption? I would appreciate a discussion at the end of the paper on what it means to implement such a system with a finite set of neurons. I may have missed this in the paper, but what were the implementation details for the experiments shown in figures 3 and 4? How does the number of neurons (i.e. resolution of the discretization of the stimulus space) influence the analysis?\n\n4) Line 22 “decode the instant orientation” did you mean instantaneous?\n\n5) Typo on line 35 of the supplement – “clear to se”\n\n[1] Ringach, D. L., Shapley, R. M., & Hawken, M. J. (2002). Orientation selectivity in macaque v1: Diversity and laminar dependence. Journal of Neuroscience, 22(13), 5639–5651.\n", "limitations": "The authors note limitations in sections 5 and 6. I note some questions that touch on limitations of the presented work (e.g. the mismatch between theory & experimentation).", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:** Of course the concept of using a recurrent circuit to encode an environment variable, such as position, is not new. Additionally, the application of Lie algebra to achieve equivariance in neural circuits is not new. However, as the authors point out and as far as I know, the connection that they provide between these two concepts is both novel and exciting. I am fairly confident that the additional clarity and rigor that the present study provides over previous work in the space is an original contribution.\n\n**Quality:** I believe the mathematics are technically sound, and the supporting experiments demonstrate the applicability of the theory. I have some minor comments below about including more discussion on weaknesses in applying the theory.\n\n**Clarity:** I include minor comments below to improve clarity, but overall the exposition is clearly communicated.\n\n**Significance:** I believe the submitted work is a significant theoretical contribution. It draws on several ideas from iconic work in neural coding [20], and the application of Lie algebra lends itself to several future extensions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-hBUcuS8RZH", "reviewer": "Reviewer_2shN", "summary": "The paper introduces a translation-equivariant neural model where the neural response to a stationary stimulus is equivalent to convolving with a Gaussian; however, through recurrence, the response is maintained in the absence of a stimulus.  Introduces the use of Lie algebra for deriving the translation generator.\n\n\n", "questions": "I am curious about challenges the authors faced - i.e., what did not work?\n", "limitations": "general limitations, see above.\nsocietal impact: n/a\n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the framing in terms of group theory, equivariant representations, and the exposition is fairly clear. \n\nWeaknesses:\n- it is unclear to me what parts of the paper are mostly for exposition (drawing on previous work of Kechen Zhang), and what parts are novel contributions. Most of the paper consists of theoretical derivations, and it’s often not made clear where results were proved in other papers.\n- given the focus on the 1D case I think the title is a bit too general.\n\nFor example:\n- Section 3.2 shows that the fixed points of continuous attractors are equivariant representations, but I’d be surprised if this result was not already known or demonstrated\n- Section 4.1 uses the derivative as the translation operator, as does Kechen Zhang’s (1996) paper which is cited here but not really discussed. It seems that the real contribution is the connection between this derivative to group actions, but I wish this were more clearly laid out.\n\nAnd some thoughts on the circuit model:\n- The discussion of biological (im)plausibility seems weak. There’s not really a problem of multiplicative modulation — synaptic weight multiplied by variable input would do fine (as the authors do in eq 19b). And the other problem they mention — Dale’s law — is easily circumvented by having excitatory and inhibitory neurons, again as the authors do.\n- The more critical biological concerns, I think, are (1) biological noise (spike timing, synaptic noise, etc.) and (2) that the Drosophila E-PG cells number about 40, whereas the attractors considered are in the infinite neuron limit. I think some acknowledgement of these points/experiments proving the analysis holds up under these constraints would strengthen the paper.\n- in particular there’s a fair amount of literature with CANs and how they deal with noise, drift, limited precision (e.g. Burak & Fiete, 2009) — and it’s surprising to see very little of this discussed here.\n\nGiven that velocity-controlled inputs are already the inputs to many CANs for path integration, and examples beyond integrators aren’t discussed here, it seems the main contribution is an interpretation of/twist on existing models, rather than a new model per se. For this reason, I would have appreciated more suggestions on what to do with this Lie group understanding: whether on novel neural circuits, new neural mechanisms that implement the transport operator, or even learning of group of structure.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept