Slow and Weak Attractor Computation Embedded in Fast and Strong E-I Balanced Neural Dynamics
Authors: Xiaohan Lin
Liyuan Li
Boxin Shi
Tiejun Huang
Yuanyuan Mi
Si Wu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Continuous attractor neural network; Excitation inhibition balance; Brain-inspired algorithms; Object tracking;
TL;DR: Continuous attractor network dynamics and excitation-inhibition balanced dynamics can coexist in one circuit with synergistic computation benefits.

Abstract: Attractor networks require neuronal connections to be highly structured in order to maintain attractor states that represent information, while excitation and inhibition balanced networks (E-INNs) require neuronal connections to be random and sparse to generate irregular neuronal firings. Despite being regarded as canonical models of neural circuits, both types of networks are usually studied in isolation, and it remains unclear how they coexist in the brain, given their very different structural demands. In this study, we investigate the compatibility of continuous attractor neural networks (CANNs) and E-INNs. In line with recent experimental data, we find that a neural circuit can exhibit both the traits of CANNs and E-INNs if the neuronal synapses consist of two sets: one set is strong and fast for irregular firing, and the other set is weak and slow for attractor dynamics. Our results from simulations and theoretical analysis reveal that the network also exhibits enhanced performance compared to the case of using only one set of synapses, with accelerated convergence of attractor states and retained E-I balanced condition for localized input. We also apply the network model to solve a real-world tracking problem and demonstrate that it can track fast-moving objects well. We hope that this study provides insight into how structured neural computations are realized by irregular firings of neurons.

Slow and Weak Attractor Computation Embedded in Fast and Strong E-I Balanced Neural Dynamics
Xiaohan Lin Liyuan Li3.4 Boxin Shi Tiejun Huang Yuanyuan Mi Si Wu?*
1 Nat'l Key Lab for Multimedia Information Processing, Nat'l Eng. Lab of Visual Technology, School of Computer Science, Peking University
Peking-Tsinghin Center for Life Sciences, Academy for Advanced Interdisciplinary Studies
School of Psychological and Cognitive Sciences
Beijing Key Laboratory of Behavior and Mental Health
IDG/McGovern Institute for Brain Research
Center of Quantitative Biology. Peking University
Department of Mathematics, Peking University
Institute of Neuroinformatics University of Zurich
Institute for Artificial Intelligence, Peking University 6 Department of Psychology, Tsinghua University
(lin xiaohan, xiwohaanniyuunnlll.,. liy.llllynnn,iiiiiim shiboxin, tijhuang,siie siwu]@pku. edu. on miyuanyuan@tsinghur edu.cn
Abstract
Attractor networks require neuronal connections to be highly structured in or- der to maintain attractor states that represent information, while excitation and inhibition balanced networks (F-INNs) require neuronal connections to be ran- dom and sparse to generate irregular neuronal firing. Despite being regarded as canonical models of neural circuits, both types of networks are usually studied independently, and it remains unclear how they coexist in the brain, given their very different structural demands. In this study, investigate the compatibility continuous attractor neural networks (CANNs) and E-INNs. cent experimental data, find that neural circuit can exhibit both the traits of CANNs and E-INNs when the neuronal synapses consist of two sets: one set is strong and fast for irregular firing, and the other set is weak and slow for attrac- tor dynamics. In addition, both simulations and theoretical analysis reveal that the network exhibits enhanced performance compared the of using only one set of synapses, with accelerated convergence of attractor states and retained E-I balanced condition for localized input. We hope that this study provides in- sight into how structured neural computations are realized by irregular firings of neurons.
line with re-
Introduction
The core of theoretical neuroscience is to understand how different neural circuit models generate different dynamical behaviors, and consequently accomplish different brain functions. Among the proposed models in the literature, continuous attractor neural networks (CANNs) and excitation- inhibition balanced neural networks (E-INNs) are two canonical models that have been widely and successfully applied to describe many brain functions and neuronal response properties. Specifically, CANNs have been applied to explain the representation of continuous features in the brain, includ- ing, for instances, orientation tuning in the primary visual cortex [D], action intention the motor
"Correspondig author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(A)
(B)
(C)
E-INN (ftst and strong)
gh ((L/vK)
B
B. (t)
20
E-NN weights
CANN weight
15
Density
Loak
E,
p
10-
tu
BE
5.
CANN (slow and weak)
BB
O(1/K)
0.0
0.4
0.8
(D)
(olon ~ o(1)
(R)
No
Eigenvauue
o(1)
Currest
2
10
((0 O(VR)
102
oly
Network
Eigenvalue
Figure 1: A neural circuit exhibiting traits of both E-INNS and CANNS. (A) E-INN dynamics are mediated by strong, unstructured connections on the order of ((1/VK) and featuring fast synaptic current dynamics (not shown). CANN dynamics rely on weak, translationally invariant connections on the order of (1//K) and exhibiting slow synaptic current dynamics (not shown). Only afferent connections to the excitatory neuron are depicted.
(B) The kinetics of single excitatory neuron are influenced by leak current, input currents from E-INN dynamics and CANN dynamics, as well as external input. (C) Weight distribution density plot illustrating the relatively weak nature of CANN weights compared to E-INN weights. N = 1000. (D) Intuitive illustration of the coupling mecha- nism between the two dynamics. Note that the same group of E' neurons shared by both dynamics. They are duplicated in this figure for better visualization.
(E) Simulation results demonstrate that E-INN dynamics operate at the scale of (VVK), while CANN dynamics operate at the scale of (1). (F) Theoretical analysis of parallel firing-rate model reveals that E-INN dynamies effec- tively decrease the first eigenvalue of the CANN dynamics projected onto its dominating motion modes, resulting in faster convergence.
cortex [10]] oculomotor signal the brainstem [25]] head direction tuning in the subicular complex 128.
2 place cell tuning in the hippocampus [23. [0], and grid cell tuning in the entorhinal cortex
The structure of CANN was also directly observed in the neural circuits of Drosophila
15
[
and zebrafish [2I]]. On the other hand, E-INNs were applied to explain the irregular firing
of neurons widely observed the experiments [26,
[ as well as the fast response property of neural system to external stimuli [33, 14. 29). The experimental evidences for the existence of
32,
E-I balance in neural systems were also reported [22]] CN 12, 20, [II]. In many modelling studies of large-scale spiking neural networks, E-I balance is imposed as default condition in the neural dynamics.
Despite being regarded as canonical models of neural circuits, the compatibility of CANNs and E- INNs in the brain remains unclear. In most modeling studies in the literature, the two models are often used separately to account for different brain functions, and there is no formal study on how the two models coexist a neural circuit. This presents a nontrivial challenge, as the two models exhibit fundamentally different structural characteristics. CANNs rely highly structured connec- tions with decaying strengths based on neuronal tuning disparity, enabling localized neural activity (bumps) for information representation. In contrast, E-INNs necessitate highly unstructured, ran- dom, and sparse connections to ensure statistical independence among input received by individual neurons, facilitating irregular firing patterns. This prompts the question of how can neural circuit reconcile these seemingly conflicting requirements?
In this study, we demonstrate that it is possible for neural circuit to concurrently incorporate the properties of CANN and E-INN, while also compensating for their respective limitations, through appropriate consideration of connection strengths and synaptic dynamics timescales. Specifically, we consider that the connections are composed
two sets, mediated by two different types of in- hibitory neurons respectively (Fig. IIA). One set comprises randomly sparse synapses, characterized by strong synaptic weights scaled with 1/ 1/VK (where K represents the number of connections) and

---

fast synaptic dynamics with time constants on the order of ms. The other set consists of synapses which are highly structured in a CANN, featuring weak synaptic strength that scales with 1/K and slow synaptic dynamics with time constants on the order of 100 ms (Fig. IIC.. This model as- sumption is supported by recent experimental data [249], where strong synapses exhibit no correlation with neuronal selectivity, while the co-activation of weak synapses determines neuronal selectivity.
The mechanism of the neural circuit exhibiting computational traits of CANN and E-INN dynamics simuitaneously is illustrated in Fig. ID-E and can be intuitively understood as follows. E-INN dynamics are mediated by strong synaptic weights that scale with 1/v K and fast synaptic dynamies, effectively processing feedforward inputs of ((VK) while always keeping network dynamics in balance; the resulting excitatory and inhibitory synaptic currents, albeit both of O(VK), ) cancel out and create a constant total synaptic current of (11) [G This constant total synaptic current serves the input to CANN dynamics, which operates on slower timescale with weaker synaptic weights that scale with 1/K, producing excitatory and inhibitory synaptic current of (1). Notably, this study introduces parallel firing-rate model that accounts for the effect of E-INN dynamics by incorporating a constant term & into the CANN connectivity. From the firing-rate model, an analytical solution is derived, allowing us to elucidate the synergistic computation between E-INN and CANN dynamics. We find that E-INN dynamics effectively reduce the eigenvalue of CANN dynamics in the height mode, without affecting other high-order modes (Fig. IF). We hope that this study can provide insight into how structured neural computations are realized by irregular firings of neurons.
2 The Network Model
We present our simulation results using a spiking model. Additionally, we have developed a firing- rate model for theoretical analysis, which is discussed in Sec. B the main text, with more compre- hensive version provided in the Supplementary Information (SI). The spiking model comprises three neuronal populations: an excitatory group (labeled E) and two inhibitory groups (shown in Fig. MA). One group of inhibitory neurons (Ia). presumably formed by somatostatin -xxressxp neu- rons (also known regular-spiking interneurons), provides distal inhibition to excitatory neurons, resulting in inhibitory currents that interact linearly with the somatic membrane potential. The other group of inhibitory neurons (Tp). presumably formed by parvalbumin -eV)-xxpns neurons (also known as fast-spiking interneurons), provides proximal inhibition to excitatory neurons which in- duces shunting inhibition. The network dynamics depend two sets of weights: weak weights from CANN dynamics and strong weights from E-INN dynamics. The E-INN connectivity pat- tern is random with connection probability p. CANN dynamics, on the other hand, have structured connections w EE with translational invariance. Otherwise, the connections
are all-to-all.
The dynamics of a neuron is given by
dVb (-gVV + +1)),
(I)
dt
where b E, Ip. Is denotes the neuron type. i denotes the neuron index, -." the membrane time constant, 9L is the leaky conductance, and 1 is the synaptic current received by the neuron. In our experiments, set membrane resistance R 1.
For simplicity, we only present the synaptic current received by an neuron. Refer to SI for the full model. The current consists of three components: a fast recurrent input B E...fatt (t), a slow recurrent input C.E.slon (L), and an external input T. e,,ext (L) (Fig IB):
I (t)(t B. fast (t) + C F.slow (t) + 1. F,ext (t).
B. Fast
The fast recurrent input B. Fustt (t) is mediated by set of strong synaptic connections g. that scales with 1/VK with fast synaptic current dynamics. f.f.fast( (t) enables the network exhibit E-INN dynamics and is given by
fast (1) = 2 + 224
(3)
where _ Pt Eld
denotes the fast excitatory synaptic input current from population E. and Qia = ExPLig fl (t) denotes the fast inhibitory synaptic input current from population La
EE ff((t
3

---

Pi, {1,0} denotes that neurons and j are connected or unconnected, respectively. 9i the synaptic strength from neuron j of population b to neuron i in the population, which of O(I/VK). (t) represents the fast synaptic current input:
Fb denotes
17 = Σ
(t- (t-(),))
b E=,,da,4p
(4)
Fast
Here f denotes the spike time of the kth spike neuron j. Tast is the fast synaptic time constant. We set pi, randomly with a probability of 0.25. We choose T 5 ms such that it's similar to the time constant of AMPA receptors.
The slow recurrent input C F.
(t) is mediated by a set of weak synaptic connections Bi that wif scales with 1/K with slow synaptic current dynamics. C.F.slo(())) enables the network to exhibit CANN dynamics. is given by
CB (t) (f)=Ie) r st),
where I E, w
FE F. (t) denotes the slow excitatory synaptic current dynamics from population ..j
E and r Ei 1 f (t) denotes the fast inhibitory synaptic current dynamics from population
Elp
Tp- "i EE = EE exp (0,-0,1" /NE. with whax being the maximal synaptic weight and 0g E [-,, π) being the preferred stimulus feature of neuron k (e.g. orientation). wi thus only depends Fl, on the tuning disparity (0, (0,-0,) and satisfies translational invariance. Wi,1 represents the connection strength from population J, to population E, which is all-to-all with connection strength scales with 1/K. (t) models the effect of the shunting inhibition the product of excitatory postsynaptic currents (EPSCs) and inhibitory postsynaptic currents (IPSCs) [[
max
2a
<s)T:(() ( + +I + I1 B,ext (1) r..
(6)
denotes the strength of shunting inhibition. (t) represents slow synaptic current and is given by
(1)
Σ e (t-tz,))/T
E.
slow
We set the slow synaptic time constant 'slow 150 such that it's similar to the time constant of NMDA receptors. For the full network dynamics, please refer to SI. 1.
3 Preservation of E-INN and CANN Dynamics
We simulate the neural eircuit with coupled dynamies of E-INNs and CANNs using a spiking model implemented in BrainPy. BrainPy uses JIT to speed up the simulation process [35, B4]] The param- eters used in our simulations can be found in SI.
3.1 Preservation of E-INN Dynamies
The coupled neural circuit preserves the dynamics and computational properties of E-INNs. Fig. DIA- demonstrate that the network exhibits irregular activity patterns when the external stimulus is spatially uniform [2G, 62]. Additionally, the network's 's activity level rapidly adjusts to continu- ously changing external input, computational feature exhibited by E-INN dynamics, whereas the counterpart with unbalanced dynamics (where B, F. is scaled by 1/K) demonstrates significantly slower adaptation (Fig. (D) [33]]
3.2 Preservation of CANN Dynamics
The equilibrium state of CANN dynamics can maintain bump-like persistent activity after offset of external stimulus, property thought to be the substrate of working memory. [37 [ Moreover, CANN dynamics exhibit neutral stability along its attractor space, endowing a neural circuit the ability to smoothly adapt to time-varyiin varying stimulus [e0 g] Our model retains these two properties

---

Nouron (A)
(B)
(D)
10a
500
Autooorr:
10
PSD
Activity
Our Unbalaaned
200
400
50000
Time (ms)
Time (ms)
Frequency (Hb)
Time (ms)
Figure 2: The network preserves E-INN properties. (A) The raster plot of excitatory neurons shows irregular spike patterns, verified by (B) average autocorrelation between neuron pairs, and (C) uni- form power spectrum density of network activity. (D) Fitted response curves show the fast response property the network versus an unbalanced dynamics.
(A)
Neuron Indox
600
800
(B)
Neurve Inde:
800
10000
400
200
0
200
300
400
2400
N-3000
Time (ms)
1.5
Neurun Index
800
0
Readout
1.0
4000
800
0.5
0
0
250
500
750
1000
1250
1500
1000 1250 1500
1000
Time (ms)
Time (ms)
Time (ms)
Figure 3: The network preserves CANN dynamics. (A) Top: A raster plot illustrating the net- work's persistent activity. The blue dashed lines indicate the stimulus onset and offset. Bottom: The network activity projected onto the stimulus, normalized. Shade represents instantaneous readout results. The solid line represents running average calculated over 1.5ms. (B) The network exhibits smooth tracking of time-varying stimulus. The light blue line represents the stimulus position, while the red line represents the position decoded from the network activity.
(C) Fast synaptic dy- namics associated with CANN dynamics can lead to synchronization, causing failure to sustain persistent activity. (D) The model maintains its computational properties across different network sizes using the same parameter set. The vertical dashed lines represent the stimulus onset and offset.
(Fig. [A- Notably, it is crucial for attractor dynamics to be mediated by slow synaptic dynamics. This is because fast synaptic dynamics could lead to synchronization, causing the network to lose persistent activity (Fig. BC)- phenomenon referred to turning-ofi saating--ff-wtt-h-ec wiff---ee-xcitttion [IG.. [ This is not problem for base- CANNs but is a biologically realistic issue introduced by spiking dynamics. Moreover, given that the attractor dynamics in model operates (1) (see Fig. IID- E), the dynamics can scale with network size without altering parameters (Fig. BD).
Synergistic Computation of CANNs and E-INNS
Spiking CANNs and E-INNs have their respective limitations. For instance, spiking CANN cannot employ fast synaptic current dynamics because of the stability issues (Fig. BC), while slow synaptic current dynamics severely restricts the neural system's 's ability adapt to external changes in timely manner. On the other hand, E-INN dynamics with global unstructured connectivity cannot maintain balance under localized input [22] [
However, we have discovered that CANN and E-INN dynamics not only can coexist and but also exhibit synergistic computation. This coexistence accelerates the convergence speed of CANN dy- namics, addressing the limitation of slow synaptic current dynamics while maintaining stability. Ad- ditionally, shunting inhibition in the CANN dynamics helps alleviate the issue of imbalance under localized input. We present the simulation results and an intuitive explanation below, and provide rigorous theoretical analysis in Sec. B and SI. 2.
5

---

(A)
(B)
(D)
Neuron
400
Position (rad)
0.4
Total Cureent
- FI-NNN
0.2
0-
Our Midic
10
200
200
400
500
600
(196)
Network
CANN
014
Densty
10
Our Model
Curoni
50
ST
0
0.2
0.4
Time (mr))
Time [mi)]
Lag (rad)
Neuzon
Figure 4: Synergistic computation of CANN and EI-NN dynamics. (A) Decoding results from a CANN (left column) and our model (right column). Top: Raster plots illustrating the activity of excitatory (E) neurons in both models. Middle: Decoded position R(t) with color indicating the readout confidence C(t) (see text). Bottom: Standard deviation of decoded position. (B) The network activity (red) in our model adapts more rapidly to sudden changes in the stimulus (black) compared to CANN with the same parameter settings (blue) (error bars should included). (Time comparison can be included if feasible). (C) A density plot demonstrates that our model consistently exhibits smaller tracking lag than the CANN model with the same parameter settings.
(D) Input current to a center neuron under spatially localized input. Top: Total current increases as the network size increases in E-INN, but remains nearly unchanged in our model. Error bars represent standard deviation over trials. Bottom: Shunting inhibition (orange bar) is strongest at the center neuron with the stimulus in the center and weakest for peripheral neurons.
4.1 Synergistic Computation
Faster population activity readout The efficiency of information processing in neural system critically relies on the speed at which the neural dynamics can be accurately and confidently inter- preted. In Fig. IA, we demonstrate that despite the inherent weakness and slowness of the CANN dynamics in our model, downstream neurons can reliably decode its representation almost imme- diately after the stimulus onset. In contrast, during the initial phase stimulus presentation, the original CANN model exhibits more fluctuation and less confidence in the decoding result.
We investigate the population decoding process by calculating the decoded position R(t) and de- coding confidence C(I). We perform the calculation in complex domain. R(1) E (-H, is given by
R(t)= = arg(v(()),
where v(I) =,
E; (1)ccs0 + isin 0,) complex number representing the weighted summa- tion of population activity. Tý represents the firing rate and 0z E (-H, represents the preferred stimulus of neuron j. The decoding confidence C(t) is given by
C(I) = |u(()/ max(ūr)
which measures the normalized length of the decoded activity vector. ör represents the smoothed collection of [u(t) using a time window T.
Faster adaptation sudden changes The rapid adaptation sudden changes stimulus is fundamental aspect of neural computation, as it signifies the neural system's 's capacity to respond to external fluctuations. In our model, we observe a faster adaptation speed to sudden changes in stimulus compared to the original CANN dynamics under identical parameter settings (Fig. (B). To quantify this, we measured the time it took for the population activity to reach 95% of the new steady-state value. Remarkably, the unbalanced CANN required almost 50% more time to converge to the new position compared to the balanced CANN. This finding highlights the computational benefits of our model in terms of enhanced responsiveness to dynamic stimuli.
Smaller tracking lag The activity of CANN dynamics can continuously adapt to a time-varying stimulus, such as head movement external moving targets, with certain amount of lag [T]]. The

---

(A)
(B)
Norm. Current
Stimulus
Our Model
Center
Center
onset
CANN
Ratio
Peripheral
E-1 Ratio
Peripheral
2-
0-
1000
1500
2000
1000
1500
2000
500
1000
1500
2000
Time (ms)
Time (ms)
Time (ms)
Figure 5: Intuitive explanation of convergence acceleration. Shades represent standard deviation over trials. (A) Visualization of the normalized total input current (see text) to peripheral neuron our model and a CANN. (B) E-I ratio in our model. (C) E-I ratio in a CANN.
existence of this lag entails the neural system to come up with solutions to alleviate the delay, for example, by negative feedback [8, 19]. The E-I balanced dynamics also help to mitigate this problem. Fig. BC shows that our model reliably exhibits smaller tracking lag compared to considering the CANN dynamics alone.
Maintenance of balanced condition One key trait of E-INNs is the production of total input cur- rent of O(1) out of large EPSCs and IPSCs of O(KK), hence the name E-I balance. However, spatially localized stimuli can disrupt this balance due to the lack of spatial structure in the connec- tions between neurons [22 Ld.. This means that the total input current
neuron is no longer of O(1). but is instead related to its afferent connection number K. When the network size is large, K PN is also large, which can drive neurons to fire pathologically. In our model, shunting inhibition, mediated through PV neurons, generates a strong inhibitory current that proportional the total EPSCs. This greatly mitigates the imbalance issue. Specifically, the neurons that receive the greatest excitatory input also receive the greatest inhibitory input, which keeps the total input to each neuron roughly constant as the network size increases (Fig. DD. These results are consistent with experimental findings [II]], which suggest that PV neurons, but not SST neurons, are essential in adjusting the E-I ratio and maintaining E-I balance.
4.2 Intuitive Explanation
To provide an intuitive explanation for the accelerated convergence in our model, we examine the total input current a neuron located the peripheral region relative the stimulus (Fig. B), as the total input current determines the firing rate of a neuron and its rate of change. To clearly see the evolution trajectory of the total input current, we normalize using the value at the stimulus onset point. We find that the total current in our model converges more quickly to its equilibrium value without undergoing drastic changes compared to a CANN model, as shown in Fig. BA We further examine the EPSCS-to-IPSCS ratio (E-I ratio), as it provides clues on the dynamics of the total input current, The E-I ratio in our model quickly settles to fixed values (Fig. SB), because the strong E-INN dynamics always keep the network under balance, whereas the E-I ratio in the CANN model slowly converges to its equilibrium value due to the unbalanced nature of the dynamics (Fig. ISC).
5 Theoretical Analysis
We present the main results of our theoretical analysis in this section. The full result the SI. 2. In SI 1.1 we present a ID firing-rate model to investigate the joint dynamics of CANNs and E- INNs. In SI 1.2, show that this model can be reduced using fast-slow dynamics and mean-field approximation to obtain the following reduced model:
F (xE( = -UE(2,)) + / [J x') + ( t)dx' + lex((x,) alow
(10)
Ø1
UE 1))
TE(x,
(11)
1 l 1 [UE(2', ()) dx!
7

---

where (z(2,t) t) represent the synaptic input to the excitatory neurons at time t for neurons whose preferred stimulus is z. r(z,t) t) represent the firing rate of these neurons, and Lext t) represent the external inputs. []]+ denotes the rectifier function and A denotes the strength of global inhibition. The recurrent neural connection matrix J(x,²') Jn exp [-(x - x') x²)² (2a)) is translationally invariant, which means that J(x, xx) is function of (x - x²).
Importantly _ل - ge /ggg represents the effect of E-INN dynamics on the CANN dynamics, and gohh represents the average connection strength from neural group b to in the E-INN dynamics. The E-I balanced condition in classical literatures [53] requires that
g
9
EE
(12)
GIE
which gives 3 < O. Intuitively, the E-INN dynamics affects the CANN dynamics by counteracting the excessive excitatory recurrent inputs through introducing a negative offset to its neural interac- tion strength. We analyze the case when the magnitude of 3 is sufficiently small for the advantage that it permits an analytical solution of the network stable state for the reduced model. Although there is theoretical guarantee that the final conclusions of our analysis applicable to general cases where |8] is large, we perform simulations to ensure that for reasonably large |+]. our conclu- sion still holds (sce Fig. S1 in SI).
We first consider the case when there no external stimulus. In this case, it is straightforward to check that, for 0 < l < ke, the network holds a continuous family of stationary states [39]]]
Ag v2
(x
Üp(# 2)
exp
(13)
4a
(xr =)²
FE(0 2) = A exp
(14)
2a
where z represents the center of the activity bump and is a constant. Previous studies have demon- strated that the CANN dynamics is dominated by small number of motion modes [40]. Therefore, to simplify the analysis, we can project the network dynamics onto these dominant modes. In this study, we adopt the first two dominant motion modes, which correspond to changes in bump height
and position, respectively. We address higher-order motion modes in SI 1.3. The expressions for the first two dominant motion modes are as follows:
(x 2
60(œ 2) = (/27)2/2a exp
(15)
4a
(x 2)
(x 2)²
d (x =)) av 1/(22))//2
exp
(16)
4a²
The corresponding eigenvalues for the first two motion modes are:
An = (1 + 2v 2
)(1
and λ = 1.
(17)
9
We denote the first eigenvalue when _ = O as Ag. which represents the eigenvalue on the height motion mode without the effects of the E-INN dynamics. For 3 < O, we have 0 < Ao < Xj < 1. indicating that the CANN will converge faster on the height motion mode when coupled with the E-INN dynamics. The second eigenvalue indicates that the network is still neutrally stable with respect to the second motion mode, i.e. positional shift, and can hold a continuous family of activity bumps. In SI 1.3, we show that 3 does not affect higher-order eigenvalues.
Faster population activity readout Given that 0 < Ao < Ao < 1 when 3 < O, it is evident that the E-INN dynamics facilitates the convergence of the network towards the height direction. Downstream neurons can thus have faster access to the equilibrium state of CANN activity.
Faster adaptation to sudden changes Following [+], in SI 1.3 we show that when the external stimulus takes the following form:
Iesa (x., t) = a Ag
(a
zo) )2
cxp
(18)
/2
4a
8

---

where a is positive number indicating the input magnitude, the speed of the network's 's activity adapting sudden change (up to the first order of perturbation) given by:
dz
a
(z
2) exp
(3o - 2)²
R(t)
(19)
dt
Sa²
slow
where
{"
dt'
1 do (t t')
(20 2 (t'))
R(t) 1++
exp
(20)
Sa
slow and 50 represents the new input location after the sudden change. Consequently, the following relation holds:
slow
de dz
(21)
di
Ao
di 1a
This implies that coupling CANN dynamics with E-INN dynamics always leads to faster conver- gence to the new equilibrium location than without E-INN dynamics.
Smaller tracking lag Assuming a constant external stimulus speed v, we can substitute 2 with ot Equation (19), resulting = et 2, where 2 represents the instantaneous activity bump location. Therefore, we have the following equation for the time derivative of the lag distance s:
ds as exp slow
(- 8a²
s2
R(t) -1
(22)
where
de'
Ao
s s(')2
R(1) = +
exp
(1 11)
(23)
E
8a²
slow
Equilibrium is reached quickly for a sufficiently strong stimulus, with s(t') = s(t) for any t'. Using this approximation, we can integrate Equation (23) and obtain:
R(()
1 +
exp
(24)
1 Au
8a² 2
We then solve for u and obtain:
T
E slow
08
exp
Sa²
R(t)-1
(25)
Denote the right- higtt side as g(s), which is a concave function. The maximum value of g(s) rep- resents the maximal trackable speed Umax- Solving g(b) = for v < Umax gives two solutions: stable fixed point 81 8(v) and an unstable fixed point 82 ū(v). The fixed point corresponds to the lag in the smooth tracking task. In 1.3, we show that when 3 < O, we have
s(r)
<
<5(r) and max g(s) > max g(s). AAp"
(26)
A5
This implies that the CANN dynamics with E-INN dynamics can tolerate a higher maximal speed than without E-INN dynamics.
Conclusion and Discussion
In this study, we investigated the coexistence of CANN and E-INN dynamics and found that to successfully fuse these two dynamics, it's crucial to set the strength and speed of their connections appropriately. In line with recent experimental data [24]], we suggest making the E-INN connections strong and fast to provide a balanced computational substrate, while keeping the CANN connec- tions weak and slow for stability. Our results demonstrate that the fused dynamics can preserve the unique features of both CANN and E-INN dynamics. Furthermore, we discovered that these two dynamics can work synergistically, allowing for faster downstream readout, faster adaptation to sudden change stimulus, smaller tracking lag and maintaining balanced condition under localized input. Finally, we developed firing-rate model to explain this synergy analytically, concluding that the E-INN dynamics essentially decreases the eigenvalue on the height motion mode, leading to faster convergence on this mode and overall dynamics.
Our work builds on two key components: fast E-I balanced dynamics and slow CANN dynamics which have been explored separately by recent studies.

---

On fast E-1 balanced dynamics Recent developments in spike- networks have demon- strated that coupling a fast E-I balanced dynamics with slow neural computation dynamics can lead to effective implementation of linear dynamical systems [3, DJ. This idea is further extended by local learning rules based on E-I balanced condition [+]] and minimax optimization objective for various cognitive computations [[ In these studies, E-I balanced dynamics serve an efficient and accurate coding strategy for continuous values from discrete spikes using specifically designed
connection weights that do not scale with 1/vK.
On slow CANN dynamics Several studies have pointed out that enforcing slow dynamics on CANNS a necessary choice for biologically realistic model for several reasons. Firstly, fast excita- tory dynamics of spiking CANNS can easily lead to turning-off- srrhiin-off-wihh--ee wit---wwitthtttionn where synchronous firing abruptly shuts down the persistent activity [TG. [Т.. This a biologically realistic problem introduced by spiking dynamics. Secondly, persistent activity in CANNS is subject to negative feedback processes such that the bump remains stable. These negative feedback processes (e.g.. x
usually slower than AMPAA-receptor-meediaee dynamics. As system with slow negative and fast positive feedback is inherently unstable, NMDA-receple receptor- mediated dynamics is thus essential for the stability of the dynamical system [36, B7
dynamics)
Overall, our study sheds light on how structured network can reconcile with the requirement of E-I balanced condition, which entails large and sparse unstructured connection weights. Our findings could offer a possible explanation as to why it's hard to find structured weights in biological circuits in experiments, myriad of unstructured neural connections may be required for maintaining balanced computation environment, and the structured connections that actually implement cogni- tive tasks hidden within. We hope this study will inspire further investigation into the complex interplay between different types of connectivity patterns in neural networks.
Acknowledgement
This work was supported by Science and Technology Innovation 2030-Brain Science and Brain- inspired Intelligence Project (No. 20212D0200204, 20212I0020370, 20212D0003305), National Natural Science Foundation China (No. 62088102, 62136001, T2122016).
References
[1] L F Abbott, Brian DePasquale, and Raoul-Martin Memmesheimer. Building functional net- works of spiking model neurons. Nature Neuroscience, 19(3):350-355 March 2016.
[2] Ben- Yishai, R L Bar-Or, and H Sompolinsky. Theory of orientation tuning in visual cortex. Proceedings of the National Academy of Sciences, (2(9)3344-3884 April 1995.
[3] Martin Boerlin, Christian K. Machens, and Sophic Denève. Predictive Coding of Dynami- cal Variables in Balanced Spiking Networks. PLoS Computational Biology, 10032588. November 2013.
[4] Ralph Bourdoukan and Sophie Denève. Enforcing balance allows local supervised learning in spiking recurrent networks. Advances Neural Information Processing Systems, page 2015.
[5] Nicolas Brunel. Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spik- ing Neurons. Journal of computational neuroscience, page 26, 2000.
[6] Neil Burgess, Caswell Barry, and John Keefe. An oscillatory interference model of grid cell firing. Hippocampus, 17(9)801-88) September 2007.
[7] M. C. Fuhs. A Spin Glass Model of Path Integration in Rat Medial Entorhinal Cortex. Journal of Neuroscience, 26(16):4266 4276, April 2006.
[8] C. C. Alan Fung, K. Y. Michael Wong, He Wang, and Si Wu. Dynamical Synapses Enhance Neural Information Processing: Gracefulness, Accuracy and Mobility. Neural Computation, 24(5): (115):1177-1188 May 2012.
10

---

[9] C. Alan Fung, K. Y. Michael Wong, and Si Wu. A Moving Bump in a Continuous Manifold: Comprehensive Study of the Tracking Dynamics of Continuous Attractor Neural Networks. Neural Computation, 22(3):752-792, March 2010.
[10] Apostolos Georgopoulos, Masato Taira, and Alexander Lukashin. Cognitive Neurophysiol- ogy of the Motor Cortex. Science,
April 1993.
[11] Boris S Gutkin, Carlo R Laing, Carol L Colby, Carson C Chow, and G Bard Ermentrout. Turning On and Off with Excitation: The Role of Binke-Timing Asynchrony and Synchrony in Sustained Neural Activity. page 14, 2001.
[12] Bilal Haider, Alvaro Duque, Andrea Hasenstaub, and David McCormick. Neocortical Net- work Activity In Vivo Is Generated through a Dynamic Balance of Excitation and Inhibition. Journal of Neuroscience, page 11, 2006.
[13] Jiang Hao, Xu-dong Wang, Yang Dan, Mu-ming Poo, and Xiao-hui Zhang. An arithmetic rule for spatial summation of excitatory and inhibitory inputs in pyramidal neurons. Proceedings of the National Academy of Sciences, (0665)))2190622111 December 2009.
[14] Longwen Huang, Yuwei Cui, Danke Zhang, and Si Wu. Impact of noise structure and net- work topology on tracking speed of neural networks. Neural Networks, 24(10)1110-1119, December 2011.
[15] Sung Soo Kim, Hervé Rouault, Shaul Druckmann, and Vivek Jayaraman. Ring attractor dy- namics in the Drosophila central brain. Science, 356(6340) May 2017.
[16] Carlo R. Laing and Carson C. Chow. Stationary Bumps in Networks of Spiking Neurons. Neural Computation, 13(7):1473-1494, July 2001.
[17] Qianyi Li and Cengiz Pehlevan. Minimax Dynamics of Optimally Balanced Spiking Networks of Excitatory and Inhibitory Neurons. Advances in Neural Information Processing Systems, page 11, 2020.
[18] Bruce McNaughton, Francesco Battaglia, Ole Jensen, Edvard I Moser, and May-Britt Moser. Path integration and the neural basis of the "cognitive map'. Nature Reviews Neuro- science, 768)6663-678, August 2006.
[19] Yuanyuan Mi, C C Alan Fung, K. Y. Michael Wong, and Si Wu. Spike Frequency Adaptation Implements Anticipative Tracking Continuous Attractor Neural Networks, page 9, 2014.
[20] Michael Okun and Ilan Lampl. Instantaneous correlation of excitation and inhibition during ongoing and sensory-evoked activities. Nature Neuroscience, 1 11(5):535 537, May 2008.
[21] Luigi Petrucco, Hagar Lavian, You Kure Wu, Fabian Svara, Vilim Štih, and Ruben Portugues. Neural dynamics and architecture of the heading direction circuit in zebrafish. Nature Neuro- science, 26(5):765-773, May 2023.
[22] Robert Rosenbaum and Brent Doiron. Balanced Networks of Spiking Neurons with Spatially Dependent Recurrent Connections. Physical Review X, 4(2):021039, May 2014.
[23] Alexei Samsonovich and Bruce I.. McNaughton. Path Integration and Cognitive Mapping in a Continuous Attractor Neural Network Model. The Journal of Neuroscience, 17(15):5900- 5920, August 1997.
[24] Benjamin Scholl, Connon Thomas, Melissa A. Ryan, Naomi Kamasawa, and David Fitz- patrick. Cortical response selectivity derives from strength in numbers of synapses. Nature, 590(7844):111-144 February 2021.
[25] H. Sebastian Seung. Continuous attractors and oculomotor control. Neural Networks, 11(7-
8): 90:1253-1258 October 1998.
[26] Michael N. Shadlen and William T. Newsome, Noise, neural codes and cortical organization. Current Opinion in Neurobiology, 4(4):569-579, August 1994.
11

---

[27] Yousheng Shu, Andrea Hasenstaub, and David A. McCormick. Turning on and off recurrent balanced cortical activity. Nature, 423(6937) May 2003,
[28] Jeffrey S. Taube. Head direction cells and the neurophysiological basis for a sense of direction. Progress in Neurobiology, 55(3):225-25) June 1998.
[29] Gengshuo Tian, Shangyang Li, Tiejun Huang, and Si Wu. Excitation-Inhibitton Balanced Neural Networks for Fast Signal Detection. Frontiers in Computational Neuroscience, 14:79, September 2020.
[30] Misha Tsodyks. Attractor neural network models of spatial maps in hippocampus. page 1999.
[31] Daniel B. Evrner-Evans, Kristopher T. Jensen, Saba Ali, Tyler Paterson, Arlo Sheridan, Robert Ray, Tanya Wolff, Scott Lauritzen, Gerald M. Rubin, Davi D. Bock, and Vivek Jayaraman. The Neuroanatomical Utrastructure and Function of Biological Ring Attractor. Neuron,
October 2020.
[32] C. van Vreeswijk and H. Sompolinsky. Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity. Science, 274(5293).1224-177 December 1996.
[33] C. Vreeswijk and H. Sompolinsky. Chaotic Balanced State in Model of Cortical Circuits. Neural Computation, 10(6):1311-1371 August 1998.
[34] Chaoming Wang, Xiaoyu Chen, Tianqiu Zhang, and Si Wu. BrainPy: A flexible, integra- tive, efficient, and extensible framework towards general brain dynamics program- ming. Preprint, Neuroscience, October 2022.
[35] Chaoming Wang, Yinggian Jiang, Xinyu Liu, Xiaohan Lin, Xiaolong Zou, Zilong Ji, and Si Wu. Just-In-Time Compilation Approach for Neural Dynamics Simulation. Teddy Mantoro, Minho Lee, Media Anugerah Ayu, Kok Wai Wong, and Achmad Nizar Hidayanto, editors, Neural Information Processing, volume 13110, pages 15-26. Springer International Publish- ing, Cham, 2021.
[36] Xiao-Jing Wang. Synaptic Basis of Cortical Persistent Activity: The Importance of NMDA Receptors to Working Memory. The Journal of Neuroscience, (9(21):9587-960) November 1999.
[37] Xiao-Jing Wang. Synaptic reverberation underlying mnemonic persistent activity. Trends in Neurosciences, 24(8):455-463, August 2001.
[38] Michael Wehr and Anthony M. Zador. Balanced inhibition underlies tuning and sharpens spike timing in auditory cortex. Nature, 426(6965):442 446, November 2003.
[39] Si Wu and Shum-ichi Amari. Computing with Continuous Attractors: Stability and Online Aspects. Neural Computation, 17(10)22215-2239 October 2005.
[40] Si Wu, Kosuke Hamaguchi, and Shun-ichi Amari. Dynamics and Computation of Continuous Attractors. Neural Computation, 20(4):994-1025, April 2008.
[41] Mingshan Xue, Bassam V. Atallah, and Massimo Scanziani. Equalizing excitation-inhibition ratios across visual cortical neurons. Nature, 511(75111:596-600, July 2014.
[42] Zhang. Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: A theory. The Journal of Neuroscience, (666)2211-2112 March 1996.
[43] Joel Zylberberg and Ben W. Strowbridge. Mechanisms of Persistent Activity in Cortical Cir- cuits: Possible Neural Substrates for Working Memory. Annual Review of Neuroscience, 40(1):603-66) 6270 July 2017.
12


Reviews:
[{"review_id": "Rp58AA2aPy", "reviewer": "Reviewer_kktH", "summary": "This study explores the compatibility of attractor networks and excitation-inhibition balanced networks (E-INNs) in neural circuits. It proposes that a neural circuit can exhibit traits of both by utilizing two sets of synapses: one set for strong and fast irregular firing and another set for weak and slow attractor dynamics. Simulations and analysis show that this approach enhances network performance, including accelerated convergence of attractor states and preserved E-I balanced conditions.", "questions": "See above.", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The approach addresses the challenge of reconciling the structural demands of attractor networks and E-INNs, which are typically studied independently. By combining two sets of synapses with different properties, the proposed approach allows for the coexistence of both attractor dynamics and irregular firing in a neural circuit.\n\n* The simulations and theoretical analysis demonstrate that the approach leads to improved network performance compared to using only one set of synapses. The enhanced performance includes accelerated convergence of attractor states and the retention of E-I balanced conditions for localized input. This suggests that the combined approach can achieve the advantages of both attractor networks and E-INNs simultaneously.\n\n* The study provides insight into how structured neural computations can be realized through the integration of irregular firings of neurons. By investigating the coexistence of attractor networks and E-INNs, the approach sheds light on the mechanisms underlying complex neural processing, contributing to a better understanding of neural computation in the brain.", "weaknesses": "* The authors do not discuss the biological plausibility of the proposed two-set synapse model. It is essential to consider whether such a system could be implemented in actual neural circuits and whether it aligns with known biological mechanisms.\n\n*  Another drawback of the paper is the lack of proper discussion of how general these findings are when the paper's assumptions are softened or when dealing with sufficiently different learning algorithms. When studying biological featuers in silico, qualitative and quantiative robustness is extremely important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the encouraging and valuable comments of the reviewer. Below are our replies to reviewer’s concerns.\n\n1.\tOn the biological plausibility of our model. We actually just found an experimental study [1] (which we did not know when writing this paper), which has already provided strong evidence for the two-set synapse assumption in our model. In the experimental data, the authors measured synapse strengths using combined two-photon and scanning electron microscopy techniques. They found that **“no evidence that strong synapses have a predominant role in the selectivity of cortical neuron responses”**, which correspond to the unstructured and strong synapses for EI balance dynamics in our model. They also found that **“spatial clustering of co-active inputs appears to be reserved for weaker synapses”**, which corresponds to the weak synapses for CANN dynamics in our model. Please see the experimental figure in the appended PDF.\n\n2.\tOn the generality of our model. The results of our model are quite general. There are no specific assumptions on the model, other than a requirement on the relationship between two set of synaptic weights: one scales in $\\mathcal{O}(1/K)$ and the other in $\\mathcal{O}(1/\\sqrt{K})$, with K the connectivity of neurons. However, this strict scaling relationship is mainly for the convenience of theoretical analysis, as done in [2]. In practice, this scaling relationship can be largely relaxed, as long as the synaptic weights for EI balance dynamics are much stronger than those for CANN dynamics. For other parameters, they are all consistent with the normal parameter settings in EI balance and CANN dynamics. In the revised manuscript, we will discuss about the generality of our model.\n\n[1] Scholl, B., Thomas, C. I., Ryan, M. A., Kamasawa, N., & Fitzpatrick, D. (2021). Cortical response selectivity derives from strength in numbers of synapses. Nature, 590(7844), 111-114. \n\n[2] van Vreeswijk, C., & Sompolinsky, H. (1998). Chaotic balanced state in a model of cortical circuits. Neural computation, 10(6), 1321-1371.\n"}, {"review_id": "45DJmd3FPz", "reviewer": "Reviewer_FMpH", "summary": "The paper proposes a new spiking neural network where structured connectivity, consistent with a CANN, is combined with random connectivity, consistent with a excitation and inhibition network, with two inhibitory subpopulations. The network depends onj two different sets of weights: weak weights for CANN dynamics and strong weights for E-INN dynamics.\nThe network has better performance on a signal tracking task compared to a CANN network because of faster convergence of attractor states.", "questions": "131: How do you justify that the final conclusions of our model are qualitatively applicable to general cases where $\\beta$ is large, except for simulations?\nThe solution of the stationary state $\\bar U_E$ in Equation S11 and through that $G^J (x; x0 j z)$ would be different.\nShow that this change does not affect the relation $\\lambda_0<\\lambda_*$.\n\nDid you perform simulations with a range of all the parameters?\n\nDoes it only hold for the particular set of parameter that is shown there?\n\nWhen $\\beta$ is sufficiently small, the stable solution is minimally affected by the E-INN. \nHow small does it need to be?\n\n254: Does the CANN have a different equilibrium than the actual perfectly tracked signal? Would the equilibrium that is reached quickly for a sufficiently strong input not be the perfectly tracked signal?\n\nWhy are and two different inhibitory groups necessary for the network?", "limitations": "They are adequately discussed.\n", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed network that the paper discusses is the first to combine CANNs with balanced excitatory-inhibitory spiking neurons. The proposed network has some desirable properties in terms of ability to quickly track an input. \nFurthermore, CANN dynamics with E-INN dynamics can tolerate a higher maximal speed for tracking compared to without E-INN dynamics.", "weaknesses": "It seems that the network does not truly have the same continuum of fixed points if $\\beta\\neq0$ as the CANN network would have.\nThis can be also seen with the diffusion of the \"persistent activity\" in Figure 3 after the stimulus is off. The CANN would have an actual persistent activity. \nWhat is the trade-off how the proposed network might be faster, but does not have persistent activity (e.g. in terms of $\\beta$)?\n\nThe CANN should also have a high readout after the stimulus is turned off because of the persistent activity.\n\nThe experiments  seems to be in contrast with the observation that the coupled network dynamics is neutrally stable in the second motion mode of the QHO.\n\n\nAs mentioned in 57, a total synaptic current of $\\mathcal{O}(1)$ serves as input to the CANN network. This would actually shift the second motion mode of the QHO, making the \"persistent\" state not persistent.\n\n\nThe theory for this network suggests that an infinite number of neurons are needed to build a CANN network. However, this is physically impossible in the brain. It is possible to build a CANN network with a finite number of neurons [1]. A comparison of the tracking of a signal with such a network would benefit the analysis.\n\n\n\nOverall, the methods used for the analysis are not novel.\n\n\n[1] Noorman, Marcella, et al. \"Accurate angular integration with only a handful of neurons.\" bioRxiv (2022): 2022-05.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the encouraging and valuable comments of the reviewer, which are very helpful for us to improve the paper. Below, we would like to address reviewer’s concerns in weakness and questions in detail. \n\n**On weakness:**\n\n1.\tWhether the model can maintain a persistent bump is mainly dependent on the strength of the recurrent connection strength $w_{max}^{EE}$ of the CANN dynamics. In reality, the brain does not need to hold forever-lasting activity bumps. The computationally meaningful parameter region is that the network holds not permanent, but slow-decaying bump states, the so-called slow point dynamics [1]. Therefore, the parameters we choose in the main text is in this slow point dynamics region. Nonetheless, in Fig. R2 of the appended PDF, we demonstrate that by increasing the $w_{max}^{EE}$ 25%, the network can maintain the persistent state.\n\n[1] Sussillo, D., & Barak, O. (2013). Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural computation, 25(3), 626-649.\n\n\n2.\tIndeed, the removal of the stimulus would result in a sudden drop in readout, but the dropped bump can last for a long or infinite time depending on the parameter setting.\n\n\n3.\tIn Fig. R2, we demonstrate that the neutral stability holds for stronger recurrent connections of CANN dynamics. For the parameters we used in the main text, we can approximate the slow-decaying states as stable states, and hence the neutral stability in the second motion mode roughly holds.\n\n\n4.\tWe presume the reviewer refers to that the total synaptic from EI balance dynamics shifts the _first_ motion mode (i.e., the bump height mode) and causes instability of the bump in the height direction. But we note that for $\\beta< 0$, the network can still hold static bump if the CANN weights are large enough. However, we think computational meaningful states of CANN dynamics are the slow-decaying states as discussed above.\n\n\n5.\tIn theoretical analysis, we assume that there are infinite number of neurons. To confirm the theoretical results, we often carry out simulations with a finite number of neurons, e.g., in our simulation, we chose $N=100$. The reviewer suggested an interesting reference. This paper proposes an interesting method to smoothen the energy barrier. As a side note, we also observed less discrete attractor space for $\\beta<0$ in our model when CANN connection strength is inhomogeneous (not shown in this paper). We will include this paper in the related work section.\n\n\n**On questions:**\n\n6.\tThanks for the insightful comments of the reviewer. It is true that to carry out theoretical analysis, we need small $\\beta$ (we use small/large here to represent the magnitude of $\\beta$). For large $\\beta$, we can only validate the results by simulation. Fig.S1 show that if $\\beta$ is not too large, $\\lambda_0^\\beta < \\lambda_0^*$ still holds. However, there is no theoretical guarantee that $\\lambda_0^\\beta < \\lambda_0^*$ always holds for arbitrarily large $\\beta$. We will modify our statement in the revised manuscript.\n\n\n7.\tWe performed simulations over a wide range of parameters. There is a very large parameter space in which our conclusion holds.\n\n\n8.\tThe model holds for a relatively wide range of parameters, for example, for theoretical analysis, we require the EI balance weights scale in $\\mathcal{O}(1/K)$ while the CANN weights in $\\mathcal{O}(1/\\sqrt{K})$. In practice, however, this scaling relationship can be largely relaxed, as long as the EI balance weights are much stronger than the CANN weights, our main results hold.\n\n\n9.\tTo keep the bump stable, it needs $\\beta$ to be smaller than a threshold, whose value depends on other parameters, such as the recurrent connection strength $J_0$ and the global inhibition strength $\\kappa_c$. However, it is difficult to analytically calculate this threshold\n\n\n10.\tSorry, we do not understand what the reviewer means by “the perfectly tracked signal”. \n\n\n11.\tTwo different inhibitory groups are necessary, since they are needed to separate the two dynamics running on different time scales.\n"}, {"review_id": "yy1uH0zL2y", "reviewer": "Reviewer_bo6j", "summary": "The authors present a model of a continuous attractor network in a balanced-state spiking network.\nThe model combines fast and slow connections. The fast connections are in charge of the balanced state and are thus strong 1/sqrt(K). The slow connections are in charge of the continuous attractor, and are weak 1/K. The model also has shunting inhibition which helps keep the network in the balance regime.\nThe resulting model has irregular activity, along with bump dynamics. The balanced state helps the network respond faster to changing inputs. The authors also provide an approximate firing rate model that gives intuition on the network dynamics.\n", "questions": "Figure 2 shows some properties of balanced networks, but not two that are often used in the literature: Fano factor and the log-normal distribution of firing rates.\nTable S4, S5: Why these numbers? Were they tuned somehow? Is the model robust to this choice?\nTable S6, is I_d zero?\nLine 98: cluster / clutter\nRef 17: missing bibliographic info\nSupporting, line 11. Scaled by 1/K (not K) ?\nSupporting, equation above line 4: sqrt(N) or sqrt(K)?\nLine 101. Should the fast inhibitory be slow inhibitory?\n", "limitations": "yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors combine two categories of models that are often studied separately, and demonstrate their harmonious coexistence.\nThe analytical approximation of the interaction between the two components is very helpful and could be generalized to other models as well.\n\n", "weaknesses": "The stability of the bump in figure 3A bottom – it seems that the bump decays after stimulus offset.\nI am not an expert on balanced networks, so I hope I didn’t miss something crucial. The authors claim that localized input disrupts balance in unstructured networks (line 137-138). The work of Hansel and van Vreeswijk 2012 shows that tuned input does not disrupt balance.\nAnother claim (line 40) is that E-I balance requires highly unstructured connectivity. This seems at odds with the work of Darshan et al (PRX 2018).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the encouraging and valuable comments of the reviewer, which are very helpful for us to improve the paper. Below, we would like to address reviewer’s concerns in weakness and questions in detail.\n\n\n1.\tLocalized inputs disrupt the balance condition, if the EI balance dynamics is mediated by global unstructured connections. To mitigate this problem, several modelling studies (e.g., Rosenbaum and Doiron, 2014; Rosenbaum et al., 2017 and also the two studies the reviewer mentioned) use local connectivity to construct balance networks. These works have not explored the co-existence of EI balance and CANN. Notably, these modelling works do not refute our model assumption. The assumption in our model **explicitly** requires that the observed local connectivity structure come from the small synapse set, while the EI balance dynamics is still mediated via global unstructured connections. While an EI balance network with local connectivity could work when faced with localized inputs as in previous literature, our hypothesis is more aligned with the recent experimental data in (Scholl et al., 2021 Nature), where in Fig2a of the paper, the data shows null relationships of large connection weights between neuronal selectivity and orientation preference difference.\n\n2.\tThanks for the suggestion. We plot the Fano factor of neuronal responses in the appended PDF, which is around $1$, indicating irregular neuronal responses in our model satisfies the Possion statictics. We will replace Fig.2C with it. \n\n3.\tWe set $I_d$ as zero because in the brain, inhibitory neurons usually serve as interneurons and do not receive feedforward inputs from other cortical areas. Nonetheless, $I_d$ can be non-negative and it would not affect our results.\n\n4.\tThe inhibition in our model is fast for both EI balance and CANN dynamics. The reasons for us using fast inhibition in CANN are two-fold: 1) slow negative feedback is unstable for a dynamical system and would easily lead to oscillations which could severely limit our choice of parameters; 2) the rate model requires fast inhibition so we can absorb the effect of $I_p$ into Equation (11) which permits an analytical solution. Notably, this setting is also biologically plausible, as the GABA dyanmics is much slower than the NMDA dynamics involved in the CANN.\n\n5.\tDuring our investigation, we first determined the parameters for EI balance dynamics (i.e. Table S4, S6). The classical EI balance constraint $\\frac{f_E}{f_I}>\\frac{w_{E I}}{w_{I I}}>\\frac{w_{E E}}{w_{I E}}$ is only a necessary condition, and we found some parameters satisfying this constraint can still lead to oscillatory activity on a population level. We chose the values in Table S4 and S6 simply because they give nice irregular activity patterns. There is a pretty large parameter space that satisfies this requirement. We then determine the parameters for CANN dynamics. The most important parameter is $w_{max}^{EE}$, i.e., the recurrent connection strength in CANN dynamics. It cannot be too large, which would lead to recurrent dynamics stronger than EI balance dynamics and thus defeat the purpose of this study. It cannot be too small as well, as otherwise the plateau activity would decay very fast (but see also Fig. R2 in the appended PDF where we show increasing $w_{max}^{EE}$ by 25% gives persistent activity). Other parameters in Table S5 are basically chosen as we see fit. We are sure there is a large parameter space that achieves the same performance. \n\n6.\tWe will also correct the typos the reviewer mentioned. \n"}, {"review_id": "mfdKXVdR6L", "reviewer": "Reviewer_Mp8Q", "summary": "The paper combines two established models in theoretical neuroscience, the continuous attractor neural network (CANN), and the excitation-inhibition balanced neural networks (E-INN). Though those models have been applied to explain various phenomena in the brain, they possess seemingly contradictory characteristics, begging the question of how they can coexist in the brain, which is what this paper is about.\nMore specifically, the paper first introduces a plausible spiking model with three neuron populations that incorporates both CANN and E-INN.\nThen, this spiking model is numerically simulated and shown to not only preserve the known properties of both CANN and E-INN, but also to benefit synergistically from one another through sped-up convergence, faster adaptation to change, and smaller tracking lag for varying stimuli.\nFinally, the paper introduces a firing-rate model to account for the benefits observed in simulations. The main insight is obtained with the derivation of the eigenvalue of the dominant motion mode as a function of the $\\beta$ coefficient, which accounts for the presence of E-I balance.", "questions": "Suggestions:\n\n- Figure 1 (F) caption could be more specific about what quantity is the eigenvalue from, since the answer comes only in section 5.\n\n- The shades in Figs 3 (A) Bottom, 5 (A,B,C) are not defined in the caption. Same for error bars in Fig 4 (D).\n\nQuestions:\n\n- For the spiking model, how is the equilibrium of the CANN part defined theoretically? Is there a way to define an energy function?\n\n- The CANN requires symmetric connectivity to function as an attractor network, which is often not considered a plausible connectivity motif. How can this be accounted for in the model?\n\n- I don't see from Fig 4 (D) bottom how one can conclude \"Shunting inhibition strength is proportional to the total EPSCs.\". It feels like another type of plot would better demonstrate proportionality. Could you please explain in more details?", "limitations": "The authors do not mention any limitations of their model.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The strengths of the paper are:\n\n- It is well-written and easy to follow despite being theory-heavy. It is also clearly structured.\n\n- It tackles the important question of how to reconcile E-I balance with attractor models, and the presented insights might transfer to other attractor models than CANNs.\n\n- It is a creative combination of existing ideas.", "weaknesses": "The current weaknesses of the paper are:\n\n- Many equations in the spiking model are not homogeneous, so some constant might be missing. E.g. in Eq 1 a voltage (left) is equal to a current (right). In Eq 4, $f_{j}^{b}$ should be a voltage but it is equal to the inverse of a time. Same for Eq. 7. The paper would gain in clarity if the units are correctly worked out, although this does not change the message of the paper.\n\n- The body text does not seem self-contained, I could only find the definitions of some quantities such as $U_E$, $r_E$, $k$ and $[\\cdot]_{+}$ (line 202) in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the encouraging and valuable comments of the reviewer, which are very helpful for us to improve the paper. Below, we would like to address reviewer’s concerns in weakness and questions in detail. \n\n**On weakness:**\n\nThank the reviewer for pointing out that there exist inconsistencies of units in some equations in our paper. We will correct all these inconsistencies in the revision. Also, we will add the missing definitions for some quantities in the main text as suggested by the reviewer.\n\n**On questions:**\n\n1.\tThe eigenvalues in Fig. 1F refer to the eigenvalues of CANN dynamics projected onto its dominating motion modes. We will modify the caption of Fig.1F to describe this clearly in the revision.\n\n\n2.\tShades in Fig. 3A are the instantaneous readout results (computed at each dt=0.01ms), and the solid line is the running average calculated over 150 dts (i.e. 1.5ms). Shades in Fig. 5 indicate std. calculated over 20 trials. Error bars in Fig. 4D are std. calculated over 10 trials. We will add these details in figure captions in the revised manuscript.\n\n\n3.\tIn the field, the equilibrium of a spiking CANN has never been theoretically defined, and people normally verify an equilibrium state by simulation. For a rate-based CANN, it can define an energy function if the input-output function is local. But for the rate-based CANN model we consider in Section 5 and Supplementary Material, since global divisive normalization (corresponding to shunting inhibition) is used, it does not have an energy function, although its equilibrium state can be analytically solved.  \n\n\n4.\tIndeed, all CANN models consider symmetrical connections, which is not fully biologically plausible. In reality, this assumption can be regarded as a good approximation in many cases, since CANNs have successfully modelled many neural system behaviors.\n\n\n5.\tFig 4D shows shunting inhibition (orange bar) is the strongest for the center neuron when the stimulus is positioned at the center, and the weakest for peripheral neurons. Indeed, this plot does not directly show *proportionality*. One has to refer to Equation (6) to see this where we define shunting inhibition as a product of total EPSCs and IPSCs from PV-expressing neurons. We will refine this statement in the revised manuscript.\n"}]

Author rebuttal:
**On the biological plausibility of our model**

We are delighted to find that an experimental study [1], which we did not know when writing this NeurIPS paper, has already provided strong evidence for the key assumption of our model, namely, the neural circuit consists of two sets of synapses: a weak one for attractor computation and a strong one for EI balance dynamics. In this experimental work, the authors measured synapse strengths using combined two-photon and scanning electron microscopy techniques. They found that **“no evidence that strong synapses have a predominant role in the selectivity of cortical neuron responses”**, which corresponds to the strong and unstructured synapses for EI balance dynamics in our model. They also found that **“spatial clustering of co-active inputs appears to be reserved for weaker synapses”**, which indicates that the set of weak synapses determines the orientation selectivity of neurons (note that orientation selectivity is often modeled by a CANN [2]). Please refer to the experimental figure in the appended PDF.

[1] Scholl, B., Thomas, C. I., Ryan, M. A., Kamasawa, N., & Fitzpatrick, D. (2021). Cortical response selectivity derives from strength in numbers of synapses. Nature, 590 (7844), 111-114. 

[2] Ben-Yishai, R., Bar-Or, R. L., & Sompolinsky, H. (1995). Theory of orientation tuning in visual cortex. Proceedings of the National Academy of Sciences, 92 (9), 3844-3848.

Meta-review: This paper introduces a balanced spiking neural network that implements an approximate, continuous bump attractor network. It incorporates the advantages of fast response to input and slow decay of the bump using a biologically plausible synaptic architecture. Reviewers are enthusiastic about this technically sound paper.
Decision: Accept (spotlight)