SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaohan Lin $^{1,2}$  Liyuan Li $^{3,4}$  Boxin Shi $^{1,5}$  Tiejun Huang $^{1,5}$  Yuanyuan Mi $^{6}$  Si Wu $^{2*}$
$^{1}$  Nat'l Key Lab for Multimedia Information Processing, Nat'l Eng. Lab of Visual Technology, School of Computer Science, Peking University
$^{2}$  Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies School of Psychological and Cognitive Sciences
Beijing Key Laboratory of Behavior and Mental Health
IDG/McGovern Institute for Brain Research
Center of Quantitative Biology, Peking University
$^{3}$  Department of Mathematics, Peking University
$^{4}$  Institute of Neuroinformatics, University of Zurich
$^{6}$  Department of Psychology, Tsinghua University
{lin.xiaohan,liyuanli,shiboxin,tjhuang,siwu}@pku.edu.cn
Attractor networks require neuronal connections to be highly structured in order to maintain attractor states that represent information, while excitation and inhibition balanced networks (E-INNs) require neuronal connections to be random and sparse to generate irregular neuronal firing. Despite being regarded as canonical models of neural circuits, both types of networks are usually studied independently, and it remains unclear how they coexist in the brain, given their very different structural demands. In this study, we investigate the compatibility of continuous attractor neural networks (CANNs) and E-INNs. In line with recent experimental data, we find that a neural circuit can exhibit both the traits of CANNs and E-INNs when the neuronal synapses consist of two sets: one set is strong and fast for irregular firing, and the other set is weak and slow for attractor dynamics. In addition, both simulations and theoretical analysis reveal that the network exhibits enhanced performance compared to the case of using only one set of synapses, with accelerated convergence of attractor states and retained E-I balanced condition for localized input. We hope that this study provides insight into how structured neural computations are realized by irregular firings of neurons.
The core of theoretical neuroscience is to understand how different neural circuit models generate different dynamical behaviors, and consequently accomplish different brain functions. Among the proposed models in the literature, continuous attractor neural networks (CANNs) and excitation-inhibition balanced neural networks (E-INNs) are two canonical models that have been widely and successfully applied to describe many brain functions and neuronal response properties. Specifically, CANNs have been applied to explain the representation of continuous features in the brain, including, for instances, orientation tuning in the primary visual cortex [2], action intention in the motor
Figure 1: A neural circuit exhibiting traits of both E-INNs and CANNs. (A) E-INN dynamics are mediated by strong, unstructured connections on the order of  $\mathcal{O}(1 / \sqrt{K})$  and featuring fast synaptic current dynamics (not shown). CANN dynamics rely on weak, translationally invariant connections on the order of  $\mathcal{O}(1 / K)$  and exhibiting slow synaptic current dynamics (not shown). Only afferent connections to the excitatory neuron are depicted. (B) The kinetics of a single excitatory neuron are influenced by leak current, input currents from E-INN dynamics and CANN dynamics, as well as external input. (C) Weight distribution density plot illustrating the relatively weak nature of CANN weights compared to E-INN weights.  $N = 1000$ . (D) Intuitive illustration of the coupling mechanism between the two dynamics. Note that the same group of  $E$  neurons is shared by both dynamics. They are duplicated in this figure for better visualization. (E) Simulation results demonstrate that E-INN dynamics operate at the scale of  $\mathcal{O}(\sqrt{K})$ , while CANN dynamics operate at the scale of  $\mathcal{O}(1)$ . (F) Theoretical analysis of a parallel firing-rate model reveals that E-INN dynamics effectively decrease the first eigenvalue of the CANN dynamics projected onto its dominating motion modes, resulting in faster convergence.
cortex [10], oculomotor signal in the brainstem [25], head direction tuning in the subicular complex [28, 42], place cell tuning in the hippocampus [23, 30], and grid cell tuning in the entorhinal cortex [18, 7, 6]. The structure of a CANN was also directly observed in the neural circuits of Drosophila [15, 31] and zebrafish [21]. On the other hand, E-INNs were applied to explain the irregular firing of neurons widely observed in the experiments [26, 32, 33, 5] as well as the fast response property of a neural system to external stimuli [33, 14, 29]. The experimental evidences for the existence of E-I balance in neural systems were also reported [27, 38, 12, 20, 41]. In many modelling studies of large-scale spiking neural networks, E-I balance is imposed as a default condition in the neural dynamics.
Despite being regarded as canonical models of neural circuits, the compatibility of CANNs and EINNs in the brain remains unclear. In most modeling studies in the literature, the two models are often used separately to account for different brain functions, and there is no formal study on how the two models coexist in a neural circuit. This presents a nontrivial challenge, as the two models exhibit fundamentally different structural characteristics. CANNs rely on highly structured connections with decaying strengths based on neuronal tuning disparity, enabling localized neural activity (bumps) for information representation. In contrast, E-INNs necessitate highly unstructured, random, and sparse connections to ensure statistical independence among input received by individual neurons, facilitating irregular firing patterns. This prompts the question of how can a neural circuit reconcile these seemingly conflicting requirements?
In this study, we demonstrate that it is possible for a neural circuit to concurrently incorporate the properties of CANN and E-INN, while also compensating for their respective limitations, through appropriate consideration of connection strengths and synaptic dynamics timescales. Specifically, we consider that the connections are composed of two sets, mediated by two different types of inhibitory neurons respectively (Fig. 1A). One set comprises randomly sparse synapses, characterized by strong synaptic weights scaled with  $1 / \sqrt{K}$  (where  $K$  represents the number of connections) and
fast synaptic dynamics with time constants on the order of  $1\mathrm{ms}$ . The other set consists of synapses which are highly structured as in a CANN, featuring weak synaptic strength that scales with  $1 / K$  and slow synaptic dynamics with time constants on the order of  $100~\mathrm{ms}$  (Fig. 1C). This model assumption is supported by recent experimental data [24], where strong synapses exhibit no correlation with neuronal selectivity, while the co-activation of weak synapses determines neuronal selectivity.
The mechanism of the neural circuit exhibiting computational traits of CANN and E-INN dynamics simultaneously is illustrated in Fig. 1D-E and can be intuitively understood as follows. E-INN dynamics are mediated by strong synaptic weights that scale with  $1 / \sqrt{K}$  and fast synaptic dynamics, effectively processing feedforward inputs of  $\mathcal{O}(\sqrt{K})$  while always keeping network dynamics in balance; the resulting excitatory and inhibitory synaptic currents, albeit both of  $\mathcal{O}(\sqrt{K})$ , cancel out and create a constant total synaptic current of  $\mathcal{O}(1)$  [33]. This constant total synaptic current serves as the input to CANN dynamics, which operates on a slower timescale with weaker synaptic weights that scale with  $1 / K$ , producing excitatory and inhibitory synaptic current of  $\mathcal{O}(1)$ . Notably, this study introduces a parallel firing-rate model that accounts for the effect of E-INN dynamics by incorporating a constant term  $\beta$  into the CANN connectivity. From the firing-rate model, an analytical solution is derived, allowing us to elucidate the synergistic computation between E-INN and CANN dynamics. We find that E-INN dynamics effectively reduce the eigenvalue of CANN dynamics in the height mode, without affecting other high-order modes (Fig. 1F). We hope that this study can provide insight into how structured neural computations are realized by irregular firings of neurons.
2 The Network Model
We present our simulation results using a spiking model. Additionally, we have developed a firing-rate model for theoretical analysis, which is discussed in Sec. 5 in the main text, with a more comprehensive version provided in the Supplementary Information (SI). The spiking model comprises three neuronal populations: an excitatory group (labeled  $E$ ) and two inhibitory groups (shown in Fig. 1A). One group of inhibitory neurons ( $I_{\mathrm{d}}$ ), presumably formed by somatostatin (SST)-expressing neurons (also known as regular-spiking interneurons), provides distal inhibition to excitatory neurons, resulting in inhibitory currents that interact linearly with the somatic membrane potential. The other group of inhibitory neurons ( $I_{\mathrm{p}}$ ), presumably formed by parvalbumin (PV)-expressing neurons (also known as fast-spiking interneurons), provides proximal inhibition to excitatory neurons which induces shunting inhibition. The network dynamics depend on two sets of weights: weak weights  $w$  from CANN dynamics and strong weights  $g$  from E-INN dynamics. The E-INN connectivity pattern is random with connection probability  $p$ . CANN dynamics, on the other hand, have structured excitatory-to-excitatory connections  $w^{EE}$  with translational invariance. Otherwise, the connections are all-to-all.
The dynamics of a neuron is given by
$$
\tau_ {b} ^ {m} \frac {d V _ {i} ^ {b}}{d t} = R _ {i} ^ {b} \left(- g _ {L} V _ {i} ^ {b} + I _ {i} ^ {b}\right), \tag {1}
$$
where  $b = E, I_p, I_d$  denotes the neuron type,  $i$  denotes the neuron index,  $\tau_b^m$  is the membrane time constant,  $g_L$  is the leaky conductance, and  $I_i^b$  is the synaptic current received by the neuron. In our experiments, we set membrane resistance  $R_i^b = 1$ .
For simplicity, we only present the synaptic current received by an  $E$  neuron. Refer to SI for the full model. The current consists of three components: a fast recurrent input  $B_{i}^{\mathrm{E,fast}}(t)$ , a slow recurrent input  $C_i^{\mathrm{E,slow}}(t)$ , and an external input  $I_{i}^{\mathrm{E,ext}}(t)$  (Fig 1B):
$$
I _ {i} ^ {\mathrm {E}} (t) = B _ {i} ^ {\mathrm {E}, \text {f a s t}} (t) + C _ {i} ^ {\mathrm {E}, \text {s l o w}} (t) + I _ {i} ^ {\mathrm {E}, \text {e x t}} (t). \tag {2}
$$
The fast recurrent input  $B_{i}^{\mathrm{E,fast}}(t)$  is mediated by a set of strong synaptic connections  $g_{i,j}^{\mathrm{Eb}}$  that scales with  $1 / \sqrt{K}$  with fast synaptic current dynamics.  $B_{i}^{\mathrm{E,fast}}(t)$  enables the network to exhibit E-INN dynamics and is given by
$$
B _ {i} ^ {\mathrm {E}, \text {f a s t}} (t) = \Omega^ {\mathrm {E}} + \Omega^ {\mathrm {I} _ {\mathrm {d}}}, \tag {3}
$$
where  $\Omega^{\mathrm{E}} = \sum_{j}p_{i,j}g_{i,j}^{\mathrm{EE}}f_{j}^{\mathrm{E}}(t)$  denotes the fast excitatory synaptic input current from population  $E$ , and  $\Omega^{\mathrm{I_d}} = \sum_l p_{i,l} g_{i,l}^{\mathrm{EI_d}} f_l^{\mathrm{I_d}}(t)$  denotes the fast inhibitory synaptic input current from population  $I_d$ .
$p_{i,j} = \{1,0\}$  denotes that neurons  $i$  and  $j$  are connected or unconnected, respectively.  $g_{i,j}^{\mathrm{Eb}}$  denotes the synaptic strength from neuron  $j$  of population  $b$  to neuron  $i$  in the  $E$  population, which is of  $\mathcal{O}(1 / \sqrt{K})$ .  $f_j^b (t)$  represents the fast synaptic current input:
$$
f _ {j} ^ {b} (t) = \sum_ {k} \frac {1}{\tau_ {\mathrm {f a s t}} ^ {s}} e ^ {- \left(t - t _ {j, k}\right) / \tau_ {\mathrm {f a s t}} ^ {s}}, \quad b = E, I _ {\mathrm {d}}, I _ {\mathrm {p}}. \tag {4}
$$
Here  $t_{j,k}$  denotes the spike time of the  $k$ th spike of neuron  $j$ .  $\tau_{\mathrm{fast}}^s$  is the fast synaptic time constant. We set  $p_{i,j} = 1$  randomly with a probability of 0.25. We choose  $\tau_{\mathrm{fast}}^s = 5$  ms such that it's similar to the time constant of AMPA receptors.
The slow recurrent input  $C_i^{\mathrm{E,slow}}(t)$  is mediated by a set of weak synaptic connections  $w_{i,j}^{\mathrm{Eb}}$  that scales with  $1 / K$  with slow synaptic current dynamics.  $C_i^{\mathrm{E,slow}}(t)$  enables the network to exhibit CANN dynamics. It is given by
$$
C _ {i} ^ {\mathrm {E}, \text {s l o w}} (t) = \Gamma^ {\mathrm {E}} + \Gamma^ {\mathrm {I} _ {\mathrm {p}}} + \kappa \mathrm {S I} _ {i} (t), \tag {5}
$$
where  $\Gamma^{\mathrm{E}} = \sum_{j}w_{i,j}^{\mathrm{EE}}s_{j}^{\mathrm{E}}(t)$  denotes the slow excitatory synaptic current dynamics from population  $E$  and  $\Gamma^{\mathrm{I_p}} = \sum_l w_{i,l}^{\mathrm{EI_p}}f_l^{\mathrm{I_p}}(t)$  denotes the fast inhibitory synaptic current dynamics from population  $I_{p}$ .  $w_{ij}^{EE} = w_{\mathrm{EE}}^{\max}\exp \left[-\frac{(\theta_i - \theta_j)^2}{2a^2}\right] / N_E$ , with  $w_{\mathrm{max}}^{\mathrm{EE}}$  being the maximal synaptic weight and  $\theta_k\in [-\pi ,\pi)$  being the preferred stimulus feature of neuron  $k$  (e.g. orientation).  $w_{ij}^{EE}$  thus only depends on the tuning disparity  $(\theta_{i} - \theta_{j})$  and satisfies translational invariance.  $w_{i,l}^{\mathrm{EI_p}}$  represents the connection strength from population  $I_{p}$  to population  $E$ , which is all-to-all with connection strength scales with  $1 / K$ .  $\mathrm{SI}_i(t)$  models the effect of the shunting inhibition as the product of excitatory postsynaptic currents (EPSCs) and inhibitory postsynaptic currents (IPSCs) [13]:
$$
\kappa \mathrm {S I} _ {i} (t) = \kappa \left(\Omega^ {\mathrm {E}} + \Gamma^ {\mathrm {E}} + I _ {i} ^ {\mathrm {E}, \text {e x t}} (t)\right) \Gamma^ {\mathrm {I} _ {\mathrm {p}}}. \tag {6}
$$
$\kappa$  denotes the strength of shunting inhibition.  $s_j^b (t)$  represents slow synaptic current and is given by
$$
s _ {j} ^ {b} (t) = \sum_ {k} \frac {1}{\tau_ {\text {s l o w}} ^ {s}} e ^ {- \left(t - t _ {j, k}\right) / \tau_ {\text {s l o w}} ^ {s}}, \quad b = E. \tag {7}
$$
We set the slow synaptic time constant  $\tau_{\mathrm{slow}}^s = 150$  ms such that it's similar to the time constant of NMDA receptors. For the full network dynamics, please refer to SI. 1.
3 Preservation of E-INN and CANN Dynamics
We simulate the neural circuit with coupled dynamics of E-INNs and CANNs using a spiking model implemented in BrainPy. BrainPy uses JIT to speed up the simulation process [35, 34]. The parameters used in our simulations can be found in SI. 1.
3.1 Preservation of E-INN Dynamics
The coupled neural circuit preserves the dynamics and computational properties of E-INNs. Fig. 2A-C demonstrate that the network exhibits irregular activity patterns when the external stimulus is spatially uniform [26, 32]. Additionally, the network's activity level rapidly adjusts to a continuously changing external input, a computational feature exhibited by E-INN dynamics, whereas the counterpart with unbalanced dynamics (where  $B_{i}^{\mathrm{E,fast}}$  is scaled by  $1 / K$ ) demonstrates significantly slower adaptation (Fig. 2D) [33].
3.2 Preservation of CANN Dynamics
The equilibrium state of CANN dynamics can maintain bump-like persistent activity after the offset of external stimulus, a property thought to be the substrate of working memory. [37, 43]. Moreover, CANN dynamics exhibit neutral stability along its attractor space, endowing a neural circuit the ability to smoothly adapt to a time-varying stimulus [40, 9]. Our model retains these two properties
(C)
(D)
Figure 2: The network preserves E-INN properties. (A) The raster plot of excitatory neurons shows irregular spike patterns, verified by (B) average autocorrelation between neuron pairs, and (C) uniform power spectrum density of network activity. (D) Fitted response curves show the fast response property of the network versus an unbalanced dynamics.
Figure 3: The network preserves CANN dynamics. (A) Top: A raster plot illustrating the network's persistent activity. The blue dashed lines indicate the stimulus onset and offset. Bottom: The network activity projected onto the stimulus, normalized. Shade represents instantaneous readout results. The solid line represents running average calculated over  $1.5\mathrm{ms}$ . (B) The network exhibits smooth tracking of a time-varying stimulus. The light blue line represents the stimulus position, while the red line represents the position decoded from the network activity. (C) Fast synaptic dynamics associated with CANN dynamics can lead to synchronization, causing a failure to sustain persistent activity. (D) The model maintains its computational properties across different network sizes using the same parameter set. The vertical dashed lines represent the stimulus onset and offset.
(Fig. 3A-B). Notably, it is crucial for attractor dynamics to be mediated by slow synaptic dynamics. This is because fast synaptic dynamics could lead to synchronization, causing the network to lose persistent activity (Fig. 3C) - a phenomenon referred to as turning-off-with-excitation [16, 11]. This is not a problem for rate-based CANNs but is a biologically realistic issue introduced by spiking dynamics. Moreover, given that the attractor dynamics in our model operates at  $\mathcal{O}(1)$  (see Fig. 1D-E), the dynamics can scale with network size without altering parameters (Fig. 3D).
4 Synergistic Computation of CANNs and E-INNs
Spiking CANNs and E-INNs have their respective limitations. For instance, a spiking CANN cannot employ fast synaptic current dynamics because of the stability issues (Fig. 3C), while slow synaptic current dynamics severely restricts the neural system's ability to adapt to external changes in a timely manner. On the other hand, E-INN dynamics with global unstructured connectivity cannot maintain balance under localized input [22, 29].
However, we have discovered that CANN and E-INN dynamics not only can coexist and but also exhibit synergistic computation. This coexistence accelerates the convergence speed of CANN dynamics, addressing the limitation of slow synaptic current dynamics while maintaining stability. Additionally, shunting inhibition in the CANN dynamics helps alleviate the issue of imbalance under localized input. We present the simulation results and an intuitive explanation below, and provide a rigorous theoretical analysis in Sec. 5 and SI. 2.
(A)
(B)
(C)
Figure 4: Synergistic computation of CANN and EI-NN dynamics. (A) Decoding results from a CANN (left column) and our model (right column). Top: Raster plots illustrating the activity of excitatory (E) neurons in both models. Middle: Decoded position  $\mathcal{R}(t)$  with color indicating the readout confidence  $\mathcal{C}(t)$  (see text). Bottom: Standard deviation of decoded position. (B) The network activity (red) in our model adapts more rapidly to sudden changes in the stimulus (black) compared to a CANN with the same parameter settings (blue) (error bars should be included). (Time comparison can be included if feasible). (C) A density plot demonstrates that our model consistently exhibits smaller tracking lag than the CANN model with the same parameter settings. (D) Input current to a center neuron under spatially localized input. Top: Total current increases as the network size increases in E-INN, but remains nearly unchanged in our model. Error bars represent standard deviation over trials. Bottom: Shunting inhibition (orange bar) is strongest at the center neuron with the stimulus in the center and weakest for peripheral neurons.
4.1 Synergistic Computation
Faster population activity readout The efficiency of information processing in a neural system critically relies on the speed at which the neural dynamics can be accurately and confidently interpreted. In Fig. 4A, we demonstrate that despite the inherent weakness and slowness of the CANN dynamics in our model, downstream neurons can reliably decode its representation almost immediately after the stimulus onset. In contrast, during the initial phase of stimulus presentation, the original CANN model exhibits more fluctuation and less confidence in the decoding result.
We investigate the population decoding process by calculating the decoded position  $\mathcal{R}(t)$  and decoding confidence  $\mathcal{C}(t)$ . We perform the calculation in complex domain.  $\mathcal{R}(t)\in (-\pi ,\pi)$  is given by
$$
\mathcal {R} (t) = \arg (v (t)), \tag {8}
$$
where  $v(t) = \sum_{j} r_{j}(t) (\cos \theta_{j} + \mathrm{i} \sin \theta_{j})$  is a complex number representing the weighted summation of population activity.  $r_{j}$  represents the firing rate and  $\theta_{j} \in (-\pi, \pi)$  represents the preferred stimulus of neuron  $j$ . The decoding confidence  $\mathcal{C}(t)$  is given by
$$
\mathcal {C} (t) = | v (t) | / \max  \left(\tilde {v} _ {T}\right) \tag {9}
$$
which measures the normalized length of the decoded activity vector.  $\tilde{v}_T$  represents the smoothed collection of  $|v(t)|$  using a time window  $T$ .
Faster adaptation to sudden changes The rapid adaptation to sudden changes in stimulus is a fundamental aspect of neural computation, as it signifies the neural system's capacity to respond to external fluctuations. In our model, we observe a faster adaptation speed to sudden changes in stimulus compared to the original CANN dynamics under identical parameter settings (Fig. 4B). To quantify this, we measured the time it took for the population activity to reach  $95\%$  of the new steady-state value. Remarkably, the unbalanced CANN required almost  $50\%$  more time to converge to the new position compared to the balanced CANN. This finding highlights the computational benefits of our model in terms of enhanced responsiveness to dynamic stimuli.
Smaller tracking lag The activity of CANN dynamics can continuously adapt to a time-varying stimulus, such as head movement or external moving targets, with certain amount of lag [40]. The
Figure 5: Intuitive explanation of convergence acceleration. Shades represent standard deviation over trials. (A) Visualization of the normalized total input current (see text) to a peripheral neuron in our model and a CANN. (B) E-I ratio in our model. (C) E-I ratio in a CANN.
existence of this lag entails the neural system to come up with solutions to alleviate the delay, for example, by negative feedback [8, 19]. The E-I balanced dynamics also help to mitigate this problem. Fig. 4C shows that our model reliably exhibits smaller tracking lag compared to considering the CANN dynamics alone.
Maintenance of balanced condition One key trait of E-INNs is the production of total input current of  $\mathcal{O}(1)$  out of large EPSCs and IPSCs of  $\mathcal{O}(\sqrt{K})$ , hence the name E-I balance. However, spatially localized stimuli can disrupt this balance due to the lack of spatial structure in the connections between neurons [22, 29]. This means that the total input current to a neuron is no longer of  $\mathcal{O}(1)$ , but is instead related to its afferent connection number  $K$ . When the network size  $N$  is large,  $K = pN$  is also large, which can drive neurons to fire pathologically. In our model, shunting inhibition, mediated through PV neurons, generates a strong inhibitory current that is proportional to the total EPSCs. This greatly mitigates the imbalance issue. Specifically, the neurons that receive the greatest excitatory input also receive the greatest inhibitory input, which keeps the total input to each neuron roughly constant as the network size increases (Fig. 4D). These results are consistent with experimental findings [41], which suggest that PV neurons, but not SST neurons, are essential in adjusting the E-I ratio and maintaining E-I balance.
4.2 Intuitive Explanation