SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To provide an intuitive explanation for the accelerated convergence in our model, we examine the total input current to a neuron located in the peripheral region relative to the stimulus (Fig. 5), as the total input current determines the firing rate of a neuron and its rate of change. To clearly see the evolution trajectory of the total input current, we normalize it using the value at the stimulus onset point. We find that the total current in our model converges more quickly to its equilibrium value without undergoing drastic changes compared to a CANN model, as shown in Fig. 5A. We further examine the EPSCs-to-IPSCs ratio (E-I ratio), as it provides clues on the dynamics of the total input current. The E-I ratio in our model quickly settles to fixed values (Fig. 5B), because the strong E-INN dynamics always keep the network under balance, whereas the E-I ratio in the CANN model slowly converges to its equilibrium value due to the unbalanced nature of the dynamics (Fig. 5C).
We present the main results of our theoretical analysis in this section. The full result is in the SI. 2.
In SI 1.1 we present a 1D firing-rate model to investigate the joint dynamics of CANNs and EINNs. In SI 1.2, we show that this model can be reduced using fast-slow dynamics and mean-field approximation to obtain the following reduced model:
$$
\tau_ {\text {s l o w}} ^ {E} \frac {\partial U _ {E} (x , t)}{\partial t} = - U _ {E} (x, t) + \int_ {- \infty} ^ {\infty} [ J (x, x ^ {\prime}) + \beta ] r _ {E} \left(x ^ {\prime}, t\right) d x ^ {\prime} + I _ {\text {e x t}} (x, t) \tag {10}
$$
$$
r _ {E} (x, t) = \frac {\left[ U _ {E} (x , t) \right] _ {+} ^ {2}}{1 + k \int_ {- \infty} ^ {\infty} \left[ U _ {E} \left(x ^ {\prime} , t\right) \right] _ {+} ^ {2} d x ^ {\prime}}, \tag {11}
$$
where  $U_{E}(x,t)$  represent the synaptic input to the excitatory neurons at time  $t$  for neurons whose preferred stimulus is  $x$ ,  $r(x,t)$  represent the firing rate of these neurons, and  $I_{\mathrm{ext}}(x,t)$  represent the external inputs.  $[.]_{+}$  denotes the rectifier function and  $k$  denotes the strength of global inhibition. The recurrent neural connection matrix  $J(x,x^{\prime}) = J_{0}\exp \left[-(x - x^{\prime})^{2} / (2a)^{2}\right]$  is translationally invariant, which means that  $J(x,x^{\prime})$  is a function of  $(x - x^{\prime})$ .
Importantly,  $\beta = \bar{g}^{EE} - \bar{g}^{IE}\bar{g}^{EI} / \bar{g}^{II}$  represents the effect of E-INN dynamics on the CANN dynamics, and  $\bar{g}^{ab}$  represents the average connection strength from neural group  $b$  to  $a$  in the E-INN dynamics. The E-I balanced condition in classical literatures [33] requires that
$$
\frac {\bar {g} ^ {E I}}{\bar {g} ^ {I I}} > \frac {\bar {g} ^ {E E}}{\bar {g} ^ {I E}}, \tag {12}
$$
which gives  $\beta < 0$ . Intuitively, the E-INN dynamics affects the CANN dynamics by counteracting the excessive excitatory recurrent inputs through introducing a negative offset to its neural interaction strength. We analyze the case when the magnitude of  $\beta$  is sufficiently small for the advantage that it permits an analytical solution of the network stable state for the reduced model. Although there is no theoretical guarantee that the final conclusions of our analysis are applicable to general cases where  $|\beta|$  is large, we perform simulations to ensure that for reasonably large  $|\beta|$ , our conclusion still holds (see Fig. S1 in SI).
We first consider the case when there is no external stimulus. In this case, it is straightforward to check that, for  $0 < k < k_{c}$ , the network holds a continuous family of stationary states [39]:
$$
\bar {U} _ {E} (x \mid z) = \frac {A g}{\sqrt {2}} \exp \left[ - \frac {(x - z) ^ {2}}{4 a ^ {2}} \right] \tag {13}
$$
$$
\bar {r} _ {E} (x \mid z) = A \exp \left[ - \frac {(x - z) ^ {2}}{2 a ^ {2}} \right] \tag {14}
$$
where  $z$  represents the center of the activity bump and  $A$  is a constant. Previous studies have demonstrated that the CANN dynamics is dominated by a small number of motion modes [40]. Therefore, to simplify the analysis, we can project the network dynamics onto these dominant modes. In this study, we adopt the first two dominant motion modes, which correspond to changes in bump height and position, respectively. We address higher-order motion modes in SI 1.3. The expressions for the first two dominant motion modes are as follows:
$$
\phi_ {0} (x \mid z) = \frac {1}{\sqrt {(2 \pi) ^ {1 / 2}} a} \exp \left[ - \frac {(x - z) ^ {2}}{4 a ^ {2}} \right] \tag {15}
$$
$$
\phi_ {1} (x \mid z) = \frac {(x - z)}{a \sqrt {(2 \pi) ^ {1 / 2}} a} \exp \left[ - \frac {(x - z) ^ {2}}{4 a ^ {2}} \right] \tag {16}
$$
The corresponding eigenvalues for the first two motion modes are:
$$
\lambda_ {0} = \left(1 + \frac {2 \sqrt {2 \pi} a \beta}{g}\right) \left(1 - \sqrt {1 - \frac {k}{k _ {c}}}\right), \text {a n d} \lambda_ {1} = 1. \tag {17}
$$
We denote the first eigenvalue when  $\beta = 0$  as  $\lambda_0^*$ , which represents the eigenvalue on the height motion mode without the effects of the E-INN dynamics. For  $\beta < 0$ , we have  $0 < \lambda_0 < \lambda_0^* < 1$ , indicating that the CANN will converge faster on the height motion mode when coupled with the E-INN dynamics. The second eigenvalue indicates that the network is still neutrally stable with respect to the second motion mode, i.e. positional shift, and can hold a continuous family of activity bumps. In SI 1.3, we show that  $\beta$  does not affect higher-order eigenvalues.
Faster population activity readout Given that  $0 < \lambda_0 < \lambda_0^* < 1$  when  $\beta < 0$ , it is evident that the E-INN dynamics facilitates the convergence of the network towards the height direction. Downstream neurons can thus have faster access to the equilibrium state of CANN activity.
Faster adaptation to sudden changes Following [9], in SI 1.3 we show that when the external stimulus takes the following form:
$$
I _ {\text {e x t}} (x, t) = \alpha \frac {A g}{\sqrt {2}} \exp \left[ - \frac {\left(x - z _ {0}\right) ^ {2}}{4 a ^ {2}} \right] \tag {18}
$$
where  $\alpha$  is a positive number indicating the input magnitude, the speed of the network's activity adapting to a sudden change (up to the first order of perturbation) is given by:
$$
\frac {d z}{d t} = \left\{\frac {\alpha}{\tau_ {\text {s l o w}} ^ {E}} (z _ {0} - z) \exp \left[ - \frac {(z _ {0} - z) ^ {2}}{8 a ^ {2}} \right] \right\} R (t) ^ {- 1} \tag {19}
$$
$$
R (t) = 1 + \alpha \int_ {0} ^ {t} \frac {d t ^ {\prime}}{\tau_ {\text {s l o w}} ^ {E}} \exp \left[ - \frac {1 - \lambda_ {0}}{\tau_ {\text {s l o w}} ^ {E}} \left(t - t ^ {\prime}\right) - \frac {\left(z _ {0} - z \left(t ^ {\prime}\right)\right) ^ {2}}{8 a ^ {2}} \right] \tag {20}
$$
and  $z_0$  represents the new input location after the sudden change. Consequently, the following relation holds:
$$
\left. \frac {d z}{d t} \right| _ {\lambda_ {0}} > \left. \frac {d z}{d t} \right| _ {\lambda_ {0} ^ {*}}. \tag {21}
$$
This implies that coupling CANN dynamics with E-INN dynamics always leads to faster convergence to the new equilibrium location than without E-INN dynamics.
Smaller tracking lag Assuming a constant external stimulus speed  $v$ , we can substitute  $z_0$  with  $vt$  in Equation (19), resulting in  $s = vt - z$ , where  $z$  represents the instantaneous activity bump location. Therefore, we have the following equation for the time derivative of the lag distance  $s$ :
$$
\frac {d s}{d t} = v - \frac {\alpha s}{\tau_ {\text {s l o w}} ^ {E}} \exp \left(- \frac {s ^ {2}}{8 a ^ {2}}\right) R (t) ^ {- 1} \tag {22}
$$
$$
R (t) = 1 + \alpha \int_ {- \infty} ^ {t} \frac {d t ^ {\prime}}{\tau_ {\text {s l o w}} ^ {E}} \exp \left[ - \frac {1 - \lambda_ {0}}{\tau_ {\text {s l o w}} ^ {E}} \left(t - t ^ {\prime}\right) - \frac {s \left(t ^ {\prime}\right) ^ {2}}{8 a ^ {2}} \right]. \tag {23}
$$
Equilibrium is reached quickly for a sufficiently strong stimulus, with  $s(t') = s(t)$  for any  $t'$ . Using this approximation, we can integrate Equation (23) and obtain:
$$
R (t) = 1 + \frac {\alpha}{1 - \lambda_ {0}} \exp \left[ - \frac {s ^ {2}}{8 a ^ {2}} \right] \tag {24}
$$
We can then solve for  $v$  and obtain:
$$
v = \frac {\alpha s}{\tau_ {\text {s l o w}} ^ {E}} \exp \left(- \frac {s ^ {2}}{8 a ^ {2}}\right) R (t) ^ {- 1}. \tag {25}
$$
Denote the right-hand side as  $g(s)$ , which is a concave function. The maximum value of  $g(s)$  represents the maximal trackable speed  $v_{\mathrm{max}}$ . Solving  $g(s) = v$  for  $v < v_{\mathrm{max}}$  gives two solutions: a stable fixed point  $s_1 = \bar{s}(v)$  and an unstable fixed point  $s_2 = \bar{u}(v)$ . The fixed point corresponds to the lag in the smooth tracking task. In SI 1.3, we show that when  $\beta < 0$ , we have
$$
\left. \bar {s} (v) \right| _ {\lambda_ {0}} <   \bar {s} (v) \Big | _ {\lambda_ {0} ^ {*}}, \text {a n d} \max  _ {\lambda_ {0}} g (s) > \max  _ {\lambda_ {0} ^ {*}} g (s). \tag {26}
$$
This implies that the CANN dynamics with E-INN dynamics can tolerate a higher maximal speed than without E-INN dynamics.
In this study, we investigated the coexistence of CANN and E-INN dynamics and found that to successfully fuse these two dynamics, it's crucial to set the strength and speed of their connections appropriately. In line with recent experimental data [24], we suggest making the E-INN connections strong and fast to provide a balanced computational substrate, while keeping the CANN connections weak and slow for stability. Our results demonstrate that the fused dynamics can preserve the unique features of both CANN and E-INN dynamics. Furthermore, we discovered that these two dynamics can work synergistically, allowing for faster downstream readout, faster adaptation to a sudden change stimulus, smaller tracking lag and maintaining balanced condition under localized input. Finally, we developed a firing-rate model to explain this synergy analytically, concluding that the E-INN dynamics essentially decreases the eigenvalue on the height motion mode, leading to faster convergence on this mode and overall dynamics.
Our work builds on two key components: fast E-I balanced dynamics and slow CANN dynamics which have been explored separately by recent studies.
On fast E-I balanced dynamics Recent developments in spike-coding networks have demonstrated that coupling a fast E-I balanced dynamics with a slow neural computation dynamics can lead to effective implementation of linear dynamical systems [3, 1]. This idea is further extended by local learning rules based on E-I balanced condition [4] and a minimax optimization objective for various cognitive computations [17]. In these studies, E-I balanced dynamics serve as an efficient and accurate coding strategy for continuous values from discrete spikes using specifically designed connection weights that do not scale with  $1 / \sqrt{K}$ .
On slow CANN dynamics Several studies have pointed out that enforcing slow dynamics on CANNs is a necessary choice for biologically realistic model for several reasons. Firstly, fast excitatory dynamics of spiking CANNs can easily lead to turning-off-with-excitation where synchronous firing abruptly shuts down the persistent activity [16, 11]. This is a biologically realistic problem introduced by spiking dynamics. Secondly, persistent activity in CANNs is subject to negative feedback processes such that the bump remains stable. These negative feedback processes (e.g.,  $\mathrm{GABA_A}$ -receptor-mediated dynamics) are usually slower than AMPA-receptor-mediated dynamics. As a system with slow negative and fast positive feedback is inherently unstable, NMDA-receptor-mediated dynamics is thus essential for the stability of the dynamical system [36, 37].
Overall, our study sheds light on how structured network can reconcile with the requirement of E-I balanced condition, which entails large and sparse unstructured connection weights. Our findings could offer a possible explanation as to why it's hard to find structured weights in biological circuits in experiments, as a myriad of unstructured neural connections may be required for maintaining a balanced computation environment, and the structured connections that actually implement cognitive tasks are hidden within. We hope this study will inspire further investigation into the complex interplay between different types of connectivity patterns in neural networks.
This work was supported by Science and Technology Innovation 2030-Brain Science and Brain-inspired Intelligence Project (No. 2021ZD0200204, 2021ZD0203700, 2021ZD0203705), National Natural Science Foundation of China (No. 62088102, 62136001, T2122016).
[1] L F Abbott, Brian DePasquale, and Raoul-Martin Memmesheimer. Building functional networks of spiking model neurons. Nature Neuroscience, 19(3):350-355, March 2016.
[2] R Ben-Yishai, R L Bar-Or, and H Sompolinsky. Theory of orientation tuning in visual cortex. Proceedings of the National Academy of Sciences, 92(9):3844–3848, April 1995.
[3] Martin Boerlin, Christian K. Machens, and Sophie Denève. Predictive Coding of Dynamical Variables in Balanced Spiking Networks. PLoS Computational Biology, 9(11):e1003258, November 2013.
[4] Ralph Bourdoukan and Sophie Denève. Enforcing balance allows local supervised learning in spiking recurrent networks. Advances in Neural Information Processing Systems, page 9, 2015.
[5] Nicolas Brunel. Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons. Journal of computational neuroscience, page 26, 2000.
[6] Neil Burgess, Caswell Barry, and John O'Keefe. An oscillatory interference model of grid cell firing. Hippocampus, 17(9):801-812, September 2007.
[7] M. C. Fuhs. A Spin Glass Model of Path Integration in Rat Medial Entorhinal Cortex. Journal of Neuroscience, 26(16):4266-4276, April 2006.
[8] C. C. Alan Fung, K. Y. Michael Wong, He Wang, and Si Wu. Dynamical Synapses Enhance Neural Information Processing: Gracefulness, Accuracy and Mobility. *Neural Computation*, 24(5):1147–1185, May 2012.
[27] Yousheng Shu, Andrea Hasenstaub, and David A. McCormick. Turning on and off recurrent balanced cortical activity. Nature, 423(6937):288-293, May 2003.
[28] Jeffrey S. Taube. Head direction cells and the neurophysiological basis for a sense of direction. Progress in Neurobiology, 55(3):225-256, June 1998.
[29] Gengshuo Tian, Shangyang Li, Tiejun Huang, and Si Wu. Excitation-Inhibition Balanced Neural Networks for Fast Signal Detection. Frontiers in Computational Neuroscience, 14:79, September 2020.
[30] Misha Tsodyks. Attractor neural network models of spatial maps in hippocampus. page 9, 1999.
[31] Daniel B. Turner-Evans, Kristopher T. Jensen, Saba Ali, Tyler Paterson, Arlo Sheridan, Robert P. Ray, Tanya Wolff, J. Scott Lauritzen, Gerald M. Rubin, Davi D. Bock, and Vivek Jayaraman. The Neuroanatomical Ultrastructure and Function of a Biological Ring Attractor. Neuron, 108(1):145-163.e10, October 2020.
[32] C. van Vreeswijk and H. Sompolinsky. Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity. Science, 274(5293):1724-1726, December 1996.
[33] C. van Vreeswijk and H. Sompolinsky. Chaotic Balanced State in a Model of Cortical Circuits. Neural Computation, 10(6):1321-1371, August 1998.
[34] Chaoming Wang, Xiaoyu Chen, Tianqiu Zhang, and Si Wu. BrainPy: A flexible, integrative, efficient, and extensible framework towards general-purpose brain dynamics programming. Preprint, Neuroscience, October 2022.
[35] Chaoming Wang, Yingqian Jiang, Xinyu Liu, Xiaohan Lin, Xiaolong Zou, Zilong Ji, and Si Wu. A Just-In-Time Compilation Approach for Neural Dynamics Simulation. In Teddy Mantoro, Minho Lee, Media Anugerah Ayu, Kok Wai Wong, and Achmad Nizar Hidayanto, editors, Neural Information Processing, volume 13110, pages 15-26. Springer International Publishing, Cham, 2021.
[36] Xiao-Jing Wang. Synaptic Basis of Cortical Persistent Activity: The Importance of NMDA Receptors to Working Memory. The Journal of Neuroscience, 19(21):9587-9603, November 1999.
[37] Xiao-Jing Wang. Synaptic reverberation underlying mnemonic persistent activity. Trends in Neurosciences, 24(8):455-463, August 2001.
[38] Michael Wehr and Anthony M. Zador. Balanced inhibition underlies tuning and sharpens spike timing in auditory cortex. Nature, 426(6965):442-446, November 2003.
[39] Si Wu and Shun-ichi Amari. Computing with Continuous Attractors: Stability and Online Aspects. Neural Computation, 17(10):2215-2239, October 2005.
[40] Si Wu, Kosuke Hamaguchi, and Shun-ichi Amari. Dynamics and Computation of Continuous Attractors. Neural Computation, 20(4):994–1025, April 2008.
[41] Mingshan Xue, Bassam V. Atallah, and Massimo Scanziani. Equalizing excitation-inhibition ratios across visual cortical neurons. Nature, 511(7511):596-600, July 2014.
[42] K Zhang. Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: A theory. The Journal of Neuroscience, 16(6):2112-2126, March 1996.
[43] Joel Zylberberg and Ben W. Strowbridge. Mechanisms of Persistent Activity in Cortical Circuits: Possible Neural Substrates for Working Memory. Annual Review of Neuroscience, 40(1):603-627, July 2017.