Title: Slow and Weak Attractor Computation Embedded in Fast and Strong E-I Balanced Neural Dynamics
Venue: NeurIPS 2023
Accepted: yes
Keywords: Continuous attractor neural network; Excitation inhibition balance; Brain-inspired algorithms; Object tracking;
Abstract: Attractor networks require neuronal connections to be highly structured in order to maintain attractor states that represent information, while excitation and inhibition balanced networks (E-INNs) require neuronal connections to be random and sparse to generate irregular neuronal firings. Despite being regarded as canonical models of neural circuits, both types of networks are usually studied in isolation, and it remains unclear how they coexist in the brain, given their very different structural demands. In this study, we investigate the compatibility of continuous attractor neural networks (CANNs) and E-INNs. In line with recent experimental data, we find that a neural circuit can exhibit both the traits of CANNs and E-INNs if the neuronal synapses consist of two sets: one set is strong and fast for irregular firing, and the other set is weak and slow for attractor dynamics. Our results from simulations and theoretical analysis reveal that the network also exhibits enhanced performance compared to the case of using only one set of synapses, with accelerated convergence of attractor states and retained E-I balanced condition for localized input. We also apply the network model to solve a real-world tracking problem and demonstrate that it can track fast-moving objects well. We hope that this study provides insight into how structured neural computations are realized by irregular firings of neurons.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper combines two established models in theoretical neuroscience, the continuous attractor neural network (CANN), and the excitation-inhibition balanced neural networks (E-INN). Though those models have been applied to explain various phenomena in the brain, they possess seemingly contradictory characteristics, begging the question of how they can coexist in the brain, which is what this paper is about.\nMore specifically, the paper first introduces a plausible spiking model with three neuron populations that incorporates both CANN and E-INN.\nThen, this spiking model is numerically simulated and shown to not only preserve the known properties of both CANN and E-INN, but also to benefit synergistically from one another through sped-up convergence, faster adaptation to change, and smaller tracking lag for varying stimuli.\nFinally, the paper introduces a firing-rate model to account for the benefits observed in simulations. The main insight is obtained with the derivation of the eigenvalue of the dominant motion mode as a function of the $\\beta$ coefficient, which accounts for the presence of E-I balance.\n\nSTRENGTHS:\nThe strengths of the paper are:\n\n- It is well-written and easy to follow despite being theory-heavy. It is also clearly structured.\n\n- It tackles the important question of how to reconcile E-I balance with attractor models, and the presented insights might transfer to other attractor models than CANNs.\n\n- It is a creative combination of existing ideas.\n\nWEAKNESSES:\nThe current weaknesses of the paper are:\n\n- Many equations in the spiking model are not homogeneous, so some constant might be missing. E.g. in Eq 1 a voltage (left) is equal to a current (right). In Eq 4, $f_{j}^{b}$ should be a voltage but it is equal to the inverse of a time. Same for Eq. 7. The paper would gain in clarity if the units are correctly worked out, although this does not change the message of the paper.\n\n- The body text does not seem self-contained, I could only find the definitions of some quantities such as $U_E$, $r_E$, $k$ and $[\\cdot]_{+}$ (line 202) in the appendix.\n\nQUESTIONS:\nSuggestions:\n\n- Figure 1 (F) caption could be more specific about what quantity is the eigenvalue from, since the answer comes only in section 5.\n\n- The shades in Figs 3 (A) Bottom, 5 (A,B,C) are not defined in the caption. Same for error bars in Fig 4 (D).\n\nQuestions:\n\n- For the spiking model, how is the equilibrium of the CANN part defined theoretically? Is there a way to define an energy function?\n\n- The CANN requires symmetric connectivity to function as an attractor network, which is often not considered a plausible connectivity motif. How can this be accounted for in the model?\n\n- I don't see from Fig 4 (D) bottom how one can conclude \"Shunting inhibition strength is proportional to the total EPSCs.\". It feels like another type of plot would better demonstrate proportionality. Could you please explain in more details?\n\nLIMITATIONS:\nThe authors do not mention any limitations of their model.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a model of a continuous attractor network in a balanced-state spiking network.\nThe model combines fast and slow connections. The fast connections are in charge of the balanced state and are thus strong 1/sqrt(K). The slow connections are in charge of the continuous attractor, and are weak 1/K. The model also has shunting inhibition which helps keep the network in the balance regime.\nThe resulting model has irregular activity, along with bump dynamics. The balanced state helps the network respond faster to changing inputs. The authors also provide an approximate firing rate model that gives intuition on the network dynamics.\n\nSTRENGTHS:\nThe authors combine two categories of models that are often studied separately, and demonstrate their harmonious coexistence.\nThe analytical approximation of the interaction between the two components is very helpful and could be generalized to other models as well.\n\nWEAKNESSES:\nThe stability of the bump in figure 3A bottom – it seems that the bump decays after stimulus offset.\nI am not an expert on balanced networks, so I hope I didn’t miss something crucial. The authors claim that localized input disrupts balance in unstructured networks (line 137-138). The work of Hansel and van Vreeswijk 2012 shows that tuned input does not disrupt balance.\nAnother claim (line 40) is that E-I balance requires highly unstructured connectivity. This seems at odds with the work of Darshan et al (PRX 2018).\n\nQUESTIONS:\nFigure 2 shows some properties of balanced networks, but not two that are often used in the literature: Fano factor and the log-normal distribution of firing rates.\nTable S4, S5: Why these numbers? Were they tuned somehow? Is the model robust to this choice?\nTable S6, is I_d zero?\nLine 98: cluster / clutter\nRef 17: missing bibliographic info\nSupporting, line 11. Scaled by 1/K (not K) ?\nSupporting, equation above line 4: sqrt(N) or sqrt(K)?\nLine 101. Should the fast inhibitory be slow inhibitory?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new spiking neural network where structured connectivity, consistent with a CANN, is combined with random connectivity, consistent with a excitation and inhibition network, with two inhibitory subpopulations. The network depends onj two different sets of weights: weak weights for CANN dynamics and strong weights for E-INN dynamics.\nThe network has better performance on a signal tracking task compared to a CANN network because of faster convergence of attractor states.\n\nSTRENGTHS:\nThe proposed network that the paper discusses is the first to combine CANNs with balanced excitatory-inhibitory spiking neurons. The proposed network has some desirable properties in terms of ability to quickly track an input. \nFurthermore, CANN dynamics with E-INN dynamics can tolerate a higher maximal speed for tracking compared to without E-INN dynamics.\n\nWEAKNESSES:\nIt seems that the network does not truly have the same continuum of fixed points if $\\beta\\neq0$ as the CANN network would have.\nThis can be also seen with the diffusion of the \"persistent activity\" in Figure 3 after the stimulus is off. The CANN would have an actual persistent activity. \nWhat is the trade-off how the proposed network might be faster, but does not have persistent activity (e.g. in terms of $\\beta$)?\n\nThe CANN should also have a high readout after the stimulus is turned off because of the persistent activity.\n\nThe experiments  seems to be in contrast with the observation that the coupled network dynamics is neutrally stable in the second motion mode of the QHO.\n\n\nAs mentioned in 57, a total synaptic current of $\\mathcal{O}(1)$ serves as input to the CANN network. This would actually shift the second motion mode of the QHO, making the \"persistent\" state not persistent.\n\n\nThe theory for this network suggests that an infinite number of neurons are needed to build a CANN network. However, this is physically impossible in the brain. It is possible to build a CANN network with a finite number of neurons [1]. A comparison of the tracking of a signal with such a network would benefit the analysis.\n\n\n\nOverall, the methods used for the analysis are not novel.\n\n\n[1] Noorman, Marcella, et al. \"Accurate angular integration with only a handful of neurons.\" bioRxiv (2022): 2022-05.\n\nQUESTIONS:\n131: How do you justify that the final conclusions of our model are qualitatively applicable to general cases where $\\beta$ is large, except for simulations?\nThe solution of the stationary state $\\bar U_E$ in Equation S11 and through that $G^J (x; x0 j z)$ would be different.\nShow that this change does not affect the relation $\\lambda_0<\\lambda_*$.\n\nDid you perform simulations with a range of all the parameters?\n\nDoes it only hold for the particular set of parameter that is shown there?\n\nWhen $\\beta$ is sufficiently small, the stable solution is minimally affected by the E-INN. \nHow small does it need to be?\n\n254: Does the CANN have a different equilibrium than the actual perfectly tracked signal? Would the equilibrium that is reached quickly for a sufficiently strong input not be the perfectly tracked signal?\n\nWhy are and two different inhibitory groups necessary for the network?\n\nLIMITATIONS:\nThey are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study explores the compatibility of attractor networks and excitation-inhibition balanced networks (E-INNs) in neural circuits. It proposes that a neural circuit can exhibit traits of both by utilizing two sets of synapses: one set for strong and fast irregular firing and another set for weak and slow attractor dynamics. Simulations and analysis show that this approach enhances network performance, including accelerated convergence of attractor states and preserved E-I balanced conditions.\n\nSTRENGTHS:\n* The approach addresses the challenge of reconciling the structural demands of attractor networks and E-INNs, which are typically studied independently. By combining two sets of synapses with different properties, the proposed approach allows for the coexistence of both attractor dynamics and irregular firing in a neural circuit.\n\n* The simulations and theoretical analysis demonstrate that the approach leads to improved network performance compared to using only one set of synapses. The enhanced performance includes accelerated convergence of attractor states and the retention of E-I balanced conditions for localized input. This suggests that the combined approach can achieve the advantages of both attractor networks and E-INNs simultaneously.\n\n* The study provides insight into how structured neural computations can be realized through the integration of irregular firings of neurons. By investigating the coexistence of attractor networks and E-INNs, the approach sheds light on the mechanisms underlying complex neural processing, contributing to a better understanding of neural computation in the brain.\n\nWEAKNESSES:\n* The authors do not discuss the biological plausibility of the proposed two-set synapse model. It is essential to consider whether such a system could be implemented in actual neural circuits and whether it aligns with known biological mechanisms.\n\n*  Another drawback of the paper is the lack of proper discussion of how general these findings are when the paper's assumptions are softened or when dealing with sufficiently different learning algorithms. When studying biological featuers in silico, qualitative and quantiative robustness is extremely important.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}