Learning better with Dale’s Law: A Spectral Perspective
Authors: Pingsheng Li
Jonathan Cornford
Arna Ghosh
Blake Aaron Richards
Venue: NeurIPS 2023 poster (2023)
Keywords: Dale's Law
RNNs
brain-inspired neural networks
DANNs
computational neuroscience
spectral properties
inhibition
TL;DR: Spectral properties matter more than sign constraints for learning in EI RNNs

Abstract: Most recurrent neural networks (RNNs) do not include a fundamental constraint of real neural circuits: Dale's Law, which implies that neurons must be excitatory (E) or inhibitory (I). Dale's Law is generally absent from RNNs because simply partitioning a standard network's units into E and I populations impairs learning. However, here we extend a recent feedforward bio-inspired EI network architecture, named Dale's ANNs, to recurrent networks, and demonstrate that good performance is possible while respecting Dale's Law. This begs the question: What makes some forms of EI network learn poorly and others learn well? And, why does the simple approach of incorporating Dale's Law impair learning?  Historically the answer was thought to be the sign constraints on EI network parameters, and this was a motivation behind Dale's ANNs. However, here we show the spectral properties of the recurrent weight matrix at initialisation are more impactful on network performance than sign constraints. We find that simple EI partitioning results in a singular value distribution that is multimodal and dispersed, whereas standard RNNs have an unimodal, more clustered singular value distribution, as do recurrent Dale's ANNs. We also show that the spectral properties and performance of partitioned EI networks are worse for small networks with fewer I units, and we present normalised SVD entropy as a measure of spectrum pathology that correlates with performance. Overall, this work sheds light on a long-standing mystery in neuroscience-inspired AI and computational neuroscience, paving the way for greater alignment between neural networks and biology.

Learning better with Dale's Law: A Spectral Perspective
Pingsheng Li* McGill University
Jonathan Cornford*
McGill University
Mila Quebec AI Institute
Mila Quebec AI Institute
pingsheng- li @ma@mmm..mm.mmmmmmmmmmm.....mmmm @0ammm.mmmii.......mmam......... mwg.ll.ccc..
cornfor @mi .emiba.
Arma Ghosh
Blake Richards
McGill University
McGill University
Mila Quebec AI Institute
Montreal Neurological Institute
Mila Quebee AI Institute & CIFAR
blake. ri www.caaaaaaamaaaacc.cccc quebec
Abstract
Most recurrent neural networks (RNNs) do not include a fundamental constraint of real neural circuits: Dale's Law, which implies that neurons must be excitatory (E) or inhibitory (I). Dale's ' Law is generally absent from RNNs because simply partitioning standard network's units E and populations impairs learning. However, here we extend a recent feedforward bio-inspired El network architecture, named Dale's ANNs, recurrent networks, and demonstrate that good performance is possible while respecting Dale's Law. This begs the question: What makes some forms of network learn poorly and others learn well? And, why does the simple approach of incorporating Dale's Law impair learning? Historically the answer was thought to be the sign constraints on El network parameters, and this was motivation behind Dale's ' ANNs. However, here show the spectral properties the recurrent weight matrix at initialisation are more impactful on network performance than sign constraints. We find that simple EI partitioning results singular value distribution that is multimodal and dispersed, whereas standard RNNs have unimodal, recurrent Dale's ANNs. We also show that the spectral properties and performance of partitioned EI networks worse for small networks with fewer units, and we present normalised SVD entropy as measure of spectrum pathology that correlates with performance. Overall, this work sheds light a long-standing mystery in neuroscience-inspired AI and computational neuroscience, paving the way for greater alignment between neural networks and biology.
clustered singular value distribution, do
Introduction
Recurrent neural networks (RNNs) are a major tool in computational neuroscience research, with numerous papers that use RNNs to model the brain published every year 2 3 4 [5]. However, when using RNNs model real neural networks there is always the question of which biological constraints to incorporate into the model. Some biological constraints are considered law-like [d. and are thus thought be particularly important. One example is Dale's Law, which says that neurons
*Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

release the same set of neurotransmitters at all of their axon terminals 77. Practically, for artificial neural networks, Dale's Law implies that synaptic ouiput weights should be all positive (excitatory, E) or all negative (inhibitory, 1) for any individual neuron ! And, though it is common to use models that respect Dale's s Law in computational neuroscience p 5), it is not universal practice, and it rare outside of neuroscience. One reason for this is that incorporating Dale's Law into neural networks tends to harm their learning performance [[C], which is widely known by those who have trained such networks, but rarely explicitly addressed in the scientific literature. This limits our ability to model the impressive learning of biological neural networks with RNNs that respect Dale's Law. It also impairs studying the unique computations played by inhibitory cells using RNN models [1 [3] and restricts comparisons between RNN models and real neural data [4 15].
Typically, Dale's 's Law is incorporated into RNNs using the most obvious solution, by constraining entire columns of the synaptic weight matrices to he of the same sign, thereby partitioning the hidden units into E and I populations (here we refer to these networks as 'Column
or "ColEI", see Figure ! for an illustration)
19] While simple and intuitive, as mentioned, this approach impairs learning performance 10 One hypothesis for why this is, is that El sign constraints impair network performance by limiting the network's s solution space 200. [C] Indeed, a recent feedforward EI ANN architecture (Dale's ANNs, or DANNs), inspired by fast inhibitory circuits in the brain, and which was designed to overcome sign constraints, was found to learn just as well standard ANNs IC 211 In this work, we show that recurrent versions of DANNs can also learn as well as standard RNNs. But, we also want to determine whether the sign constraints really the primary cause of impaired learning in ColEI networks. Are there other reasons that ColEl networks learn worse than standard RNNs and DANNs?
5
Here we show that the spectral properties of El networks that are most important for their performance. We find that the singular value spectrum of ColEI networks as in 2 is multimodal, dispersed. and includes a notable mode of large singular values. In contrast, DANNs and standard RNNs exhibit unimodal spectra as expected from random matrix theory 22 Next, we transplant spectra between networks and find the spectrum is more impactful than sign constraints on performance though both contribute. Furthermore, we find that small ColEI networks with biologically realistic E to units ratio learn less effectively, and this is reflected in their spectral properties. In contrast, the spectrum and learning of DANNs is robust to both the ratio of E and I units, and network size: even small networks with only inhibitory neuron have good spectral properties and learn well. Finally, show that normalised SVD entropy, a measure of unimodal singular value clustering, is predictive of EI RNN learning performance.
Altogether, our results provide evidence that the reason for the impaired performance of traditional ColEI RNN networks is predominantly their spectral properties, and less so their sign constraints. Therefore, the improved performance of DANNs should be understood as primarily resulting from them having spectral properties that are similar to standard RNNs, in addition to their being robust to sign constraints. This is a critical insight for designing EI networks that are architecturally different from DANNs, which may be desirable when modelling specific subtypes of interneurons 23.. Finally, more broadly we present an answer to the long-standing mystery of poor performance of RNNs that respect Dale's Law, thereby paving the way for their future development and use in neuroscience-inspired AI.
2 Methods
2.1 Model Definitions
In this section we first define "standard" RNNs, then the two different kinds of RNNs that obey Dale's Law, namely ColEI and DANNs. To obey Dale's Law, ColEI and DANN networks were first initialised in line with Dale's law (see initialisation details below), and projected gradient descent was used to maintain parameter signs during training. As such, after a candidate gradient descent update, parameters that had changed sign were set to 0, corresponding to euclidean projection onto to the allowed solution space.
The reader can find the code for all our experiments available here
SSee g for rare exception of excitatory and inhibitory neurotransmitter co-release.

---

Standard Recurrent Neural Networks (RNN)
For multilayer network, define the hidden state activations of "standard" RNN layer & at time I, hy, via:
h = 4 + W Wf b²)
(1)
where a-)) (.) is the activation function, 1 is the activation vector for the lower layer, 1-11 is the hidden state activations at the previous time-step. W and W rec are the input and recurrent synaptic weight matrices, and be the bias. Note h is equivalent to the input, Iy. for the first layer, i.. hy It. The output, Vr. is a function of the final layer's 's (L) hidden state:
y/ = 9- + bout)
where Wout is the output weight matrix, bout is the output bias, and g(-) is optional output non-linearity, We set g(-) depending on the task: softmax function for classification and language modelling tasks, and identity for the adding problem.
Column Excitation-Inhibitton (ColET)
The ColEI model discussed in this work has the same formulation as a standard RNN (equations
M and 2 except that the weight matrices sign constrained. Specifically: (1) the input weights
of the first recurrent layer W(()
in are positive; (2) all other weight matrices are constrained to have columns whose entries are the same sign, either all positíve or all negative. In combination
with ReLU activations, this constraint makes positive columns correspond to the output weights of presynaptic excitatory neurons, and negative columns inhibitory neurons. An illustration is provided Figure A.
layer L+1
layer
layer L+1
layer
layer
layer L-1
layer
layer
1=
1=1
1=0
t=1
Figure 1: Illustration of ColEI and DANN weight matrix partitions with excitatory (red) and inhibitory (blue) populations. (A) The ColEI partition, which separates the neurons within layer into E and populations. (B) The DANN partition with a population of neurons placed between E layers.
Dale's Artificial Neural Networks (DANN)
DANNs El architecture first introduced in 110 for feedforward networks that were inspired by fast feedforward inhibition the brain [24]. Here we extend DANNs to the recurrent setting. A DANN layer reparameterises the weight matrices of standard ANNs using three separate non-negative matrices. Specifically, for a given weight matrix W,, we have:
EE WEIWIE
where WEE, WIE and W denote E-to-E, E-to-I, and I-to-E projections respectively (see Figure B for an illustration). Note that such parameterisation allows the sign of each element of W to be unconstrained, although the signs of individual weight matrices, i.e WEE II. and W/ are constrained by clipping illegal weights to 0 after each update.

---

2.2 Initialisation details
Standard RNNs:
The weights of standard RNNs were sampled from a uniform distribution U vo J. where n is the number of hidden units (i.e. the PyTorch default initialisation), unless otherwise stated. Biases were initialised as O.
The parameters of ColEI networks were initialised following 10 The weights in each column were sampled from either the exponential distribution corresponding to inhibitory (negative) or orexiitatory (positive) weights. The distribution means were set (i) to ensure that the sum of excitatory and inhibitory inputs to each neuron was balanced in expectation (i.e. Σjeeze lug] Jeann ldj] for inputs indexed by j) and, (ii) following [25, 26), so that activation variance did not scale with depth (see Appendix 6.6 for details). Next recurrent weights were adjusted constrain their spectral radius, in line with 19. by setting W ree +
ColEI:
rear where Po is the spectral radius of W before the adjustment, and p = 1.5 unless stated. Biases were initialised as O, except for the first layer, where they were set to centre the activations.
DANNs: Unless explicitly mentioned, all entries of W matrices were independently sampled from an exponential distribution of variance 3m(n-15. where n is the number of hidden units. Next, to balance inhibitory and excitatory inputs, each row W was initialised as the mean row of W WEE (therefore each row of W is the same at initialisation), and rows of WR were sampled from an exponential distribution and then squashed so that the sum of each row was 1. As such, the variance of W is the same Pytorch's 's default initialisation for RNNs. Biases were initialised as O.
2.3 Experiments
We tested the three different types of networks (standard RNN, ColEI, and DANN) on three classical tasks for RNNs: the adding problem 277,, sequential MNIST classification 28 and language modelling using the Penn Tree Bank [2 All experiments were run with PyTorch version 1.5.0 on a RTX 8000 GPU cluster. Unless otherwise stated, all presented results were averaged over seeds and shaded error bars denote standard deviation. Performance metrics are plotted after the first 100 updates.
The adding problem serves as a test to evaluate the ability of RNNs to learn long-term dependencies pT. The objective is to compute the sum of two numbers selected from sequence of random numbers E [0, 1] (of length 20 our experiments). At each time step the input to the network a a 2-dimensional vector, comprising element from the sequence of random numbers and a boolean, which is one only for the two numbers that should be added and zero otherwise.
Adding problem:
The sequential MNIST problem is common test for the ability of RNNs to engage classification. In this task, the network must classify MNIST digits (i.e. handwritten digits of numbers) when the rows of the image are presented sequentially to the network from top to bottom (resulting in 28 timesteps).
Sequential MNIST:
The Penn Tree Bank task is a common baseline for natural language processing that more challenging than either the adding problem or sequential MNIST. In this task, input words from sentences taken from newspaper corpus are embedded as 300 200-dimensional vectors using GloVe [30 At each time step, the network must predict the next word.
Penn Tree Bank:
Natural object recognition task: Following 31 we designed the following naturalistic object recognition task. At the first time step, the network's input is down-sampled, flattened achromatic image of one of 64 natural objects from categories (e.g. fruits, faces, etc.) with a random natural- scene background. The network receives no further inputs and must retain information about the image over 5 time steps to output the classification result at the last time step. Please see [3 and Appendix Fig. for more details.
2.3.1 Spectrum Transplant
Singular value decomposition (SVD) decomposes a matrix into matrices of singular values and singular vectors: W = UEVT, where U and V are orthonormal matrices whose columns are singular vectors, and Σ is a matrix of singular values. Therefore, network weight matrices can be decomposed as: W WRNN= = U ENNN V2 N- WCA = UCaET ECCaEr VT VOLEE- To disentangle the impact of singular values and singular vectors performance constructed networks initialised with weight

---

matrices W1 = URNN EOOEE RNN and W2 = UCOLEEENN N ColEI. such that a standard RNN was given the spectral properties ColEI network, and vice-versa.
2.3.2 Normalised SVD Entropy (NSE)
To quantify the clustering strength of the singular values of a N x N recurrent weight matrix we used normalised SVD entropy (NSE), also denoted as H
N
di log(N)> Σ o,log(or), where 6, Σ, Øj
where o; are the singular values. The normalisation term (Ng(N) ensures H [0. 1] regardless of network size, where H = 1 if all singular values are the same (clustered) and H z 0 implies the distribution of singular values is extremely dispersed. For example, in orthogonal RNNs H = 1, since all singular values of the recurrent weights are 33
3 Results
3.1 Network performance on benchmark tasks
We first assessed the performance of standard, ColEI, and DANN RNNs on four benchmark tasks for RNNs: the adding problem, row-wise sequential MNIST, the Penn Tree Bank dataset, and a naturalistic object recognition task (Section 2.3 We found that ColEI networks did not learn as well as DANNs and standard RNNs. Their final performance was worse across all four tasks, and more variable across random seeds (Figure 2
A-D). Furthermore, observed that gradient clipping benefitted ColEI networks substantially more than the other networks (Appendix Figure I0
A
DoNN (pe-31
RoMine nAM or!
240
(po-3)
90
(pc-
Test MSE Loss
nen (o+1)
Colt (gc=?)
Test Error 1%6)
Cut 1po-101
Perplexity
200
Cott (gowti
(%)
5
160
20
0
4k
8k Updates
12k
0
15
30
45
120
0
10
20
30
40
10
20
30
k Updates
# Updates
# Updates
Figure 2: Performance of networks benchmark datasets. (A) Adding problem (1 layer of 10 neurons). (B) Sequential MNIST (3 layers of 100 neurons). (C) Penn Tree Bank (3 layers of 500
neurons). (D) Naturalistic Object Recognition (1 layer of 1000 neurons). ge denotes gradient clipping.
confound that could explain the discrepancy performance between ColEI networks and DANNs & standard RNNs is the distinct strategies used for their initialisation (Section 2.2 Specifically, ColEI recurrent weights are initialised such that the greatest norm of eigenvalues, L.e. the spectral radius p. 1.5 [ In contrast, the initialisation of DANNs standard RNNs results in p ~ 1/v3 (PyTorch default RNN initialisation). Therefore, we assessed model performance across a range of initial values of p (Appendix D. Again, we found that ColEI networks learned poorly, and there was no value of p for which ColEI networks matched the performance of the best standard RNNs and DANNs. However, while standard RNNs and DANNs had similar relationship between P and performance, ColEI networks performed better than regular RNNs and DANNs at larger values of and worse for smaller values (Appendix Figure Dh. Additionally, we also ran a control experiment to verify that the inhibitory parameter scaling of DANNs did not improve ColEI network performance (Appendix Figure
3.2 Spectral properties of networks
The different relationship between performance and of ColEI networks vs standard & DANN RNNs Frigure[D) indicative different spectral properties, such as the eigenvalue distribution of the recurrent weights. For example, eigenvalues greater (or less) than one indicate activity directions that
Note this quantity is not the entropy of the probability distribution of singular values.
5

---

lead to exploding (or vanishing) activations and gradients, thereby impairing learning, Inspecting the eigenvalues of the initial weight matrices in the complex plane (Figure B A-C) shows that the eigenvalues of ColEI networks follow non-uniform distribution that is clustered around the origin, In contrast, the eigenvalue distributions of DANNS as has been previously reported 334 R5
standard RNNs converge to a uniform circle in the complex plane f We note that these spectral properties provide an informal explanation for the better learning of ColEI networks at larger values of p (Figure D), as there will still be majority of smaller eigenvalues in addition to the minority of large eigenvalues.
A
eigenvals of ColEI (p 1.5)
B
of eigenvals RNN
C
Dist eigenvals DANN
1.51
1.5
1.5
Imaginary
Imaginary
0.0
0.0
0.0
-1.5
-1.5
0.0 Real
1.5
-1.51.5
0.0 Real
1.5
-1.5
0.0
1.5
Real
D
Sing Vals of ColEl (p = 1.5)
E
Dist of Sing Vals
F
Sing Vais
10
10ap
Prob Density
10a
Prob Density
Prob Density
hub Denalt
0.6
10-2
10-2
10" 2
0.0
SinGIAN Vake
Singu Value
12 Singular Value
12
18
12 Singular Value
6
18
Singular Value
Figure 3: Visualisation of eigenvalues and singular values for ColEI, standard, and DANN recurrent weight matrices. (A-C): Eigenvalues plotted in the complex plane for 100 different initialisations. Red circle in A has radius of 1.5; for B.C radius 1/v3. (D-F): Histograms of singular values for 100 different initialisations. Insets in E,F show blow-ups of the red lines in main plot.
However, there are limitations to using eigenvalues for understanding learning dynamics. Importantly, the eigenvectors of non-symmetric matrices are always non-orthogonal, and eigendecomposition may fail to capture certain features of matrices with highly non-orthogonal columns such as ColEI networks [36]. In contrast, singular vectors always orthogonal. Thus, for ColEI networks reasoned that although the maximum eigenvalue is constrained to be 1.5, the maximum singular value may be large and unconstrained. Indeed, while the maximum singular value for DANNs & standard RNNs was approximately twice the maximum radius of their eigenvalues, the ColEI singluar value spectrum had notable mode of large singular values (Figure 3 D-F). F. Also, the distribution of singular values for ColEI networks was multimodal (Figure F-),, whereas the distribution of standard RNNs and DANNs converged to quarter circle, in agreement with the Marchenko-Pane Pastur Quarter Circle Law 5 These results tell us that even when the eigenvalue radius is constrained for ColEI networks, there will still be directions in activity space corresponding the large singular values along which the activations and gradients will be heavily deformed and stretched, leading to poor learning.
"This corollary of Girko's circle law, which states that, large eigenvalues of an random matrix are distributed uniformly within the complex unit circle, if the elements are independently sampled from a mero-mean distribution with variance N- Pytorch default initialisation (U[- J )) has a variance of sw which scales down the radius of the circle by V
"This phenomenon is unique to exponential distribution, which used in the initialisation of ColEI RNNs. We also observed similar spectral properties for gamma and uniform distributions see Appendix 14 'This result is for N x N matrix with i.i.d. elements drawn from a distribution with zero mean and variacce and is agnostic to the exact distribution from which the elements are drawn.
22

---

3.3 Spectral properties matter more than sign constraints for learning
common hypothesis is that the poor performance of EI networks is due to the parameter sign constraints applied during learning, and addressing this was a motivation behind the architectural design of DANNs TO 20- Given the different spectral properties of ColEI and standard RNNs that we observed, we wondered: are sign constraints the main reason for impaired learning in ColEI networks, or are the spectral differences from standard RNNs the most critical factor? To investigate this, we first explored the impact of sign constraints: we trained ColEI networks without sign constraint during learning, and we trained standard RNNs with parameter signs frozen after initialisation (Figure 4 A, sequential MNIST), As expected, for both networks we found that sign constraints impaired learning (0.7% = 0.3% for RNNs, 1% + 1% for ColEI), but they did not account for the full difference in performance between standard RNNs and ColEIs (2.3% + 0.6%).
Next, we designed spectrum "transplant" experiments. to investigate the impact of singular value spectra on learning (Section 2.3.Ip p. We constructed hybrid networks with combination of spectral TD. D. specifically: 1) RNN or ColEI spectrum, 2) RNN or ColEI singular vectors, 3) presence or absence of sign constraints during learning. In total, with these three factors we had eight different network configurations (23 networks). These experiments revealed that the singular value spectra contribute more than sign constraints and singular vectors to the performance gap between ColEI and RNNs (Figure 4 B,C 1.5% 0.3% for RNNs, 2% + 1% for ColET). In addition to sequential MNIST, we also ran the same set of experiments on the Naturalistic Object Recognition task, and observed very similar results (Appendix Table 2 Finally, we investigated if the empirical EI balance initialisation (i.e. the mean of the sum of incoming weights to each neuron) was altered in these experiments. We found it was not: for ColEI network with 10% inhibitory neurons and 1000 hidden units, the mean balance across units was -0.0006 with std. deviation 1. 1.12, and for ColEI with the RNN spectrum it was 0.0696 with std. deviation 0.88 (t test, p=0.12). Thus, sign constraints are only one component, and the singular value spectrum at initialisation has a stronger impact on learning performance.
properties at initialisation and sign constraints (see Section 2.3.
A
Sign Constraint
B
Spectrum Singular Vecs
C
RNN Ablation
ColEl Ablation
10
10
2.5
Decreade in Yest Enor 2
Test Test Error (%)
Test Emor (%)
lecce
Incresse in Test Errer
1.5
2.5
2
0.5
0.5
0
15
30
45
0
k Updates
A k Updates
Speciun
Senlia Sin COMPER
Figure 4: Impact of spectrum, singular vectors and sign constraints on learning (A) Impact of sign constraints on standard and ColEI RNNs (note that DANNs are always sign constrained). (B) Impact of transplanting singular vectors and values between ColEI and RNN networks. (C) Summary of final test error changes induced by spectrum, singular vectors, and sign constraints. RNNs 5 seeds (p < 0.05); ColEI 30 seeds due to higher variance (p 0.01). Mann-Whitney U test
3.4 Network performance and spectrum with respect to changes in EI ratio and network size
Having established that the spectrum of singular values is important for ColEI learning, we inves- tigated how architectural hyperparameter choices, such as the El ratio and number of hidden units, change the spectral properties and learning performance. Here, to quantify changes in the singular value distribution, we leveraged normalised SVD entropy (NSE) as measure of spectrum "pathology" (see Section 2.3.2 2. Notably, NSE captures the degree of unimodal clustering in the singular value spectrum, and we hypothesise that a unimodal spectrum of clustered singular values is desirable as is the case for orthogonal RNNs
3 28
We found that reducing the percentage of inhibitory neurons (thereby increasing the El ratio) decreased the NSE and impaired learning performance (Figure 5) Also, despite constraining p = 1.5 in each case, decreasing the number of inhibitory neurons increased the largest singualar value, amaz (Figure D-F). In contrast, we found that the spectra and performance of DANNs were robust to changes in the El ratio, and they learned well even with inhibitory neuron (NSE remained ~ 0.95 for both DANNs and standard RNNs, Figure A-C, Appendix Figure B5 Appendix Table g. This
7

---

result further supports the idea that the spectral properties of El networks predict their learning performance, and shows that the E-to-I ratio impacts the spectral properties of ColEI networks.
A
Percent of inhib 1%
B
Percent of inhib 10%
C
Percent of inhib 50%
20
20
Rest E 68
15-
DANN
15
DAM
PAN
Tost Error (99
15
G
10
10
10
Regress Line
5
Test Error (%)
10-
1% inhibtrry
10% inhibtoryy
0
52% intibturyy
15
30
0
Updates
Updatas
Updates
D
Percent of inhib 1%
m
Percent of inhib 10%
F
Percent inhia 50%
Pro Deraty
20
prot Deretty
Froh Denatty
20
20
0.9 Normalized SVD Entropy
0.5
0.7
10-1
Singuaa
Value
Singular Va
Figure 5: Performance and spectral properties of DANN and ColEI RNNs with different El ratios. (A-C) Performance sequential MNIST. Dashed line indicates the final error of standard RNNs of same size. (D-F) Distribution of singular values for ColEI networks y-axis log scaled, inset plots are x-axis 80. See equivalent for DANNs in Figure B and Appendix Figure 15 (G) ColEI performance v.s. initial normalised SVD entropy of the recurrent weights across different El ratios. (I-layer RNN,
100 units, PColEs = 1.5, PDANN
v3
A
Num of Hidden Units:
B
Num Hidden Units: 300
C
Num Hidden Units: 1000
Com
Test Eroor (S)
Colfi
ColEI
Test Frtor (8a
DONN
DANN RMWN
Emor
DANN
RNN
RNN
Teat Error (9)
4.5
50
100
150
50
100
150
D
Num of Hidden Units: 100
Updates
Hidden Units:
Updales
Updates
2.5
Regression L ne
n=100
1 300
Prot Denstt
Pelo Damaty
prob Demin
0.5
no1000
0.7
0.8
0.9
20-1
Normalized SVD Entropy
65
Singuiar Value
Singuar Value
Singular Value
Figure 6: Performance and spectral properties of DANN and ColEI RNNs with different network sizes (column-wise) wise). (A-C) Performance on sequential MNIST. Dashed line indicates the final error of RNNN of same size. (D-F) Distribution of singular values for ColEI networks y-axis log scaled, inset plots are x-axis [0, 8). See equivalent for DANNs in Appendix Figure [5 (G) ColEI performance initial normalised SVD entropy of the recurrent weights across different network sizes. (1-layer RNN, 10% inhibitory units, PCoLEI = 1.5, PDANN
v3
Next, investigated how the number of hidden units affects the spectral properties and learning performance of El networks. As shown in Figure 5 A-F, we found that the bulk of singular values became more clustered for larger ColEl networks (E [0, 8] Figure D-F), and the NSE increased despite an increasing Tmaz. Correspondingly, these changes in NSE were anti-correlated with test error (Figure similar to the performance trends for EI ratio. In contrast, DANN spectra were similar across network sizes (Appendix Figure 15). Taken together with the El ratio experiments, these results suggest that for ColEI networks, the best performance will be obtained for larger networks with balanced EI ratios. because these networks have spectral properties more similar to standard RNNs. However,
interesting that large ColEI networks learn well despite a large mwars which indicates that the overall effect of all singular values, rather than omans is what matters most for learning- We futher discuss this result in the discussion, but put another way, in larger networks even if some directions have large singular values, there many more directions with small singular
8

---

values. As a result, activities will likely be rotated away from the directions with large singular values, ameliorating the exploding gradients problem. Therefore. for metric to be predictive of the network performance it should capture the overall effect of the entire spectrum, which we do with NSE.
A
B
H
0.95
C
Mcan Test Error
5.0
our High sunay
Network Size
200
Network Size
200
Test Test Error (86)
10-
600
-075
600
3.5
Test Emar
2.0
2.0
1000
0.55
30
23%
45%
1.0 Normalized SVD Entropy H
0.6
0.8
Updates
Percent of inhib
Percent of inhib
Figure 7: Normalised SVD entropy predicts ColEI network performance. (A) Performance on sequential MNIST for networks indicated by circles in B&C. (B) Normalised SVD entropy for networks of different sizes and EI ratios. (C) Mean test error for networks of different sizes and El ratios. Top left pixel value clipped at 5 for better visualisation. (D) Final mean test error vs Normalised SVD entropy for all ColEI networks. 1 outlier network (corresponding to the top left pixel in C, B) is removed for better visualisation.
The above experiments suggest that the spectral properties of ColEI networks at initialisation, summarised by NSE, can be used to estimate their performance across a range of architectural hyperparameters. As the NSE at initialisation is dependent on both the network size and El ratio, networks can have a high NSE from either being large, or having a balanced El ratio. Therefore, we hypothesised that small networks with balanced E & I units will outperform larger networks with a
skewed EI ratio that results in a smaller NSE at initialisation. To test this prediction, we generated networks of different EI ratios and numbers of hidden units and calculated their associated NSE at
initialisation Figure 7 B. Again, we found that NSE anti-correlates with final test error (Figure 7 D).
and small networks with high NSE performed better than large networks with lower NSE (Figure A).
Similar trends are observed for the natural object recognition and language modelling tasks (Appendix Figure 12 & Figure 13 respectivelyy. This supports the hypothesis that the NSE at initialisation is of
primary importance for determining learning performance. Moreover, it allows us to characterise the
link between spectrum pathology and performance and explain why different hyperparameter choices
may affect learning in EI RNNs.
4 Discussion
In this work, we have provided evidence that both sign constraints and the spectral properties of the weights at initialisation can cause poor learning in EI RNNs. As well, we compared the relative contributions of sign constraints and spectral properties for EI network performance, and found that the distribution of singular values are more important for learning. We found that the simplest method of incorporating Dale's Law into RNNs (by constraining columns of the weight matrix, ColEI, following (9]) results in networks with multi-modal spectra that are dispersed, especially for small networks with large ratios of
units. In contrast, we found that recurrent versions of the DANN EI architecture (10) have similar spectra to standard RNNs, and learn just as well. We also presented NSE as a measurement of spectrum pathology that can be used as diagnostic tool to predict performance before training. The higher the NSE at initialisation, the more likely an El RNN to train well. This work therefore explains why DANN RNNs perform as well as standard RNNs, but not ColEI RNNs. Overall, by highlighting the importance of El network spectral properties at initialisation we provide a map for designing networks that respect Dale's law and learn well.
In addition to their poor learning, ColEI networks also showed more variance during training and over different initialisations when compared to the other networks. This likely due to them having fewer activity modes with appropriate singular values for learning computations over time (the motivation behind SVD entropy) and therefore small differences at initialisation can strongly impact learning. Similarly, we also found that without gradient clipping ColEI networks are highly unstable, again line with a pathological distribution of singular values initialisation [37. However, other open theoretical questions arise from the empirical results presented above. First, what is the origin of the unusual multi-modal spectra of CoIEI networks? And second, why do large ColEI networks learn well despite large maximum singular value, Umaz?

---

For simple intuitive explanation of ColEI spectrum pathology, let us first consider the difference between the recurrent weight matrix of ColEI and standard RNNss For both the standard & DANN RNNs, this a zero-mean random matrix. However, the ColEI recurrent weight matrix can be written as the sum of a rank-one nilpotent matrix (of column means) and zero-mean random matrix. While technically the singular and eigen value distributions of two matrices can't simply be added when two matrices are added, this already provides some intuition as to the origin of the unusual spectrum: it likely stems from the nilpotent matrix of column means. Here we note that the eigenvalues of any nilpotent matrix are O, and hence we expect they do not significantly change the eigenvalue radius. If so, constraining the spectral radius of the recurrent matrix will not constrain the nilpotent matrix's ' contribution towards the singular value spectrum (see the Appendix 6.5 for a more detailed discussion).
Regarding the second open question, we found that large ColEI networks learned well, despite an increasing maa- Intuitively this should harm learning, as some activity directions will result in exploding gradients 37. However, we can understand this by writing the operation performed by a linear network in terms of the SVD of its recurrent weight matrix:
M++2 = W rec Wi hf = USVV
is clear that between each scaling operation Σ, there are two rotation steps VT, U. These rotation steps could allow the network to avoid pathological growth by rotating the activity vectors aligned to directions with large singular values, such that they instead align with directions with small singular values. Furthermore, the network width increases, the number of rotations where this effect occurs increases, as there exist more directions with smaller singular values, as highlighted by NSE.
There are several avenues for future work that build on the results we present here. First, there may be other ways to design RNNs that respect Dale's Law while learning well. Our results suggest that a guiding principle for investigations of this sort should be the consideration of the spectral properties of the weight matrices. And further, we note that this insight likely applies broadly to all El network architectures, such as convolutional, feedforward, and potentially spiking models, not just RNNs. Similarly, the models we present here could be made more biologically faithful, for example inhibitory to inhibitory connections could be added to DANNs, and cell-type specific time constants included in ColEI networks. Furthermore, the two El models have different ncurobiological interpretations, the populations of inhibitory neurons that they naturally model are different. ColEI RNNs are a more general model of inhibitory cells (eg. SOM+, CCK+ etc.), whereas DANNS specifically model fast PV+ interneurons. Therefore promising future line of work could be to hybridise the two the models together. As our central motivation was to understand why EI networks can fail to learn well, we didn't explore these directions in the experiments we present here. However, our results indicate that the network's s spectral properties should be carefully considered whenever such architectural choices made for modelling purposes.
Additionally, from a spectral properties of RNNs can help ameliorate problems of exploding/vanishing gradients (36]. and so, an interesting question for neuroscience research into biologically plausible gradient estimation [8] is whether real brains, which largely obey Dale's 's Law, have spectral properties that are well-suited to gradient descent or not. Likewise, there are interesting questions regarding the balance between excitation and inhibition in RNNs (a major topic in neuroscience) that flow from the observation that such balance can impact the spectral properties of networks [
generating perspective. appropriately constraining the
In summary, our work builds on tools from deep learning theory to provide guidance on how to construct RNNs that incorporate Dale's Law. We presented evidence that the reason for poor learning of ColEI networks and good learning of DANNs is, in large part, their respective spectral properties. Therefore, we hope our work serves as the basis and inspiration for an alternative initialization for ColEI networks that improves their learning. More generally, by providing explanation for EI network learning differences, we have opened up new directions of research for RNNs in computational neuroscience.
10

---

5 Acknowledgements
This work was supported by a NSERC (Discovery Grant: RGPIN-2020-05105; Discovery Accelerator Supplement: RGPAS-20200 2020000031;; Arthur B. McDonald Fellowship: 566355-2022) and CIFAR (Canada AT Chair; Learning in Machine and Brains Fellowship). This research was enabled in part by support provided by (Calcul Québec)
and the Digital Research Alliance of Canada (htpps//allianceaan.c ca/ea. The authors acknowledge the material support of NVIDIA in the form of computational resources. JC: IVADO Postdoctoral Fellowship and the Canada First Research Excellence Fund. AG: Vanier Canada Graduate scholarship. PL. wishes to convey his deepest gratitude to Ann Z.X. Huang for her support.
http::/wwwww.llullueeeee.co
References
[1] Hari Teja Kalidindi, Kevin P Cross, Timothy P Lillicrap, Mohsen Omrani, Egidio Falotico, Philip N Sabes, and Stephen H Scott. Rotational dynamics motor cortex are consistent with feedback controller. Elife, 10:e67256, 2021.
[2] Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome, Context-dependent computation by recurrent dynamics in prefrontal cortex. nature, 503(74744:77-84) 2013.
[3] Abigail A Russo, Ramin Khajeh, Sean R Bittner, Sean M Perkins, John P Cunningham, Lau- rence F Abbott, and Mark M Churchland. Neural trajectories in the supplementary motor area and motor cortex exhibit distinct geometries, compatible with different classes of computation. Neuron, 107(4))745-758, 2020.
[4] Omri Barak. Recurrent neural networks as versatile tools of neuroscience research. Current opinion in neurobiology, 16:1-6, 2017.
[5] Guangyu Robert Yang and Xiao-Jing Wang. Artificial neural networks for neuroscientisss: primer. Neuron, 107(6) 1070, 2020.
[6] Mahi Hardalupas. How neural neural net? Bio-inspired computational models and their impact on the multiple realization debate. PhD thesis, University of Pittsburgh, 2021.
[7] John Carew Eccles. From electrical to chemical transmission in the central nervous system: the closing address of the sir henry dale centennial symposium cambridge, 19 september 1975. Notes and records of the Royal Society of London, 30(2):219-234 1976.
[8] SeulAh Kim, Michael L. Wallace, Mahmoud EI-Rifai, Alexa R. Knudsen, and Bernardo Sabatini. Co-packaging of opposing neurotransmitters in individual synaptic vesicles in the central nervous system. Neuron, 111(8)):371-13844
2022.
[9] Francis Song, Guangyu Yang, and Xiao-Jing Wang. Training excitatory-inhibitryy recurrent neural networks for cognitive tasks: a simple and flexible framework. PLoS computational biology, 1004)2e 2016.
[10] Jonathan Cornford, Damjan Kalajdzievski, Marco Leite, Amélie Lamarquette, Dimitri Michael Kullmann, and Blake Aaron Richards. Learning to live with dale's s principle: Anns with separate excitatory and inhibitory units. In International Conference on Learning Representations, 2021.
[11] Guillaume Hennequin, Everton J Agnes, and Tim P Vogels. Inhibitory plasticity: balance, control, and codependence. Annual review of neuroscience, 40:557-579, 2017.
[12] Helen C Barron, Tim P Vogels, Timothy E Behrens, and Mani Ramaswami. Inhibitory engrams in perception and memory. Proceedings of the National Academy of Sciences, 114(2))6666- 6674, 2017.
[13] Tim Vogels, Henning Sprekeler, Friedemann Zenke, Claudia Clopath, and Wulfram Gerst- ner. Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks. Science, (56-0066)))16--117777 2011.
[14] Martin Schrimpf, Jonas Kubilius. Ha Hong. Najib J Majaj. Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Ron Roy, Kailyn Schmidt, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
[15] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. perfornance-optimizee hierarchical models predict neural responses higher visual cortex. Proceedings of the National Academy of Sciences, 1 11(23):8119886) 2014.
11

---

[16] Yuxiu Shao and Srdjan Ostojic. Relating local connectivity and global dynamics in recurrent networks. bioRxiv, pages 2022-08, 2022.
[17] Alessandro Ingrosso and LF Abbott, Training dynamically balanced excitatory-inhibioryy networks. PloS one, 14(8)=e022044, 2019.
[18] Thomas Miconi. Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks. Elife, 6:c20899, 2017.
[19] Sun Minni, Li Ji-An, Theodore Moskovitz, Grace Lindsay, Kenneth Miller, Mario Dipoppa, and Guangyu Robert Yang. Understanding the functional and structural differences across excitatory and inhibitory neurons. bioRxiv, 2019.
[20] Daniel J Amit, C Campbell, and KYM Wong. The interaction space of neural networks with sign-constrained synapses. Journal of Physics A: Mathematical and General, (2(21))468)) 1989.
[21] Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. A study of biologically plausible neural network: The role and interactions of brain-inspired mechanisms in continual lcarning, arXiv preprint arXiv:2304.06738 2023.
[22] Charles Bordenave and Djalil Chafai. Around the circular law. Probability Surveys, 9none), jan 2012.
[23] Robin Tremblay, Soohyun Lee, and Bernardo Rudy, Gabaergic interneurons in the neocortex: from cellular properties to circuits. Neuron, 91(2):60--292, 2016.
[24] Hua Hu, Jian Gan, and Peter Jonas. Fast-spiking, parvalbumin+ gabaergic interneurons: From cellular design to microcircuit function. Science, 345(6196): (1001199::120000 1255263, 2014.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[26) Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256. 2010.
[27] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1735:1735-7780, 1997.
[28] Quoc V. Le, Navdeep Jaitly, and Geoffrey E. Hinton. A simple way to initialize recurrent networks of rectified linear units, 2015.
[29] Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Comput. Linguist., 19(2):313-330, jun 1993.
[30] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[31] Najib J Majaj, Ha Hong, Ethan A Solomon, and James DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 35(39))13402-1344
2015.
[32] Tanya Strydom, Giulio V. Dalla Riva, and Timothée Poisot. Svd entropy reveals the high complexity of ecological networks. Frontiers in Ecology and Evolution, 9, 2021.
[33] Scott Wisdom, Thomas Powers, John R. Hershey Jonathan Le Roux, and Les Atlas. Full- capacity unitary recurrent neural networks, 2016.
[34] Kanaka Rajan and L Abbott. Eigenvalue spectra of random matrices for neural networks. Physical review letters, 18800, 12 2006.
[35] Isabelle D Harris, Hamish Meffin, Anthony N Burkitt, and Andre DH Peterson. Eigenvalue spec- tral properties of sparse random matrices obeying dale's law. arXiv preprint arXiv:2212.01559, 2022.
[36] Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, and Guillaume Lajoie. Non-normal recurrent neural network (nnrnn): learning long time dependencies while improving expressivity with transient dynamics. Advances in neural information processing systems, 32, 2019.
12

---

[37] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. Pmlr, 2013.
[38] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, pages 1-12, 2020.
[39] Paul Werbos. Generalization of backpropagation with application to recurrent gas market model. Neural networks, 1(4):339-355) 1988.
13


Reviews:
[{"review_id": "3bzuLNzGIj", "reviewer": "Reviewer_TD5G", "summary": "This paper presents a comparison between the performance of standard RNNs and two classes of models: the ColEI network and the DANN, which incorporate Dale's Law with the constraint that units in a circuit should be excitatory or inhibitory but not both. The DANN achieves similar performance to a non-signed RNN, but ColEI shows inferior performance. The paper demonstrates that the spectral properties of the recurrent weight matrix at initialization have a more significant impact on network performance than sign constraints, which may explain why some forms of EI network learn better than others.", "questions": "Considering the biological motivation of having dedicated excitatory and inhibitory units, it would be valuable to provide a deeper validation or interpretation of the results in the context of neurobiology. For example, could the finding about the different effects of changing the ratio of E/I units on ColEI networks and DANNs offer insights into the biological relevance of these two models to the neural circuit in the brain?", "limitations": "While the focus on RNNs is relevant to the paper's objectives, it would be helpful to briefly acknowledge potential implications for other types of networks.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper uses analytical approaches from machine learning to explore network models that adhere to biological principles, which may inspire the development of more biologically realistic models that also have good performance.\n- A novel contribution is the introduction of normalised SVD entropy as a measurement of spectrum pathology during the initialization stage which predicts the final performance of a network before training.\n- It also includes extensive experiments that test three RNN architecture on three different tasks with progressive difficulty levels.", "weaknesses": "This work builds upon existing research about EI networks with incremental advancements, offering interesting insights into how the spectral properties of the recurrent weight matrix at initialization may impact network performance. However, the practical application for designing EI networks that perform well requires further clarification. It is unclear to me whether DANNs can be effectively improved with this insight, and even with a more appropriate spectrum, ColEI falls short of standard RNN or DANN performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review. Please note that the reference numbers refer to the references list in the general comment.\n\n>**Weaknesses:** However, the practical application for designing EI networks that perform well requires further clarification. It is unclear to me whether DANNs can be effectively improved with this insight, and even with a more appropriate spectrum, ColEI falls short of standard RNN or DANN performance.\n\nThank you for this comment, we agree that adding a paragraph to the discussion that clarifies and expands on the practical applications of our results to future work would improve the paper.\n\nFirst, the novel insight that the spectral properties are the main issue for EI network learning vs standard networks can indeed be applied to DANNs. We can improve the DANN spectrum by carefully selecting Wee, Wei, and Wie to approach an orthogonal net-effect matrix W=Wee - Wei Wie with singular values~=1, which has been shown to be helpful for RNNs [10,11]. We have data showing improved performance on sequential MNIST from this strategy. We did not include this direction in the original draft due to the page limitation and our focus on explaining poor training in ColEI networks, but we can add it to the appendix if helpful. \n\nSecond, it is true that ColEI networks fall short of standard RNN and DANN performance, even with hyperparameter choices that make the spectrum more appropriate (i.e. larger sizes and ratios of E to I). However, the insight that it is the spectrum that underlies the poor ColEI performance is a foundational one. We hope our work serves as the basis and inspiration for an alternative initialization for ColEI networks that improves their learning. We will add a comment in the discussion highlighting this as a future research direction.\n\nFinally, another exciting future direction is to hybridize DANNs and ColEI networks, as DANNs will remove the far-outlying singular values of the ColEI spectrum and will have the effect of enabling sign changes in the effective recurrent weight matrix (which still have a minor contribution to poor ColEI performance). This hybridization would also be satisfying from a neuroscientific standpoint as the inhibitory neurons in ColEI networks can be thought of as modeling different inhibitory populations (eg. SOM+, CCK+ etc. cells) than the fast inhibitory populations modeled by DANNs (PV+ cells). Again, we omitted this discussion originally due to the page limit but believe this is a very promising direction and would be happy to add some discussion of it.\n\n> **Questions:** Considering the biological motivation of having dedicated excitatory and inhibitory units, it would be valuable to provide a deeper validation or interpretation of the results in the context of neurobiology. For example, could the finding about the different effects of changing the ratio of E/I units on ColEI networks and DANNs offer insights into the biological relevance of these two models to the neural circuit in the brain?\n\nWe agree, and will strengthen the neurobiological focus for the camera ready version of the paper. Related to our response above, we think that the two models have slightly different neurobiological interpretations as the populations of inhibitory neurons that they naturally model are different: ColEI RNNs are a more general model of I cells, whereas DANNs specifically model fast PV+ interneurons. Interestingly, and to the reviewer’s point, our finding that ColEI networks learn better with more inhibitory cells (a smaller E/I ratio), and that DANNs learn fine even with very large E/I ratios, fits with the small number of PV+ cells compared to the total pool of inhibitory cells (~20%) [12] in real neural circuits. Furthermore, recent work has found that the human and macaque cortex contains approximately three times the amount of inhibitory interneurons than mice [13], which is consistent with our finding that a more balanced ratio of E to I results in better spectrum and learning for ColEI networks. \n\n>**Limitations:**\nWhile the focus on RNNs is relevant to the paper's objectives, it would be helpful to briefly acknowledge potential implications for other types of networks.\n\nThank you for pointing this out. We will edit the text highlighting how we expect these insights to apply to all EI networks: e.g. feedforward MLPs, convolutional networks and potentially spiking neural networks. \n"}, {"review_id": "wO6fahGI6G", "reviewer": "Reviewer_YJwG", "summary": "The paper investigated the problem of why columnEI networks have impaired learning, and they experimentally found that instead of sign constraint, the spectral property of weight at initialization contributed the most; they further experimentally showed that E/I ratio and network size change the spectral properties thus lead to different learning performances. Additionally, they showed DANN in the RNN form show similar performance as normal RNN in 3 different tasks.", "questions": "My main concern is on the spectral discussion \n- Does it extend to other tasks with different data distribution?\n- Why clusterness of singular values matter for learning? The intuition currently given in the paper seems to mainly pertain to large singular values; yet it is shown large singular value ColEI can learn well. Then what is the intuition behind SVD entropy tracking performance? This lack of intuition is what prompted me to ask the first question as it is not immediately clear to me how generalizable the current results are. And thus a current score of 5.", "limitations": "See questions", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Originality** This is a follow-up work on applying DANN to RNN. I appreciate the detailed discussion on why it is hard to train E-I separated RNNs. \n\n**Quality**The effectiveness of DRNN is tested in three different tasks which is great! And all hyperparameter selections are listed in the appe ndix. The discussion on initialization spectral property contain proper ablation experiments and extended discussion on E-I ratio and network size.   \n\n**Clarity** The paper is clearly written, with details in the appendix. Some part of the result explanation can be confusing (see second question)\n\n**Significance** I believe the paper will be of interest to the computational neuroscience community.", "weaknesses": "- The discussion of initialization spectral property is only done for sequential MNIST. Does the observation on SVD entropy hold across data distributions? I'm a bit concerned on how generalizable the results are (details see question). \n- For the sign constrained training, how are the gradients rectified? Set to zero? If set to zero, it may leads to silent unit problem. Did the authors exclude the possibility that sign constrained RNN learn worse due to increased number of silent units? \n- Typo: line 110 citation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful review. Please see the general comment for the references list.\n\n>**Weaknesses** The discussion of initialization spectral property is only done for sequential MNIST. Does the observation on SVD entropy hold across data distributions? I'm a bit concerned on how generalizable the results are (details see question).\n\nThank you for raising this. We agree it is very important to verify that SVD entropy is predictive of performance across different data distributions. We have therefore run more experiments (Penn Treebank and the new naturalistic image task) to verify the generalizability of our results. Please see the global response and the Rebuttal Figures. Thank you for this comment, it has helped us improve the paper. \n\n>For the sign constrained training, how are the gradients rectified? Set to zero? If set to zero, it may leads to silent unit problem. Did the authors exclude the possibility that sign constrained RNN learn worse due to increased number of silent units?\n\nWe would like to clarify that we do not rectify the gradients. Sign constraints are realized using projected gradient descent, i.e. after a parameter update via gradient descent as normal, parameters were then clamped to be all positive or all negative depending on whether the parameters were E or I. We will make this clear in the methods section.\n \nIn response to the reviewer’s second comment we investigated the distribution of weights and found that the sparsity of the weight matrices for ColEI networks and regular RNNs are very similar, making silent units from sign constraints unlikely to be an issue. This observation also aligns with our observation that sign constraints have a minimal effect on learning performance. \n\n> **Questions:** My main concern is on the spectral discussion.  Does it extend to other tasks with different data distribution?\n\nPlease see our response to this question above. \n\n> Why clusterness of singular values matter for learning? The intuition currently given in the paper seems to mainly pertain to large singular values; yet it is shown large singular value ColEI can learn well. Then what is the intuition behind SVD entropy tracking performance? This lack of intuition is what prompted me to ask the first question as it is not immediately clear to me how generalizable the current results are. And thus a current score of 5.\n\nThank you for communicating this point. We originally shared the reviewer’s puzzlement, as larger ColEI networks have even larger maximum singular values (sigma_max) and yet train better.  Indeed this observation led us to propose normalized SVD entropy as a candidate metric for tracking performance and spectrum pathology. As explained in the main text (lines 279 - 287), the intuition behind SVD entropy tracking performance is that the bulk of the singular values also matters [10,11]. Put another way, for the large networks, although there are a small number of directions with a large singular value, the bulk of them are smaller, which helps ameliorate the problem of exploding gradients, by rotating the activity away from the direction corresponding to the large singular values. As such, to understand learning performance we require a metric that captures the overall effect of the entire singular value spectrum, rather than just a few singular values, which we do using the SVD entropy metric.\n\nThanks to the reviewer’s comment, we realize that the current manuscript may benefit from describing this insight earlier in the results, rather than in the discussion (lines 279 - 287) after the results are presented. We will also note that the additional experiments with the Penn Treebank and the naturalistic images further demonstrate the validity of using SVD entropy to track ColEI performance.\n"}, {"review_id": "lksl84aqqq", "reviewer": "Reviewer_gdj1", "summary": "The authors apply Dale's law (that neurons provide exclusively excitatory or inhibitory outputs) to RNNs with two different architectures: ColEIs which are the “straightforward” way of applying sign constraints per neuron, and recurrent DANNs based on an architecture with two layers per recurrent step. They compare singular value spectrums, network sizes and Excitation/Inhibition ratios in all three architectures and analyze how they affect performance. They conclude that the reason why DANNs perform as well as RNNs, while ColEIs perform worse is largely due to differences in the SVD distribution (as measured by Normalised SVD Entropy).", "questions": "Are the inhibitory units in a DANN strictly linear? This seems necessary if the DANN weight matrix is actually an exact reparameterization of the RNN weight matrix, as advertised (L95). Either way, this is unclear. And this has important implications for the story: if the DANN is an exact reparameterization, then isn’t it trivial that RNN and DANN spectra are identical?\nWhy the variance of line 115? Why ρ=1.5 and ρ ~= 1/√3 in Section 3.1?\nCan you explain the 3-layer architectures used in the Sequential MNIST figures? Do the eigenvalues and singular value visualizations of Figures 5,6 include values from weights from all layers? If so, are there any differences between the eigen/singular values between layers?\nLine 200: Is initialization the only difference between a ColEI network without sign constraints and a standard RNNs?\n", "limitations": "Adequately addressed", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Dale’s law is an important and ubiquitous property of biological neural networks, but its consequences have not been thoroughly explored. It is interesting that a simple architecture can enforce Dale’s law while remaining trainable, and while keeping many properties of unconstrained RNNs. This architecture features feedforward inhibition, a biologically plausible motif. Understanding consequences of such biological structure is a key topic of Neuro-AI.\n\nThe work offers some nice empirical results, both in performance and in spectral properties.\n\nThe writing and figures are clear, and good efforts are made to provide explanations and interpretations.\n", "weaknesses": "While being advertised as a biological neural network, the architectural constraints that make the recurrent DANN trainable may be the very same ones that make it biologically unrealistic. For instance, there is no direct recurrent inhibition (I to I), and there cannot be direct reciprocal connectivity between E and I units.\n\nThere is not much mathematical analysis of the results, so the insights are somewhat limited. Clearly there are differences in the spectral distributions, but why these appear and most importantly how these are “directly responsible” for decreased performance is not deeply discussed. There is some discussion of the low-rank nilpotent component of ColEI matrices but there is more related literature to connect to, most notably from the Ostoijc group (no, this reviewer is not from that group). For example, Mastrogiuseppe and Ostojic 2018, Schuessler et al 2022, in addition to the Shao and Ostoijc 2022 paper the authors do cite (now evidently published in PLoS comp bio).\n\nMany of the analyses are done only on the sequential MNIST task, which is not particularly natural. Claims are being made on the basis of small differences in errors, and some qualitative analysis. The small differences suggest that the networks are not being pushed very far to demonstrate their inductive biases.\n\nWhere did the ρ=1.5 and ρ ~= 1/sqrt(3) come from in Section 3.1?\n\nHow were the sign constraints implemented?\n\nMinor: watch out for punctuation errors and typos.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful review. Please see the general comment for the references list.\n\n> **Strengths**: Dale’s law is an important and ubiquitous property...\n\nMany thanks!\n\n**Weaknesses:**\n\n>While being advertised…\n\nThank you for highlighting this aspect of DANNs, we will add to the text, noting that the lack of I-I connections in DANNs is not biologically realistic and that this could be changed in the future to make them more realistic, an important point of clarification. However, we want to highlight that the central motivation of this paper is to understand why EI networks can fail to train well, and our data supports the conclusion that the spectral properties of the weights is the main reason. Importantly, this central conclusion is not dependent on the specific DANN formulation being completely biologically faithful. The reviewer is correct that adding I-to-I connectivity would increase the biological faithfulness of recurrent DANNs. But the spectral properties would be the same, because fast-recurrent inhibition provides a “winner-take-all” mechanism, which would simply sparsify the activity. Therefore for simplicity we did not explore this aspect in our work.\n But, the reviewer’s point holds, and we will be sure to highlight this biological infidelity of DANNs in our camera ready manuscript. We think it is an exciting direction of future research.\n\nOne thing: we are unsure what the reviewer means by “there cannot be direct reciprocal connectivity between E and I units”, because the absence of I-I connections should not affect reciprocal E-I connectivity.\n\n>There is not much…\n\nWe agree with the reviewer that our work is largely an empirical analysis rather than a mathematical analysis of the origins of poor learning in ColEI networks. We provide a discussion of the low-rank nilpotent matrices mainly for the purpose of intuitions. However, we would strongly contest that the empirical nature of our work means the insights are limited. As far as we know, the understandings we present here through our experiments are extremely novel and this is the first work that directly associates spectrum pathology with poor ColEI learning performance. Overall, this work is a major shift in the field’s understanding of why recurrent networks of E and I units perform poorly. \n\nWe also wish to distinguish our study from prior works like Mastrogiuseppe and Ostojic 2018, and Schuessler et al 2022. First, our work focused on RNNs respecting Dale’s law whereas the mentioned works did not. Also, the mentioned works mainly focused on exploring the relationship between low-rank connectivity, (low-rank) network dynamics and how network computations are related to them, rather than how to understand the learning performance of E-I RNNs. In contrast, in this work we aim to directly link singular value spectrum with the learning performance of E-I RNNs when we train them with gradient descent. Past research also mainly looks at the spectral properties of random networks, with little focus on learning performance [7,8] Therefore, we hope this work can lay the foundation for future research on optimizing network performance without breaking biological constraints. But we agree the mentioned works are relevant and we are happy to cite them. \n\n>Many of the analyses …\n\nThank you for highlighting this. We agree it is very important to verify that our results and analyses are consistent across data distributions, especially more naturalistic ones given this work’s connections to neurobiology. As a result of your comment we have included a new naturalistic image task and also ran the analyses presented in Figures 5 and 6 on the Penn Treebank. Please see the general response for more details. We believe these experiments substantially improve the paper - thank you again for suggesting this.  \n\n>Where did the ρ=1.5 and ρ ~= 1/sqrt(3) come from in Section 3.1?\n\nRho = 1.5 is from [2]. 1/root(3) comes from  Pytorch’s default initialization for RNNs. \nHow were the sign constraints implemented?\nSign constraints are implemented via projected gradient descent. After a parameter update outgoing parameters from E or I neurons are clamped to either be all positive or all negative respectively. \n\n**Questions:**\n>Are the inhibitory units…\n\nThank you for raising this. As noted in [1], the nonlinearity used for inhibitory units in DANN is ReLU. The subtlety here is that the post-activation from the previous layer of DANN is non-negative due to ReLU,  rendering the pre-activation of inhibitory units of DANN non-negative as well (because Wei is a non-negative matrix). Therefore, ReLU and linear activation functions are equivalent for the inhibitory units in DANN. The expression in line 95 is a simplification of the subtlety explained above. In addition, we wish to emphasize that the equivalence of RNN and DANN spectra is nontrivial because both the low-rank matrix Wei*Wie and the fully positive matrix Wee in DANNs can lead to outliers in the spectrum at initialization [9] and must be carefully chosen in order to cancel out each other. Rebuttal Fig. 5 shows the spectrum of a poorly initialized DANN weight matrix although all elements of the net-effect W=Wee - Wei*Wie matrix have mean zero.\n\n>Why the variance of line 115? Why ρ=1.5 and ρ ~= 1/√3 in Section 3.1?\n\nPlease see above and also footnote in the main text (page 5) for the correspondence between variance and rho.\n\n>Can you explain the 3-layer architectures…\n\nBy three layers we mean that there are 3 RNN modules stacked on top of each other, \n\n>Do the eigenvalues …between layers? \n\nAll layers have the same number of recurrent units and are initialized with the same random initialization scheme, therefore they are the same. \n\n>Line 200: Is initialization …RNNs?\n\nYes, correct! \n"}, {"review_id": "A2iKGsxHed", "reviewer": "Reviewer_SnGV", "summary": "The author present a simulation study that investigates gradient-based optimisation of RNNs that obey Dale’s Law, i.e. networks with neurons that are strictly excitatory or inhibitory at initialisation and during training. In particular, the authors disentangle the effect of enforcing Dale’s Law during training (synaptic weights can not flip sign) and the initial spectral properties that result from initialisation with separate excitatory and inhibitory populations of neurons.", "questions": "1) As written in appendix 5.3, the learning rate of DANNs are scaled independently for E and I weights to balance the impact of E and I and their updates. As ColEI networks seem not to be trained with a similar scaling scheme: How does the IE weight scaling influence simulation results and conclusions; in particular with respect to changes in EI ratio and network size?\n\n2) In line 109 the authors write “[…] the activation variance did not scale with depth […]”. Did the authors control for mean and variance shift across layers and increasing depth? It would help to better understand the three different training regimes if there would be a plot that is visualising the mean and variance shift at initialisation across layers and depth for random data for RNNs, DANNs and ColEIs for varying EI ratios.\n\n3) Related to 2 – In order to better understand the learning dynamics and in order to detect anomalies, a plot that visualises the mean and variance of synaptic weights and biases and how they evolve during training would be helpful. Do you expect the mean and variance of the weight matrices and biases within the E and I population to evolve similarly for RNNs, DANNs and ColEIs?\n\n4) Related to 2 – Judging from the learning trajectories (e.g. in Figure 2B/C), to me, it looks like that ColEI networks have a significantly higher initial error (at t=0) – however, since the y-axis is cut – I might be wrong? If the initial errors are different, why are they different?\n\n5) In line 116 the authors write “[…] each row of W^IE was initialised as the mean row of W^EE […]”. What does that mean? Do all entries in the row have the same value?\n\n6) In Figure 3 it looks like ColEI networks have a large variance in performance. Do you have an explanation or intuition why some of the initialisations fail and others succeed?\n\n7) From equation 3, I conclude that DANNs have 3x more free parameters in the recurrent part of the network than ColEIs. This makes comparison tricky, especially when it comes to network size. How does performance look like as a function of free parameters (instead of number of neurons in the hidden layer)?\n\n8) I would assume that spectrum transplants alter the EI balance. Did you correct for that effect?", "limitations": "The authors have adequately addressed limitations of their work", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper investigates an important open question: What are the origins of the performance gap between RNNs that do / do not obey Dale’s Law. Through a large set of simulation studies and numerical analysis, the authors show in a convincing way that the performance gap does not solely stem from enforcing Dale’s Law during training (and thus restricting the solution space by enforcing signs of weights) but also significantly depends on how networks are initialised and parametrised. Initialising RNNs with columns of positive / negative weights to create populations of excitatory / inhibitory neurons (“ColEI networks”) causes a skewed and multi-modal singular value spectrum – leading to the well known effect of exploding neural activations / gradients that hamper training and performance. The authors further provide empirical evidence that the Normalised SVD Entropy thus provides an easy to compute predictor of network performance before the onset of training. Further, the authors provide analytical intuition for their simulation results in the discussion. The paper is well structured, written and accessible. The simulation details are explained in great detail; which is very helpful for understanding the research question, results and potential limitations. The figures are generally clear and accessible.", "weaknesses": "- My main concern is, that as written in appendix 5.3, the learning rate of DANNs is scaled independently for E and I weights to balance the impact and updates of E and I populations. As ColEI networks seem not to be trained with a similar scaling scheme, this has the potential to explain away or confound results and observations. I am happy to revise my score if this is adequately addressed.\n- The authors base their analysis on ColEI networks that are initialised following [5], however, the authors do not provide insight (through simulations or analytically) if there generally can or cannot exist a weight initialisation scheme for ColEI networks which does not lead to a skewed and multimodal spectrum of singular values. As thus, the results seem to be limited to a specific initialisation scheme and not to ColEI networks in general.\n- The authors apply gradient clipping during network training. This may significantly reduce the effects of skewed, multimodal singular value spectra and outliers in the singular value spectrum. As the authors try to disentangle the effects of enforcing Dale’s Law and the spectral properties of initial recurrent weight matrices, gradient clipping may confound results and conclusions.\n- If I understand correctly, recurrent weight matrices in DANNs are parametrised by a linear combination of three weight matrices. As such, it seems like DANNs have three times more free parameters than ColEI networks. If true, this would pose the question if comparisons between DANNs and ColEIs as presented in figure 6 are fair.\n\nMinor:\n- In figure 2D it is difficult to see what is going on – maybe log-axis and a higher alpha value for the scatter point’s colour would help?\n- In figure 2B it looks like the networks haven’t been trained until convergence\n- Missing “in” in line 94\n- Broken reference in line 110", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to the reviewer for their thoughtful review. We are pleased they share our perspective that this paper investigates an important, unanswered question: the origins of the performance gap between EI and standard RNNs. Furthermore, it is our desire that this paper contributes to a broad spectrum of computational neuroscience research, so we are pleased that you find it well written, accessible and clear. \n\nPlease see the general comment for the references list.\n\n**Weaknesses:**\n> My main concern is... …I am happy to revise my score if this is adequately addressed.\n\nIn short, the scalings used for DANN models are specific to its architecture, not the parameters being E or I, and they do not apply to the ColEI network. In [1], the learning rates for DANN I parameters are scaled based on an analysis of how much an update to each parameter type (Wee, Wie, Wie) will change a layer’s output distribution, as quantified by KL divergence of the layer before and after the update. \n\nIn a ColEI network, the E and I parameters have the same impact on the network’s output function (ColEI E and I parameters are like DANN Wee parameters), hence we did not originally run experiments with such scalings. However we agree this is valid scientific concern, and have ran additional ColEI experiments on sequential MNIST with outgoing (I->I, I->E) and incoming (E->I) I parameter-learning scaled as in DANNs (Rebuttal Fig 4).  We find that this scaling impairs learning, in line with our theoretical intuitions. We will add the figure to the appendix. We hope this addresses this important concern. \n\n>The authors base their analysis on ColEI...\n\nThanks for this comment. To our knowledge, there is no better method for initializing ColEI networks [2, 3]. Nonetheless, we share the reviewer’s suspicion that an initialization exists for ColEI networks that results in better learning. As such, one contribution of our work is providing a potential foundation for this (currently) unknown method of initialization. Our results imply that if one can initialize ColEI networks with better spectral properties, then they should train well. We will add to the discussion highlighting this point. \n\n>The authors apply gradient clipping… \n\nThank you for raising this. We missed an opportunity to highlight just how problematic the standard ColEI spectrum is! Gradient clipping is a common technique for training RNNs [4] and we applied it under the assumption that it would benefit ColEI networks. Indeed, we find that without gradient clipping ColEI networks perform very poorly, please see (Rebuttal Figure 3). \n\n>If I understand…\n\nDANNs do have more parameters than ColEI networks, but not as many as 3 times. Wei and Wie^T are both of dimension # hidden* # inhibitory. So for the case of 10% inhibitory units (Fig 6) there are only 20% more parameters. This is a relatively small difference, but if the reviewer wishes we can run and include experiments in which the total number of free parameters are equal. However in this case the number of hidden units and the dimensions of the hidden to hidden connectivity matrix would no longer be the same between DANNs and ColEI networks, and therefore these simulations would be unfair in a different way. \n\nMinor:\n>In figure 2D …\n\nWe’ll add Fig 2d with zoomed in axes to the appendix (Rebuttal Fig 6A). \n\n>In figure 2B …\n\nPlease see Rebuttal Fig 6B, we have increased the number of parameter updates from 5 to 18 M. \n\n**Questions**\nThank you for this list, we will update the camera-ready version to address all of these points.\n\n>As written in appendix 5.3…\n\nPlease see our comment in the weaknesses section. If desired we can include experiments with different ratios and network size, but we predict the ColEI performance will track the trends of the current results but be worse. \n\n>In line 109 the authors…\n\nHere, we just meant that we adopt the same general initialization strategy as found in [5,6]. \n\nRegarding differences between hidden layer variance and mean over timesteps. We expect that there will indeed be differences, but that these will be another fingerprint of the spectral properties. \n\n>Related to 2 – In order…\n\nWe expect there will be differences but expect it will be unclear how they relate to spectral properties. We are happy to discuss this and the previous point further if the reviewer wishes. \n\n>Related to 2 – Judging…\n\nAll untrained networks have the same initial error, but the first point in the plot is the error after 100 updates. i.e. the error is not logged at the beginning. \n\n>In line 116 the authors…\n\nWe apologize for an error in our text. We will change the text to “each row of W^EI was initialized as the mean row of W^EE”.  By this we mean that W^EI_ji = 1/n \\sum_k W^EE_ki for all j, and I unit equivalence is then broken with W_ie weights. Therefore all entries in the columns of W^IE have the same values. \n\n>In Figure 3…\n\nThank you for highlighting this aspect of ColEI training, we did reference it in a previous draft of the text and will reinsert this observation in the updated text. Our intuition is that variance in ColEI networks is due to them having fewer activity modes with appropriate singular values for learning computations over time (the motivation behind SVD entropy) and therefore small differences at initialisation can strongly impact learning.We also find that without gradient clipping ColEI networks are highly unstable, likely due to the distribution of singular values at initialisation [4], and this can also lead to variable performance given small differences at initialization. \n\n\n>From...\nPlease see our response to this issue above. \n\n>I would…\nThe spectrum transplant experiments convert EI RNNs into networks without separate populations of E and I units. We didn’t correct for this because to do so would require changing the spectral properties of the donor networks, and we would not be directly testing the impact of the different network spectral properties. \n"}]

Author rebuttal:
We thank all reviewers for their time and effort in providing a set of positive, high-quality helpful reviews.

We would like to take this opportunity to reiterate the main contribution of our work. We investigate the following question: why do recurrent neural networks (RNNs) that have separate excitatory (E) and inhibitory (I) neurons often fail to train as well as regular RNNs? We present evidence that strongly supports the theory that, instead of sign constraints, the traditional column E-I network design is handicapped because the synaptic weight spectrum is multimodal and disperse, unlike standard RNNs.  We also show that there are E-I networks based on fast inhibition (Dale’s ANNs or DANNs) that have spectral properties akin to standard RNNs, and these networks train well. This work substantially shifts our understanding of the origins of poor learning in column EI networks, and suggests that contrary to popular assumptions, the imposition of sign constraints on a neural network is not the main impediment to good training. 

Here we communicate the more general and most substantial improvements we have made as a result of reviewer feedback:

* We have added a new naturalistic image categorization task and verified that all our results and analyses apply in this new setting (see “Natural object recognition task description” below and Rebuttal Fig 1). Consistent with our previous findings, spectrum transplant experiments decrease/increase error by ~40% whereas adding/removing sign constraints only account for ~0.3 and 8% error differences respectively (Rebuttal Fig 1C & Table 1). 
* Similarly, we have extended the sequential MNIST results presented in Fig 5, 6 to the Penn Treebank dataset which is both more challenging and more naturalistic (as it is a natural language processing dataset). Again, results here are similar to sequential MNIST; we find that normalized SVD entropy is anti-correlated with perplexity.
* We have run additional experiments without gradient clipping, which showcase more obviously the problematic spectral properties of ColEI networks. 

We believe that these additions attend to some of the most pressing questions raised by the reviewers and strengthen the conclusions of the paper. We also hope each of the individual, more directed responses adequately address the additional reviewer comments, and we look forward to the discussion in the next phase. Due to the new character limit per response we have had to prioritize brevity for some of the discussion points, so please do not hesitate to request further clarification. 

## Natural object recognition task description: 

Following Majaj & Hong, et al., 2015, we designed the following naturalistic object recognition task for RNN: at the first time step, the input to the RNN is a flattened achromatic image of one of 64 natural objects from 8 categories (e.g., Fruits, Faces) with a random natural scene background. Then the network receives no inputs and must retain information about the image over 5 timesteps to output the classification result at the last time step. The task presents two main challenges: retaining the information over time without additional inputs and correctly mapping different natural objects, like raspberry and watermelon, to the same category, such as Fruits. Please see
Majaj & Hong, 2015 for more details. https://www.jneurosci.org/content/jneuro/35/39/13402.full.pdf

## References:

[1] Cornford et al 2021
[2] Song et al, 2016
[3] Yang & Wang, 2021
[4] Pascanu et al., 2012
[5] He et al., 2015
[6] Glorot & Bengio, 2010
[7] Rajan & Abbott, 2006 
[8] Harris et al., 2023
[9] Benaych-Georges & Nadakuditi 2012 
[10] Arjovsky et al., 2016
[11] Le et al., 2015
[12] Bezaire & Soltesz, 2016
[13] Loomba et al., 2022


Meta-review: The study investigates why training recurrent neural networks (RNNs) with excitatory (E) and inhibitory (I) neurons is harder compared to regular RNNs. Authors demonstrate that the problem is not sign constraints, but rather the multimodal and disperse synaptic weight spectrum in the traditional E-I network design. Reviewers and the AC found this result to be important. Most criticism was addressed during the rebuttal period and additional experiments strengthened the paper.
Decision: Accept (poster)