Title: Learning better with Dale’s Law: A Spectral Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: Dale's Law

RNNs

brain-inspired neural networks

DANNs

computational neuroscience

spectral properties

inhibition
Abstract: Most recurrent neural networks (RNNs) do not include a fundamental constraint of real neural circuits: Dale's Law, which implies that neurons must be excitatory (E) or inhibitory (I). Dale's Law is generally absent from RNNs because simply partitioning a standard network's units into E and I populations impairs learning. However, here we extend a recent feedforward bio-inspired EI network architecture, named Dale's ANNs, to recurrent networks, and demonstrate that good performance is possible while respecting Dale's Law. This begs the question: What makes some forms of EI network learn poorly and others learn well? And, why does the simple approach of incorporating Dale's Law impair learning?  Historically the answer was thought to be the sign constraints on EI network parameters, and this was a motivation behind Dale's ANNs. However, here we show the spectral properties of the recurrent weight matrix at initialisation are more impactful on network performance than sign constraints. We find that simple EI partitioning results in a singular value distribution that is multimodal and dispersed, whereas standard RNNs have an unimodal, more clustered singular value distribution, as do recurrent Dale's ANNs. We also show that the spectral properties and performance of partitioned EI networks are worse for small networks with fewer I units, and we present normalised SVD entropy as a measure of spectrum pathology that correlates with performance. Overall, this work sheds light on a long-standing mystery in neuroscience-inspired AI and computational neuroscience, paving the way for greater alignment between neural networks and biology.

Learning better with Dale's Law: A Spectral Perspective

Abstract

Most recurrent neural networks (RNNs) do not include a fundamental constraint of real neural circuits: Dale's Law, which implies that neurons must be excitatory (E) or inhibitory (I). Dale's Law is generally absent from RNNs because simply partitioning a standard network's units into E and I populations impairs learning. However, here we extend a recent feedforward bio-inspired EI network architecture, named Dale's ANNs, to recurrent networks, and demonstrate that good performance is possible while respecting Dale's Law. This begs the question: What makes some forms of EI network learn poorly and others learn well? And, why does the simple approach of incorporating Dale's Law impair learning? Historically the answer was thought to be the sign constraints on EI network parameters, and this was a motivation behind Dale's ANNs. However, here we show the spectral properties of the recurrent weight matrix at initialisation are more impactful on network performance than sign constraints. We find that simple EI partitioning results in a singular value distribution that is multimodal and dispersed, whereas standard RNNs have an unimodal, more clustered singular value distribution, as do recurrent Dale's ANNs. We also show that the spectral properties and performance of partitioned EI networks are worse for small networks with fewer I units, and we present normalised SVD entropy as a measure of spectrum pathology that correlates with performance. Overall, this work sheds light on a long-standing mystery in neuroscience-inspired AI and computational neuroscience, paving the way for greater alignment between neural networks and biology.

1 Introduction

Recurrent neural networks (RNNs) are a major tool in computational neuroscience research, with numerous papers that use RNNs to model the brain published every year [1, 2, 3, 4, 5]. However, when using RNNs to model real neural networks there is always the question of which biological constraints to incorporate into the model. Some biological constraints are considered law-like [6], and are thus thought to be particularly important. One example is Dale's Law, which says that neurons

release the same set of neurotransmitters at all of their axon terminals [7]. Practically, for artificial neural networks, Dale's Law implies that synaptic output weights should be all positive (excitatory, E) or all negative (inhibitory, I) for any individual neuron. 1 And, though it is common to use models that respect Dale's Law in computational neuroscience [9, 5], it is not universal practice, and it is rare outside of neuroscience. One reason for this is that incorporating Dale's Law into neural networks tends to harm their learning performance [10], which is widely known by those who have trained such networks, but rarely explicitly addressed in the scientific literature. This limits our ability to model the impressive learning of biological neural networks with RNNs that respect Dale's Law. It also impairs studying the unique computations played by inhibitory cells using RNN models [11, 12, 13], and restricts comparisons between RNN models and real neural data [14, 15].

Typically, Dale's Law is incorporated into RNNs using the most obvious solution, by constraining entire columns of the synaptic weight matrices to be of the same sign, thereby partitioning the hidden units into E and I populations (here we refer to these networks as 'Column Excitation-Inhibition' or 'ColEI', see Figure 1 for an illustration) [9, 5, 16, 17, 18, 19]. While simple and intuitive, as mentioned, this approach impairs learning performance [10]. One hypothesis for why this is, is that EI sign constraints impair network performance by limiting the network's solution space [20, 10]. Indeed, a recent feedforward EI ANN architecture (Dale's ANNs, or DANNs), inspired by fast inhibitory circuits in the brain, and which was designed to overcome sign constraints, was found to learn just as well as standard ANNs [10, 21]. In this work, we show that recurrent versions of DANNs can also learn as well as standard RNNs. But, we also want to determine whether the sign constraints are really the primary cause of impaired learning in ColEI networks. Are there other reasons that ColEI networks learn worse than standard RNNs and DANNs?

Here we show that the spectral properties of EI networks that are most important for their performance. We find that the singular value spectrum of ColEI networks as in [9] is multimodal, dispersed, and includes a notable mode of large singular values. In contrast, DANNs and standard RNNs exhibit unimodal spectra as expected from random matrix theory [22]. Next, we transplant spectra between networks and find the spectrum is more impactful than sign constraints on performance - though both contribute. Furthermore, we find that small ColEI networks with biologically realistic E to I units ratio learn less effectively, and this is reflected in their spectral properties. In contrast, the spectrum and learning of DANNs is robust to both the ratio of E and I units, and network size: even small networks with only 1 inhibitory neuron have good spectral properties and learn well. Finally, we show that normalised SVD entropy, a measure of unimodal singular value clustering, is predictive of EI RNN learning performance.

Altogether, our results provide evidence that the reason for the impaired performance of traditional ColEI RNN networks is predominantly their spectral properties, and less so their sign constraints. Therefore, the improved performance of DANNs should be understood as primarily resulting from them having spectral properties that are similar to standard RNNs, in addition to their being robust to sign constraints. This is a critical insight for designing EI networks that are architecturally different from DANNs, which may be desirable when modelling specific subtypes of interneurons [23]. Finally, more broadly we present an answer to the long-standing mystery of poor performance of RNNs that respect Dale's Law, thereby paving the way for their future development and use in neuroscience-inspired AI.

2 Methods

2.1 Model Definitions

In this section we first define 'standard' RNNs, then the two different kinds of RNNs that obey Dale's Law, namely ColEI and DANNs. To obey Dale's Law, ColEI and DANN networks were first initialised in line with Dale's law (see initialisation details below), and projected gradient descent was used to maintain parameter signs during training. As such, after a candidate gradient descent update, parameters that had changed sign were set to 0, corresponding to a euclidean projection onto to the allowed solution space.

The reader can find the code for all our experiments available here.

Standard Recurrent Neural Networks (RNN)

For a multilayer network, we define the hidden state activations of a 'standard' RNN layer /lscript at time t , h /lscript t , via:

where σ ( · ) = ReLU ( · ) is the activation function, h /lscript -1 t is the activation vector for the lower layer, h /lscript t -1 is the hidden state activations at the previous time-step, W /lscript in and W /lscript rec are the input and recurrent synaptic weight matrices, and b /lscript is the bias. Note h /lscript t -1 is equivalent to the input, x t , for the first layer, i.e. h 0 t = x t . The output, y t , is a function of the final layer's ( L ) hidden state:

where W out is the output weight matrix, b out is the output bias, and g ( · ) is an optional output non-linearity. We set g ( · ) depending on the task: softmax function for classification and language modelling tasks, and identity for the adding problem.

Column Excitation-Inhibition (ColEI)

The ColEI model discussed in this work has the same formulation as a standard RNN (equations 1 and 2) except that the weight matrices are sign constrained. Specifically: (1) the input weights of the first recurrent layer W (1) in are all positive; (2) all other weight matrices are constrained to have columns whose entries are the same sign, either all positive or all negative. In combination with ReLU activations, this constraint makes positive columns correspond to the output weights of presynaptic excitatory neurons, and negative columns inhibitory neurons. An illustration is provided in Figure 1 A.

A

B

layer L+1

layer L+1

layer L+1

h,

h

h,

h,

h

h.

h.

h

h.

e e e

n-

n-

h

ne

n

ne

h

h.

n

h;

h!

ni

layer L-1

layer L-1

layer L-1

layer L-1

t=0

t = 1

t=0

t = 1

Dale's Artificial Neural Networks (DANN)

DANNs are an EI architecture first introduced in [10] for feedforward networks that were inspired by fast feedforward inhibition the brain [24]. Here we extend DANNs to the recurrent setting. A DANNlayer reparameterises the weight matrices of standard ANNs using three separate non-negative matrices. Specifically, for a given weight matrix W , we have:

where W EE , W IE , and W EI denote E-to-E, E-to-I, and I-to-E projections respectively (see Figure 1 B for an illustration). Note that such a parameterisation allows the sign of each element of W to be unconstrained, although the signs of individual weight matrices, i.e W EE , W IE , and W EI , are constrained by clipping illegal weights to 0 after each update.

2.2 Initialisation details

Standard RNNs: The weights of standard RNNs were sampled from a uniform distribution U [ -1 √ n , 1 √ n ] , where n is the number of hidden units (i.e. the PyTorch default initialisation), unless otherwise stated. Biases were initialised as 0.

ColEI: The parameters of ColEI networks were initialised following [10]. The weights in each column were sampled from either the exponential distribution corresponding to inhibitory (negative) or excitatory (positive) weights. The distribution means were set (i) to ensure that the sum of excitatory and inhibitory inputs to each neuron was balanced in expectation (i.e. ∑ j ∈ exc | µ j | = ∑ j ∈ inh | µ j | for inputs indexed by j ) and, (ii) following [25, 26], so that activation variance did not scale with depth (see Appendix 6.6 for details). Next recurrent weights were adjusted to constrain their spectral radius, in line with [9], by setting W /lscript rec ← ( ρ / ρ 0 ) W /lscript rec , where ρ 0 is the spectral radius of W /lscript rec before the adjustment, and ρ = 1 . 5 unless stated. Biases were initialised as 0, except for the first layer, where they were set to centre the activations.

DANNs: Unless explicitly mentioned, all entries of W EE matrices were independently sampled from an exponential distribution of variance n 3 n ( n -1) , where n is the number of hidden units. Next, to balance inhibitory and excitatory inputs, each row of W IE was initialised as the mean row of W EE (therefore each row of W IE is the same at initialisation), and rows of W EI were sampled from an exponential distribution and then squashed so that the sum of each row was 1. As such, the variance of W is the same as Pytorch's default initialisation for RNNs. Biases were initialised as 0.

2.3 Experiments

We tested the three different types of networks (standard RNN, ColEI, and DANN) on three classical tasks for RNNs: the adding problem [27], sequential MNIST classification [28], and language modelling using the Penn Tree Bank [29] All experiments were run with PyTorch version 1.5.0 on a RTX 8000 GPU cluster. Unless otherwise stated, all presented results were averaged over 5 seeds and shaded error bars denote standard deviation. Performance metrics are plotted after the first 100 updates.

Adding problem: The adding problem serves as a test to evaluate the ability of RNNs to learn long-term dependencies [27]. The objective is to compute the sum of two numbers selected from a sequence of random numbers ∈ [0 , 1] (of length 20 in our experiments). At each time step the input to the network is a 2-dimensional vector, comprising an element from the sequence of random numbers and a boolean, which is one only for the two numbers that should be added and zero otherwise.

Sequential MNIST: The sequential MNIST problem is a common test for the ability of RNNs to engage in classification. In this task, the network must classify MNIST digits (i.e. handwritten digits of numbers) when the rows of the image are presented sequentially to the network from top to bottom (resulting in 28 timesteps).

Penn Tree Bank: The Penn Tree Bank task is a common baseline for natural language processing that is more challenging than either the adding problem or sequential MNIST. In this task, input words from sentences taken from a newspaper corpus are embedded as 300-dimensional vectors using GloVe [30]. At each time step, the network must predict the next word.

Natural object recognition task: Following [31], we designed the following naturalistic object recognition task. At the first time step, the network's input is a down-sampled, flattened achromatic image of one of 64 natural objects from 8 categories (e.g. fruits, faces, etc.) with a random naturalscene background. The network receives no further inputs and must retain information about the image over 5 time steps to output the classification result at the last time step. Please see [31] and Appendix Fig. 8 for more details.

2.3.1 Spectrum Transplant

Singular value decomposition (SVD) decomposes a matrix into matrices of singular values and singular vectors: W = U Σ V T , where U and V are orthonormal matrices whose columns are singular vectors, and Σ is a matrix of singular values. Therefore, network weight matrices can be decomposed as: W RNN = U RNN Σ RNN V T RNN , W ColEI = U ColEI Σ ColEI V T ColEI . To disentangle the impact of singular values and singular vectors on performance we constructed networks initialised with weight

matrices W 1 = U RNN Σ ColEI V T RNN and W 2 = U ColEI Σ RNN V T ColEI , such that a standard RNN was given the spectral properties of a ColEI network, and vice-versa.

2.3.2 Normalised SVD Entropy (NSE)

To quantify the clustering strength of the singular values of a N × N recurrent weight matrix we used normalised SVD entropy (NSE), also denoted as H [32]: 2

where σ i are the singular values. The normalisation term log ( N ) ensures H ∈ [0 , 1] regardless of network size, where H = 1 if all singular values are the same (clustered) and H ≈ 0 implies the distribution of singular values is extremely dispersed. For example, in orthogonal RNNs H = 1 , since all singular values of the recurrent weights are 1 [33].

3 Results

3.1 Network performance on benchmark tasks

We first assessed the performance of standard, ColEI, and DANN RNNs on four benchmark tasks for RNNs: the adding problem, row-wise sequential MNIST, the Penn Tree Bank dataset, and a naturalistic object recognition task (Section 2.3). We found that ColEI networks did not learn as well as DANNs and standard RNNs. Their final performance was worse across all four tasks, and more variable across random seeds (Figure 2 A-D). Furthermore, we observed that gradient clipping benefitted ColEI networks substantially more than the other networks (Appendix Figure 10).

A

B

C

D

LOSS

1

ColEI(gc=7)

RNN (gC=7)

DANN (gC=3)

Test Error (%)

8

RNN (no gc)

DANN (no gc)

ColEI(gc=10)

Perplexity

240

RNN (gc=1)

ColEI (gc=1)

DANN (gC=1)

(%)

901

RNN (gC=10)

DANN (gC=10)

ColEI(gc=1)

Test MSEI

10-1

4

200

Test Error(

55

Test

160

20

10-2

0

4k

8k

12k

0

0

15

30

45

120

0

1020

30

40

0

10

20

30

#Updates

#k Updates

#k Updates

#k Updates

A confound that could explain the discrepancy in performance between ColEI networks and DANNs & standard RNNs is the distinct strategies used for their initialisation (Section 2.2). Specifically, ColEI recurrent weights are initialised such that the greatest norm of eigenvalues, i.e. the spectral radius ρ , is 1.5 [9]. In contrast, the initialisation of DANNs & standard RNNs results in ρ /similarequal 1 / √ 3 (PyTorch default RNN initialisation). Therefore, we assessed model performance across a range of initial values of ρ (Appendix 9). Again, we found that ColEI networks learned poorly, and there was no value of ρ for which ColEI networks matched the performance of the best standard RNNs and DANNs. However, while standard RNNs and DANNs had a similar relationship between ρ and performance, ColEI networks performed better than regular RNNs and DANNs at larger values of ρ , and worse for smaller values (Appendix Figure 9). Additionally, we also ran a control experiment to verify that the inhibitory parameter scaling of DANNs did not improve ColEI network performance (Appendix Figure 11).

3.2 Spectral properties of networks

The different relationship between performance and ρ of ColEI networks vs standard & DANN RNNs (Figure 2 D) is indicative of different spectral properties, such as the eigenvalue distribution of the recurrent weights. For example, eigenvalues greater (or less) than one indicate activity directions that

lead to exploding (or vanishing) activations and gradients, thereby impairing learning. Inspecting the eigenvalues of the initial weight matrices in the complex plane (Figure 3 A-C) shows that the eigenvalues of ColEI networks follow a non-uniform distribution that is clustered around the origin, as has been previously reported [34, 35]. In contrast, the eigenvalue distributions of DANNs & standard RNNs converge to a uniform circle in the complex plane. 3 We note that these spectral properties provide an informal explanation for the better learning of ColEI networks at larger values of ρ (Figure 2 D), as there will still be a majority of smaller eigenvalues in addition to the minority of large eigenvalues.

A

Dist of eigenvals of ColEl (p = 1.5)

B

Dist of eigenvals of RNN

C

Dist of eigenvals of DANN

1.5

1.51

1.51

Imaginary

Imaginary

Imaginary

0.0

0.0

0.0

-1.5

-i.5

0.0

1.5

0.0

1.5

0.0

1.5

Real

Real

Real

D

Dist of Sing Vals of ColEl (p = 1.5)

E

Dist of Sing Vals of RNN

F

Dist of Sing Vals of DANN

100↓

100

1.2

Prob Density

100

Prob Density

1.2

Prob Density

10-2

10-2

0.0

10-2

0.0

0.0

0.4

0.8

1.2

0.0

0.4

0.8

1.2

Singular Value

Singular Value o

0

6

12

18

0

6

12

18

6

12

18

SingularValueo

SingularValueo

Singular Value o

However, there are limitations to using eigenvalues for understanding learning dynamics. Importantly, the eigenvectors of non-symmetric matrices are always non-orthogonal, and eigendecomposition may fail to capture certain features of matrices with highly non-orthogonal columns - such as ColEI networks [36]. In contrast, singular vectors are always orthogonal. Thus, for ColEI networks we reasoned that although the maximum eigenvalue is constrained to be 1.5, the maximum singular value may be large and unconstrained. Indeed, while the maximum singular value for DANNs & standard RNNs was approximately twice the maximum radius of their eigenvalues, the ColEI singluar value spectrum had a notable mode of large singular values (Figure 3 D-F). 4 Also, the distribution of singular values for ColEI networks was multimodal (Figure 3 D-F), whereas the distribution of standard RNNs and DANNs converged to a quarter circle, in agreement with the Marchenko-Pastur Quarter Circle Law 5 . These results tell us that even when the eigenvalue radius is constrained for ColEI networks, there will still be directions in activity space corresponding to the large singular values along which the activations and gradients will be heavily deformed and stretched, leading to poor learning.

3.3 Spectral properties matter more than sign constraints for learning

A common hypothesis is that the poor performance of EI networks is due to the parameter sign constraints applied during learning, and addressing this was a motivation behind the architectural design of DANNs [10, 20]. Given the different spectral properties of ColEI and standard RNNs that we observed, we wondered: are sign constraints the main reason for impaired learning in ColEI networks, or are the spectral differences from standard RNNs the most critical factor? To investigate this, we first explored the impact of sign constraints: we trained ColEI networks without sign constraint during learning, and we trained standard RNNs with parameter signs frozen after initialisation (Figure 4 A, sequential MNIST). As expected, for both networks we found that sign constraints impaired learning ( 0 . 7% ± 0 . 3% for RNNs, 1% ± 1% for ColEI), but they did not account for the full difference in performance between standard RNNs and ColEIs ( 2 . 3% ± 0 . 6% ).

Next, we designed spectrum 'transplant' experiments, to investigate the impact of singular value spectra on learning (Section 2.3.1). We constructed hybrid networks with a combination of spectral properties at initialisation and sign constraints (see Section 2.3.1), specifically: 1) RNN or ColEI spectrum, 2) RNN or ColEI singular vectors, 3) presence or absence of sign constraints during learning. In total, with these three factors we had eight different network configurations ( 2 3 = 8 networks). These experiments revealed that the singular value spectra contribute more than sign constraints and singular vectors to the performance gap between ColEI and RNNs (Figure 4 B,C 1 . 5% ± 0 . 3% for RNNs, 2% ± 1% for ColEI). In addition to sequential MNIST, we also ran the same set of experiments on the Naturalistic Object Recognition task, and observed very similar results (Appendix Table 2). Finally, we investigated if the empirical EI balance at initialisation (i.e. the mean of the sum of incoming weights to each neuron) was altered in these experiments. We found it was not: for a ColEI network with 10% inhibitory neurons and 1000 hidden units, the mean balance across units was -0.0006 with std. deviation 1.12, and for ColEI with the RNN spectrum it was 0.0696 with std. deviation 0.88 (t test, p=0.12). Thus, sign constraints are only one component, and the singular value spectrum at initialisation has a stronger impact on learning performance.

A

EffectofSign Constraint

B

EffectofSpectrum andSingularVecs

C

RNN Ablation

ColEIAblation

(%)

Test Error

10

6

Orginal RNN+Frozen Sign

DANN

Original ColEI

Original ColEl + Sign Not Frozen

Orginal RNN

（%)

Error

10

6

DANN

ColEI Vecs with RNN Spectrum

RNN Vecs with RNN Spectrum

RNN Vecs with ColE1 Spectrum

@2.5

Error

Test

1.5

Test

Error

4.5

***

2.5

Test

.=

.=

Increase

Decrease

2

2

0.5

0.5

0

15

30

45

0

15

30

45

#k Updates

Change

Change Add Sign

Change

Change Remove Sign

Spectrum

Sing Vec Constraint

Spectrum Sing Vec

Constraint

3.4 Network performance and spectrum with respect to changes in EI ratio and network size

Having established that the spectrum of singular values is important for ColEI learning, we investigated how architectural hyperparameter choices, such as the EI ratio and number of hidden units, change the spectral properties and learning performance. Here, to quantify changes in the singular value distribution, we leveraged normalised SVD entropy (NSE) as a measure of spectrum 'pathology' (see Section 2.3.2). Notably, NSE captures the degree of unimodal clustering in the singular value spectrum, and we hypothesise that a unimodal spectrum of clustered singular values is desirable - as is the case for orthogonal RNNs [33, 28].

Wefound that reducing the percentage of inhibitory neurons (thereby increasing the EI ratio) decreased the NSE and impaired learning performance (Figure 5). Also, despite constraining ρ = 1 . 5 in each case, decreasing the number of inhibitory neurons increased the largest singualar value, σ max (Figure 5 D-F). In contrast, we found that the spectra and performance of DANNs were robust to changes in the EI ratio, and they learned well even with 1 inhibitory neuron (NSE remained ∼ 0 . 95 for both DANNs and standard RNNs, Figure 5 A-C, Appendix Figure 15, Appendix Table 3). This

result further supports the idea that the spectral properties of EI networks predict their learning performance, and shows that the E-to-I ratio impacts the spectral properties of ColEI networks.

A

Percentofinhib1%

B

Percentofinhib10%

Percentofinhib50%

201

201

201

ColEI

ColEI

ColEI

Test Error (%)

15

DANN

RNN

Test Error (%)

15

DANN

RNN

(%)

Test Error (

15

DANN

RNN

G

10

10

10

Regression Line

5

5

5

%)

10

10% inhibitory

1% inhibitory

0

0

15

30

45

0

0

15

30

45

0

0

15

30

45

Test Error

50% inhibitory

#k Updates

#k Updates

#k Updates

D

Percentofinhib1%

E

Percent of inhib 10%

F

Percent of inhib 50%

6

100

100↓

Prob Density

10-1

10-3

Prob Density

10-1

2

10-3

Singular Value

Singular Value

10-2

Z

Singular Value o

4

6

Normalized SVD Entropy

0.5

0.7

0.9

10-3

0

25

50

75

0

6

12

18

0

4

8

12

SingularValueo

SingularValueo

SingularValuea

A

Num of Hidden Units:100

B

Num ofHidden Units:300

Num of Hidden Units:1000

4

4

CoIEI

Test Error (%)

ColEI

Test Error (%)

ColEI

Test Error (%)

3

DANN

3

DANN

3

DANN

RNN

RNN

RNN

2

2

2

G

1

1

1

Test Error (%)

4.5

0

0

50

100150

0

0

50

100

150

0

0

50

100150

#k Updates

#k Updates

#k Updates

2.5

D

Num of Hidden Units:100 E

Num of Hidden Units:300

F

Num of Hidden Units: 1000

一

Regression Line

n=100

100

100

n=300

Prob Density

ProbDensity

10-1

0.5

n=1000

0.7

0.8

0.9

10-2

2

anjeAjejnbuis

6

Singular Value o

6

8

10-3

2

Singular Value o

4

6

8

NormalizedSVD Entropy

Singular Value

6

12

18

0

Singular Value α

8

16

24

0

Singular Value

15

30

45

Next, we investigated how the number of hidden units affects the spectral properties and learning performance of EI networks. As shown in Figure 6 A-F, we found that the bulk of singular values became more clustered for larger ColEI networks ( ∈ [0 , 8] Figure 6 D-F), and the NSE increased despite an increasing σ max . Correspondingly, these changes in NSE were anti-correlated with test error (Figure 6 G), similar to the performance trends for EI ratio. In contrast, DANN spectra were similar across network sizes (Appendix Figure 15). Taken together with the EI ratio experiments, these results suggest that for ColEI networks, the best performance will be obtained for larger networks with balanced EI ratios, because these networks have spectral properties more similar to standard RNNs. However, it is interesting that large ColEI networks learn well despite a large σ max , which indicates that the overall effect of all singular values, rather than σ max , is what matters most for learning. We futher discuss this result in the discussion, but put another way, in larger networks even if some directions have large singular values, there are many more directions with small singular

values. As a result, activities will likely be rotated away from the directions with large singular values, ameliorating the exploding gradients problem. Therefore, for a metric to be predictive of the network performance it should capture the overall effect of the entire spectrum, which we do with NSE.

A

15

B

H

0.95

C

MeanTestError

5.0

D

5.0

Small Net High Entropy

NetworkSize

200

Size

200

(%)

Large Net Low Entropy

Error

10

-0.75

Network

3.5

Error

3.5

600

600

5

Test

2.0

2.0

0

15

30

45

1000

1%

23%

45%

0.55

1000

1%

23%

45%

0.6

0.8

1.0

#k Updates

Percentofinhib

Percentofinhib

Normalized SVDEntropyH

The above experiments suggest that the spectral properties of ColEI networks at initialisation, summarised by NSE, can be used to estimate their performance across a range of architectural hyperparameters. As the NSE at initialisation is dependent on both the network size and EI ratio, networks can have a high NSE from either being large, or having a balanced EI ratio. Therefore, we hypothesised that small networks with balanced E & I units will outperform larger networks with a skewed EI ratio that results in a smaller NSE at initialisation. To test this prediction, we generated networks of different EI ratios and numbers of hidden units and calculated their associated NSE at initialisation Figure 7 B. Again, we found that NSE anti-correlates with final test error (Figure 7 D), and small networks with high NSE performed better than large networks with lower NSE (Figure 7 A). Similar trends are observed for the natural object recognition and language modelling tasks (Appendix Figure 12 & Figure 13 respectively). This supports the hypothesis that the NSE at initialisation is of primary importance for determining learning performance. Moreover, it allows us to characterise the link between spectrum pathology and performance and explain why different hyperparameter choices may affect learning in EI RNNs.

4 Discussion

In this work, we have provided evidence that both sign constraints and the spectral properties of the weights at initialisation can cause poor learning in EI RNNs. As well, we compared the relative contributions of sign constraints and spectral properties for EI network performance, and found that the distribution of singular values are more important for learning. We found that the simplest method of incorporating Dale's Law into RNNs (by constraining columns of the weight matrix, ColEI, following [9]) results in networks with multi-modal spectra that are dispersed, especially for small networks with large ratios of E to I units. In contrast, we found that recurrent versions of the DANN EI architecture [10] have similar spectra to standard RNNs, and learn just as well. We also presented NSE as a measurement of spectrum pathology that can be used as a diagnostic tool to predict performance before training. The higher the NSE at initialisation, the more likely an EI RNN is to train well. This work therefore explains why DANN RNNs perform as well as standard RNNs, but not ColEI RNNs. Overall, by highlighting the importance of EI network spectral properties at initialisation we provide a map for designing networks that respect Dale's law and learn well.

In addition to their poor learning, ColEI networks also showed more variance during training and over different initialisations when compared to the other networks. This is likely due to them having fewer activity modes with appropriate singular values for learning computations over time (the motivation behind SVD entropy) and therefore small differences at initialisation can strongly impact learning. Similarly, we also found that without gradient clipping ColEI networks are highly unstable, again in line with a pathological distribution of singular values at initialisation [37]. However, other open theoretical questions arise from the empirical results presented above. First, what is the origin of the unusual multi-modal spectra of ColEI networks? And second, why do large ColEI networks learn well despite a large maximum singular value, σ max ?

For a simple intuitive explanation of ColEI spectrum pathology, let us first consider the difference between the recurrent weight matrix of ColEI and standard RNNs. For both the standard & DANN RNNs, this a zero-mean random matrix. However, the ColEI recurrent weight matrix can be written as the sum of a rank-one nilpotent matrix (of column means) and a zero-mean random matrix. While technically the singular and eigen value distributions of two matrices can't simply be added when two matrices are added, this already provides some intuition as to the origin of the unusual spectrum: it likely stems from the nilpotent matrix of column means. Here we note that the eigenvalues of any nilpotent matrix are 0, and hence we expect they do not significantly change the eigenvalue radius. If so, constraining the spectral radius of the recurrent matrix will not constrain the nilpotent matrix's contribution towards the singular value spectrum (see the Appendix 6.5 for a more detailed discussion).

Regarding the second open question, we found that large ColEI networks learned well, despite an increasing σ max . Intuitively this should harm learning, as some activity directions will result in exploding gradients [37]. However, we can understand this by writing the operation performed by a linear network in terms of the SVD of its recurrent weight matrix:

It is clear that between each scaling operation Σ , there are two rotation steps V T , U . These rotation steps could allow the network to avoid pathological growth by rotating the activity vectors aligned to directions with large singular values, such that they instead align with directions with small singular values. Furthermore, as the network width n increases, the number of rotations where this effect occurs increases, as there exist more directions with smaller singular values, as highlighted by NSE.

There are several avenues for future work that build on the results we present here. First, there may be other ways to design RNNs that respect Dale's Law while learning well. Our results suggest that a guiding principle for investigations of this sort should be the consideration of the spectral properties of the weight matrices. And further, we note that this insight likely applies broadly to all EI network architectures, such as convolutional, feedforward, and potentially spiking models, not just RNNs. Similarly, the models we present here could be made more biologically faithful, for example inhibitory to inhibitory connections could be added to DANNs, and cell-type specific time constants included in ColEI networks. Furthermore, the two EI models have different neurobiological interpretations, as the populations of inhibitory neurons that they naturally model are different. ColEI RNNs are a more general model of inhibitory cells (eg. SOM+, CCK+ etc.), whereas DANNs specifically model fast PV+ interneurons. Therefore a promising future line of work could be to hybridise the two the models together. As our central motivation was to understand why EI networks can fail to learn well, we didn't explore these directions in the experiments we present here. However, our results indicate that the network's spectral properties should be carefully considered whenever such architectural choices are made for modelling purposes.

Additionally, from a biological-hypothesis generating perspective, appropriately constraining the spectral properties of RNNs can help ameliorate problems of exploding/vanishing gradients [36], and so, an interesting question for neuroscience research into biologically plausible gradient estimation [38] is whether real brains, which largely obey Dale's Law, have spectral properties that are well-suited to gradient descent or not. Likewise, there are interesting questions regarding the balance between excitation and inhibition in RNNs (a major topic in neuroscience) that flow from the observation that such balance can impact the spectral properties of networks [11].

In summary, our work builds on tools from deep learning theory to provide guidance on how to construct RNNs that incorporate Dale's Law. We presented evidence that the reason for poor learning of ColEI networks and good learning of DANNs is, in large part, their respective spectral properties. Therefore, we hope our work serves as the basis and inspiration for an alternative initialization for ColEI networks that improves their learning. More generally, by providing an explanation for EI network learning differences, we have opened up new directions of research for RNNs in computational neuroscience.

References

Hari Teja Kalidindi, Kevin P Cross, Timothy P Lillicrap, Mohsen Omrani, Egidio Falotico, Philip N Sabes, and Stephen H Scott. Rotational dynamics in motor cortex are consistent with a feedback controller. Elife , 10:e67256, 2021.

Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. nature , 503(7474):78-84, 2013.

Abigail A Russo, Ramin Khajeh, Sean R Bittner, Sean M Perkins, John P Cunningham, Laurence F Abbott, and Mark M Churchland. Neural trajectories in the supplementary motor area and motor cortex exhibit distinct geometries, compatible with different classes of computation. Neuron , 107(4):745-758, 2020.

Omri Barak. Recurrent neural networks as versatile tools of neuroscience research. Current opinion in neurobiology , 46:1-6, 2017.

Guangyu Robert Yang and Xiao-Jing Wang. Artificial neural networks for neuroscientists: a primer. Neuron , 107(6):1048-1070, 2020.

Mahi Hardalupas. How neural is a neural net? Bio-inspired computational models and their impact on the multiple realization debate . PhD thesis, University of Pittsburgh, 2021.

John Carew Eccles. From electrical to chemical transmission in the central nervous system: the closing address of the sir henry dale centennial symposium cambridge, 19 september 1975. Notes and records of the Royal Society of London , 30(2):219-230, 1976.

SeulAh Kim, Michael L. Wallace, Mahmoud El-Rifai, Alexa R. Knudsen, and Bernardo L. Sabatini. Co-packaging of opposing neurotransmitters in individual synaptic vesicles in the central nervous system. Neuron , 110(8):1371-1384.e7, 2022.

HFrancis Song, Guangyu R Yang, and Xiao-Jing Wang. Training excitatory-inhibitory recurrent neural networks for cognitive tasks: a simple and flexible framework. PLoS computational biology , 12(2):e1004792, 2016.

Jonathan Cornford, Damjan Kalajdzievski, Marco Leite, Amélie Lamarquette, Dimitri Michael Kullmann, and Blake Aaron Richards. Learning to live with dale's principle: Anns with separate excitatory and inhibitory units. In International Conference on Learning Representations , 2021.

Guillaume Hennequin, Everton J Agnes, and Tim P Vogels. Inhibitory plasticity: balance, control, and codependence. Annual review of neuroscience , 40:557-579, 2017.

Helen C Barron, Tim P Vogels, Timothy E Behrens, and Mani Ramaswami. Inhibitory engrams in perception and memory. Proceedings of the National Academy of Sciences , 114(26):66666674, 2017.

Tim P Vogels, Henning Sprekeler, Friedemann Zenke, Claudia Clopath, and Wulfram Gerstner. Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks. Science , 334(6062):1569-1573, 2011.

Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv , page 407007, 2018.

Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences , 111(23):8619-8624, 2014.

Yuxiu Shao and Srdjan Ostojic. Relating local connectivity and global dynamics in recurrent excitatory-inhibitory networks. bioRxiv , pages 2022-08, 2022.

Alessandro Ingrosso and LF Abbott. Training dynamically balanced excitatory-inhibitory networks. PloS one , 14(8):e0220547, 2019.

Thomas Miconi. Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks. Elife , 6:e20899, 2017.

Sun Minni, Li Ji-An, Theodore Moskovitz, Grace Lindsay, Kenneth Miller, Mario Dipoppa, and Guangyu Robert Yang. Understanding the functional and structural differences across excitatory and inhibitory neurons. bioRxiv , 2019.

Daniel J Amit, C Campbell, and KYM Wong. The interaction space of neural networks with sign-constrained synapses. Journal of Physics A: Mathematical and General , 22(21):4687, 1989.

Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. A study of biologically plausible neural network: The role and interactions of brain-inspired mechanisms in continual learning. arXiv preprint arXiv:2304.06738 , 2023.

Charles Bordenave and Djalil Chafaï. Around the circular law. Probability Surveys , 9(none), jan 2012.

Robin Tremblay, Soohyun Lee, and Bernardo Rudy. Gabaergic interneurons in the neocortex: from cellular properties to circuits. Neuron , 91(2):260-292, 2016.

Hua Hu, Jian Gan, and Peter Jonas. Fast-spiking, parvalbumin+ gabaergic interneurons: From cellular design to microcircuit function. Science , 345(6196):1255263, 2014.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256, 2010.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Quoc V. Le, Navdeep Jaitly, and Geoffrey E. Hinton. A simple way to initialize recurrent networks of rectified linear units, 2015.

Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of english: The penn treebank. Comput. Linguist. , 19(2):313-330, jun 1993.

Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532-1543, 2014.

Najib J Majaj, Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience , 35(39):13402-13418, 2015.

Tanya Strydom, Giulio V. Dalla Riva, and Timothée Poisot. Svd entropy reveals the high complexity of ecological networks. Frontiers in Ecology and Evolution , 9, 2021.

Scott Wisdom, Thomas Powers, John R. Hershey, Jonathan Le Roux, and Les Atlas. Fullcapacity unitary recurrent neural networks, 2016.

Kanaka Rajan and L Abbott. Eigenvalue spectra of random matrices for neural networks. Physical review letters , 97:188104, 12 2006.

Isabelle D Harris, Hamish Meffin, Anthony N Burkitt, and Andre DH Peterson. Eigenvalue spectral properties of sparse random matrices obeying dale's law. arXiv preprint arXiv:2212.01549 , 2022.

Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, and Guillaume Lajoie. Non-normal recurrent neural network (nnrnn): learning long time dependencies while improving expressivity with transient dynamics. Advances in neural information processing systems , 32, 2019.

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310-1318. Pmlr, 2013.

Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience , pages 1-12, 2020.

Paul J Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural networks , 1(4):339-356, 1988.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author present a simulation study that investigates gradient-based optimisation of RNNs that obey Dale’s Law, i.e. networks with neurons that are strictly excitatory or inhibitory at initialisation and during training. In particular, the authors disentangle the effect of enforcing Dale’s Law during training (synaptic weights can not flip sign) and the initial spectral properties that result from initialisation with separate excitatory and inhibitory populations of neurons.\n\nSTRENGTHS:\nThe paper investigates an important open question: What are the origins of the performance gap between RNNs that do / do not obey Dale’s Law. Through a large set of simulation studies and numerical analysis, the authors show in a convincing way that the performance gap does not solely stem from enforcing Dale’s Law during training (and thus restricting the solution space by enforcing signs of weights) but also significantly depends on how networks are initialised and parametrised. Initialising RNNs with columns of positive / negative weights to create populations of excitatory / inhibitory neurons (“ColEI networks”) causes a skewed and multi-modal singular value spectrum – leading to the well known effect of exploding neural activations / gradients that hamper training and performance. The authors further provide empirical evidence that the Normalised SVD Entropy thus provides an easy to compute predictor of network performance before the onset of training. Further, the authors provide analytical intuition for their simulation results in the discussion. The paper is well structured, written and accessible. The simulation details are explained in great detail; which is very helpful for understanding the research question, results and potential limitations. The figures are generally clear and accessible.\n\nWEAKNESSES:\n- My main concern is, that as written in appendix 5.3, the learning rate of DANNs is scaled independently for E and I weights to balance the impact and updates of E and I populations. As ColEI networks seem not to be trained with a similar scaling scheme, this has the potential to explain away or confound results and observations. I am happy to revise my score if this is adequately addressed.\n- The authors base their analysis on ColEI networks that are initialised following [5], however, the authors do not provide insight (through simulations or analytically) if there generally can or cannot exist a weight initialisation scheme for ColEI networks which does not lead to a skewed and multimodal spectrum of singular values. As thus, the results seem to be limited to a specific initialisation scheme and not to ColEI networks in general.\n- The authors apply gradient clipping during network training. This may significantly reduce the effects of skewed, multimodal singular value spectra and outliers in the singular value spectrum. As the authors try to disentangle the effects of enforcing Dale’s Law and the spectral properties of initial recurrent weight matrices, gradient clipping may confound results and conclusions.\n- If I understand correctly, recurrent weight matrices in DANNs are parametrised by a linear combination of three weight matrices. As such, it seems like DANNs have three times more free parameters than ColEI networks. If true, this would pose the question if comparisons between DANNs and ColEIs as presented in figure 6 are fair.\n\nMinor:\n- In figure 2D it is difficult to see what is going on – maybe log-axis and a higher alpha value for the scatter point’s colour would help?\n- In figure 2B it looks like the networks haven’t been trained until convergence\n- Missing “in” in line 94\n- Broken reference in line 110\n\nQUESTIONS:\n1) As written in appendix 5.3, the learning rate of DANNs are scaled independently for E and I weights to balance the impact of E and I and their updates. As ColEI networks seem not to be trained with a similar scaling scheme: How does the IE weight scaling influence simulation results and conclusions; in particular with respect to changes in EI ratio and network size?\n\n2) In line 109 the authors write “[…] the activation variance did not scale with depth […]”. Did the authors control for mean and variance shift across layers and increasing depth? It would help to better understand the three different training regimes if there would be a plot that is visualising the mean and variance shift at initialisation across layers and depth for random data for RNNs, DANNs and ColEIs for varying EI ratios.\n\n3) Related to 2 – In order to better understand the learning dynamics and in order to detect anomalies, a plot that visualises the mean and variance of synaptic weights and biases and how they evolve during training would be helpful. Do you expect the mean and variance of the weight matrices and biases within the E and I population to evolve similarly for RNNs, DANNs and ColEIs?\n\n4) Related to 2 – Judging from the learning trajectories (e.g. in Figure 2B/C), to me, it looks like that ColEI networks have a significantly higher initial error (at t=0) – however, since the y-axis is cut – I might be wrong? If the initial errors are different, why are they different?\n\n5) In line 116 the authors write “[…] each row of W^IE was initialised as the mean row of W^EE […]”. What does that mean? Do all entries in the row have the same value?\n\n6) In Figure 3 it looks like ColEI networks have a large variance in performance. Do you have an explanation or intuition why some of the initialisations fail and others succeed?\n\n7) From equation 3, I conclude that DANNs have 3x more free parameters in the recurrent part of the network than ColEIs. This makes comparison tricky, especially when it comes to network size. How does performance look like as a function of free parameters (instead of number of neurons in the hidden layer)?\n\n8) I would assume that spectrum transplants alter the EI balance. Did you correct for that effect?\n\nLIMITATIONS:\nThe authors have adequately addressed limitations of their work", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors apply Dale's law (that neurons provide exclusively excitatory or inhibitory outputs) to RNNs with two different architectures: ColEIs which are the “straightforward” way of applying sign constraints per neuron, and recurrent DANNs based on an architecture with two layers per recurrent step. They compare singular value spectrums, network sizes and Excitation/Inhibition ratios in all three architectures and analyze how they affect performance. They conclude that the reason why DANNs perform as well as RNNs, while ColEIs perform worse is largely due to differences in the SVD distribution (as measured by Normalised SVD Entropy).\n\nSTRENGTHS:\nDale’s law is an important and ubiquitous property of biological neural networks, but its consequences have not been thoroughly explored. It is interesting that a simple architecture can enforce Dale’s law while remaining trainable, and while keeping many properties of unconstrained RNNs. This architecture features feedforward inhibition, a biologically plausible motif. Understanding consequences of such biological structure is a key topic of Neuro-AI.\n\nThe work offers some nice empirical results, both in performance and in spectral properties.\n\nThe writing and figures are clear, and good efforts are made to provide explanations and interpretations.\n\nWEAKNESSES:\nWhile being advertised as a biological neural network, the architectural constraints that make the recurrent DANN trainable may be the very same ones that make it biologically unrealistic. For instance, there is no direct recurrent inhibition (I to I), and there cannot be direct reciprocal connectivity between E and I units.\n\nThere is not much mathematical analysis of the results, so the insights are somewhat limited. Clearly there are differences in the spectral distributions, but why these appear and most importantly how these are “directly responsible” for decreased performance is not deeply discussed. There is some discussion of the low-rank nilpotent component of ColEI matrices but there is more related literature to connect to, most notably from the Ostoijc group (no, this reviewer is not from that group). For example, Mastrogiuseppe and Ostojic 2018, Schuessler et al 2022, in addition to the Shao and Ostoijc 2022 paper the authors do cite (now evidently published in PLoS comp bio).\n\nMany of the analyses are done only on the sequential MNIST task, which is not particularly natural. Claims are being made on the basis of small differences in errors, and some qualitative analysis. The small differences suggest that the networks are not being pushed very far to demonstrate their inductive biases.\n\nWhere did the ρ=1.5 and ρ ~= 1/sqrt(3) come from in Section 3.1?\n\nHow were the sign constraints implemented?\n\nMinor: watch out for punctuation errors and typos.\n\nQUESTIONS:\nAre the inhibitory units in a DANN strictly linear? This seems necessary if the DANN weight matrix is actually an exact reparameterization of the RNN weight matrix, as advertised (L95). Either way, this is unclear. And this has important implications for the story: if the DANN is an exact reparameterization, then isn’t it trivial that RNN and DANN spectra are identical?\nWhy the variance of line 115? Why ρ=1.5 and ρ ~= 1/√3 in Section 3.1?\nCan you explain the 3-layer architectures used in the Sequential MNIST figures? Do the eigenvalues and singular value visualizations of Figures 5,6 include values from weights from all layers? If so, are there any differences between the eigen/singular values between layers?\nLine 200: Is initialization the only difference between a ColEI network without sign constraints and a standard RNNs?\n\nLIMITATIONS:\nAdequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigated the problem of why columnEI networks have impaired learning, and they experimentally found that instead of sign constraint, the spectral property of weight at initialization contributed the most; they further experimentally showed that E/I ratio and network size change the spectral properties thus lead to different learning performances. Additionally, they showed DANN in the RNN form show similar performance as normal RNN in 3 different tasks.\n\nSTRENGTHS:\n**Originality** This is a follow-up work on applying DANN to RNN. I appreciate the detailed discussion on why it is hard to train E-I separated RNNs. \n\n**Quality**The effectiveness of DRNN is tested in three different tasks which is great! And all hyperparameter selections are listed in the appe ndix. The discussion on initialization spectral property contain proper ablation experiments and extended discussion on E-I ratio and network size.   \n\n**Clarity** The paper is clearly written, with details in the appendix. Some part of the result explanation can be confusing (see second question)\n\n**Significance** I believe the paper will be of interest to the computational neuroscience community.\n\nWEAKNESSES:\n- The discussion of initialization spectral property is only done for sequential MNIST. Does the observation on SVD entropy hold across data distributions? I'm a bit concerned on how generalizable the results are (details see question). \n- For the sign constrained training, how are the gradients rectified? Set to zero? If set to zero, it may leads to silent unit problem. Did the authors exclude the possibility that sign constrained RNN learn worse due to increased number of silent units? \n- Typo: line 110 citation\n\nQUESTIONS:\nMy main concern is on the spectral discussion \n- Does it extend to other tasks with different data distribution?\n- Why clusterness of singular values matter for learning? The intuition currently given in the paper seems to mainly pertain to large singular values; yet it is shown large singular value ColEI can learn well. Then what is the intuition behind SVD entropy tracking performance? This lack of intuition is what prompted me to ask the first question as it is not immediately clear to me how generalizable the current results are. And thus a current score of 5.\n\nLIMITATIONS:\nSee questions", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a comparison between the performance of standard RNNs and two classes of models: the ColEI network and the DANN, which incorporate Dale's Law with the constraint that units in a circuit should be excitatory or inhibitory but not both. The DANN achieves similar performance to a non-signed RNN, but ColEI shows inferior performance. The paper demonstrates that the spectral properties of the recurrent weight matrix at initialization have a more significant impact on network performance than sign constraints, which may explain why some forms of EI network learn better than others.\n\nSTRENGTHS:\n- This paper uses analytical approaches from machine learning to explore network models that adhere to biological principles, which may inspire the development of more biologically realistic models that also have good performance.\n- A novel contribution is the introduction of normalised SVD entropy as a measurement of spectrum pathology during the initialization stage which predicts the final performance of a network before training.\n- It also includes extensive experiments that test three RNN architecture on three different tasks with progressive difficulty levels.\n\nWEAKNESSES:\nThis work builds upon existing research about EI networks with incremental advancements, offering interesting insights into how the spectral properties of the recurrent weight matrix at initialization may impact network performance. However, the practical application for designing EI networks that perform well requires further clarification. It is unclear to me whether DANNs can be effectively improved with this insight, and even with a more appropriate spectrum, ColEI falls short of standard RNN or DANN performance.\n\nQUESTIONS:\nConsidering the biological motivation of having dedicated excitatory and inhibitory units, it would be valuable to provide a deeper validation or interpretation of the results in the context of neurobiology. For example, could the finding about the different effects of changing the ratio of E/I units on ColEI networks and DANNs offer insights into the biological relevance of these two models to the neural circuit in the brain?\n\nLIMITATIONS:\nWhile the focus on RNNs is relevant to the paper's objectives, it would be helpful to briefly acknowledge potential implications for other types of networks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}