SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Impact of spectrum, singular vectors and sign constraints on learning (A) Impact of sign constraints on standard and ColEI RNNs (note that DANNs are always sign constrained). (B) Impact of transplanting singular vectors and values between ColEI and RNN networks. (C) Summary of final test error changes induced by spectrum, singular vectors, and sign constraints. RNNs 5 seeds  $(p < 0.05)$ ; ColEI 30 seeds due to higher variance  $(p < 0.01)$ . Mann-Whitney U test
3.4 Network performance and spectrum with respect to changes in EI ratio and network size
Having established that the spectrum of singular values is important for ColEI learning, we investigated how architectural hyperparameter choices, such as the EI ratio and number of hidden units, change the spectral properties and learning performance. Here, to quantify changes in the singular value distribution, we leveraged normalised SVD entropy (NSE) as a measure of spectrum "pathology" (see Section 2.3.2). Notably, NSE captures the degree of unimodal clustering in the singular value spectrum, and we hypothesise that a unimodal spectrum of clustered singular values is desirable - as is the case for orthogonal RNNs [33, 28].
We found that reducing the percentage of inhibitory neurons (thereby increasing the EI ratio) decreased the NSE and impaired learning performance (Figure 5). Also, despite constraining  $\rho = 1.5$  in each case, decreasing the number of inhibitory neurons increased the largest singular value,  $\sigma_{max}$  (Figure 5D-F). In contrast, we found that the spectra and performance of DANNs were robust to changes in the EI ratio, and they learned well even with 1 inhibitory neuron (NSE remained  $\sim 0.95$  for both DANNs and standard RNNs, Figure 5A-C, Appendix Figure 5, Appendix Table 3). This
result further supports the idea that the spectral properties of EI networks predict their learning performance, and shows that the E-to-I ratio impacts the spectral properties of ColEI networks.
Figure 5: Performance and spectral properties of DANN and ColEI RNNs with different EI ratios. (A-C) Performance on sequential MNIST. Dashed line indicates the final error of standard RNNs of same size. (D-F) Distribution of singular values for ColEI networks y-axis log scaled, inset plots are x-axis [0, 8]. See equivalent for DANNs in Figure3F and Appendix Figure15(G) ColEI performance v.s. initial normalised SVD entropy of the recurrent weights across different EI ratios. (1-layer RNN, 100 units,  $\rho_{ColEI} = 1.5$ ,  $\rho_{DANN} = \frac{1}{\sqrt{3}}$
Figure 6: Performance and spectral properties of DANN and ColEI RNNs with different network sizes (column-wise). (A-C) Performance on sequential MNIST. Dashed line indicates the final error of RNNs of same size. (D-F) Distribution of singular values for ColEI networks y-axis log scaled, inset plots are x-axis [0, 8]. See equivalent for DANNs in Appendix Figure [15] (G) ColEI performance v.s. initial normalised SVD entropy of the recurrent weights across different network sizes. (1-layer RNN,  $10\%$  inhibitory units,  $\rho_{ColEI} = 1.5$ ,  $\rho_{DANN} = \frac{1}{\sqrt{3}}$
Next, we investigated how the number of hidden units affects the spectral properties and learning performance of EI networks. As shown in Figure 6A-F, we found that the bulk of singular values became more clustered for larger ColEI networks  $(\in [0,8]$  Figure 6D-F), and the NSE increased despite an increasing  $\sigma_{max}$ . Correspondingly, these changes in NSE were anti-correlated with test error (Figure 6G), similar to the performance trends for EI ratio. In contrast, DANN spectra were similar across network sizes (Appendix Figure 15). Taken together with the EI ratio experiments, these results suggest that for ColEI networks, the best performance will be obtained for larger networks with balanced EI ratios, because these networks have spectral properties more similar to standard RNNs. However, it is interesting that large ColEI networks learn well despite a large  $\sigma_{max}$  which indicates that the overall effect of all singular values, rather than  $\sigma_{max}$ , is what matters most for learning. We further discuss this result in the discussion, but put another way, in larger networks even if some directions have large singular values, there are many more directions with small singular
values. As a result, activities will likely be rotated away from the directions with large singular values, ameliorating the exploding gradients problem. Therefore, for a metric to be predictive of the network performance it should capture the overall effect of the entire spectrum, which we do with NSE.
Figure 7: Normalised SVD entropy predicts ColEI network performance. (A) Performance on sequential MNIST for networks indicated by circles in B&C. (B) Normalised SVD entropy for networks of different sizes and EI ratios. (C) Mean test error for networks of different sizes and EI ratios. Top left pixel value is clipped at 5 for better visualisation. (D) Final mean test error vs Normalised SVD entropy for all ColEI networks. 1 outlier network (corresponding to the top left pixel in C, B) is removed for better visualisation.
The above experiments suggest that the spectral properties of ColEI networks at initialisation, summarised by NSE, can be used to estimate their performance across a range of architectural hyperparameters. As the NSE at initialisation is dependent on both the network size and EI ratio, networks can have a high NSE from either being large, or having a balanced EI ratio. Therefore, we hypothesised that small networks with balanced E & I units will outperform larger networks with a skewed EI ratio that results in a smaller NSE at initialisation. To test this prediction, we generated networks of different EI ratios and numbers of hidden units and calculated their associated NSE at initialisation Figure 7B. Again, we found that NSE anti-correlates with final test error (Figure 7D), and small networks with high NSE performed better than large networks with lower NSE (Figure 7A). Similar trends are observed for the natural object recognition and language modelling tasks (Appendix Figure 12 & Figure 13 respectively). This supports the hypothesis that the NSE at initialisation is of primary importance for determining learning performance. Moreover, it allows us to characterise the link between spectrum pathology and performance and explain why different hyperparameter choices may affect learning in EI RNNs.
In this work, we have provided evidence that both sign constraints and the spectral properties of the weights at initialisation can cause poor learning in EI RNNs. As well, we compared the relative contributions of sign constraints and spectral properties for EI network performance, and found that the distribution of singular values are more important for learning. We found that the simplest method of incorporating Dale's Law into RNNs (by constraining columns of the weight matrix, ColEI, following [9]) results in networks with multi-modal spectra that are dispersed, especially for small networks with large ratios of E to I units. In contrast, we found that recurrent versions of the DANN EI architecture [10] have similar spectra to standard RNNs, and learn just as well. We also presented NSE as a measurement of spectrum pathology that can be used as a diagnostic tool to predict performance before training. The higher the NSE at initialisation, the more likely an EI RNN is to train well. This work therefore explains why DANN RNNs perform as well as standard RNNs, but not ColEI RNNs. Overall, by highlighting the importance of EI network spectral properties at initialisation we provide a map for designing networks that respect Dale's law and learn well.
In addition to their poor learning, ColEI networks also showed more variance during training and over different initialisations when compared to the other networks. This is likely due to them having fewer activity modes with appropriate singular values for learning computations over time (the motivation behind SVD entropy) and therefore small differences at initialisation can strongly impact learning. Similarly, we also found that without gradient clipping ColEI networks are highly unstable, again in line with a pathological distribution of singular values at initialisation [37]. However, other open theoretical questions arise from the empirical results presented above. First, what is the origin of the unusual multi-modal spectra of ColEI networks? And second, why do large ColEI networks learn well despite a large maximum singular value,  $\sigma_{max}$ ?
For a simple intuitive explanation of ColEI spectrum pathology, let us first consider the difference between the recurrent weight matrix of ColEI and standard RNNs. For both the standard & DANN RNNs, this is a zero-mean random matrix. However, the ColEI recurrent weight matrix can be written as the sum of a rank-one nilpotent matrix (of column means) and a zero-mean random matrix. While technically the singular and eigenvalue distributions of two matrices can't simply be added when two matrices are added, this already provides some intuition as to the origin of the unusual spectrum: it likely stems from the nilpotent matrix of column means. Here we note that the eigenvalues of any nilpotent matrix are 0, and hence we expect they do not significantly change the eigenvalue radius. If so, constraining the spectral radius of the recurrent matrix will not constrain the nilpotent matrix's contribution towards the singular value spectrum (see the Appendix 6.5 for a more detailed discussion).
Regarding the second open question, we found that large ColEI networks learned well, despite an increasing  $\sigma_{max}$ . Intuitively this should harm learning, as some activity directions will result in exploding gradients [37]. However, we can understand this by writing the operation performed by a linear network in terms of the SVD of its recurrent weight matrix:
$$
h _ {t + 2} ^ {\ell} = W _ {r e c} ^ {\ell} W _ {r e c} ^ {\ell} h _ {t} ^ {\ell} = U \Sigma V ^ {T} U \Sigma V ^ {T} h _ {t} ^ {\ell} \tag {5}
$$
It is clear that between each scaling operation  $\Sigma$ , there are two rotation steps  $V^T, U$ . These rotation steps could allow the network to avoid pathological growth by rotating the activity vectors aligned to directions with large singular values, such that they instead align with directions with small singular values. Furthermore, as the network width  $n$  increases, the number of rotations where this effect occurs increases, as there exist more directions with smaller singular values, as highlighted by NSE.
There are several avenues for future work that build on the results we present here. First, there may be other ways to design RNNs that respect Dale's Law while learning well. Our results suggest that a guiding principle for investigations of this sort should be the consideration of the spectral properties of the weight matrices. And further, we note that this insight likely applies broadly to all EI network architectures, such as convolutional, feedforward, and potentially spiking models, not just RNNs. Similarly, the models we present here could be made more biologically faithful, for example inhibitory to inhibitory connections could be added to DANNs, and cell-type specific time constants included in ColEI networks. Furthermore, the two EI models have different neurobiological interpretations, as the populations of inhibitory neurons that they naturally model are different. ColEI RNNs are a more general model of inhibitory cells (eg. SOM+, CCK+ etc.), whereas DANNs specifically model fast PV+ interneurons. Therefore a promising future line of work could be to hybridise the two the models together. As our central motivation was to understand why EI networks can fail to learn well, we didn't explore these directions in the experiments we present here. However, our results indicate that the network's spectral properties should be carefully considered whenever such architectural choices are made for modelling purposes.
Additionally, from a biological-hypothesis generating perspective, appropriately constraining the spectral properties of RNNs can help ameliorate problems of exploding/vanishing gradients [36], and so, an interesting question for neuroscience research into biologically plausible gradient estimation [38] is whether real brains, which largely obey Dale's Law, have spectral properties that are well-suited to gradient descent or not. Likewise, there are interesting questions regarding the balance between excitation and inhibition in RNNs (a major topic in neuroscience) that flow from the observation that such balance can impact the spectral properties of networks [11].
In summary, our work builds on tools from deep learning theory to provide guidance on how to construct RNNs that incorporate Dale's Law. We presented evidence that the reason for poor learning of ColEI networks and good learning of DANNs is, in large part, their respective spectral properties. Therefore, we hope our work serves as the basis and inspiration for an alternative initialization for ColEI networks that improves their learning. More generally, by providing an explanation for EI network learning differences, we have opened up new directions of research for RNNs in computational neuroscience.
This work was supported by a NSERC (Discovery Grant: RGPIN-2020-05105; Discovery Accelerator Supplement: RGPAS-2020-00031; Arthur B. McDonald Fellowship: 566355-2022) and CIFAR (Canada AI Chair; Learning in Machine and Brains Fellowship). This research was enabled in part by support provided by (Calcul Quebec) (https://www.calculquebec.ca/en/) and the Digital Research Alliance of Canada (https://alliancecan.ca/en). The authors acknowledge the material support of NVIDIA in the form of computational resources. JC: IVADO Postdoctoral Fellowship and the Canada First Research Excellence Fund. AG: Vanier Canada Graduate scholarship. PL wishes to convey his deepest gratitude to Ann Z.X. Huang for her support.
[37] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. Pmlr, 2013.
[38] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, pages 1-12, 2020.
[39] Paul J Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural networks, 1(4):339-356, 1988.