How connectivity structure shapes rich and lazy learning in neural circuits
Authors: Yuhan Helena Liu
Aristide Baratin
Jonathan Cornford
Stefan Mihalas
Eric Todd SheaBrown
Guillaume Lajoie
Venue: ICLR 2024 poster (2024)
Keywords: Computational neuroscience
recurrent neural networks
learning
connectivity structure
inductive bias
rich and lazy learning
deep learning theory
neural representations
Primary area: applications to neuroscience & cognitive science
TL;DR: We examine how initial connectivity structures influence the inclination toward different learning regimes in neural circuits.

Abstract: In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield a rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity generally has a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights — in particular their effective rank — influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally-driven initial connectivity in recurrent neural networks. Conversely, low-rank initialization biases learning towards richer learning. Importantly, however, as an exception to this rule, we find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research highlights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting.

Published as a conference paper at ICLR 2024
HOW CONNECTIVITY STRUCTURE SHAPES RICH AND LAZY LEARNING IN NEURAL CIRCUITS
Yuhan Helena Liu 1,3,3% Aristide Baratin", Jonathan Cornford Stefan Mihalas 12, Eric
Shea-Brown and Guillaume Laj0
University of Washington, Seattle, WA, USA
Allen Institute for Brain Science, Seattle WA, USA
3 Mila Quebee AI Institute. Montreal, QC, Canada "Samsung SAIT AI Lab, Montreal, QC, Canada
*MeGill University, Montreal, QC, Canada "Canada CIFAR AI Chair, CIFAR, Toronto, ON, Canada
Universitéé de Montréal, Montreal, QC, Canada "Correspondence: hyliu24@ @ uw.edu, g.lajoie @ uunoerrmmoooncccc
ABSTRACT
In theoretical neuroscience, recent work leverages deep learning tools to explore how some network attributes critically influence its learning dynamics. Notably, initial weight distributions with small (resp. large) variance may yield rich (resp. lazy) regime, where significant (resp. minor) changes to network states and representation are observed over the course of learning. However, in biology, neural circuit connectivity could exhibit a low-rank structure and therefore differs markedly from the random initializations generally used for these studies. As such, here we investigate how the structure of the initial weights in particular their effective rank influences the network learning regime. Through both empirical and theoretical analyses, we discover that high-rank initializations typically yield smaller network changes indicative of lazier learning, a finding we also confirm with experimentally -driiee initial connectivity in recurrent neural networks. Con- versely, low-rank initialization biases learning towards richer learning. Importantly, however, exception to this rule, find lazier learning can still occur with a low-rank initialization that aligns with task and data statistics. Our research high- lights the pivotal role of initial weight structures in shaping learning regimes, with implications for metabolic costs of plasticity and risks of catastrophic forgetting.
INTRODUCTION
Structural variations can significantly impact learning dynamics in theoretical neuroscience studies of animals. For instance, studies have revealed that specific neural connectivity patterns can facilitate faster learning of certain tasks (Braun al. [0022] Raman & O'Leary 2021 [iiard et al. [0055]
Canatar et al., 2021 Xie et al. 2022; Goudar et al 2023 Chang et al. 2023). In deep learning, structure, encompassing architecture and initial connectivity, crucially dictates learning speed and effectiveness [Richards et al., 2019: Zador
2019
Yang & Molano-Mazon
2021 Braun et al. (022).
key structural aspect the initial connectivity prior to training. Specifically, the initial connection weight magnitude can significantly bias learning dynamics, pushing them towards either rich or lazy regimes (Chizat et af., 2019 Flesch et al. (02TT. Lazy learning often induces minor changes in the network during the learning process. Such minimal adjustments are advantageous given that plasticity is metabolically costly (Mery & Kawecki (2005] Plaçais & Preat 2013), and significant changes in representations might lead to issues like catastrophic forgetting (McCloskey & Cohen 1989 Kirkpatrick et al. 2017). On the other hand, the rich learning regime can significantly adapt the network's internal representations task statistics, which can be advantageous for task feature acquisition and has implications for generalization (Flesch et al.) [02T]] George et al.) 2022). Most research on initial weight magnitude's 's role in learning dynamics has focused on random Gaussian

---

Published as a conference paper at ICLR 2024
or Uniform initializations (Woodworth al.) 2020 Flesch et al. 2021
|| Braun et al. 2022). These
patterns stand in contrast to the connectivity structures observed in biological neural circuits, which could exhibit more pronounced low-rank cigenstructure (Song et al. 2005). This divergence prompts a pivotal question: how does the initial weight structure, given a fixed initial weight magnitude, bias the learning regime?
This study examines how initial weight structure, particularly the effective rank, modulates the effective richness or laziness of task learning within the standard training regime. We note that rich and lazy learning regimes have well established meanings in deep learning theory. The latter being defined as a situation where the Neural Tangent Kernel (NTK) stays stationary during training, while the former refers to the case where the NTK changes. In this work, we slightly extend these definitions and introduce effective learning richness/laziness. Unlike the traditional definition, which is based upon initialization, effective learning richness/laziness is defined in terms of post-training adjustment measurements. From this perspective, a learning process is deemed effectively "lazy" if the measured NTK movement is small. For example, consider network whose initialization puts it in standard rich regime, but for a given task, its NTK moves very little during training. We define learning for this specific situation as effectively lazy. In other words, while the standard regime definition informs us (prior to training) whether the network can adapt significantly to task training or not, our "effective" definition lies in the post-training effects.
1.1 CONTRIBUTIONS
Our main contributions and findings can be summarized as follows:
Through theoretical derivation in two-layer feedforward linear network, we demonstrate that higher-rank initialization results in effectively lazier learning on average across tasks (Theorem T. We note that the emphasis of the theorem is on the expectation across tasks.
We validate our theoretical findings in recurrent neural networks (RNNs) through numerical experiments on well-known neuroscience tasks (Figure I and demonstrate the applicability to different initial connectivity structures extracted from neuroscience data (Figure (
We identify scenarios where certain low-rank initial weights still result in effectively lazier learning for specific tasks (Proposition) and Figure 3 We postulate that such patterns emerge when neural circuit is predisposed perhaps due evolutionary factors or post-development - to certain tasks, ingraining specific inductive biases in neural circuits.
1.2
RELATED WORKS
An extended discussion on related works can also be found in Appendix A
Theoretical Foundations of Neural Network Regimes and Implications for Neural Circuits:
The deep learning community has made tremendous strides in developing theoretical groundings
for artificial neural networks (Advani et al. 2020; Jacot et al. 2018: [Alemohammad et al. 2020
Agarwala et al. 2022 Atanasov al. 2021 Azulay al. 2021 Emaai al. (021). 1). A focal point
is the 'rich' and 'layy" learning regimes dichotomy, which have distinct impacts on representation
and generalization (Chizat et al. 2019 Flesch et al. 2021 Geiger et al. 2020 George al. 2022 Ghorbani et al. 2020 Woodworth et al. 22200] Paccolat et al. 202 N et al. [022] 2 HaoChen cct a [202 Flesch et al. 2023). The 'lazy" regime results in minimal weight changes, while the 'rich' regime fosters task- adaptations. The transition between these is influenced by various factors, including initial weight scale and network width (Chizat et al. 2019 Geiger al., 2020).
Deep learning theories increasingly inform studies of biological neural network learning dynamics
(Bordelon & Pehlevan 2022 [iu et al. 2022a Braun et al. 2022 Ghosh al. 2023 Saxe et al. 2019)
Farrell et al. [2222] Papyan et al., 2020 Tishby & Zaslavsky (015). For the rich/lazy regime theory, the existence of diverse learning regimes in neural systems is evident through the resource-intensive
plasticity-drivee transformations prevalent in developmental phases, followed by more subdued adjustments (Lohmann & Kessels 2014), and previous investigations characterized neural network behaviors under distinct regimes (Bordelon & Pehlevan 2022 (Schuessler et al.) 2023) and discerning
which mode yields solutions mimicking neural data (Flesch et 2021). Our work extends these studies by examining how initial weight structures affect learning.
2

---

Published as a conference paper at ICLR 2024
Neural circuit initialization, connectivity patterns and learning: Extensive research has explored
the influence of various random initializations on deep network learning (Saxe et al., 2013: Bahri
et al. 2020 Glorot & Bengio 2010 He et al. 2015: Arora al. 2019). The literature predominantly focuses on random initialization, but actual neural structures exhibit markedly different connectivity patterns, such as Dale's s law and enriched cell-type-specife connectivity motifs (Rajan & Abbott
2006; [psen & Peterson 2020; Harris et al.) [022 Dahmen et al. 2200;; Aljadeff et al. 2015)) Motivated by existing evidence of low-rankedness
the brain (Thibeault et al. 2024) and the
overrepresentation of local motifs in neural circuits (Song et al. (005), which could be indicative of low-rank structures due their influence on the eigenspectrum (Dahmen et al. 2020, Shao & Ostojic 2023), our study explores the impact of connectivity effective rank on learning regimes. This focus is driven by the plausible presence of such low-rank structures in the brain, potentially revealed through these local motifs. With emerging connectivity data (Campagnola et al.] [002]] MICrONS Consortium et al., 2021, Dorkenwald et al. 2022; Winnubst et 2019;; Scheffer et al. 2020). future work is poised to encompass rich additional features of connectivity.
SETUP AND THEORETICAL FINDINGS
2.1
RNN SETUP
We examine recurrent neural networks (RNNs) because they are commonly adopted for modeling
neural circuits (Barak] 2017 Song et 2016). We consider RNN with Nin input units, N hidden
units and Nout readout units (Figure A). The update formula for h E IR (the hidden state at time
governed by (Ehrlich al. 2021 Molano-Mazon al. 2022):
h++1 ph. (1 (W,, Wa.s),
where an exponential Euler approximation made with p= edt-dt/Tm E denoting the leak factor for simulation time step di and Tm denoting the membrane time constant; f(.): RV -> IRN is the activation function, for which we use ReLU'; W₁, E IR NNN (resp. W2 E R NNNN)) is the recurrent (resp. input) weight matrix and 2; E IR-Vin is the input at time step t. Readout ÿt R. with readout weights w E R. Nout x is defined as
i S(h)))
The objective is minimize scalar loss L R, for which we the cross-entropy loss for classification tasks and mean squared error for regression tasks. L is minimized by updating the parameters using variants of gradient descent:
AW VW
for learning rate 7 E R and [Wh W₂ TT) NN. +No.. contains all the trainable parameters. Details of parameter settings be found in Appendix
2.2 EFFECTIVE LAZINESS MEASURES
As mentioned above, we introduce effective richness and laziness, with effectively lazier (resp. richer) learning corresponding less (resp. greater) network change over the course of learning. To quantify network change, we adopt the following three measures that have been used previously (George et al.) 2022). We note that these measures can be sensitive to other architectural aspects that bias learning regimes, such as network width, so throughout we hold these variables constant when making the comparisons.
Weight change norm quantifies the vector norm of change in W. Effectively lazier learning should result in lower weight change norm, and it is quantified as:
|W = |W( (' 1.
(4)
where || || ||s; (0 (resp. W(f) are the weights before (resp. after) training.
Representation alignment (RA) quantifies the directional change in a representational similarity matrix (RSM) before and after training. RSM focuses on the similarity between how pairs
3

---

Published as a conference paper at ICLR 2024
of input are represented by computing the Gram matrix R of last step hidden activity. Greater representation alignment indicates effectively lazier learning in the network, and it is obtained by
Tr(B
RA R = |R RO
where :== 1HTH.
(5)
where H. E NN is the hidden activity at the last time step; R and R()) E Rmxm are the initial and final RSM, respectively; m is the batch size.
Tangent kernel alignment (KA) quantifies the directional change in the neural tangent kernel (NTK) before and after training; effectively lazier learning should result in higher tangent kernel alignment. The NTK computes the Gram matrix K of the output gradient. Greater tangent kernel alignment points to effectively lazier learning, and it is obtained by
Tr(KU)K()) k())
KA(KO), K(O)= = |KU K
where K == Vwi Vw9
(6)
where k(O) and K(O mm (for the Nout case) denote the initial and final NTK, respectively,
2.3 THEORETICAL FINDINGS
This subsection derives the theoretical impact of initial weight effective rank on tangent kernel alignment. First, Theorem focuses on task-agnostic settings, treating task definition as random variables and computing the expected tangent kernel alignment across tasks. With some assumptions, tangent kernel alignment is maximized when the initial weight singular values are distributed across all dimensions (i.e. high-rank initialization).
In this section, our theoretical results are framed in a simplified feedforward setting, as we use a two- layer network with linear activations. However, return to RNNs (Eq. [ for the rest of the paper, and verify the generality of our theoretical findings with numerical experiments for both feedforward and recurrent architectures. Our choice is motivated by the need for theoretical tractability. While research on RNN learning in the NTK regime exists (Yang] 2020 Alemohammad et al.) [202 Emami al., 2021). l) we are not aware of any studies featuring the final converged NTK that could serve as a basis for our comparison of the initial and final kernel. Consequently, we have chosen to focus on RNNs for neural circuit modeling and employ linear feedforward networks for theoretical derivations, a strategy also adopted by Farrell et al. (2022); numerous other studies, including Saxe et al. (219)) (Atanasov et al. 202T), (Arora et al., ), and (Braun al.) (002), have similarly concentrated on extracting theoretical insights from linear feedforward networks.
For a two-layer linear network with input data X E R W1 E Nxd and W2 RIIN as weights for layers and 2, respectively, the NTK throughout training. K, is:
K = (WTTWTWW + 2
Without the loss of generality, suppose the output target Y E Rixm is generated from a linear teacher network as Y = x, for some Gaussian vector 3 E Rd, with 3 ~ (N0. 1/d).
Theorem 1. (Informal) Consider the network above with its corresponding NTK in Eq. trained under MSE loss with small initialization and whitened data. The expected kernel alignment across tasks is maximized with high-rank initialization, i.e. the singular values of W( are distributed across all dimensions. (Formal statement and proof are in Appendix B
The intuition Theorem result is that, when two random vectors are drawn in high-dimensional spaces, corresponding to the low-rank initial network and the task, the probability of them being nearly orthogonal is very high; this then necessitates greater movement to eventually learn the task direction. We emphasize again that Theorem is task-agnostic, i.e. it focuses the expected tangent kernel alignment across onput-output definitions. This is in contrast to task-specific settings (e.g- Woodworth al. (2020) that focus given task. In such task-specific settings, certain low-rank initializations can in fact lead to lazier learning. The following proposition predicts that if the task structure known, low-rank initialization that is already aligned with the task statisties (input/output covariance) can lead to kernel alignment. We revisit this proposition again in Figure 3
We remark that initializing this way still have high initial error because of randomized w!
4

---

Published as a conference paper at ICLR 2024
Proposition 1. (Informal) Following the setup and assumptions in Theorem 7 rank-1 initializations of the form w( a a /||3||| 0
leads a high tangent kernel alignment. (Formal
statement and proof are in Appendix B
Above, state technical results in terms of one metric of the effective laziness of learning based on the NTK; our proof in Appendix B easily extends also to the representation alignment metric. The impact weight change is also assessed in Appendix Proposition This is in line with simulations with RNNs, which will show similar trends for three of the metrics introduced in Section 2.2
3
SIMULATION RESULTS
A
RNN model for cognitive task learning
B
Weight change ( Effectively Lazier)
Rep. alignment ( Effectively Lazier)
Kernel alignment (f) Effectively Lazier)
Sp spcaman's fho 0.7 p-cal
Spearman's 0.87 p-tal
Spooman's mo 36 soal
Before training
After training
Task=
Am30
Degert booee aa prone Sod
Target
Target
10
107
Output
Output
Intal phtrork
9
Spearmants the
Spearmants
Spearments p vol 1e
Task CKT
piva 5.6c 12
vo 2.30 15.
Learning,
AW
M
Intia
Weight
Weight
Scearman's
pivll - 0,00042.
Sptarman's 0.60
pivil 1621.
Spearmants
pivil 1.30
Win
Wini
Task=DMS DMS
IAWES H&W
Input
Input
In wiiowwcite.oom care
10rk
Werrerrrnn sggrrert
Sarve dagenor.
p-vol 3.le-12
p-val = $1-005
paval 8 9.36.07
How does W impact effective learning laziness (e.g. ||AW]]).
Task=aMNISS
Sio
given fixed ||W*|||?
Initial weight rank
Initial weight rank
Initial weight rank
Figure 1: Low-rank initial recurrent weights, generated using SVD, lead to greater changes (or effectively richer learning) in the recurrent neural network. A) Schematic of RNN training setup. B) Mcasurements of effective richness vs laziness of learning (metrics as defined in Section 2.2). for RNN trained on several cognitive tasks in Neurogym (Molano- Mazonn et al.) 2022) as well as the sequential MNIST task (sMNIST). For details SVD weight creation, points were used for sMNIST due to computational time. Each dot represents single le training run, with each using a different random initialization (10 runs total for each setting).
Appendix C Fewer rank
In this section we empirically illustrate and verify our main theoretical results, which are: (1) on average, high-rank initialization leads to effectively lazier learning (Theorem [] (2) it still possible for certain low-rank initializations that are already aligned to the task statistics to achieve effectively lazier learning (Proposition I).
As a proof-of-concept, we start in Figure [ with low-rank initialization in RNNs by truncating an initial Gaussian random matrix via Singular Value Decomposition (SVD), which enables us to precisely control the rank, and rescale to ensure that the comparison across the same weight magnitude (Schuessler et al.. 2020). Additionally, all comparisons were made after training was completed, and all these training sessions achieved comparable losses. For our investigations, we applied this initialization scheme across a variety of cognitive tasks - including two-alternative forced choice (2AF), delayed-match to-sample (DMS), context- context-dependent decision-making (CXT) tasks implemented with Neurogym (Molano-Mazze et al. (022) and the well-known machine learning benchmark sequential MNIST (sMNIST). Figure indicates that low-rank initial weights result in effectively richer learning and greater network changes.
Impact on effective laziness by low-rank initialization via SVD in RNNs:
5

---

Published as a conference paper at ICLR 2024
A Block-specific
Effective rank
Weight change (+ Effectively Lazier)
Rep. alignment (* Effectively Lazier)
Kernel alignment (*) Effectively Lazier)
Relttve Bank
variance
ock
null
0,018
voll 0.0011
val 1.3c 06
Elgenspectrum
Egga mag (relatuve)
Block
Post
"Srgert
Eival Index trelative)
thoce
B
Pio settt
EP Rank
From
9.
Dval 2e
05
Tegger same algreeet
val.
Allen Institute
2,00
EM Data
Eloval mag (reiative)
hull
0.5
1.0
Eigval Irelative)
Dale's law
Raladve Rank
0.5
poal - 5.6e-10
Porortaaee wwworttiir--oeoe strret
p-val i 0.0042
p-vul - 0,0002
0.6-
nu
cxr mag [relative)
OE cells
Cellss
1.0
Egval (relative)
D Chain Motif Overrepresentation
Relotive EIT Rank
9.5
0,02
chain
Egral mag (reletuv))
chain
0.025
chen
121
Cher
mal
Eival Index (relative)
Figure 2: Low-rank initial weight structures, inspired by biological examples, lead to effectively richer learning. We present the eigenspectrum and the relative effective rank of connectivity in A) structures with cell-type-specifie statistics, B) structures derived from EM data, C) structures obeying Dale's law, and D) structures with over-represintttion of chain motifs; we also present the effective learning laziness for networks initialized with these connectivity structures. These structures exhibit lower effective rank compared standard random Gaussian initialization (null). We plotted the magnitude of the eigenvalues (Eigval mag) - scaled by the dominant eigenvalue's 's magnitude against their indices normalized by the network size N (Eigval index). We apply the effective laziness measures described in Section 2.2 to compare the effective laziness of experimentalll-drivve initial connectivity versus standard random Gaussian initialization (null). See Appendix [i details on network initialization. The boxplots are generated from 10 independent runs with different initialization seeds. Due to space constraints, we include only the 2AF task here, but Appendix Figures 5 and show that similar trends hold for the DMS and CXT tasks.
These numerical trends are line with Theorem I which focused on an idealized setting of a two- layer linear network with numerical results in Appendix Figure 44). We also demonstrated this trend for non-idealized feedforward setting in Appendix Figure and more explorations in feedforward settings and across a broader range of architecture left for future exploration due our focus on RNNs. In the Appendix, we show the main trends observed in Figure | also hold for Uniform initialization (Figure 7 soft initial weight rank (Figure [.. various network sizes (Figure D.. learning rates (Figure 10]. gans (Figure []p and finer time step dt (Figure 12). We note that, in addition to fixing the weight magnitude across comparisons, the dynamical regime might also confound learning regimes. A common method for controlling the dynamical regime is through the leading weight cigenvalue, which affects the top Lyapunov exponent. Controlling in this manner led to similar trends (Appendix Figure 13). Investigating the relationship between learning regimes and various concepts of dynamical regimes further is a promising direction for future work. Moreover, since our emphasis is on the effective learning regime, which based on post-training changes, concentrated on the laziness measures computed from networks after training, rather than during the learning process. However, we also tracked the alignment with the initial kernel and task kernel alignment during

---

Published as a conference paper at ICLR 2024
A
Two layer linear net
B Aligned initialization,
C Aligned initialization,
featurel
two layer linear net
RNN
feature
Weight change
(,Effctively Lazion
Tosk
feature
11
2.
RNN())
1.5
W
W
7.5
W
Kernel alignment Rep. aliinment
(" Effectirely Lazier)
Targun sand algemeen Nepesseraaiin tannaeetttt
5.0
y
1.0
11.5
2.5
Teacher net
Student net
0.5
random nerie- in
0.8
107
32
partially
Tangort samel algemennt Nonesomaatinn pemmrt
Condition number
Anisotropic features
107
10:
Cond Uon number
( Efffecctivyly L Lazier)
0.5
candom raneed
aigned
partia al
10
10
Condition
107
10a
20'
1.0
Cand tion
Increasing condition number
Aligned W, x
0.5
Dominant feature Other feature
0.0
Partial W 12)
Cundition number
Figure Low-rank initializations can still achieve high alignment for specific tasks (see Proposition [] A) The student-teache two-layer linear network setup as described in Section 2.3 but with feature anisotropy controlled by a feature modulation matrix F, i.e. Fx. The condition number of F dictates the relative feature strength. We set the top half of the singular values of F are set to K, while the bottom half are set to 1, where x represents the condition number of F. B) The aligned initialization (green) is achieved by setting W1 described in Proposition | (with 3 = w F. w is as illustrated), so that the initialization aligns with the task statistics. The partial alignment (blue) mirrors the aligned case, but F is substituted with its rdnk-(d/22) truncation, causing the network to align only with the dominant features. We observe that a considerably higher alignment can be achieved when the initialization aligns solely with the dominant features, especially when the relative strength of these dominant features is high. C) The analysis from B) is replicated for RNNs learning the sMNIST task. As the ground truth network function is elusive, we use teacher network with pre- trained weights. Once again, we replace F with its rank- (d/)) truncation for partial alignment. Details on the input/output definitions and initializations, as well as other simulation specifics, are available in Appendix C We note that in all scenarios presented here, the initial errors are high since the readout weights are initialized randomly, rendering it a valid learning problem.
training (Appendix Figure 14). We also examined how the kernel's effective rank evolves throughout the training period (Appendix Figure(15
To establish closer connection with biological neural circuits, we have tested our predictions on low-rank initialization using variety of biologically motivated structures capable of resulting in low-rank connectivity. Here are some of the examples: (A) connectivity with cell-type-specifee statistics (Aljadeff et al. 2015), where each block in the weight matrix corresponds to the connections between neurons of two distinct cell types, with the variance of these connections differing from one block to another. In terms of block-specific connectivity statistics, there are infinite possibilities for defining the blocks, each resulting in unique eigenspectrum. For the example provided here, we adopted the setup from Figure S3 in AAjjadeff et al. (2015), with parameters set as a = 0.02, 7 = 10, and - e 0.8; these correspond to the fraction of hyperexcitable neurons, gain of hyperexcitable connections and gain of the rest, respectively. We follow this particular setup because it has been demonstrated to create outlier leading eigenvalue, thereby reducing the effective rank. We also consider (B) connectivity matrix derived from the electron microscopy (EM) data (Allen Institute| (002)), where the synaptic connections between individual neurons meticulously mapped to create detailed
Low-rank initialization via biologically motivated connectivity in RNNs:
7

---

Published as a conference paper at ICLR 2024
and comprehensive representation of neural circuits. Also, we consider (C) connectivity obeying Dale's ' law, where each neuron is either excitatory or inhibitory, meaning it can only send out one type of signal either increasing decreasing the activity of connected neurons a principle inspired by the way neurons behave in biological systems (Song et al. 2005). Additionally, (D) the over-representation of certain localized connectivity patterns (or network motifs) such as the chain motif, where two cells are connected via a third intermediary cell - creates outliers in the weight eigenspectrum, subsequently lowering the effective rank (Zhao al. 2011 Hu et al. 2018 Dahmen
et al. 2020). Details of these initial connectivity structures are provided in Appendix
As illustrated in Figure 2 these connectivity structures, motivated by known features of biological neural networks, exhibit a lower effective rank compared to standard random Gaussian initialization, thereby serving as natural testbeds for our theoretical predictions. To quantify (relative) effective rank, we used (E; [Ail)((Aa which indicates the fraction of eigenvalues on the order of the dominant one and captures the (scaled) area under the curve of the eigenspectrum plots. We also tried effective
rank based on singular values, i.e. (E: [s4] ([[(()(( [.V), in Appendix Figure 16 and observed similar trends. Importantly, Figure 2 show that these different low-rank biologically motivated structures can lead to effectively richer learning compared to the standard random Gaussian initialization. This finding supports our overarching prediction, that lower rank initial weights leads to effectively richer learning. We note that to test our theoretical predictions based on gradient-descent learning without specific constraints on the solutions, the structures are enforced only at initialization and not constrained during training. In Appendix Figure 17 we also constrained Dale's Law throughout training and found similar trends.
Low-rank initialization aligned with task statistics: These simulations may be considered to be within our task-agnostic framework. That is, we have chosen a "random" battery of tasks that is not directly matched to the initial network connectivity structures. Thus, our findings that lower rank initializations lead to richer leamning are expected from our theoretical prediction on the task-averaged alignment (Theorem TT. rather than something task-specific. However, Proposition T also predicts that low-rank initialization can lead to lazy learning if the initialization is already aligned to the task structure. To test this, we observe in Figure 3 that a considerably higher alignment can be achieved when the initialization aligns solely with the dominant task features, especially when the relative strength of these dominant features is high. We postulate that such alignment may occur in biological settings if the circuit has evolved to preferentially learn specific tasks.
4 DISCUSSION
Our investigation casts light on the nuanced influence of initial weight effective rank on learning dynamics. Anchored by Theorem I. our theoretical findings underscore that high-rank random initialization generally facilitates effectively lazier learning on average across tasks. This focus on the expectation across tasks can provide insights into the circuit's ss flexibility in learning across broad range of tasks as well as predict the effective learning regime when the task structure is uncertain. However, certain low-rank initial weights, when naturally predisposed to specific tasks, may lead to effectively lazier learning, suggesting an interesting interplay between evolutionary or developmental biases and learning dynamics (Proposition D. Our numerical experiments on RNNs further validate these theoretical findings illustrating the impact of initial rank in diverse settings.
Potential implications to neuroscience: We investigate the impact of effective weight rank on learning regimes due its relevance in neuroscience. Learning regimes reflect the extent of change through learning, implicating metabolic costs and catastrophic forgetting (McCloskey & Cohen 1989: Plaçais& & Preat 2013 Mery & Kawecki, 2005). The presence of different learning regimes is demonstrated in neural systems, since during developmental phases where neural circuits undergo extensive, plasticity-dii driven transformations. In contrast, mature neural circuits exhibit more subtle synaptic adjustments (Lohmann & Kessels, 2014). We hypothesize that circuit's 's task-specific alignment might be established either evolutionarily or during early development. The specialization of neural circuits, such as ventral versus dorsal (Bakhtiari et al.) (0221). may arise from engaging in tasks with similar computational demands. Conversely, circuits with high-rank structures may be less specialized, handling wider array of tasks. Our framework could be used to compare connectivities across brain regions and species in order to predict their function and flexibility, assessing their functional specialization based effective connectivity rank. Additionally, our framework predicts
8

---

Published as a conference paper at ICLR 2024
that connectivity rank will affect the degree of change in neural activity during the learning of new tasks. This hypothesis could be tested through BCI experiments, as shown in [Saddteer et al. (2014) and Golub et al. (2018), to explore how learning dynamics vary with connectivity rank.
Regarding deep learning, low-rank initialization is not a common practice, yet adaptations like low-rank updates have gained popularity in training large models (Hu et al., (021). LoRA, the study cited, concentrates on parameter updates rather than initializations, but understanding how update rank affects learning regimes is crucial. Our results offer a starting point for further exploration in this area. Although different rank initializations are less explored, with some exceptions like Vodrahalli al. (2022), our findings suggest that this area should receive more attention due to potential effects on learning regimes and, consequently, on generalization (George et al., (022).
Limitations and future directions: Our study predominantly focused on the weight (effective) rank, leaving the exploration of other facets of weight on the effective learning regime as an open avenue. Also, the ramifications of effective learning regimes on learning speed - given the known results on kernel alignment and ease of learning (Bartlett et al. 2021) and present mixed findings in the existing literature (Flesch et al. 2021 George et al. 2022 warrant further exploration.
Expanding the scope of our study calls for examining a wider variety of tasks, neural network architectures, and learning rules. Although our work is based on the backpropagation learning rule, its implications for biologically plausible learning rules remain unexplored. Our primary criterion for selecting tasks was their relevance to neuroscience, aligning with our main objectives. However, given the diverse range of tasks performed by various species, future research could benefit from exploring a more extensive array of tasks. Exploring more complex neuroscience tasks, such as those in Mod-Cog (Khona et al. 2023), could provide valuable insights. On that note, we tested the pattern generation task from Bellec et al. (2020). a neuroscience task differing in structure from the Neurogym tasks, and observed similar trends (refer to Appendix Figure 18).
Additionally, we ensured the consistency of outcomes against factors like width, learning rate, and initial gain (see Appendix[D] but other factors such as dynamical regime and noise (HaoChen et al.) 202T) remain underexamined. On that note, the study's focus on RNNs with finite task duration prompts further investigation into the implications for tasks with extended time steps and how conclusions for feedforward network depth (Xiao et al. [020 Seleznova & Kutyniok (022) translate to RNN sequence length. Examining several mechanisms at once is beyond the scope of one paper, but our theoretical work constitutes the foundation for future investigations.
Moreover, it is crucial to further explore the neuroscientific implications of effective learning regimes, as well as their diverse impacts aspects such representation, including kernel-task alignment (see Appendix Figure 14), and generalization capabilities (Flesch et al. 2021 George et al,, 2022] Schuessler al. 2023 Our current study did not delve into how initial weight rank affects these facets of learning, representing an essential future direction in connecting weight rank to these theoretical implications in both biological and artificial neural networks.
Furthermore, while there exists evidence for low-rankedness in the brain (Thibeault et al., 2024), the extent which the brain uses low-rank structures remains an open question, especially as neural circuit structures can vary across regions and species. While local connectivity statistics (Song al. 2005) can offer some predictive insight into the global low-rank structure, this relationship is not always immediately apparent (Shao & Ostojic 2023). Our theoretical results contribute to understanding the role of connectivity rank the brain by linking effective connectivity rank with learning dynamics.
Lastly, we have primarily examined low-rank tasks and there remains unexplored terrain regarding the interplay between the number of task classes and weight rank, which is pivotal to uncovering more precise relationship between the effective learning regime and the initial weight rank (Dubreuil et al., 2022, Gao et al. (011)). Overall, this dynamic area of learning regimes is ripe for many explorations, integrating numerous factors; our work contributes to this exciting area with new tools.
5 ACKNOWLEDGEMEN
We thank Andrew Saxe, Stefano Recanatesi, Kyle Aitken and Dana Mastrovito for insightful discus- sions and helpful feedback. This research was supported by NSERC PGS-D (Y.H.L.); FRONT B2X

---

Published as a conference paper at ICLR 2024
(Y.H.L.); Pearson Fellowship (Y.H.L.):: NSF AccelNet IN-BIC program, Grant No. 2019970 AM02 (Y.H...):: NIH BRAIN, Grant No. R01 IRFIDA055669 (Y.H..,, E.S.B., S.M.;; Mitacs Glob- alink Research Award (Y.HL.)); IVADO Postdoctoral Fellowship (J.C); the Canada First Research Excellence Fund (J.C.); NSERC Discovery Grant RGPIN-2018 201220881111111 (G.L); Canada Research Chair in Neural Computations and Interfacing (G.L.;; Canada CIFAR AI Chair program (G.I.). We also thank the Allen Institute founder, Paul G. Allen, for his vision, encouragement, and support.
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of general- ization error in neural networks. Neural Nenworks, 132:428 446, 2020.
Atish Agarwala, Fabian Pedregosa, and Jeffrey Pennington. Second-order regression models exhibit progressive sharpening to the edge of stability. arXiv preprint arXiv:2210.04860, 2022.
Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The recurrent neural tangent kernel. arXiv preprini arXiv:2006. 10246, 2020.
Johnatan Aljadeff, Merav Stern, and Tatyana Sharpee. Transition to chaos in random networks with cell-type-specific connectivity. Physical review letters, 1 100
Allen Institute. Allen institute for brain science, 2023, http://tpppppottta...aaaaaa maaeraiinmma....mmmm mappnnmmpppogg.... org/
Raman Arora, Sanjeev Arora, Joan Bruna, Naday Cohen, Simon Du, Rong Ge, Suriya Gunasekar, Chi Jin, Jason Lee, Tengyu Ma, et al. Theory of deep learning, 2019.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan, Neural networks as kernel learners: The silent alignment effect. arXiv preprint wxXiv:2111.00034 2021.
Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake E Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In International Conference on Machine Learning. pp. 468 467-477. PMLR, 2021.
Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11:501-528, 2020.
Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. Advances in Neural Information Processing Systems, 34: 25164-25178, 2021.
Omri Barak. Recurrent neural networks as versatile tools of neuroscience research. Current opinion in neurobiology, 46: 1,6,,6,22 2017.
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In International Conference on Artificial Intelligence and Statisties, pp. 2269-2277 PMLR, 2021.
Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica, 30:87-201, 2021.
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications, 11(1):3625, 2020.
Blake Bordelon and Cengiz Pehlevan.
The influence of learning rule on representation dynamics in 2022.
wide neural networks. arXiv preprint
Lukas Braun, Clémentine Dominé, James Fitzgerald, and Andrew Saxe. Exact learning dynamics of deep linear networks with prior knowledge. Advances in Neural Information Processing Systems, 35:6615-6629, 2022.
10

---

Published as a conference paper at ICLR 2024
Luke Campagnola, Stephanie C Seeman, Thomas Chartrand, Lisa Kim, Alex Hoggarth, Clare Gamlin, Shinya Ito, Jessica Trinh, Pasha Davoudian, Cristina Radaelli, et al. Local connectivity and synaptic dynamics in mouse and human neocortex. Science, 375(65855ccabj88611 2022.
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model align- ment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12(1))2914, 2021.
Joanna C Chang, Matthew G Perich, Lee E Miller, Juan A Gallego, and Claudia Clopath. De novo motor learning creates structure in neural activity space that shapes adaptation. bioRxiv, pp. 2023-05, 2023.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
David Dahmen, Stefano Recanatesi, Gabriel K Ocker, Xiaoxuan Jia, Moritz Helias, and Eric Shea- Brown. Strong coupling and local control of dimensionality across brain areas. Biorxiv, pp- 2020-11, 2020.
Sigurd Diederich and Manfred Opper. Learning of correlated patterns in spin-glass networks by local learning rules. Physical review letters, 58(9):949, 1987.
Sven Dorkenwald, Claire E McKellar, Thomas Macrina, Nico Kemnitz, Kisuk Lee, Ran Lu, Jingpeng Wu, Sergiy Popovych, Eric Mitchell, Barak Nehoran, et al. Flywire: online community for whole-brain connectomics. Nature methods, 19(1):199-228, 2022.
Alexis Dubreuil, Adrian Valente, Manuel Beiran, Francesca Mastrogiuseppe, and Srdjan Ostojic. The role of population structure in computations through neural dynamics. Nature neuroscience, 25(6): 783-794, 2022.
Daniel B Ehrlich, Jasmine Stone, David Brandfonbrener, Alexander Atanasov, and John D Murray. Psychmmn: An accessible and flexible python package for training recurrent neural network models on cognitive tasks. Eneuro, 8(1), 2021.
Melikasadat Emami, Mojtaba Sahraee- Ardakan, Parthe Pandit, Sundeep Rangan, and Alyson K Fletcher. Implicit bias of linear mns. In International Conference on Machine Learning, pp- 2982-2992. PMLR, 2021.
Matthew Farrell, Stefano Recanatesi, Timothy Moore, Guillaume Lajoie, and Eric Shea-Brown Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion. Nature Machine Intelligence, 4(6):564-573, 2022.
Timo Flesch, Keno Juechems, Tsvetomira Dumbalska, Andrew Saxe, and Christopher Summerfield. Rich and lazy learning of task representations in brains and neural networks. BioRxiv, pp. 2021-04, 2021.
Timo Flesch, Andrew Saxe, and Christopher Summerfield. Continual task learning in natural and artificial agents. Trends in Neurosciences, 46(3):199 2023.
Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. BioRxiv, pp- 214262, 2017.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020 (11): (133:113301, 2020.
Thomas George, Guillaume Lajoie, and Aristide Baratin. Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty. TMLR, 2022.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33: 14820-14830) 2020.
11

---

Published as a conference paper at ICLR 2024
Arna Ghosh, Yuhan Helena Liu, Guillaume Lajoie, Konrad Kording, and Blake Aaron Richards. How gradient estimator variance and bias impact learning in neural networks. In The Eleventh International Conference Learning Representations, 2023.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings. 2010.
Matthew D Golub, Patrick T Sadtler, Emily R Oby. Kristin M Quick, Stephen Ryu, Elizabeth C Tyler-Kabaras Aaron P Batista, Steven M Chase, and Byron M Yu. Learning by neural reassociation. Nature neuroscience, 21(4):607-616 2018.
Vishwa Goudar, Barbara Peysakhovich, David J Freedman, Elizabeth Buffalo, and Xiao-Jing Wang. Schema formation in a neural population subspace underlies learning-to-learn in flexible sensorimotor problem-solving. Nature Neuroscience, 26(5):879-890 2023.
Will Greedy, Heng Wei Zhu, Joseph Pemberton, Jack Mellor, and Rui Ponte Costa. Single-phast deep learning in cortico-cor cortico-cortical networks. Advances in Neural Information Processing Systems, 35:24213-2422 2022.
Jeff Z HaoChen, Colin Wei, Jason Lee, and Tengyu Ma. Shape matters: Understanding the implicit bias of the noise covariance. Conference on Learning Theory, pp. 2315-2357. PMLR, 2021.
Isabelle D Harris, Hamish Meffin, Anthony N Burkitt, and Andre DH Peterson. Eigenvalue spectral properties of sparse random matrices obeying dale's law. arXiv preprint arXiv:2212.015549 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 0rXiv:2/06.09685, 2021.
Yu Hu, Steven L Brunton, Nicholas Cain, Stefan Mihalas, Nathan Kutz, and Eric Shea-Brown. Feedback through graph motifs relates structure and function in complex networks. Physical Review E, 98(6):06332,, 2018.
Jesper R Ipsen and Andre DH Peterson. Consequences of dale's law on the stability-complexity relationship of random neural networks. Physical Review E, 101(5):052412, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Mikail Khona, Sarthak Chandra, Joy Ma, and lla R Fiete. Winning the lottery with neural connectivity constraints: Faster learning across cognitive tasks with spatially constrained sparse mns. Neural Computation, 35(1 1850--)8888,1, 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Grabska-Barwinskar Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Axel Laborieux and Friedemann Zenke. Holomorphic equilibrium propagation computes exact gradients through finite size oscillations. arXiv preprint arXiv:2209.005330 2022.
Yann LeCun. The mnist database of handwritten digits. hutp://yann. lecun. con/exdb/mmist/, 1998.
Timothy P Lillicrap. Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backprop- agation and the brain. Nature Reviews Neuroscience, 21(63333--346, 2020.
12

---

Published as a conference paper at ICLR 2024
Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric
and Uygar Sümbül. Cell- type-specific neuromodulation guides synaptic credit assigmment in a spiking neural network. Proceedings of the National Academy of Sciences. 118(51):e211 18211111111111111 18, 2021.
Yuhan Helena Liu, Arma Ghosh, Blake Richards, Eric Shea-Brow Shea-Brown, and Guillaume Lajoie. Beyond accuracy: generalization properties of bio- dio-plausible temporal credit assignment rules. Advances in Neural Information Processing Systems, 35:23077-23099 2022a.
Yuhan Helena Liu, Stephen Smith, Stefan Mihalas, Eric Shea-Brou Bhea-Brown, and Uygar Sümbül. Biologically-plaustee backpropagation through arbitrary timespans via local neuromodulators. arXiv preprint arXiv:2206.0/ 233,,,33,, 2022b.
Christian Lohmann and Helmut W Kessels. The developmental stages synaptic plasticity. The Journal of physiology, 592(1):13-31, 2014.
Owen Marschall, Kyunghyun Cho, and Cristina Savin. A unified framework of online learning algorithms for training recurrent neural networks. The Journal of Machine Learning Research, 21 (0::5320-5353, 2020.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Frederic Mery and Tadeusz J Kawecki. A cost of long-term memory in drosophila. Science, 308 (5725):1148-1144 148-1148, 2005.
Alexander Meulemans, Nicolas Zucchet, Seijin Kobayashi, Johannes Von Oswald, and João Sacra- mento. The least-control principle for local learning at equilibrium. Advances in Neural Information Processing Systems, 35:33603-33617, 2022.
MICrONS Consortium, J Alexander Bae, Mahaly Baptiste, Caitlyn A Bishop, Agnes I. Bodor, Derrick Brittain, JoAnn Buchanan, Daniel J Bumbarger, Manuel Castro, Brendan Celii, et al. Functional connectomics spanning multiple areas of mouse visual cortex. BioRxiv, pp. 2021-07, 2021.
Manuel Molano-Mazon, Joao Barbosa, Jordi Cistor-Ciurana, Marta Fradera, Ru- Yuan Zhang, Jeremy Forest, Jorge del Pozo Lerida, Li Ji-An, Christopher J Cueva, Jaime de la Rocha, et al. Neurogym: An open resource for developing and sharing neuroscience tasks. 2022.
James M Murray, Local online learning in recurrent networks with random feedback. Elife, 8:e43299. 2019.
Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference Machine Learning, pp. (6270-16295.. PMLR, 2022.
Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression of invariant manifolds in neural networks. Journal of Statistical Mechanics: Theory and Experiment, (024(4)044 2021.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652 246663, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-peeformance deep learning library. Advances in neural information processing systems, 32, 2019.
Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A Richards, and Richard Naud. Burst- dependent synaptic plasticity can coordinate learning in hierarchical circuits. Nature neuroscience, pp. 1-10, 2021.
13

---

Published as a conference paper at ICLR 2024
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Pierre-Y Yves Plaçais and Thomas Preat To favor survival under food shortage, the brain disables costly memory. Science, 339(6118):440 442, 2013.
Roman Pogodin, Jonathan Cornford, Arma Ghosh, Gauthier Gidel, Guillaume Lajoie, and Blake Richards. Synaptic weight distributions depend on the geometry of plasticity. arXiv preprint arXiv:2305. 19394, 2023.
Kanaka Rajan and Larry F Abbott. Eigenvalue spectra of random matrices for neural networks. Physical review letters, 97818)1188104 2006.
Dhruva V Raman and Timothy O'Leary. Frozen algorithms: how the brain's wiring facilitates learning. Current Opinion in Neurobiology, 67:207-214, 2021.
Blake A Richards, Timothy P Lillicrap. Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, al. A deep learning framework for neuroscience. Nature neuroscience, 22(11) 22(11):76111170, 2019.
Pieter R Roelfsema and Anthony Holtmaat. Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience, 1963):166-180, 2018
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. arXiv preprint arXiv:1810.11392 2018.
Patrick T Sadtler, Kristin M Quick, Matthew Golub, Steven M Chase, Stephen I Ryu, Elizabeth Tyler-Kabara, Kabara, Byron M Yu, and Aaron P Batista. Neural constraints on learning. Nature, 512 (7515):423 426, 2014.
Darjan Salaj, Anand Subramoney, Ceca Kraisnikovic, Guillaume Bellec, Robert Legenstein, and Wolfgang Maass. Spike frequency adaptation supports network computations temporally dispersed information. Elife, 10:e65459, 2021.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv: 132.6622, 2013.
Andrew M Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):122 12:12200 2019.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Frontiers in computational neuroscience, 11 11:24, 2017.
Louis K Scheffer, Shan Xu, Michal Januszewski, Zhiyuan Lu, Shin-ya Takemura, Kenneth Hayworth, Gary B Huang, Kazunori Shinomiya, Jeremy Maitlin-Shepard, Stuart Berg, al. A connectome and analysis of the adult drosophila central brain. Elife, 9:e57443, 2020.
Friedrich Schuessler, Francesca Mastrogiuseppe, Alexis Dubreuil, Srdjan Ostojic, and Omri Barak. The interplay between randomness and structure during learning in mns. Advances neural information processing systems, 33:1 13:11322-1336. 2020.
Friedrich Schuessler, Francesca Mastrogiuseppe, Srdjan Ostojic, and Omni Barak. Aligned and oblique dynamics in recurrent neural networks. arXiv preprint arXiv:2307.07654 2023.
Mariia Seleznova and Gitta Kutyniok. Neural tangent kernel beyond the infinite- width limit: Effects of depth and initialization. In International Conference on Machine Learning, pp. 19522-19560. PMLR, 2022.
Yuxiu Shao and Srdjan Ostojic. Relating local connectivity and global dynamics in recurrent excitatory-inhibioory networks. PLOS Computational Biology, 1111e10018855 2023.
Simard, L Nadeau, and H Kröger. Fastest learning in small-world neural networks. Physics Letters A, 336(1) 1): 2005.
14

---

Published as a conference paper at ICLR 2024
H Francis Song, Guangyu R Yang, and Xiao-Jing Wang. Training excitatory-inhiiitoyy recurrent neural networks for cognitive tasks: a simple and flexible framework. PLoS computational biology, 12(2): 1004)922 2016.
Sen Song, Per Jesper Sjöström, Markus Reigl, Sacha Nelson, and Dmitri B Chklovskii. Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS biology. 3(3):e68, 2005.
Vincent Thibeault, Antoine Allard, and Patrick Desrosiers. The low-rank hypothesis of complex systems. Nature Physics, pp. 1-9, 2024.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 ieee information theory workshop (inw), pp. 1-5. IEEE, 2015.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-272, 2020.
Kiran Vodrahalli, Rakesh Shivanna, Maheswaran Sathiamoorthy, Sagar Jain, and Ed H Chi. Nonlinear initialization methods for low-rank neural networks. arXiv preprint arX1v:2202.00334, 2022.
Johan Winnubst, Erhan Bas, Tiago A Ferreira, Zhuhao Wu, Michael N Economo, Patrick Edson, Ben J Arthur, Christopher Bruns, Konrad Rokicki, David Schauder, al. Reconstruction of 1,000 projection neurons reveals new cell types and organization long-range connectivity in the mouse brain. Cell, (79(1):268-281, 2019.
Chloe N Winston, Dana Mastrovito, Eric Shea- thea-Brown, and Stefan Mihalas. Heterogeneity in neuronal dynamics is learned by gradient descent for temporal processing tasks. Neural Computation, 35 (4):555-592, 2023.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametriiee models. In Conference on Learning Theory, pp. 5635-3673. PMLR, 2020.
Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz.
Disentangling trainability and generaliza- tion in deep neural networks. In International Conference on Machine Learning, pp. 10462-10472.
PMLR, 2020.
Marjorie Xie, Samuel Muscinelli, Kameron Decker Harris, and Ashok Litwin-Kuman Task-dependent optimal representations for cerebellar learning. bioRxiv, pp. 2022-08, 2022.
Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006. 14548, 2020.
Guangyu Robert Yang and Manuel Molano-Mazón. Towards the next generation of recurrent network models for cognitive neuroscience. Current opinion in neurobiology, 70:182-192, 2021.
Guangyu Robert Yang and Xiao-Jing Wang. Artificial neural networks for neuroscientisss: primer. Neuron, 107(6): 1048-10700 2020.
Anthony M Zador. critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, 1011):1-7,2011
Liqiong Zhao, Bryce Beverlin, Theoden Netoff, and Duane Q Nykamp. Synchronization from second order network connectivity statistics. Frontiers in computational neuroscience, 5:28, 2011.
15


Reviews:
[{"review_id": "pLId3si0bY", "reviewer": "Reviewer_RNHT", "summary": "This paper studies how structure in the weight matrices -- focusing on the recurrent weight matrices in RNNs, and feedforward matrix in two layer linear feedforward networks -- shapes rich and lazy learning (roughly whether the weights change a lot or negligibly through learning respectively). The paper finds that high-rank initializations lead to so-called \"lazier\" learning, whereas lower rank initializations tend to lead to \"richer\" learning, or larger changes in the weights. If the low-rank intialization is already suitable for the task, there can be negligible changes to the weights as well.", "questions": "- For the proof in the Appendix of Thm 1, while do you need the scale $\\sigma = \\||W_i(0)||_F$ to be small? Also small relative to what?\n- Is there evidence that the initial connectivity structures in biology are low-rank?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The results (analysis in linear feedforward networks, and experiments in RNNs) appear technically correct, and are in line with existing literature on how weights change during learning. Note that I skimmed through the Appendix and did not closely check the proofs.", "weaknesses": "I felt like the paper could provide a more finer-grained description and insight into how the initialization scheme affects the final solution. This could be by providing insight into how the existing metrics (from Sect 2.3) were changing during learning as a function of training epoch (rather than just applied on the final solution, esp. when in the rich regime) and/or through the use of additional finer-grained metrics. As it stands, it feels like there could be different explanations for how the weights are evolving during learning (that I think are interesting and important); for example is the largest rank component changing first or changing in magnitude the most during learning? Does this happen even if the task is low-rank but the (low rank) initialization is not perfectly aligned with the task?\n\nI think this could be addressed both in experiments as well as in the linear setting where the authors make analytical statements. For example, Prop 1 appears considers a low-rank 1 initialization completely aligned with the task (initialized with $\\beta$) -- but I think it would be helpful to understand in general how does a low rank initialization compare with a high rank initialization for learning some fixed (low rank) task, rather than an expectation over all tasks? In other words in general would a low-rank initialization be better for a low rank task and why/why not? also how are weights/singular components evolving to facilitate the learning? \n\nAs an aside, the paper could be made stronger if the case that the theory studied was closer to the experiments (though I understand that it is easier to analyze in the linear FF setting and that the theory still provided insight for the RNN experiments). \n\nThe presentation and writing was generally clear but can be improved. While reading the paper, it felt like, at times, the writing was too verbose and could have been made clearer. For example, \"A nexus between deep learning and neuroscience has expanded the applications of deep learning theoretical frameworks to study the learning dynamics of biological neural network\", and \" Our work builds on these studies by further exploring the precursors of these regime\" could be more clear.  Minor: Fig 3B, left title not visible", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ftRsMUHauh", "reviewer": "Reviewer_Cn5Y", "summary": "This study examines how the initial connectivity of a neural network affects its learning regime (lazy vs rich). In particular. the authors focus on the rank of the connectivity, in contrast to e.g. weight variance studied in previous work. The main motivation for this is the claim that biological networks have low-rank connectivity. Both in theory using feedforward linear networks, and in experiments using recurrent non-linear networks, it is shown that low-rank initialisation lead to richer learning than full-rank initialisation (except, if the low-rank initialisation aligns with the task). This is intuitive as high rank initialisation means it is likely that some (linear combination of the) columns of weight matrix are already aligned with the task matrix, whereas a low-rank matrix will most likely be orthogonal with the task at initialisation.", "questions": "The questions are all related to the initial dynamic regime versus initial connectivity \n\n1.  I would like to the authors to comment on whether the difference in lazy versus rich learning is potentially confounded by the regime of the initial dynamics (stable, unstable or chaotic), instead of the rank of the initial matrix. Three results that lead me to hypothesise there is at least some influence of dynamic regime on the learning:\n\n- For Figure 1, the large full rank networks will likely be initially be chaotic, given a gain of 1.5 (transition at $\\sqrt(2)$ for ReLU networks; Kadmon & Somplinsky 2015). However at least part of the truncated networks (despite rescaling by the norm) will likely be not.\n- The outlier eigenvalue in the networks of Figure 2 might lead to (unstable) dynamics initially exploding in the direction of the corresponding eigenvector, instead of following a chaotic regime. \n- Finally, in Figure 9, the differences between low-rank and full-rank networks diminishes when both are initialised in the stable regime.\n\n2. Were the Dale’s law networks balanced? E.g. increasing inhibitory strength if less then 50% of neurons are inhibitory, so that the expected input to neurons stays zero, would put the Dale’s law networks in similar regimes as the controls (see Rajan & Abbott, 2006). \n\n3. The Eigenspectrum, and as a result the initial dynamics, of the full-rank random networks are most likely approximately constant between networks, whereas the (non-zero part of the) eigenspectrum of the low-rank networks will vary more over initialisations - is this related to the observed high variance in richer learning between the low-rank models?\n\nNote that getting to same dynamics could be partly ameliorated by scaling by the largest singular value instead of the norm (which are realistically only approximately proportional for the large full rank networks). However this also is not a guarantee, as e.g. as stated above balancing also has an influence on dynamics", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The question of how connectivity influences representation and learning dynamics is of significance, both for the neuroscientists, as well as for machine learning scientists. How low-rank initialisation affects the learning regime has as far as I am aware not been studied in depth, yet networks with low-rank weight matrices are used by both the deep-learning and computational neuroscience communities.\n\nThe paper is well written and the main idea is clearly presented. Multiple experiments are performed that highlight the main result and the supplementary contains a large amounts of controls. The experimental results are supported by theoretical results on linear feedforward networks.\n\nOverall I lean on accepting this paper, and I would increase my score if the concerns below are addressed", "weaknesses": "1. While many controls are done, the initial dynamic regime of the network (stable, unstable or chaotic) is not controlled for (see questions below).\n2. I am not entirely convinced by the evidence given for the statement that the brain has low-rank connectivity. The paper repeatedly cites Song et al., 2005 and Mastrogiuseppe & Ostojic, 2018 as evidence for low-rank connectivity in the brain. The latter is a purely theoretical study, which I am not sure why it used as evidence for biology. The former indeed studies biological neural circuits, and found that the distribution of synaptic connection strength can be fitted by a lognormal distribution, as well as the circuits having overrepresented bidirectional connectives. While my intuition is that this implies low-rank connectivity, it can (and likely should) be made clear if this is the case. In general the relation between local connectivity patterns and (global) rank is not always obvious a priori (potentially, see Shao & Ostojic 2023).\n3. Biological constraints are only used as initialisation and not enforced during training. This decreases the relation of the simulations to biology, as neurons in the brain can (generally) not just decide to forget about e.g. Dale’s law during learning.\n4. The authors claim to study continuous-time networks, however, $dt$ is set equal to $\\tau_m$, and tasks have around 10 time steps after discretisation. It could be questioned how well the simulations reflect the continuous-time equations.\n\nMinor:\n1. Equation 10, last line should say d instead of 3 above the equality sign.\n2. Figure 3B left, title is cut off.\n3. The 5th references seems formatted wrong.\n4. Equation 1. -> It could be named explicit that you are making an exponential Euler approximation (which is not used in the studies cited here).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7URtMiR3F8", "reviewer": "Reviewer_SwWy", "summary": "The work investigates how the initial weight structure, especially its (effective) rank, influences network learning dynamics, and in particular whether the network learns in the \"lazy\" (small change in tangent kernel) or \"rich\" (substantial evolution of tangent kernel) regimes. The paper is written using neuroscience as a motivation, citing the fact that connectivity in the brain is substantially more structured and lower-rank than standard initializations used in ML, and thus existing work on lazy/rich regimes may not apply directly to biological learning. The authors find that low-rank initializations typically lead to richer learning, unless the low-rank initialization happens to align with the structure of the task.", "questions": "Do the kernels associated with low-rank-initialized networks grow more aligned with the target function than those of high-rank-initialized networks?  Or do they merely grow \"as aligned\" as the lazier networks with high-rank initializations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written and technically sound.  The experiments provide solid evidence for the main takeaway of the paper (that lower-rank initializations generally lead to richer learning dynamics), using a variety of tasks and complementary measures of laziness/richness.  Figure 4 provides helpful intuition for understanding this conclusion, suggesting that low-rank initializations tend to incentivize richer learning because there is a greater need for it when the model's kernel at initialization has low alignment with the task structure.", "weaknesses": "I have two main concerns about this paper:\n\n1. Given that the proof of Theorem 1 is left to an Appendix, the paper would benefit from providing more explanation / intuition for the main result.  I can imagine an intuition that the network \"needs to\" adjust its representation more to address the misalignment.  But a more thorough presentation of the key proof steps in the main text, and/or illustrative examples to provide intuition for the learning dynamics, would be very helpful.\n\n2. It is not clear to me how the findings of this paper will be useful to either the ML or neuroscience communities.  I do not mean to claim that they aren't useful, but rather that the paper does not present a clear case for why they are.  From an ML standpoint, low-rank initializations are uncommon.  From a neuroscience standpoint, the applicability of this theoretical framework seems uncertain given that the learning algorithms used in the brain are not well characterized and may differ substantially from SGD.  Moreover, it is not clear to me what we gain by knowing that low-rank neural connectivity gives rise to richer learning than a counterfactual in which brains had higher-rank connectivity.  Is the idea to compare between brain regions with connectivities of different ranks to make predictions about different representation learning dynamics?  Or across species?  More concrete and precise proposals for how these findings could be used would be helpful.\n\n3. The richness/laziness of learning dynamics by itself tells us very little about the representations a network learns.  If the goal is to understand the impact of weight initialization on representation learning, the paper would benefit from more direct consideration of this question.  For instance, see the question below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e6LkQIH361", "reviewer": "Reviewer_rK41", "summary": "This paper probes the effect of weight initialization rank on feature learning in recurrent neural networks. The authors study in particular how the alignment between a particular low-rank initialization and the task affect how much the kernel moves during training.", "questions": "1. In the block of citations in the second sentence of the introduction, the work of Canatar, Bordelon, and Pehlevan (2021) should be cited before Xie et al 2022, as the latter is based on the results developed in the former work. \n\n2. Is \"tabula rasa\" the appropriate way to describe Gaussian or Uniform random initialization with non-vanishing variance? \n\n3. In-text citations to the work of the Allen Institute and the MICrONS Consortium are not formatted correctly. \n\n4. Why do the eigenvalues in Figure 2 appear not to be sorted by magnitude? Also, why is $\\frac{\\sum\\_{i} |\\lambda|\\_{i}}{|\\lambda\\_{1} N}$ the relevant notion of effective rank in this setting?\n\n5. Figures 2 and 3 each occupy more than half of a page, but both contain a significant amount of whitespace. Is it possible to combine them side-by-side into a single figure?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In the large, I think this paper is timely, and the core idea is (to the best of my knowledge) novel and interesting.", "weaknesses": "1. Upon reaching the conclusion of the paper, I found myself confused as to why the authors did not perform any experiments with feedforward networks. Though I appreciate the neuroscience-inspired focus on recurrent networks, including some tests in the feedforward setting would be valuable. In particular, this would show more directly how departures from the idealized setting of their theoretical results (linear networks with whitened data) affect the phenomenology. Demonstrating that these ideas are applicable to deep feedforward networks  and more standard machine learning tasks would in my view substantially improve the impact of the paper by tying it more closely to the main body of theoretical work on feature learning. \n\n2. The metrics introduced in Section 2.2 are restricted to measuring changes in weights or kernels over the course of training. Particularly given the fact that the stated goal is to investigate how task structure affects feature learning, it would be useful to also track a measure of task-kernel alignment, for instance the kernel-target alignment $y^{\\top} K y / |y|^2 tr K$ or the task norm $y^{\\top} K^{-1} y$ throughout training. \n\n3. The criteria used for task selection are unclear. The authors choose three of the original tasks from the neurogym environment (I note that this task suite has been critiqued by Khona et al., \"Winning the Lottery With Neural Connectivity Constraints: Faster Learning Across Cognitive Tasks With Spatially Constrained Sparse RNNs\") along with sequential MNIST. Why are these four tasks relevant? The paper could be made more compelling either by using a more diverse suite of tasks or by justifying why these four are appropriate. \n\n4. On the whole, I found the Discussion to be a weak point of the manuscript. Section 4.1 does not answer the vital question of precisely what experimentally-measurable biological phenomena the present work can explain, contextualize, or predict. As it stands, this section is in some sense a restatement of Zador's ideas around the importance of task-aligned initialization in the language of lazy learning. Moreover, the discussion of implications for deep learning promised by the section title seems to be missing. Section 4.2 contains far too many ideas to be jammed into its single paragraph; for clarity the authors should split it into at least two paragraphs. Finally, the relevance to neuroscience of neural collapse and of Tishby's proposals regarding the information bottleneck needs to be justified. The sentence where they are mentioned is much too long as it is, and these are both somewhat subtle and controversial topics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)