Learning dynamics in linear recurrent neural networks
Authors: Alexandra Maria Proca
Clémentine Carla Juliette Dominé
Murray Shanahan
Pedro A. M. Mediano
Venue: ICML 2025 oral (2025)
Keywords: deep learning
learning dynamics
RNNs
rich and lazy learning
teacher-student
neuroscience
Primary area: theory->deep_learning
TL;DR: We study the learning dynamics of linear RNNs using a novel framework that accounts for task dynamics.

Abstract: Recurrent neural networks (RNNs) are powerful models used widely in both machine learning and neuroscience to learn tasks with temporal dependencies and to model neural dynamics. However, despite significant advancements in the theory of RNNs, there is still limited understanding of their learning process and the impact of the temporal structure of data. Here, we bridge this gap by analyzing the learning dynamics of linear RNNs (LRNNs) analytically, enabled by a novel framework that accounts for task dynamics. Our mathematical result reveals four key properties of LRNNs: (1) Learning of data singular values is ordered by both scale and temporal precedence, such that singular values that are larger and occur later are learned faster. (2) Task dynamics impact solution stability and extrapolation ability. (3) The loss function contains an effective regularization term that incentivizes small weights and mediates a tradeoff between recurrent and feedforward computation. (4) Recurrence encourages feature learning, as shown through a novel derivation of the neural tangent kernel for finite-width LRNNs. As a final proof-of-concept, we apply our theoretical framework to explain the behavior of LRNNs performing sensory integration tasks. Our work provides a first analytical treatment of the relationship between the temporal dependencies in tasks and learning dynamics in LRNNs, building a foundation for understanding how complex dynamic behavior emerges in cognitive models.

Learning dynamics in linear recurrent neural networks
Alexandra M. Proca' Clémentine C.J. Dominé? Murray Shanahan
Pedro A.M. Mediano
Abstract
1. Introduction
Recurrent neural networks (RNNs) are important tools in both machine learning and neuroscience for learning tasks with temporal dependencies. Recently, (linear) recurrent architectures (state space models), have had a resurgence of popularity in long-range sequence modeling (Gu et al., 2020; 2022; Orvieto et al., 2023; Gu & Dao, 2024). In tandem with the success of dynamical systems theory in describing neural activity related to motor control, working memory, and decision-making (Remington et al., 2018a; Vyas et al., 2020; Khona Ficte, 2021), RNNs have also become popular choice for cognitive models of neural dynamics (Barak, 2017), as they not only replicate recurrent dynamics recorded in animals but are also capable of performing ab- stractions of the same cognitive tasks used in experiments (Mante et al., 2013; Engel et al., 2015; Chaisangmongkon et al., 2017: Wang et al., 2017; Masse et al., 2018; Reming- ton et al., 2018b; Orhan & Ma, 2018; Masse et al., 2020; Beirán et al., 2023). More generally. RNNs present an in- teresting model of study due the complex computational capabilities given by their hidden layer that evolves with time and which is a universal approximator of any open dynamical system (Doya, 1993; Schäfer & Zimmermann, 2007).
Recurrent neural networks (RNNs) are power- ful models used widely in both machine learn- ing and neuroscience to learn tasks with tempo- ral dependencies and to model neural dynamics. However, despite significant advancements in the theory of RNNs, there still limited understand- ing of their learning process and the impact of the temporal structure of data. Here, we bridge this gap by analyzing the learning dynamics of linear RNNs (LRNNs) analytically, enabled by a novel framework that accounts for task dynamics. Our mathematical result reveals four key properties of LRNNS: (1) Learning of data singular values ordered by both scale and temporal precedence, such that singular values that are larger and oc- cur later are learned faster. (2) Task dynamics impact solution stability and extrapolation ability. (3) The loss function contains an effective reg- ularization term that incentivizes small weights and mediates a tradeoff between recurrent and feedforward computation. (4) Recurrence encour- ages feature learning, as shown through a novel derivation of the neural tangent kernel for finite- width LRNNs. As final proof-of-concept, apply our theoretical framework to explain the behavior of LRNNS performing sensory integra- tion tasks. Our work provides a first analytical treatment of the relationship between the tempo- dependencies in tasks and learning dynamics in LRNNs, building a foundation for understand- ing how complex dynamic behavior emerges in cognitive models.
Accompanying the popularity of RNNs, there have been significant efforts dedicated to their thcoretical understand- ing, both from deep learning theoreticians (Cohen-Karlik et al., 2023; Orvieto et al., 2024; Zucchet & Orvieto, 2024) and neuroscientists relating these findings to observations about the brain (Sussillo & Barak, 2013; Mastrogiuseppe Ostojic, 2018; Yang et al., 2019; Schuessler et al., 2020a;b; Turner et al., 2021: Dubreuil al., 2022: Farrell al., 2022; Tumer & Barak, 2023; Driscoll al., 2024; Liu et al., 2024). However, most theoretical studies of RNNs are done at the end of training lutions they find, but ignoring the learning process itself (Saxe et al., 2020). Of the work that does study learning, the focus is often related to practical considerations about training, such as learning long-range dependencies. Overall, despite the widespread use and known complex computa- tional abilities of RNNs, it is still unknown how their under- lying functional structures emerge as result of training on temporally--truccurre tasks.
analyzing properties of the so-
Deparrmment of Computing, Imperial College London, Lon- don, United Kingdom Gatsby Computational Neuroscience Unit, University College London, London, United Kingdom Diivision of Psychology and Language Sciences, University College Lon- don, London, United Kingdom. Correspondence to: Alexandra M. Proca <a.proca22@@ @ imperial.ac.uks
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning dynamics in linear recurrent neural networks
One related line of previous work has focused on using deep linear networks to analyze learning dynamics (Saxe al., 2014; 2018; Braun et al., 2022; Dominé et al., 2025). Al- though unable to solve nonlinear problems (note however that there has been progress to overcome this limitation (Saxe et al., 2022; Sandbrink et al., 2024)), these networks exhibit complex nonlinear learning dynamics and are analyt- ically tractable. providing a useful framework for theoretical investigation. Applied to cognitive neuroscience, the study of learning dynamics has been used to propose theory of semantic development (Saxe et al., 201 8), cognitive flexi- bility (Sandbrink et al., 2024), and localization in receptive fields (Lufkin et al., 2024), among other work. Despite its successes, the analytical treatment of learning dynamics in linear networks has primarily remained in the domain of feedforward networks. In order more broadly characterize learning, however, theory needs to account for the impact of dynamic task settings and the rich structure endowed by recurrent networks, especially since it is such a critical com- ponent of neural computation. Of the few prior studies of learning dynamics in linear RNNs, Schuessler et al. (2020b) showed that networks make low-rank changes to their con- nectivity during learning and Smékal et al. (2024) showed how overparameterization accelerates convergence time by studying the frequency domain. However, the influence of temporally structured data on learning has not been studied analytically to our knowledge.
network achieves 0 loss.
We identify an effective regularization term in the en- ergy function that incentivizes small weights, and a phase transition in the connectivity modes that leads to low-rank solutions.
We derive the neural tangent kernel (Jacot et al., 2018) for finite-width LRNNs and show that recurrence facil- itates feature learning.
We demonstrate the generalizability of our results by applying theoretical framework to describe the be- havior of LRNNs trained on sensory integration tasks, relaxing our prior assumptions.
2. Mathematical setup
2.1. Model
We study a LRNN (Figure 1) parameterized by matrices Wa E N&NN W, N W, E N.N, with hidden state h; E R& that receives an input æ, E R. at each timestep / and updates its hidden state. For sim- plicity, in the main text we study the single-output case, where the network only produces an output ýr E R at the last timestep T. In Appendix M, we generalize our ap- proach to networks trained to produce outputs ÿ, at every timestep (the autoregressive T-output case). The network is characterized by the equations
In this work, we study the learning dynamics of linear RNNs (LRNN) to better understand the influence of temporal data on learning in recurrent cognitive systems, unifying the areas of RNN theory and learning dynamics. Our theoreti- cal results contribute to explanations of many phenomena spanning both topics, including low-rank connectivity (Mas- trogiuseppe & Ostojic, 2018), rich and lazy learning (Farrell et al., 2023), extrapolation capabilities (Cohen-Karlin et al., 2023), and network stability (Sompolinsky et al., 1988). Taken together, this represents a substantial step towards the theoretical understanding of learning in recurrent deep learning models, building foundation for new theories and hypotheses of learning in neural networks and the brain.
h Wnh,, Wy
(1)
ÿr W Wyrtri
(2)
We initialize the hidden state h1 vector of zeros, yield- ing
h Σ W w.
W
(3)
We analyze learning in the LRNN when trained using back- propagation through time on the squared error over P tra- jectories (T.p,1 2p.2, p. Up.r }p=1
Our contributions are as follows:
We provide, for the first time, a closed-form analytical expression for the energy function of LRNNS decou- pled along singular/eigen-value dimensions. We use this result, together with a novel framework describe task dynamics, accurately predict solutions found by LRNNs.
T
C
Up.T
W
W W 221
(4)
p=1
2.2. Temporal singular values
With the model and loss function fixed, our next step is to specify task for the model to learn. In this linear set- ting, the task is fully specified by the sequence of matrices Y X, = Σp= 3p.r the input-output correlation ma- trix between the input pp.t at timestep and the final output
We identify how both the magnitude and temporal or- dering of data singular values affect learning speed.
We describe how task dynamics impact network stabil- ity and extrapolation ability, even in cases where the
2

---

Learning dynamics in linear recurrent neural networks
Ely
8p)
singular/eigen--ecoors are constant if data is generated by diagonalizable LRNN teacher (Appendices B and N), and in fact, because we don't restrict the dynamics of the singular/eigen-vallee our form captures more general set- tings than the standard teacher-student setup (which con- strains task dynamics to the form 8x"-1)) (3) prior work has shown that model alignment occurs early in training for networks initialized with small random weights (Atanasov et al., 2022) and there are theoretical and practical justifi- cations for diagonalizable state spaces (Hazan et al., 2018; Gupta & Berant, 2022).
Wy
11
>
h22
W x
X1
XT
XX Varr U.S.V
Figure 1. Linear RNN model captures task dynamics through lemporally-ependerr singular values. The data correlation ma- trices Xxx have constant left and right singular vectors, varying only in their singular values St across time.
3. Results
3.1. LRNN energy function
target yp.T- Extending the approach by Saxe al. (2014; 2018), we can represent these matrices through either their singular value decomposition (SVD; case 1 in the assump- tions below), or their eigendecomposition (case 2), which enables us to account for task dynamics. In the main text, we follow the precedence from prior work and base our analysis around a derivation using SVD, which, although more restrictive with task dynamics, simplifies the setting and allows for non-square networks. Then, in Appendices N and O, we use an eigendecomposition to derive similar but more general form (accounting for complex cigenvalues thus allowing for rotational dynamics) and show that our framework and results extend naturally to this case. To sim- plify our derivations, we make the following assumptions:
With the aforementioned assumptions, we can diagonalize the network, eliminating cross-terms. Let da bas Ca be the at diagonal entry of W x, W h. W y. respectively, and Sa,t be the tt singular value (SV) of St. Assuming small learning rate 1/T (i.e., the gradient flow regime), we can write the gradients of the network parameters as a set of differential equations in terms of these variables, or connec- fivity modes, the dynamics of which decouple across SV dimensions a (Appendix B). We refer to a, as the input, ba the recurrent, and co the output connectivity mode. Their dynamics are given by
d =
a,
cab Gaba-1(( (S - b/.b%-aan
(5)
dtg
iml l
d ba
Assumption (whitened input): Inputs are uncorrelated and whitened across all timesteps and dimensions, such that Xx I; X. 0, f l'.
Σ (T i)cabl bhi -la- -oobb
dtn
Assumption (constant singular vectors or eigenvectors):
d
T
Σ
35- an (80,) bobb
All onput-output correlation matrices have either (1) constant left and right singular matrices Uy Ve and only vary in their singular values S, over a trajectory, such that Σ X. cigenvectors P and only vary in their eigenvalues D over a trajectory, such that yx x. = PD,Pt.
Co = dtg
where to refers to timesteps of gradient- learning as opposed to the trajectory timesteps t. Our first result shows that these dynamics arise from gradient descent on an energy function.
V.V. VI; or (2) constant
Assumption 3 (aligned model): The model is aligned to either (1) the data singular vectors at initialization such that UJ (O) R2 W;, Rz, R2_ W* (O)V, yield di- agonal matrices W, (O). W., (V((O)). (0) for some thogonal matrices Ry, Rz, or (2) the data eigenvectors P such that (O) (())) P.) P,, yield diagonal matrices W (O), (0), (0).
Lemma 3.1. Given Assumptions 1-3, the energy function of the LRNN is given by
T
E(Ga: bas Ca) =
Σ
(Sa.i - ba cabTiiaa)
(8)
2T
To ease notation, we omit specifying when referring to connectivity modes in the remainder of the paper, although note that all terms (St, a, b, c) still refer to a particular SV dimension a. We also generally refer to the input-output modes (ac) together and treat them as a single term since there isn't 't any meaningful distinction between them.
Although seemingly restrictive, we argue these assump- tions still capture meaningful learning scenarios. For ex- ample: (1) data can be whitened using the innovations form of Kalman filter (Durbin & Koopman, 2012); (2)
3

---

Learning dynamics in linear recurrent neural networks
leamning input- outpu, fixed recurrent
learning recurrent, fixed input outut-
over T SVs in each dimension (as opposed one) and the network must optimize for SVs across time. A consequence is that SVs at different timesteps are weighted differently in the gradient. SV trajectories that are larger and have SVs occurring later in the trajectory are learned faster (assuming recurrent connectivity modes are initialized < 1). We can see this effect by looking at the gradients of the connectivity modes (Equations (5) to (7)): gradients from early trajectory timesteps are weighted by the recurrent mode b exponen- tially with trajectory length. Since we initialize connectivity modes to be less than 1, this has the effect of downscaling the gradient contribution from earlier trajectory timesteps compared to later timesteps. Thus, we see a more complex portrait of the effect of both SV magnitude and SV dynam- ies (time) playing into the ordering of learning in recurrent networks. This effect can be seen in Figure 2 (right) - the blue curve converges to smaller solution than the orange curve but is initially learned faster, which differs from the behavior of feedforward networks (e.g., the left plot), and is driven by the fact that the singular value at the last timestep is larger for the blue curve than the orange curve (from 6). We study this further in Appendix E.
input-outout mode (ac)
recu mode (b)
0.5
theory
RAA
0.7'+t
0.7
theory
10002
25000
30:00
45000
10300
20000
30300
00000
training steps Rt))
training steps (ks)
Figure 2. Learning dynamics of input-output and recurrent connectivity modes in a linear RNN. Data singular values 8, can be decomposed into a scaling term and temporal term f(, t). (Left) The input-output modes Iearn the scaling component, while (right) recurrent modes learn the dynamic component. The col- ored lines simulations and dashed lines are the corresponding theoretical predictions.
To provide some intuition about the reduced form, the mag- nitude of the data SVs (Sa,;) correspond to the strength of correlation between the input (2x,) trajectory timestep and the output target (yr) in different SV dimensions a. In this work, we're re interested in understanding how recurrence and the task dynamics (given by $1:7) impacts the LRNN's learning dynamics.
3.3. Task dynamics determine solution stability and extrapolation ability
RNNs suffer from problems related to stability during train- ing and inference. By stability we refer to the state of the RNN parameters which may lead to exploding gradients or diverging hidden layer activity. Because of the expo- nential effect of the recurrent layer, (nonlinear) RNNs with eigenvalues larger than I exhibit chaotic behavior (Som- polinsky et al., 1988). By looking at the energy function (Equation (8)), we can further see the well-known effect of vanishing (exploding) gradients, given by [6] < 1 (b) > 1) as T - (Bengio et al., 1994; Hochreiter et al., 2001; Pascanu et al., 2012), which makes training on tasks with long-range dependencies challenging and to which there have been numerous methods introduced to alleviate these difficulties (Hochreiter & Schmidhuber, 1997; Le et al., 2015; Orvieto et al., 2023; Zucchet et al., 2023). Another open problem in RNNs is their ability to extrapolate (or in- terpolate) to sequence lengths that differ from those trained on, which is not well understood (Cohen-Karlik et al., 2023; Beirán et al., 2023). Here, we study how network stabil- ity and extrapolation ability are impacted by an additional factor: the underlying task dynamics a RNN is trained on.
3.2. Solutions to LRNN learning dynamics
While perhaps trivial, we can think of the LRNN as perform- ing two functions: the input-output mode (ac) performs a constant scaling and the recurrent mode (b) Icarns a time- dependent function. Thus, by decomposing the data SVs ($1:7) into a constant and temporal component, we might better understand the solutions LRNNS converge to. We de- compose each data SV as se = (A,t), where 8 is constant across all data SVs, and fA,, t) is some function parameter- ized by À that is dependent on trajectory timestep L.
We derive a full solution for the learning dynamics of the input-output modes when recurrent modes are frozen (Ap- pendix C), as well as local approximation to the recurrent modes when input-output modes are frozen (Appendix D). Intuitively, the network should use recurrent modes to learn the dynamic component of the data since it varies in its con- tribution to the output through time, whereas input-output modes do not vary with time and thus can only contribute some form of scaling. By studying the learning dynamics of the recurrent and ouput- modes separately, we confirm that they indeed learn these different components (Figure 2).
To do this, we study task dynamics that are perfectly learn- able (provably the only task dynamics with 0-loss solutions; Appendix F), but differ in their hidden layer stability or abil- ity to extrapolate. Recall that we can decompose data SVs as s; = ff t). We distinguish between three cases with known analytical O-loss solutions (Appendix F.3), which offer natural settings to study how perfectly-learabble data
The distinction between dynamic and scaling components of data SVs also highlights an important difference in learning dynamics between (deep) feedforward and recurrent linear networks. Feedforward linear networks learn the largest SVs first. In recurrent networks, however, the loss is computed
4

---

Learning dynamics in linear recurrent neural networks
extrapolate to other trajectory lengths T
constant dynamics (MA, () 1)
inverse-exponential dynamics (RA, A7
exponential dynamics (MA. A9
cata singuiar vollues (sc)
valli (50)
1.0
A=1
Ap1
A<1
0.0
0.6
8 30
0.4
021
trajectory timesteps (t)
trajectory timestaps (t)
trajectory timestops (t)
stable (b 1.
dynamics with early-importance are unstable
unstable (b= L.cw
Figure 3. Task dynamics determine solution stability and extrapolation ability. For RNNs trained on (left) constant dynamies (f((A,) () 1). solution stability is dependent on the scaling term d, where d - 0 is stable and 8 > 0 is less stable. (Middle) Inverse- exponential tf( x"-1) and (right) exponential (f(), t) = X²) dynamics produce unstable solutions when respectively, which correspond to early-importance dynamies (st 8t41). Further, the solution to the input-output modes (ac 5x²) for exponential dynamics depends on trajectory length, so solutions learned for one length will not extrapolate to other trajectory lengths.
and 1.
impacts solution stability and extrapolation ability. We con- sider cases where the data SVs are constant tf 1), change inverse-exponentialy (f(),t) = XT--),, or change exponentially (f(),t) = X²). By varying 5, X, we can pa- rameterize the task dynamics differently and elicit particular network behavior. We note that technically all of these dynamics can be reparameterized as inverse- eveereeexponential dy- namics when the trajectory length fixed (Appendix F.1), but for simplicity, we will keep these separate.
mode (b) approaches I; however, when the scaling term is large (6 >> 0), optimization is more challenging as solutions approach unstable solutions near b = 1 (left in Figure 3), This observation about 5 also generalizes to other dynamics when the solution for the recurrent mode is not close to 0 (b > 0).
Although each case of task dynamics we consider here has O-loss solution, not all of these solutions extrapolate per- fectly to other trajectory lengths T. In particular, the global solution for exponential task dynamics is dependent on the trajectory length T (ac = 5A"). As such, this solution will not perfectly extrapolate to trajectory lengths that differ to the one trained on, and in fact the error will grow as the difference in trajectory length increases (see Appendix F.1).
For constant task dynamics, the global solution exists at b = 1, ac = d; this can be understood as 'equally weight- ing' the input at each timestep. while the input-output connectivity modes learn an appropriate scaling d. For exproeneepponentil task dynamies, the minimum found at b = X, ac 8, as the dynamics of the singular values (st (f( t), f(),t) = XT) correspond exactly to the dynamics of the LRNN (cbT aa). Finally, for exponential task dynamics, the minimum exists at b = L/A, ac = 5x".
In conclusion, can that even for data that is perfectly learnable, properties of the task dynamics crucially impact the stability of solutions and the ability to extrapolate to other trajectory lengths. In particular, we find that (1) data with correlations that decrease over trajectory time produce unstable solutions, (2) task dynamics with a large scaling term (6) are less stable, and (3) task dynamics with solu- tions that depend on trajectory length (such as exponential dynamics) do not extrapolate with 0-loss to other trajectory lengths. We show in Section 3.6 and Appendix I. that these findings hold for RNNs without our theoretical assumptions.
By studying these solutions, we can first observe that inverse-exponential and exponential task dynamics yield un- stable solutions (where the recurrent mode b 1) for and À 1, respectively (green dashed lines in Figure 3). In both cases, the data SVs decrease across the trajectory (st > 1); we hence refer to this as early-imporiance. Due to the instability of exploding gradients as the recurrent mode b increases over 1, early-importance dynamics are more challenging, if not impossible, to learn as trajectory length increases (T + oo).
3.4. Connectivity modes exhibit phase transitions between recurrent and feedforward computations
Constant dynamics essentially an intermediary between exponential and inverse-exponentii dynamics (i.e., because f(A = 1,t) = 1' = 1T- = 1). Constant dynamics are common, as they correspond to basic integration of input. They also present a way to study the influence of the scaling term (6) on solution stability. Constant dynamics have stable solutions when the scaling term is small + 0) because it keeps SVs and input-output modes small as the recurrent
In the previous section we studied task dynamics with per- fect solutions. However, most real-world tasks will, natu- rally, not exhibit inverse-cxponential dynamics and may not have perfect 0-loss solutions (although the loss may be low in practice). A separate observation is that RNNs initialized with small random weights seem to learn low-rank solutions along effective "task dimensions' (Schuessler et al., 2020b;
5

---

Learning dynamics in linear recurrent neural networks
constant task dynamics
exponential task dynamics
Dirac delta task dynamics
recurret mode (b)
recurrnnt mode (b)
2.0
recureen mode (b)
1.-
1.2
04-
1.-
10
0.2
0-loss solution
0.5
-08
0.0
0.0
1/2
input-outout mode [ac)
recurrent computation
putttttuuuuttt mode (ac)
recurrent computation (A)
Input ooputouuouut mode (ac)
recurrent computation (8)
-06
1.25
2.3
1.25
1.00
2.00
1.00-
0.75
0.75
02
0.50
0.50
0.25
025
0.55
0.00
0.00
0.00
1
12
2.4
recurrent computation
recurrent computation (A)
recurrent computation (8)
Figure 4. Task dynamics modulate phase transitions of network connectivity modes. We plot the learned solution of the connectivity modes for (leff) constant, (middle) exponential, and (righr) Dirac delta task dynamics. When the recurrent computation of the task dynamics (x-axis) is small, (top) recurrent modes (b) remain near 0 and (bottom) input-output modes (ac) learn the fcedforward computation (k; indicated by color). As the recurrent computation increases, the network transitions to instead learn the recurrent computation (dotted line the O-loss solution), manifesting as a phase transition both the recurrent and input-output connectivity modes.
Liu et al., 2024). This motivates to ask question that is two-fold: (1) how task dynamics impact what LRNN learns when it cannot perfectly fit (all of) the data, and (2) when pruning connectivity modes (and hence low-rank so- lutions) may be a useful strategy. To gain intuition, we can rewrite the energy function as sum of two terms.
where + (A, T). Here, there is way for the network to perfectly fit the task dynamics (which would only be the case if K 6f (T)... We can think of the two cases as recurrent contribution to the task dynamics (t < 2) and a feedforward contribution (t = T).
Using this setup, we experiment with varying each of the different parameters to show that each one can affect the underlying task dynamics and consequently influence the network's 's behavior. For example, when studying constant task dynamics (where À 1 by default), we change the value of the scaling term (d). Instead, when studying expo- nential task dynamics, we change the dynamic term (A).
Lemma 3.2. The energy function of a LRNN can be decom- posed into data-driven term and an effective regularization term given by
T Σ
1
aa 2
E
ot
2s,cbt-ia
2T
27
B2
data-driven term
offective regularization term
Finally, in all settings, we vary the feedforward computa- tion (s) independently of the other (recurrent) parameters, to study how the network deals with the tradeoff between learn- ing solutions for t < T and t = T, which cannot be learned simultancously. In particular, this construction forces the network to either approximate the recurrent dynamics (con- stant: ac - 6, b + X; exponential: ac - 6X" b - 1/2) or the feedforward computation (ac - ni, b - 0). each of which will ineur non-zero error from its counterpart.
where, as -> oo, the second term goes to infinity for cba 0.
For task dynamics that are learnable with low loss, both terms cancel out. However, when the LRNN cannot fit the data, the second term acts as an effective regularizer that incentivizes connectivity modes to remain close to O (Appendix G). This suggests that LRNNS might have an implicit bias towards effectively low-rank solutions, both when tasks span only few dimensions (i.e., Sa,t 0, Vit for most ar) and when tasks are not perfectly learnable.
We run simulations varying across these different task dy- namics and plot the final solutions the connectivity modes converge to (Figure 4). We find that the aforementioned tradeoff between recurrent and feedforward computation manifests as rapid phase transition of connectivity mode values across different task dynamics and becomes sharper as trajectory length 7' increases. We show that this phase transition can be induced by varying either the scaling term (6; left in Figure 4) or the dynamic term (); middle in Figure 4). When the error term is dominated by the feed- forward computation (k is large, ff, t) is small), the net- work effectively prunes the recurrent mode (b - 0) rather
To further investigate how data impacts what LRNN learns, we modify the task dynamics we studied earlier (constant, exponential, cweers--expoenniine timestep, 8T, that does not follow the task dynamics as the rest of the trajectory. More specifically,
to have a SV at the last
(f(()t) i T
8t
(10)
6

---

Learning dynamics in linear recurrent neural networks
aligned weights
unaligned weights
of the setting, we emphasize that the cases we study here are likely not the only task dynamics with representational tradeoff's when there are I singular values to fit to in single dimension. It's unclear how RNNs might prioritize learning certain computations over others in various scenarios. These results, together with those on learning speed (Section 3.2), suggest a recency bias, although the cumulative effect of the recurrent computation can outweigh this.
trajectry kegghh (7)
3
trajectory
6-
0.0
Inttializtton
Figure 5. Recurrence encourages feature learning. Phase plots illustrating the kernel distance of the NTK from initialization as a function of trajectory length and initialization scale for LRNN initialized with weights that are (left) aligned and (right) unaligned.
3.5. Recurrence facilitates rich learning
Prior work has identified two distinct learning regimes in neural networks: feature learning (rich learning,, where networks learn structured task-relevant representations, and non-feature learning (lazy learning), where networks per- form high- -igh-dimensional projections of the input (Heij et al., 2007; Yang, 2020; Farrell et al., 2023); rich learning typ- ically occurs in networks initialized with small random weights and lazy in networks with large weights. Significant progress has been made in the theoretical understanding of these regimes, particularly in feedforward architectures (Arora et al., 2019; Azulay et al., 2021; Braun al., 2022; Saxe et al., 2022; Kunin et al., 2024; Dominé et al., 2025). However, research into how non- non-feedforward architectures affect these learning regimes remain limited. Notably, Liu et al. (2024) examined the role of weight connectivity in shaping learning regimes in RNNs and Schuessler et al. (2024) showed that RNNs have different learning regimes characterized by either aligned oblique recurrent dynam- ies. Building on this line of inquiry, we explore how re- currence impacts feature learning dynamics. Specifically, we investigate whether recurrent architectures impose addi- tional constraints on the learning problem. thereby biasing the network towards the rich learning regime.
than approximating the recurrent computation (11:T-1) and the input-output mode learns the feedforward computation (ac se k). As 8 (A, t) increases and the recurrent computa- tion becomes more important (has a greater contribution to the loss), the network rapidly transitions to a regime where it approximates the task dynamics and approaches the 0- loss solution for the case where k = (2, T), following the dynamic trajectory to the last timestep (dashed line in Figure 4) and ignoring the feedforward computation.
To further illustrate the tradeoff between feedforward and recurrent computation, we simulate networks trained on task dynamics produced by two Dirac delta functions, which have no 0-loss solution:
B
= = 1 T
K
(11)
0
otherwise
As before, we vary the recurrent and feedforward compu- tations separately by independently changing and k. As shown in Figure 4 (right), again see a sharp transition as the recurrent computation (3) increases, where the re- current mode becomes non-zero, while simultaneously, the input-output mode decreases in magnitude. In Appendix H, we show that the phase transition depends only on the ra- tio of recurrent to feedforward computation (8/K), such that the recurrent computation is pruned when this ratio is small. Using Landau theory, we show analytically that this corresponds to a first-order phrase transition for T > 3.
The rich and lazy learning regimes are typically evaluated using the neural tangent kernel (NTK) (Jacot et al., 2018), which is constant during lazy learning and non-constant during rich learning. We derive the NTK for finitewiidth LRNNS (Appendix I), which we then use to study what learning regimes emerge in LRNNS with different initial- izations and trajectory lengths. Importantly, our derivation does not place any assumptions the alignment of LRNN weights as in the prior sections.
Taken altogether, these results suggest an implicit bias to- wards small weights and low-rank connectivity in RNNs, mediated by an effective regularization term. They fur- ther illustrate a tradeoff between feedforward and recurrent computations, and show cases where the network prunes connectivity modes to deal with task dynamics that are not perfectly learnable, leading to low-rank connectivity. If the recurrent part of the computation is small, and/or there is a strong correlation with the input at the final timestep and the output, the network will prune that dimension, leading to a low-rank RNN. While this behavior might seem an artifact
To quantify feature learning, we measure the kernel distance between the NTK at initialization and the end of training for LRNNS trained on constant task dynamics as a function of trajectory length and weight initialization scale both the aligned and unaligned case (Figure 5). As expected, we see that the kernel moves further in networks with smaller initializations relative the target (= 1), but surprisingly, the NTK still moves substantially even across larger initial- izations (Appendix J). We also find that the kernel distance increases as the network transitions from a feedforward net-
7

---

Learning dynamics in linear recurrent neural networks
yr= m
yr sum(K
Our theory also predicts that task dynamics that produce solutions dependent trajectory length will be unable to extrapolate to other trajectory lengths. Thus, since input- output modes should learn 1/T for the mean-integration task, we do not expect it to extrapolate to other trajectory lengths, while we would expect the sum aum-integration task to extrapolate perfectly.
2.
singular values
singubr
00-
24-
04
recureent
iuput-oupput glatel solution
20-
1000
$000
3,00
420
12:00
T=4
By simulating networks on the sensory integration tasks and plotting the network SVs, we see that our theory in- deed predicts the solutions found by networks for both mncan-nteegratoon and sum-integration (top row in Figure 6). As expected, we also find that the networks trained on sum-integration tasks are able to extrapolate to other tra- jectory lengths perfectly, while networks trained on mean- integration accumulate error as a function of the difference in trajectory length from that trained on (bottom row in Fig- ure 6). In Appendix I., we further extend this setting to show that our predictions about stability (early-importance versus late-importance dynamics) are validated in networks with- out our assumptions. In summary, these results illustrate the application of our theoretical framework for understanding the behavior and capabilities of LRNNs more generally.
1030
extrapolation T=
(C)
(C)
830
axtrapalation 75
actraccitionn to T 100
1200
1900
2002
2000
100
200
301
430
training stops (fe)
training steps (tg)
Figure 6. Theory predicts learned solutions and extrapolation ability in sensory-integratin producing an output that is the (left) mean of the input in cach dimension, or the (right) sum. (Top) The LRNN singular values converge to our theoretical predictions for constant task dynam- ics. (Bottom) We accurately predict extrapolation ability for sum- integration, and lack thereof for mean-integration.
task. We train LRNNs on either
work (i.e., == 1) recurrent network, indicating greater feature learning. In Appendix J. we further study the effects of rotational tasks, larger initializations and widths, and independentty-inittaiieed modes.
4. Discussion & Related Work
Summary of results. In this work, we extend the grow- ing literature on learning dynamics to a new architecture, linear RNNs. We derive an analytical solution to the energy function and learning dynamies of LRNNs under certain conditions, using novel approach that accounts for task dynamics. Unlike feedforward networks, LRNNS learn data singular values ordered by both their scale and tempo- ral precedence, with larger and later singular values being learned first. We identify how task dynamics impact solu- tion stability and extrapolation ability, an often understudied aspect of RNN dynamics. We further reveal a tradeoff be- tween recurrent and feedforward computation that leads to low-rank solutions, mediated by effective regularization term in the energy function. We extend existing work on rich and lazy learning in RNNs beyond the effect of initial connectivity by deriving the NTK for finite-width LRNNS and showing that recurrence encourages feature learning. Finally, we demonstrate an application of our results in sen- sory integration task where we relax our prior assumptions and find that our theory explains the behavior of LRNNS.
3.6. Sensory integration task
Although in this work we study a simplified setting that cannot fully capture all of the rich neural dynamics exhibited in animals, LRNNs can still learn and perform some basic computations studied in neuroscience, such as temporal integration of input and rotational dynamics (Khona & Fiete, 2021). Here, as a proof of concept, we study sensory integration task where we remove our prior assumptions about whitened data and aligned weights, and show that the insights developed in our theoretical model generalize to predict behavior in this setting.
We consider two versions of a sensory integration task where the network is given noisy input in several dimensions and tasked with producing either the mean input activity in each dimension, or the sum of the input in each dimension. We train LRNNs with small random weights, making no other architectural assumptions. In such a task, the output is equally correlated across inputs at all trajectory timesteps, thus exhibiting constant dynamics. Our theory predicts that networks trained on tasks with constant dynamics produce recurrent modes equal to one, and that the input-output modes learn to scale these dynamics. In the case where the output is a sum of inputs, no additional scaling is necessary so input- modes should become one, while in the case where the output is the mean, the input-output modes should become 1 1/T to appropriately scale.
Learning dynamics in linear networks. Differing from prior work on learning dynamics in linear networks (Saxe et al., 2014; 2018; 2022; Braun et al., 2022; Sandbrink et al., 2024; Dominé et al., 2025), we study a recurrent network, allowing us to analyze how other architectures constrain optimization in ways that differ from feedforward ones. No- tably, Schuessler al. (2020b) previously studied learning
8

---

Learning dynamics in linear recurrent neural networks
dynamics in LRNNS study how networks make low-rank changes to their connectivity, but used task with constant input in the limit of infinite trajectory length, and Smékal et al. (2024) studied learning in the frequency domain but focused on the effects of overparameterization on conver- gence time. Instead, our work accounts for the effect of task dynamics, which are critically important for modeling and understanding dynamic cognitive behavior.
factor impacting feature learning: the effect of recurrence. Although lazy learning is still possible in recurrent networks (e.g., with larger weight initializations or widths), we find that recurrence induces substantial NTK movement.
Limitations and future directions. In this work, we per- form our theoretical analysis on LRNNs with data-aligned weights, trained on tasks with input-output correlations that have constant singular/cigen- based on SVD severely restricts the expressivity of the net- work, our derivation based on an eigendecomposition (Ap- pendix N) relieves many of these limitations, whereby our framework and results naturally extend to this case. Inter- estingly, other work has shown that there are some practical justifications for using diagonal state spaces (Gupta & Be- rant, 2022; Orvieto et al., 2023), and it's also a common choice theoretical work (Hazan et al., 2018; Zucchet & Orvieto, 2024).
While our form
Stability and extrapolation. The problem of stability in training RNNS well-studied problem (Bengio et al., 1994; Hochreiter et al., 2001; Pascanu et al., 2012; Zuc- chet & Orvieto, 2024) with numerous proposed solutions (Hochreiter& & Schmidhuber, 1997; Le et al., 2015; Orvieto et al., 2023; Zucchet et al., 2023). Here, we highlight an additional, understudied factor - the impact of task dy- namics. We show how certain task dynamics (those with carly-importance) can lead unstable training regimes as a result of the solutions they drive the network to. This sug- gests that practical approaches to such problems should take task dynamics into account when designing new solutions. Although less theoretically understood (Emami et al., 2021; Cohen-Karlik et al., 2022; Beirán et al., 2023), our frame- work sheds light on how task dynamics impact LRNN's extrapolation to sequence lengths different to those in the training set and how this is driven by a mismatch between architecture and the latent structure of the data.
In this paper, make several choices when constructing the setting we study, including our focus on the single-output case (rather than an autoregressive one), our initialization of the hidden layer at O, and our use of small square networks. While we do extend our main derivations to the autoregres- sive (T (T-output) case, fully characterizing the behavior of RNNs will require charting these different settings.
Finally, although linear networks are more tractable, many computations of interest can only be implemented in RNNs with nonlinear dynamics. Thus, an important future direc- tion of theory will be to find new ways to study learning in networks with nonlincarities, potentially through gating (Saxe et al., 2022; Sandbrink et al., 2024; Jarvis al., 2025). It's an open question to what extent the findings in this work will generalize to other settings, but we believe the frame- work we have constructed is flexible and will support new research inquiries in this direction.
Low-rank connectivity. Networks with low-rank con- nectivity have been used as more interpretable models from which to study dynamics related to cognition (Mas- trogiuseppe & Ostojic, 2018; Schuessler et al., 2020a;b; Dubreuil et al., 2022), motivated by the fact that neural population activity is often low-dimensional, and it's been shown that RNNs learn low-rank solutions along task di- mensions (Schuessler et al., 2020b). Complementing this work, we identify an effective regularization term in the energy function that incentivizes small-wein weight solutions and demonstrate specific cases of task dynamics where RNNs prune connectivity modes resulting in low-rank connectivity.
5. Conclusion
This work presents theoretical study of learning dynam- ies in linear RNNs and the effect of temporally data. It presents one of the few studies of learning dynam- ics in recurrent networks, and, to our best knowledge, the first to account for the effect of task dynamics and to more explicitly connect recurrence to feature learning by study- ing the transition from feedforward to recurrent networks using trajectory length. This study generates new insights into the learning process of RNNs and encourages further theoretical developments to consider the learning process and the impact of temporal data when studying RNNs. We hope future work can characterize how complex dynamics, such as those in the brain, are developed during learning and ultimately, help us better understand cognition from a dynamic perspective.
Rich and lazy learning. Neural networks can lie in two different learning regimes (so-called rich or lazy) depend- ing on their weight initialization and width, and there is increasing evidence that these regimes are related to the representational geometry of different brain regions (Rigotti et al., 2013; Bernardi et al., 2020; Flesch al., 2022; Farrell et al., 2023; Payeur et al., 2023). Most theoretical studies of rich and lazy learning have been done in feedforward networks, with the exception of Liu et al. (2024), which showed that connectivity rank impacts features learning, and Schuessler et al. (2024), which showed that the scale of the readout acts as a control parameter between aligned and oblique dynamics in RNNs. Here we reveal an additional

---

Learning dynamics in linear recurrent neural networks
Acknowledgements
Braun, L.,, Dominé, C., Fitzgerald, J., and Saxe, A. Exact learning dynamics of deep linear networks with prior knowledge. Advances in Neural Information Processing Systems, 35:6615 1-26611-6629,, 2022.
AP funded by the Imperial College London President's PhD Scholarship. CD was supported by the Gatsby Chari- table Foundation (GAT3755). This research was funded in part by the Wellcome Trust [216386/2/19/Z]
Chaisangmongkon, W., Swaminathan, S. K., Freedman, D. and Wang. X.-J Computing by robust transience: How the fronto-parietal network performs sequential, category-base decisions. Neuron, 93:1504-1517.e4, 2017.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Cohen-Karlik, E., David, B., Cohen, N., and Globerson, A. On the implicit bias of gradient descent for tempo- ral extrapolation. International Conference Artificial Intelligence and Statistics, 151, 2022.
References
Cohen-Karli E., Menuhin Grrmman,, I., Giryes, R., Co- hen, N., and Globerson, Learning low dimensional state spaces with overparameterized recurrent neural nets. International Conference Learning Representations, 2023.
Alemohammad, S., Wang, Z., Balestriero, R., and Baraniuk, R. The recurrent neural tangent kernel. International Conference on Learning Representations, 2021.
Arora, S., Cohen, N., Golowich, N., and Hu, W. A conver- gence analysis of gradient descent for deep linear neural networks. International Conference on Learning Repre- sentations, 2019.
Dominé, C. C., Anguita, N., Proca, A. M., Braun, L., Kunin, D., Mediano, A., and Saxe, M. From lazy to rich: Exact learning dynamics in deep linear networks. Inter- national Conference Learning Representations, 2025.
Atanasov, A., Bordelon, B., and Pehlevan, C. Neural net- works as kemee learners: The silent alignment effect. International Conference on Learning Representations, 2022.
Doya, K. Universality of fully-connected recurrent neural networks. IEEE Transactions on Neural Nenworks, 1993.
Driscoll. L., Shenoy, K., and Sussillo, D. Flexible multi- task computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, 2024.
Azulay, S., Moroshko, E., Nacson, M. S., Woodworth, B. E., Srebro, N., Globerson, A., and Soudry, D. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. International Conference on Machine Learning,
Dubreuil, M., Valente, A., Beirán, M., Mastrogiuseppe, F., and Ostojic, S. The role of population structure in com- putations through neural dynamics. Nature Neuroscience, 25:783 794, 2022.
pp. 468-477, 477, 2021.
Barak, O. Recurrent neural networks as versatile tools of neuroscience research. Current Opinion in Neurobiology, 46:1-6, 2017.
Durbin, J. and Koopman, J. Time Series Analysis by State Space Methods. OUP Oxford, 2012.
Beirán, M., Meirhaeghe, N., Sohn, H., Jazayeri, M., and Ostojic, S. Parametric control of flexible timing through low-dimensional neural manifolds. Neuron, 11::739- 753.e8, 2023.
Emami, M. M., Sahraee-Ardakan, M., Pandit, P., Rangan, S., and Fletcher, A. K. Implicit bias of linear mns. Inter- national Conference on Machine Learning, 2021.
Engel, T. A., Chaisangmongkon, W., Freedman, D. J., and Wang, X.-. Choice activity fluctuations under- lie learning of neuronal category representation. Nature Communications, 6, 2015.
Bengio, Y., Simard, P. Y., and Frasconi, P. Learning long- term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5 2:157-66, 1994.
Bernardi, S., Benna, M. K., Rigotti, M., and Salzman, C. D. The geometry of abstraction in the hippocampus and prefronnal cortex. Cell, 183:954-967.c22 c,1, 2020.
Farrell, M., Recanatesi, S., and thoa-B From lazy to rich exclusive task representations in neural networks and neural codes. Current opinion in neurobiology, 83: 102780, 2023.
Bordelon, B., Cotler, J., Pehlevan, C., and Zavatone- Veth, A. Dynamically learning to integrate in recurrent neural networks. 2025.
Farrell, M. T., Recanatesi, S., Moore, T., Lajoie, G., and Shea-Brown E. Gradient- based learning drives robust
10

---

Learning dynamics in linear recurrent neural networks
representations in recurrent neural networks by balancing compression and expansion. Nature Machine Intelligence, 4:564 573, 2022.
Khona, M. and Fiete, I. R. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23:744 766, 2021.
Kunin, D., Raventós, A., Dominé, C.,, Chen, F., Klindt, D., Saxe, A., and Ganguli, S. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. Advances in Neural Information Processing Systems, 2024.
Flesch, T., Juechems, K., Dumbalska, T., Saxe, A., and Summerfield, C. Orthogonal representations for robust context-dependent task performance in brains and neural networks. Neuron, 110())1258--270,, 2022.
Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S. Deep learning versus kernel learning: an empirical study of loss landscape geome- try and the time evolution the neural tangent kernel. Advances in Neural Information Processing Systems, 33: 5850-5861, 2020.
Le, Q V., Jaitly, N., and Hinton, G. E. A simple way initialize recurrent networks of rectified linear units. ArXiv, 2015.
Liu, Y. H., Baratin, A., Cornford, J., Mihalas, S., Shea- Brown, E., and Lajoie, G. How connectivity structure shapes rich and lazy learning in neural circuits. Interna- tional Conference on Learning Representations, 2024.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. Conference on Language Modelling, 2024.
Lufkin, L., Saxe, A. M., and Grant, E. Nonlinear dynamics of localization in neural receptive fields. In Advances in Neural Information Processing Systems, 2024.
Gu, A., Dao, T., Ermon, S., Rudra, A., and Ré, C. Hippo: Re- current memory with optimal polynomial projections. Ad- vances in Neural Information Processing Systems, 2020.
Mante, V., Sussillo, D., Shenoy, K. V., and Newsome, W. T. Context-dependent computation by recurrent dynamics in prefrontal cortex. Nature, 503:78 - 84, 2013.
Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. International Conference on Learning Representations, 2022.
Masse, N. Y., Yang, G. R., Song. H. F,, Wang, X.-J, and Freedman, D. J. Circuit mechanisms for the maintenance and manipulation of information in working memory.
Gupta, A. and Berant, J. Diagonal state spaces are ef- fective as structured state spaces. Advances in Neural Information Processing Systems, abs/2203. 14343, 2022.
Nature Neuroscience, 22:1 159 1167, 2018.
Masse, N. Y., Rosen, M. C.,, and Freedman, J. Reevaluat- ing the role of persistent neural activity in short-term memory. Trends in Cognitive Sciences, 24:242-255 258, 2020.
Hazan, E., Lee, H., Singh, K., Zhang. C., and Zhang, Y. Spectral filtering for general linear dynamical sys- tems. Advances in Neural Information Processing Sys- tems, 2018.
Mastrogiuseppe, F. and Ostojic, S. Linking connectivity, dynamics, and computations in low-rank recurrent neural networks. Neuron, 99(3):609 603-623,33,, 2018.
Heij, C., Ran, A. C., and Van Schagen, F. Introduction to Mathematical Systems Theory. Springer, 2007.
Molano-Mazón, M., Barbosa, J., Pastor- Cistor-Ciurana, J., Fradera, M., Zhang, R.-Y., Forest, J., del Pozo, J., Ji-An, L.,, Cueva, C., de la Rocha, J., Narain, D., and Yang, G. R. Neu- roGym: An open resource for developing and sharing neuroscience tasks. 2022.
Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 98): 1780, 1997.
Hochreiter, S., Bengio, Y., Frasconi, P., and Schmidhuber, J. Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Term Dependencies. IEEE, 2001.
Orhan, A. E. and Ma, J. A diverse range of factors affect the nature of neural representations underlying short-term memory. Nature Neuroscience, 22:275 283, 2018.
Jacot, A., Gabriel. F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, pp. 8571-8580, 2018
Orvieto, A., Smith, S. L., Gu, A., Fernando, A., Gulcehre, C., Pascanu, R., and De, S. Resurrecting recurrent neural networks for long sequences. International Conference on Machine Learning. 2023.
Jarvis, D., Klein, R., Rosman, B., and Saxe, A. M. Make haste slowly: A theory of emergent structured mixed selectivity in feature learning relu networks. International Conference on Learning Representations, 2025.
Orvicto, A., De. S., Gulcehre, C., Pascanu, R., and Smith, S. L.. Universality of linear recurrences followed by non- linear projections: Finite-width guarantees and benefits of complex eigenvalues. 2024.

---

Learning dynamics in linear recurrent neural networks
Pascanu, R., Mikolov, T., and Bengio, Y. On the difficulty of training recurrent neural networks. In International Conference Machine Learning, 2012.
Schuessler, F,, Mastrogiussppe, F., Ostojic, S., and Barak, O. Aligned and oblique dynamies in recurrent neural networks. eLife, 2024.
Payeur, A., Orsbom, A. L., and Lajoie, G. Neural manifolds and learning regimes in neural-inde curra-interrfce tasks. bioRxiv, 2023.
Schäfer, A. M. and Zimmermann, H. G. Recurrent neu- ral networks are universal approximators. International Journal of Neural Systems, 17(4):253-63, 2007.
Smékal, J., Smith, J., Kleinman, M., Biderman, D., and Linderman, S. Towards a theory of learning dynamics in deep state space models. ICML 2024 Next Generation of Sequence Modeling Architectures Workshop, 2024.
Remington, E. D., Egger, S. W., Narain, D., Wang, J., and Jazayeri, M. A dynamical systems perspective on flexible motor timing. Trends in Cognitive Sciences, 22(10))938- 952, 2018a.
Sompolinsky, H., Crisanti, A., and Sommers, H.-J. Chaos in random neural networks. Physical review letters, 61 (3):259, 1988.
Remington, E. D., Narain, D., Hosseini, E. A., and Jazayeri, M. Flexible sensorimotor computations through rapid reconfiguration of cortical dynamics. Neuron, 98:100: 1005- 1019.e5, 2018b.
Sussillo, D. and Barak, O. Opening the black box: Low- dimensional dynamics in high-dimensional recurrent neu- ral networks. Neural Computation, 25:626-649, 2013.
Rigotti, M., Barak, O., Warden, M. R., Wang, X.-J,, Daw, N. D., Miller, E. K., and Fusi, The importance of mixed selectivity in complex cognitive tasks. Nature, 497:585- 590,,5599..000 2013.
Turner, E. and Barak, O. The simplicity bias in multi-task rnns: Shared attractors, reuse of dynamics, and geomet- ric representation. In Advances in Neural Information Processing Systems, 2023.
Sandbrink, K. J., Bauer, J. P., Proca, A. M., Saxe, A. M., Summerfield, C., and Hummos, A. Flexible task abstrac- tions emerge in linear networks with fast and bounded units. 2024.
Turner, E., Dabholkar, K., and Barak, O. Charting and navigating the space of solutions for recurrent neural networks. In Advances in Neural Information Processing Systems, 2021.
Saxe, A., Sodhani, S., and Lewallen, S. J. The neural race reduction: Dynamics of abstraction in gated networks. In International Conference on Machine Learning, pp- 19287-19309, PMLR, 2022.
Vyas, S., Golub, M. D., Sussillo, D., and Shenoy, K. V. Computation through neural population dynamics. An- nual Review of Neuroscience, 43:249-275, 2020.
Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solu- tions to the nonlinear dynamics of learning in deep linear neural networks. International Conference on Learning Representations. 2014.
Wang, J., Narain, D., Hosseini, E. A., and Jazayeri, M.
Flexible timing by temporal scaling of cortical responses. Nature Neuroscience, 21:102 - 110, 2017.
Yang, Tensor programs ii: Neural tangent kernel for any architecture. ArXiv, 2020.
Saxe, A. M., McClelland, J. L.,, and Ganguli, mathe- matical theory of semantic development deep neural networks. Proceedings of the National Academy Sci- ences, 116:11537 11546. 2018.
Yang, G. R., Joglekar, M. R., Song, H. F., Newsome, W. T., and Wang, X.-J Task representations in neural networks trained to perform many cognitive tasks. Nature Neuro- science, 22:297 306, 2019.
Saxe, A. M., Nelli, S., and Summerfield, C. If deep learning is the answer, what is the question? Nature Reviews Neuroscience, 22:55 67, 2020.
Zucchet, N. and Orvieto, A. Recurrent neural networks: vanishing and exploding gradients are not the end of the story. Advances in Neural Information Processing Systems, 2024.
Schuessler, F., Dubreuil, A., Mastrogiuseppe, F., Ostojic, S., and Barak, O. Dynamics of random recurrent networks with correlated low-rank structure. Physical Review Re- search, 2(1) :):011111, 11, 2020a.
Zucchet. N., Meier, R., Schug, S., Mujika, A., and Sacra- mento, J. Online learning of long-range dependencies. Ad- vances in Neural Information Processing Systems, 2023.
Schuessler, F. Mastrogiuseppe, F., Dubreuil, A., Ostojic, S., and Barak, O. The interplay between randomness and structure during learning in mns. Advances in Neural Information Processing Systems, 2020b.
12

---

Learning dynamics in linear recurrent neural networks
Appendix
Table of Contents
Notation
B Derivation of gradient flow equations and energy function
15
Exact solution of input-output connectivity modes
D Local approximation of recurrent connectivity modes
17
D.1 Analytical approximation using Faà Bruno formula and Bell polynomials
18
E Effeet of task dynamics on the ordering of learning
19
F Zero-loss solutions only exist for inverse-exponential task dynamies
20
E1 Discussion on exponential task dynamics as reparameterizatioo inverse-exponential task dynamics
20
F.2 Proof
20
F3 Global solutions dynamics
21
G Effective regularization term incentivizes small-weights
23
Connectivity modes exhibit phase transition as function of task dynamics
23
H.1 T case.
25
H.2 T>> case.
25
I Finite-width neural tangent kernel of LRNN
26
J Analyzing the impact of recurrence feature learning
28
Impact connectivity modes on the energy function
31
L. Farly-importance task dynamics lead to unstable solutions
32
Extending the (autoregressive) 7 Toutppu case
33
M. Exact solution of input-output connectivity modes
34
M.2 Local approximation recurrent connectivity modes
M.3 Zero-loss solutions only exist for unverse-exponeett task dynamics
35
M.4 Existence effective regularization term
................................
M.5 Neural tangent kernel
35
N Generalizing gradient flow equations the eigenspace capture rotations
35
Learning dynamics of rotations in the complex plane
37
P Simulations
41
P.I LRNN initialization
41
P.2 Training
41
P.3 Recovering connectivity modes
42
P.4 Tasks
42
13

---

Learning dynamics in linear recurrent neural networks
A. Notation
Table Al. Notation
Symbol
Description
trujectory timestep
T
trajectory length (final timestep)
tg
learning timestep
learning timescale (inverse learning rate)
learning rate
P
datasel size
data sample index
N
input size
No
hidden size
Ny
output size input sample p at timestep
Pp.t E R.
h4t1 Wa + a. e RRN
hidden state at timestep model oulput at timestep I output target at timestep f inpul weight matrix recurrent weight matrix outpur weight matrix
#: Wyhte E R
Mps E R.,
W. BNAXN.
W,, R.....
W, RN,,,
xXX,- = 2p=1 Ma
DP
input-output correlation matrix between input it, at trajectory timestep and final output target Br input-input correlation matrix between input 2; at trajectory timestep and æ singular value decomipsition of input-outpul correlation matrix for inpul at timestep / cigendecomposition of input-output correlation matrix for input at timestep diagonalized input matrix
NNNY DP p
V. D
PD,PÍ = X.
W.
WA
diagonalized recurrent matrix diagonalized output matrix
&
singular/eigen-van value dimension
imput connectivity mode at dimension recurrent connectivity mode at dimension oulput connectivity mode at dimension o singular value of S; at dimension eigenvalue of D; at dimension
ba
Bo.t
da.t
E
energy function of connectivity modes decoupled along singularicigen-value dimensions constant component/jarameee of data singularleigen-valle (when s,,4, 85(A, ())
dynamic componen//parameet of data singular/cigen-valll values (when 8t, de 5f(),t) t))
((A,1) =
constant task dynamics
[(A.t) = -t
inverse-expomential task dynamics
[(Att) =
exponential task dynamics
'feedforward computation' (= sy) in phase transition experiments
'recurrent computation' (= in Dirac delta task dynamics
indicating teacher parameters
single variable for input-output modes indicating global solution
Xy = Ep 3
input-outpur correlation matrix between input at timestep 1' and output at timestep (1' s () for autoregressive case conjugate transpose matrix complex conjugate radial component of (= Rieeti)
Rs
do
angle component 5 radial component of À (= Rye angle component of A
Rx
14

---

Learning dynamics in linear recurrent neural networks
B. Derivation of gradient flow equations and energy function Recall our model definition as
h Whhr +
(12)
Σ
W
(13)
==1 9: = Wy+++
(14)
with loss of
P
T Wg
L
Up.T
W7
pp:ill
(15)
2
p=1
By taking the derivative
loss with respect to each set of parameters Wz, Who Wy, we get the following equations
aw,
7c
W
WTT-OTWT..
h
W.
yr
Σ
Wy WTWWT.WWo.... w.p))
ZiP
(16)
p=1
T
x W W(T W( )
7C
Σ
Σ
Σ
w W
yr
Wy W WWa
(17)
aW, p=1
T
OC
Σ
yr
Σ
W W -
T Di W W (T-
(18)
ow,
We define the input-output correlation matrices between an input at trajectory timestep and the final output as
yp. p.t
(19)
(20)
and the input-input correlation matrices between two inputs at trajectory timesteps t, t' as
XX
Σ ap. p.u'
(21)
p=1
(22)
Under the assumption of whitened input with 0 mean, the input-input correlation matrices become XX.X, 0, 14 4 L' and
XXXXi T. Substituting the correlation matrices and assuming the gradient flow regime where the learning rate (17 1/T) is small, we can rewrite the gradient equations above as set of differential equations over training time to
d
5 W.
W h
WW, W WT - - (W.)
(23)
dtg
T-1T-i-1 (r) W (Σ W.W T
Wi- =
(24)
dtg
W, dtg
d
E W,W, -w.--W,) W.W.--WWN W W!T-OT
(25)
We assume that the input-output correlation matrices have constant left and right singular vectors across trajectory timesteps, such that only their singular values vary through time. Although this may seem like restrictive assumption, note that this assumption holds for any data generated by a teacher linear RNN with weights that can be diagonalized.
15

---

Learning dynamics in linear recurrent neural networks
Proof. Data generated by linear RNN teacher parameterized by Wa. Who Wa that can be diagonalized with SVD has constant left and right singular vectors across trajectory timesteps.
P Σ p=1
yVX,
Up T p.t
(26)
Σ WyW' p=1
(27)
WWW
(28)
We assume W, is diagonalized by orthogonal matrices Rg R2 such that W,, R. Wa RyS.VV Wn R Then,
Wy WT SVS.Va
(29)
We place additional assumptions on the temporal dynamics of the singular values through time S,, such that they could be generated by any dynamic process. Substituting the singular value decomposition (SVD) of the data-correlation matrix into the gradient flow equations yields
d
W2
Wh
TT TT
WT (U,SVV. WWW.-'W.)
(30)
dtg
i=1
T-1T-i-1
T
d
Wh
Σ
W W (U,SVVV W,W.- W. W. WT.. ((T---1-M)
(31)
dto
t=1
d
T
Wy
Σ
(USS.VV. - W, wwW-wWW.. W. (TT i=1
(32)
dla
Similarly to Saxe et al. (2014; 2018), we assume the LRNN is data-aligned at initialization such that for some orthogonal matrices Ry. Rz, R, W, W, (0) R₂ = (O), R.J w- (, W_ (0). U, W,) = (O), where W z: W W, are diagonal matrices. Atanasov et al. (2022) showed that this alignment happens early in training for networks initialized with small random weights. Performing a change of variables in the gradient flow equations and simplifying yields,
d
W.
W W
- W W W.) W
(33)
dtg
d
W h
T-17-i-1
Σ
Σ
W y (S, W W W W. W Wh T-i----)
(34)
dtg
t=1
T'
T dtg
W y
= Σ (S. -
W W W W W W(T
(35)
1=1
Let ag. by: ca the th diagonal entry W WA, h, y. respectively, and Sa,t be the th singular value of S.. We can then rewrite the above equations in terms these variables, or connectivity modes that decouple along singular value dimensions
16

---

Learning dynamics in linear recurrent neural networks
0,
d
T
ba
(8a.)
(a0)
(36)
dig
1=1
d
T dig
ba
Σ
(er (5a, -aabaaaan aanaaaa aanaaaba ((T-----)))
(37)
0
T-1
Σ
(T' i)ca (Sa, - cata aaaaaaaaaab ((T--))
(38)
T
T
Ca
Σ
Sa,i Caba b aa a aaba
(39)
dig
1=1
These dynamics arise from gradient descent on the energy function
E
80. - Co b2 1-2--' an))
(40)
2T
To ease notation, we omit specifying a when referring to connectivity modes, although note that all terms (ss, a, b, still refer to particular singular value dimension a.
C. Exact solution of input-output connectivity modes
We solve for the learning dynamics of the input-output connectivity modes when the recurrent connectivity mode is frozen. If we assume balanced weights such that c, we can solve for both modes u ac together.
d
d
u c(T a) + a(T c)
(41)
dlg
dla
dig
T
T ab"
c(ss ob a))
cb--1a))
+ a
T --((s
(8 ccbT+aa)
(42)
2u
T - -15T-(u)
(43)
This equation can be integrated to yield
((()
du
to ((0)) 1 2ubF T (88 - (ubT-1)
(44)
log(u) log ΣT= i= br --62(T-- u(to)
(45)
Σ -i-0
x(0)
u(to) (tt)() -i=1 (2 6- 13; - (00622(r-) 0)
T log
(46)
2 2 6 -i;
s;
u EE) i=1
T 6T 6- -3; - u(to) ((to) -0)
2 ET. b7-1si)/T (EF= 6² -s))
u(ta)= EE bt (T-is:)/u(0) (ET ()( + 2200 (2in 6r 20:/0 (ET, (01/
(47)
(2(T-00)
D. Local approximation of recurrent connectivity modes
Due to the exponential term, the learning dynamics of the recurrent mode b are difficult to solve for. Instead, take an approach similar to Schuessler et al. (2020b), by performing Taylor expansion the learning dynamics of b through training time (with input-output modes held constant),
d'b(0) (ta/T)"
b(to/T)
(48)
dtp
17

---

Learning dynamics in linear recurrent neural networks
First solve for the rith partial derivative of the energy function E with respect to the recurrent mode b which has an explicit closed-form solution given by
E diy
T-n
(2T n//2
I (2T - 2i 1)) -
(T - i)
II [T ( s;cab"-inn
Σ
(T- - i)
20262r
(49)
Recall that in the gradient flow regime, the recurrent mode b changes continuously according to higher-order derivatives of 6,
db dE de
Thus, to compute
d5
d
d
dm-1b
dtg
=
dtg
dtg
(50)
d dm-ib b.
db
(51)
db
d
dtg
d b
dE
(52)
db dtg-1 db
Note that this is a recursive operation and does not give a simple closed-form expression.
Applying this to higher-orders and using chain rule, we compute the dime-derivatives of up to 5th order,
db dE dtg db
(53)
dbb d
dE
(54)
db db dbb
dt
F
& dE
dF
(55)
dt3
db2
db3
db
db
dtb
E
d E JE
d
E
dE
dE
(56)
dtg
db2
db
db
db
db
db
dbb
df=FF (BR) db2
F
d
F.
dF
8 F,
) /2 )² +
dF
d' F. de d.
d
dF (
dF
11
+ 4
y²
)3
(57)
dt5
db3
db
db
db3
db
db
db2
db
dbs
db
db
We then approximate the learning dynamics of the recurrent mode b and substitute the formula above for the nth partial derivative of the energy function
dE db
FE (Lg/T)2
PF db2
FF
dE
(Lo/T)3 3!
b(to/T) R2 ((0) +
(to
(58)
db
db
2!
db3
db
db
E 8
dE
CE
da E
dE
dE
(ta/T)²
(59)
db
db
db
db
db
4!
( ) + 11
L
dF
F.
d..
d d
E
dE
F.
dE
d.
(lo/T)
4(
2 + 7
)2
(60)
db2
db
db
db
db
db
db
db
db
dbs
db
db
5!
+ 066)
(61)
In practice, when simulating the learning dynamics using this approximation, we apply the solution locally across a window of size A and iterate over each window b(tp to + A). The window-size is dependent on the smoothness of the connectivity mode dynamics (i.e., how sharp the gradient is).
D.I. Analytical approximation using Faà di Bruno formula and Bell polynomials
Here we use the Faà di Brumo formula/Bell polynomials to write out a combinatorial solution for the nihh derivative b, which can be used to expand the learning dynamics of b higher orders without repeated recursive chain rule. Using this
18

---

Learning dynamics in linear recurrent neural networks
approach, the nth derivative of is
b
dk+ E
db
dbb
dm-bb bb dim-k h
B
(62)
dty
dbè+1
dtg"
dt
d E
(n 1)!
d dtp
Σ
my
(63)
dbb+1
my !mn! ... mm- b
j!my
k=1
{m1 (m Rn k)
where the summation over {m1, m2...... mp-kk indicates summation over all l partitions of nonnegative integers satisfying
mi + m2 + ... mn-k = k:
(64)
1m, + 2mg +.... + (n k)mn-k 11
(65)
Although useful, we note that this approach still requires substitution of other lower-order terms of b (because of the d'b term). The equation can be substituted back into the Taylor expansion through training time,
de.
0 b(te/T)= Σ
d0) (ta/T)"
(66)
dt
b(0)
dE
ta T) +
n
d E
Σ
(n 1)!
n
1
d'
m3
(ta/T)"
db
dbk+1
mi mm M. k
jim
dig
{mi Ma.
j=1
(67)
E. Effect of task dynamics on the ordering of learning
To illustrate the effect of the ordering of singular values on learning speed, compare task dynamics where the network connectivity modes learn solutions of the same magnitude (for different modes such that input-output and recurrent modes "swap" solutions), but the ordering of singular values is either ascending or descending. More specifically, we consider the case of nnverse-exponential task dynamics given by s, Ó, f(A t) = X -t, The solution for inverse-exponentia dynamics are ac = 5, b = X. Thus, we switch the values for 6, X (orange: 5 = 1.1, = 0.5, blue: = 0.5, À = 1.1) for two simulations so that the network connectivity modes learn solutions of the same magnitude (so that we somewhat control for the effect that larger singular values has on accelerating learning speed), but the task dynamics either have ascending or descending singular values over the trajectory length, and, more importantly, the magnitude of the SVs at the end of the trajectory differ. We indeed see that modes trained task dynamics with larger singular values occurring later in the trajectory learn faster. As we discuss in the main text, this is due to the fact that the recurrent connectivity mode i scales the gradient contribution for early trajectory timesteps exponentially. Since b is initialized to be less than 1, this has the effect of downscaling the gradient contribution of earlier timesteps compared to later ones. This manifests singular values occurring later in the trajectory to "contribute more" to learning (when b < 1), such that modes trained on task dynamics with larger and later singular values learn faster (because they have larger gradient updates).
19

---

Learning dynamics in linear recurrent neural networks
5
1.0
1.0
S; > St+1
connectivity modes
task dynamics (s))
0.8
0.8
0.6
0.6
0.4
0.4
0.2
recurrent (b)
0.2
onputt (ac)
0.0
0.0
100
200
300
400
500
training timesteps ta
trajectory timesteps (t)
Figure AI. Large singular values occurring later in trajectory of the task dynamics accelerate learning compared to earlier ones. Networks trained on task dynamics with larger singular values occurring later the trajectory learn faster.
This relates the well-studied problems of learning of long-range dependencies and vanishing gradients in RNNs, as data from earlier timesteps are harder to learn trajectory length increases because of the exponential downscaling effect of the recurrent mode. Here, we build these ideas to understand learning speed and its dependency task dynamics, including both the ordering of singular values and their scale.
This also relates to our study on the tradeoff' between feedforward versus recurrent computation in LRNNs in Section 3.4. Because later singular values have a greater effect on the gradient of the loss, the feedforward computation is effectively favored in learning, although the cumulative effect of the recurrent computation can course outweigh this.
F. Zero-loss solutions only exist for inverse-exponential task dynamics
F1. Discussion on exponential task dynamics as reparameterization of inverse-exponential task dynamics
In the main text, we refer to inverse-expon eepoeeelepponntiiall and exponential task dynamics separately. Here, we make the distinction that although we refer to these separately. they can both be rewritten as reparameterizations of each other (i.c., exponential task dynamics can be rewritten in the inverse-expomential form and vice versa) specifically when the trajectory length is held constant. We refer to eevers-expooentte and exponential dynamics separately in the main text primarily to distinguish between cases that extrapolate (or don't) and illustrate how RNNs can learn perfect solutions that do not match the ground-truth data generating process.
In particular, for fixed T, exponential task dynamics given by (A, where f(A,t) X can equivalently be written as = _A g(,, t) for g(X,t) t) = TT
This illustrates how and why LRNNS can still learn a perfect solution to exponential task dynamics for a fixed trajectory length (by overfitting), but because their architecture does not match the latent structure of the ground-truth data generating process, the network will not extrapolate to other trajectory lengths. More generally, we can see how mismatches between latent task dynamics and the network's recurrent dynamics can lead non-extrapolating solutions. Of course, this does occur when data is generated by teacher network with matched architecture because the latent form of the task dynamics and recurrent dynamics are the same.
F.2. Proof
Here prove that zero-loss solutions only exist for inverse-exponential task dynamics. Recall the energy function is given by
1
E
(8,
cb
a)2
(68)
2T
20

---

Learning dynamics in linear recurrent neural networks
The only task dynamics with zero-loss solutions in aligned LRNNs are those with inverse-exponential task dynamics (st -0
Proof. The term (s; ob²- is a quadratic function and thus is nonnegative for real numbers, meaning that the O-loss solution to the energy function must satisfy (s; cb"-=a) 0 vi. Furthermore, because (s; cb"-ia) -a)² 2 is quadratic function, there exists unique global minimum when = cb. Thus, the only zero-loss solutions exist at cb cbT-1s 8t which can only be all satisfied simultaneously when = 5x"--
F.3. Global solutions of task dynamics
Here, we solve for the global solutions of constant, inverse-exponentill and exponential task dynamics.
F.3.1. CONSTANT TASK DYNAMICS
Constant task dynamics are defined as singular values 8; = (f(,, where f(A,t) = 1, vt, such that 81:7 = 5. The global solution for constant task dynamics is given by 1, aC 6.
Proof. The 0-loss solution corresponding to the global minimum given by ac, must satisfy Σi= (s: of This condition will be satisfied cb"-'a s; Vi. For singular values following constant dynamies, this becomes
Σ (8 cb-
(69)
At T, the expression becomes
T
(70)
6
(71)
Then, substituting ca when i = T
cb T- (T -1) a = 8
(72)
cba = 8
(73)
b =
(74)
Substituting the solution into our original expression
T Σ (6- - -(1
(75)
T Σ 1=1
(6 - 6) =0
(76)
F.3.2. WWVERSE-EXPONENIAA TASK DYNAMICS
nnverse-exponentii task dynamics are defined as singular values s; = 8f(A,t) where f(A,t) = XT The global solution for inverse-exponential task dynamics is given by X. ac
Proof. The 0-loss solution corresponding to the global minimum given by ac, must satisfy condition will be satisfied if ct 8; Vi. For singular values following inverse-exponential dynamics, this becomes
This
T
(8AT cb--aa)
(77)
21

---

Learning dynamics in linear recurrent neural networks
At i T. the expression becomes
cb I a = -8 8X
(78)
ca
(79)
Then, substituting ca = when i = - 1,
cb² rb (T 101) 8X --T--17 1T
(80)
cba 8λ
(81)
b b= X
(82)
Substituting the solution into original expression
Σ -
-8
(83)
F.3.3. EXPONENTIAL TASK DYNAMICS
Exponential task dynamics are defined as singular values 8 6f(A,t) where f(A,t) = At. The global solution for exponential task dynamics given by b 1/2, ac 8X7.
Proof. The O-ooss solution corresponding to the global minimum given by ac, must satisfy Σ (8: cbf-+aa = 0. This condition will be satisfied cbT-'a = s; Vi. For singular values following exponential dynamics, this becomes
Σ (o0 - cb
(84)
1=1
At i T, the expression becomes
ob 8X
(85)
ca = 8A
(86)
Then, substituting ca _x when 1,
ob T- (T 1) = 8A --
(87)
cba 8 AT
(88)
b X
(89)
Substituting the solution into original expression
(8X (1/A()---
(90)
- -X
(91)
Σ
(ox² - -xx) =
(92)
1=1
(93)
22

---

Learning dynamics in linear recurrent neural networks
dota-driven torm
effective regularization term
error
recurentt mode (6)
$1 8 0.5
0.12
1.5
1.5
1.5
51 = B 1.5
recurent mode (b)
0.06
0.04
1.5
1.5
1.5
1.5
$1 B 2.5
recurnent mode (b)
input output (ac)
Input output
input output mode (ac)
Figure A2. Effective regularization term is constant across task dynamics and incentivizes small weights. The phase plots show the energy function terms: (leff) the data-driven term, (middle) effective regularization, and (right) the error/energy, a function of the connectivity modes for Dirac delta task dynamics with (top) $1 0.5, (middle) = 1.5, and (bottom) $1 2.5.
G. Effective regularization term incentivizes small-
As discussed in Section 3.4, the energy function be split into data-driven term and effective regularization term. Note that the form of the effective regularization term comes from the fact that Σ (2(T-() geometric series that be rewritten as 1-²² We plot the two terms, as well as their sum (the error), across different connectivity mode values to visualize the energy landscape (Figure A2). We this for Dirac delta task dynamics where we keep the singular value at the last timestep constant (ST = 1) and change the singular value at the first timestep across each row (s1 = (). We can how the effective regularization term affects the energy landscaping, forcing parameters remain in small-weight regime and how it mediates between the transition from feedforward computation to recurrent computation by forcing the input-output mode to decrease in magnitude when the recurrent mode increases in magnitude.
1-
H. Connectivity modes exhibit phase transition a function of task dynamics
Here, we use Landau theory to show that the qualitative behavior we observe in Section 3.4 is characterized by phase transition in the connectivity modes as function of the ratio between recurrent and feedforward computations. We prove that for trajectory length T 3. there is smooth second-order phase transition and that for larger trajectory lengths 7 3, there are first-order phase transitions.
To simplify the problem, we consider Dirac delta task dynamics and show that the ratio of $ acts as control parameter mediating the phase transition of connectivity modes up to a scaling.
Proof. The phase transition of the Dirac delta task dynamics is solely controlled by the ratio 8 $1/sr (up to a scaling). Treating the onput-output weights as one term ac, the energy function for Dirac delta task dynamics can be written out
T-1 (-ub
E (81 1112 (ST 2)2+
Σ
T-ip
(94)
23

---

Learning dynamics in linear recurrent neural networks
Let u' 8r. Then, by dividing the energy function by s²z, we get
E
81 u'b" 1122"-12- + (1 w²2² + Σ (-u'b-
T-1
(95)
ST
We can see from this form that the impact of the data on the energy function can be reduced to ratio of s1/ST, up to a scaling.
Because only the ratio matters, make the substitution 91/5T f to reduce the number of parameters. This is equivalent to the Dirac delta task dynamics given by
B
ift = 1
1/B
= T
(96)
0
otherwise
In this form, as 3 increases, the recurrent computation increases and the feedforward computation decreases. This simplification still captures the phenomena we're 're interested in: how the ratio between the recurrent and the feedforward computation affects the connectivity mode phase transition.
Returning back to the original form of the energy function,
1
E (B ub² 11²
u)²
+
(97)
3
T-
1
2u
= 32 28rb T-
+ u²
2
+
+ 2
+ n2
(T
(98)
32
B
1=2
T-1
= 32 B2
236 T-
+
1 (TT-(T-1)
+
(2(T-i)
(99)
B
1=
ET 626
We can then solve for the minimum of the energy function with respect to the input-output modes (letting x* denote the optimum)
0E
T-1
2(300
+ 3 +
2n
2 = 0
(100)
du
==0
TbT
+ 1
u*
(101)
T-1 === 821
Substituting back into the energy function and simplifying yields the effective energy E E*(B) where u is always at its minimum:
(96T- + 1
(*) = -2
(102)
g2
0- b2i
The trivial (symmetric) solution to minimize E is given by = 0, yielding E = E39 A phase transition requires that for some range of there is another solution where b + and F. ,
24

---

Learning dynamics in linear recurrent neural networks
H.1. 7' case
We first consider the case where T' 3 and show that there is continuous second-order phase transition as 3 increases. We expand the energy function for near 0 to identify critical point.
(962 + ))
1
(3)
B²
+
(103)
1+ + b
+ 26 k264
1
+
(104)
8
1 62 + 64
Using a binomial approximation (1+12 + b4)
- 132 14
1
* (B)
=
3²
-
+
26 112bt(11 b1- 6*)
(105)
g2
82
1
1
= 32
2}1²
+
2 -B2 - (2 + 128
+
(106)
B2
82
which takes on the Landau form of F(3,b) 6) Fo + A(3)62 + 2 B1 + .... We identify continuous second-order phase transition given by the quadratic coefficient A(5) = 3 the coefficient of b²
2. Solving for the critical point where there is sign change in
1
(107)
V2
We can see that at this critical point, A(3) = 0 and the system becomes unstable to fluctuations in b. Then, for V2' A(3) and the minimum of E' exists at some value b + 0. This also marks a transition u* due to its dependence on b.
H.2. T
case
Recall the effective energy function given by
21 (2)2(T-1)
E (B) = 32
3"
(108)
B3
621
1i=0
We rewrite the geometric sum ET- T-1
b26
1-627 as
Substituting in and expanding the terms yields
62
+ 26
26 + (2 -82266
* (3) 32
(109)
g2
1 62T
observe that for b near 0 (i.e., the denominator 62T R 1), the quadratic coefficient A(3) = zZ is always positive, indicating that there is small-h instability and no second-order phase transition as B changes. Hence, we instead consider the large case to study whether there is a point where large b overtakes the trivial solution. Rewriting the energy function in terms of the dominating terms,
From this expression,
1
122
E* (8)
82
(110)
122
(111)
g2
Thus for large b, the energy becomes E gr. Comparing to the energy for the trivial solution = 33, we that for large b and B > 1, the energy is lower than the trivial solution J < 82), indicating that the system has a first-order transition at 1 nonzero and corresponding change x*
25

---

Learning dynamics in linear recurrent neural networks
optimum Input output made (u* (A11
optmum recurrent mode (6-161
1/82
88a
eneray (())
1.5
3-
6-
1.0
0.5
7-3
7
2-
100
0.0
0-
0.5
1.0
1.5
2.0
1.5 recurrent computation (8)
1.0
2.0
0.5
1.0
1.5
2.0
recurrent computation g
recurrent computation (B)
Figure A3. Connectivity modes undergo phase transition the recurrent and feedforward computation is varied. The change (left) recurrent and (center) input-output connectivity modes as a function of 8 are plotted, as well as (right) the minimum effective energy. The vertical dash-dot lines denote the critical points identified at 1/v2 and 1. The black dashed line in the center figure denotes the trivial solution for the input-outpil modes at u Ø2 and in the right figure denotes the energy for the trivial solution at E = B².
We verify our work in simulation by plotting the minimizing recurrent modes and onput- modes, and the corresponding energy, for different values of / in Figure A3. As predicted, a second-order phase transition in b occurs at the critical point v2 for 7 = 3 and a first-order phase transition occurs near the critical point for > 3. This accompanied by the deviation 1 of the input-output modes from the trivial solution z and with an energy lower than the trivial solution ,82, For larger trajectory lengths, the simplifying approximations we make accumulate some error such that the critical point occurs at values slightly larger than 1, but we observe empirically that the critical point stabilizes around = 1.08 across larger trajectory lengths.
This matches what we observe empirically in Figure 4, whereby the recurrent mode b remains close to 0 and the input-output modes ac match the singular value at the last timestep (3T = Λ)) for small values of 8 (8). As 33 increases and the ratio of $1 31/5T changes, the energy function reaches critical point marking a first-order phase transition, where becomes nonzero, accompanied by a rapid change in ac.
Here adopted the Dirac delta setting for mathematical simplicity, but that similar phase transition of connectivity modes occurs for other task dynamics dependent on the relationship of the 'recurrent" computation (i.e., $1.T-1) to the 'feedforward' computation (sT) (Figure 4). This suggests an inherent interplay or trade-off between recurrent and feedforward computation in LRNNS (in the single-output case), dependent on the underlying task dynamics.
I. Finite-width neural tangent kernel of LRNN
Here derive the finite-width neural tangent kernel (NTK) (Jacot et al., 2018) for LRNN where the loss is computed over the final output of the network, following similar approach Braun al. (2022) in deep linear networks.
Recall the network function of the LRNN at training step to is
T Fr,to (X1:T) W, Σ W W.-WW,X
(112)
After taking training step with learning rate the network function becomes
OL T-i
JL
T1.001(X1.1) (W, aw,
Σ (w
)x;
(113)
JW,
''ow,
26

---

Learning dynamics in linear recurrent neural networks
Using the binomial expansion (a- ( a0
(Wn. "oWn
y
T-i
-i
Wh-
Oc "Wn
)*
(114)
k=U
T-
0L yh
+
wT--i-k (1) W
(115)
h
k
h2
A=
Substituting back,
dL
T
T-i Σ
-1
1)
WT-i>* h "11 Oc >WW, -Waw )x
OL
Yrso+1 (X1:) -W, 17 aw, y
> (WT-'
(11
(116)
k
aw
1=1
T
1)
Wh wi--i (1 ac
Σ W, W WT. WW, + WWO
W',,,,
(117)
k
awn
1=1
-Won 0L Ow, X; Wyn Σ
T - i
W (1 ac )b) 2C X, aWn oo
Wh i
(118)
k
DL WT-iW, "W, X,- 11 Oc
-i WT WT-i- (1 Oc >W, Xx
"ow, 9
aw, " k=1
OW
(119)
+ 11 ow, 4
Ac WW. OC Oc
2; + ow,
n
5 (1)* k=1
T-i
(Tx
(T=))
T k
W i (1
OL yh) OC
x,]
(120)
ow, x
oWn
Ow, x
The gradient flow equation describing the dynamics of the network function is then
Yr.t+11
VT.
T
T-í
( 1)
W w k- ( ac )k W - W,WT OL X; OW
1
(Tx
(121)
17
k
ow, x
h=1
W
T-i
1-1)*
--
W - (n awn h Dw, X, ow, i=1
ac )d) Oc
Σ Wh
W NT-'W,.. X,
(122)
k.
OC
1-1) (T-1) - (17 awn
Oc
OC
w Oc
)WW W. Ow, ow, X;
(123)
aw, y
Oc
(-1 k T -
W/: W -ick (1) dc )k) OC
+ ow,"
Xi)
(124)
k
oW, Ow,
y
As the learning rate 77 + 0 (the gradient flow regime),
Y, dta
T
Oc Wo
OC W₂ Oc
T
aw - (T - -W,,
WT WTT'',,)x,
(125)
OWn
ow, y
Substituting the partial derivatives of the loss
Yr dig
W TT
WT (Yr W,, Σ WT W Wx,x)x,
(126)
- -T-1)/6
-TT-
i)W WT
w WT (Yr Wy Σ
WT-
X W. W (T (T---1-)) TW,X (127)
k=1
-
-(Yr
Wy v
WT
Σ
x
W W J
(128)
3=1
27

---

Learning dynamics in linear recurrent neural networks
Finally, we use the identity vec(A (B" AX)ec(X) to derive the NTK VVovec(Yr) the left-side of the vectorizing function
dvec(Yr))
T T-1
ΣΣ
-X X, & W. W, W
T
(129)
dtg
j=1
T T-i-1
Σ
-T- xx
W W, W., W WT)
w/ W
(130)
T IN, x, W W W TT (T Wxx)voo w
w, Σ WWT
(131)
T T-1
Vovec(Y))Vooee()))
x X; 2 www. ( W
(132)
1=1 j=1
T-1-1
Σ
-(T 1) x W w W.X, R W, W, " w
(133)
IN, @ x, W (T
T) wh w W2X,)
(134)
J. Analyzing the impact of recurrence on feature learning
Here we study how increasing trajectory length T' (becoming more 'recurrent') and the scale of initialization weights influence the learning dynamics of the network, as measured by the movement of the NTK and illustrated Figure 5. We conduct several different experiments in networks with aligned and unaligned weights and observe that both settings yield similar qualitative behavior. Networks with unaligned weights are initialized with weights drawn from normal distribution centered at O with specified variance (initialization scale), and networks with aligned weights have singular vectors (or eigenvectors for the rotational task below) aligned with the data correlation matrix singular vectors (eigenvectors) and connectivity modes initialized at the corresponding initialization scale. Hence, both settings have the same expected spectral radius (equal to the initialization scale), but different means and variance (unaligned: mean is O, variance is initialization scale; aligned: mean is initialization scale, variance is 0). During training, we calculate the kernel distance of the NTK at training step to (K(t)) from its initialization (K (0)) quantify feature learning. Following the definition in Fort et al. (2020), the kernel distance (ta) is defined
(K()),((L)) ((6)) |[(o]]
(135)
We visualize the kernel distance between initialization and the end of training as heatmaps, varying the trajectory length and the initialization scale to study how these two factors impact the NTK. In the main text (Figure 5), we show the case where networks trained perform perfect integration (summation) of input (i.e., constant task dynamics where = 8 8, = 1). For completeness, in Figure A4 we also show networks trained on rotational task dynamics of angle Φi = z/12 and radius RA = 1 (i.e., inverse-exponential task dynamics where X = f 8 1, = TAT-t, see Appendices N and O for details on rotational task dynamics).
We consistently see that recurrence (T > 1) leads to kemee movement across a large range of initialization strengths which contrasts with feedforward networks (T = 1). This occurs in both aligned and unaligned networks.
28

---

Learning dynamics in linear recurrent neural networks
aligned weights
unaligned weights
trajectory length G
2-
E2-
0.8
kernel distance
trajectory length
3
5
5
0.2
7-
7
0.0
0.01
0.06
0.12
0.17
0.23
0.39
0.45
0.01
0.06
0.12
0.17
0.23
0.39
0.45
0.5
initialization scale
initialization scale
Figure A4. NTK distance is qualitatively similar rotational task. Phase plots illustrating the kernel distance of the NTK from initialization to the end of training. Networks are trained to perform rotations of angle Φi = x/12 and show qualitatively similar behavior as networks trained on constant task dynamics in Figure
Infinite width. As stated in the main text, a network's learning regime is determined by its initialization scale and its hidden layer width. It's known that as networks increase in width (infinite-width limit) and/or initialization scale of the weights, feature learning is reduced and instead lazy learning occurs. Since our first experiments (Figures 5 and A4) show that the NTK changes more as networks transition from feedforward recurrent networks, a natural next question whether lazy learning is still induced by large initializations and network widths in recurrent networks. To study the impact of increasing network width, repeat the same experiment networks with unaligned weights trained on constant task dynamics (Figure 5), keeping all parameters identical except for the hidden layer size, which is set to 300 (rather than 4 as before), while the task itself only spans 4 singular value dimensions. While the RNNs we study are still small compared to the size of networks used for typical applications, they are still significantly overparameterized relative to the task we train on and serve as a qualitative comparison between different network widths. In Figure A5, we see that, although there still substantial kernel movement, especially as trajectory length increases, this occurs primarily at smaller initializations compared the prior experiments. Although we only study network of size 300, it's possible that this pattern continues with larger network widths (Alemohammad et al., 2021), as in feedforward networks.
1
trajectory length (
2
0.8
kernl distance
0.6
4
5
0.4
6
0.2
7
8
0.01
0.06
0.12
0.17
0.23
45
0.5
initialization scale
Figure A5. Effects of recurrence on NTK movement are reduced with increasing width. Phase plots illustrating the kernel distance of the NTK from initialization when the hidden layer size set to compared size in LRNNs with unaligned weights. There still movement in the NTK, but smaller initialization scales compared to networks with smaller widths.
29

---

Learning dynamics in linear recurrent neural networks
Large initialization scale. Next, we study the effect of larger initializations (which typically induce lazy learning) on kernel distance. This is more challenging do in recurrent networks than feedforward networks because of the instability of training initializations approach 1. However, networks with aligned weights are comparatively more stable during training because their optimization is slightly simplified from its initialization. We show the results in Figure A6. As we can see, the effect of longer trajectory length yielding greater kernel distance persists across large initializations as well, especially in unaligned networks. However, we also see, especially in aligned networks, that this kernel distance decreases as initialization approaches 1. This is expected. Under the aligned configuration, (singular/eigen-) vector rotation (which typically occurs early in training in unaligned networks in the rich learning regime) align with the task unnecessary and only the scaling from initialization changes. In this case, when the network's 's initialization is close to the target value, the adjustments required to fit the target are minimized and the kernel distance is smaller. In the tasks we study here, the target has magnitude of 1, explaining why smaller initialization scales result in more pronounced NTK movement and why initializations close to result in minimal movement for aligned networks. In contrast, unaligned networks still have pronounced kernel distance at large initializations likely because vector rotation still substantially changes the NTK beyond scaling of (singular/eigen-) values. The fact that kernel distance still substantially changes at large initialization scales quite surprising, but supports the idea that the optimization of the recurrent layer might force some form of feature learning. In parallel with the publication of this work, Bordelon et al. (2025) has shown how weight initialization scale and outlier eigenvalues affect feature learning.
aligned weights
unaligned weights
trajectory length E
E
kernel distance
trajectory length
0.6
5
5-
0.4
0.2
7
7
0.02
0.13
0.24
0.35
46
0.56
67
0.89
1.0
0.02
0.13
0.24
0.35
0.46
0.56
6.
0.78
0.89
1.0
initialization scale
initialization scale
Figure A6. Effects of recurrence on NTK movement persist across larger initializations. Phase plots illustrating the kernel distance of the NTK from initialization across larger initializations for (left) aligned and (right) unaligned weights. The white boxes indicate NaNs. As expected, networks with aligned weights decrease in kernel distance as their initialization approaches the target value at because there is no (singular/cigen-) vector alignment occurring and the change in parameters to fit the target is minimal. However, networks with unaligned weights still exhibit substantial kernel movement even for large weight initializations, which is particularly surprising.
Independently initialized modes Our final experiment studies how the initialization of input-output modes and recurrent modes affect kernel distance, and whether there is any meaningful difference between them. This experiment is motivated by Schuessler et al. (2024), which found that the initialization strength of the readout weight (analogous to output modes in our case) acts as a control parameter in RNNs that induces either aligned ('rich") dynamics, or oblique ( ('lazy") dynamics. Although we do not explicitly study the alignment of the dynamics (note that aligned dynamics differs from what we here refer to as aligned weights), we are interested in seeing how the scaling of the input-output mode might impact kernel distance when varied independently from the recurrent mode. We note, however, that our setting differs from Schuessler al. (2024) in that we study linear RNNs, which might substantially change network behavior.
Interestingly, by independently varying the initialization strength of the onput-output modes and recurrent modes, see that the initialization of the recurrent mode more strongly impacts kernel distance than the input-output modes- the recurrent mode mostly determines the NTK distance. Intuitively, this seems reasonable given that the NTK distance increases with trajectory length (corresponding to the 'recurrent mode' entering into the function performed by the network more). This result also supports the idea that the increase in NTK distance with trajectory length driven by the repeated application of the recurrent mode and modulated by its initialization. We suggest that the constrained optimization RNNs might
30

---

Learning dynamics in linear recurrent neural networks
T=2
T=
input-output initialization Scale
input-output initialiiztioo scale
1.0
1.0
aligned weights
0.78
0.78
0.67
0.67
0.45
0.45
0.8
0.34
0.34
0.12
-0.6
kernel distance
0.01
0.01
0.01
0.12
0.23
0.34
45
0.56
0.67
0.78
0.89
1.0
0.01
0.12
0.23
0.34
0.45
0.56
0.67
0.78
0.89
1.0
input-output initialization scale
Input-output initialization scale
0.4
1.0
1.0
unaligned weights
0.78
0.78
0.67
0.67
-0.2
0.56
0.45
0.45
0.34
0.34
0.0
0.01
0.01
0.01
0.12
0.34
0.67
0.78
0.01
0.34
recurrent initialization scale
recurrent initialization scale
Figure A7. Initialization of recurrent modes more strongly impacts NTK movement. Phase plots illustrating the kernel distance the NTK from initialization for different trajectory lengths ((left) = 2, (right) T = as function of initialization strength of input-output and recurrent connectivity modes, which are independently varied. Visualized for RNNs with (top) aligned and (bottom) unaligned weights.
encourage or force feature learning in recurrent modes, but more work is needed to understand this. In Appendix K, we show that the energy function is impacted differently by errors in recurrent connectivity modes versus input-output modes, which might provide some intuition for the differences we discuss here.
A note of caution that the continuous increase in kernel distance with trajectory length in these experiments does necessarily indicate that the network continues becoming 'richer' learner as T - co, but is likely, at least in part, due to the fact that changes in the weights are amplified more by the NTK for longer trajectory lengths. This makes it more challenging to compare NTK movement across different trajectory lengths. Nevertheless, the NTK is still non-constant in the cases with non-zero kernel distance, indicating some feature learning is occurring.
Returning back to Schuessler et al. (2024), we see that in our setting the initialization the input-output mode does not appear substantially affect the NTK behavior. One possible reason may simply be that this effect arises specifically in nonlinear networks, in networks trained particular tasks (not constant integration the case that the distinction between aligned and oblique dynamics are not adequately captured by the NTK distance, and that separate analyses are required to evaluate these differences. We leave this to future work.
consider here). It might also be
K. Impact of connectivity modes on the energy function
Here, we analyze how the energy function is impacted differently by errors in the recurrent connectivity mode versus the input-output connectivity modes.
Recall the energy function is given
1
E
(s; ct 22
(136)
2T
and that we can decompose the data singular values as 8; = ff(Att). We make change of variables such that ac = 8+ Anc
and b = ((,, (/(T-)) + Ab.- When Age Ab, = 0 Vi, ac, b the global solution. Substituting into the energy
31

---

Learning dynamics in linear recurrent neural networks
function,
1 T 2T
Σ
- - (6+A) + Ao,)"=aj2
(137)
To analyze the effect of error in the input-output modes, we first consider the case where Ab; = 0 Vi.
T E
Σ (6f(A, - (6+
(138)
2T
1=1
(A,
(139)
27
We can see that the error of the input-output connectivity modes Dec enters into the energy function quadratically. Instead, for the case where Age 0 and Ab. + 0,
E - (( D 2T
(140)
t=1
(-x (1
T (T-))
Σ (8f(A.)) -
(( 1(
(141)
2T
T-i (T (T-1) (T-)) (
-(f(),))(((()))))
+ Σ
(142)
2T
k
k=
T (T- (
(143)
2T
k
Here we can see that the error of the recurrent connectivity mode As, enters into the energy function exponentially with time.
L. Early-importance task dynamics lead to unstable solutions
We extend our sensory integration task in the main text to consider the impact of increasing versus decreasing task dynamics on stability of the network. Similar to before, we remove prior assumptions about whitened data and aligned weights, and show that our predictions made in aligned linear RNNs generalize. Our setting identical that in Section 3.6 except that rather than taking the output be the mean or the sum of the input in each dimension, we instead vary the relation be scaled with increasing weight over time (late-importance: yr 0.1 Σn lax,) or decreasing weight over time (early-imper carly-importance: yr = Σ a.))) We select tasks that do not have inverse- exproneniilll dynamics to illustrate the generality our theory (and also verify that this behavior occurs with inverse-exponentall dynamics), Our theory predicts that carly-importance task dynamics (if the scaling term reflected in the ouput- modes is sufficiently large) are unstable.
To visualize stability of learning, we plot the gradient of the hidden layer (# throughout training. We see that in the late- aate-importance case, the network singular values converge the same solutions in each dimension and the hidden layer
gradient remains small (< 10-3). Conversely, for early-importance task dynamics, we observe that optimization is more challenging as only some of the network singular values converge to the solution and others fluctuate at local minima. In parallel, the hidden layer gradient is larger (~~ 0.3) and fluctuates throughout training. We are only able to visualize this effect because we study a network trained on a trajectory length of (T = 6): for large T. exploding gradients make training impossible.
32

---

Learning dynamics in linear recurrent neural networks
lale importence U/ 0.1 ((t)))
early importance (V+ ((x)0
0.8
1.0
0.6
1.2
enguervaless
0.4
0.8
recument oupuuu
0.0
0.0
2000
4000
5020
10000
2900
5000
7600
10000
12500
16300
17500 20000
hobin loyee guddee (3)
055
0.0000
-0.5
-0.0033
2000
4003
0000
1000
10000
2000 5000
7000 10000
12500 15000
17500 20000
takhing
Figure A8. Early-importance task dynamics yield unstable solutions. We train LRNNs with small, random weights on either task with (left) late-importance dynamics (yr 0.1 Σ (twx)) (right) sarly-importance dynamics (yr ( (Top) All of the LRNN singular values converge to the same solution when task dynamics are stable (late-importance), but not when they are unstable (early-importance). (Bottom) The hidden layer gradient illustrates the (in)stability of each network, smoothly converging to 0 through training with small magnitude in the late late- ieeeipottaance case and fluctuating throughout training with large magnitude in the carly-importance
M. Extending the (autoregressive) T- T-output case
Although our main analyses in this work focus on the single-output case for simplicity, here we extend our main theoretical framework to the T -outuut case for completeness. The network receives an input æ; E IR at each timestep and produces an output at every timestep ÿt E R.N, In this case, the loss is computed over the output at every timestep over trajectories ( 2.......... T,, 1p Up. }
P
T
Up.t
Wo
W
(144)
2
1=1
The derivation is the same the single-output case, except that now, there is addition summation the input-output correlation matrices for the output at different timesteps
EVXe Σ 7
T
(145)
Under the same assumptions as before of whitened input with mean and the gradient flow regime, the gradient equations
33

---

Learning dynamics in linear recurrent neural networks
become
W₂
Wh
-T W -W,
W.
(146)
dlg
T t-1t-i-1
T dtg
Wh = ΣΣ
Wh
W W. (E - w W. W. W
(147)
t=1 i=1
1=0
T ΣE
Wy
(,wW)W ((--))
(148)
dig
Again, assuming the input-output correlation matrices have constant left and right singular vectors across trajectory timesteps and that the LRNN is data-aligned initialization, get diagonal matrices W.. WA. h: W
d
W
W
(t T W -
W WoWn W.)
(149)
dtg
h
t-1t-i11
T dlg
Wb h
ΣΣ
Σ
Wht W y (St, - W " ".,, W W,),, W " ((----1-))]
(r]
(150)
T
T dig
W,, "
Σ Σ (Se,
W, W, W W w W W
(151)
t=1
where St,i is the singular value matrix of the input-output correlation matrix between Y, at trajectory time t and X, trajectory time i.
Rewriting the equations in terms of the decoupled connectivity modes along each singular value dimension
T
aa
Ca (Sa, -aa cabl-'aa aa)
(152)
dta
ba
( ca (8a,t. cabi (05-'aaaac (ig
(153)
dta
i)ca cabi aabaa'aa (t-i--)
(154)
d
Co
(8o.t.
ba b ' Yanaa
(155)
dtg
These dynamics arise from gradient descent on the energy function
(8a,t, - aaa
(156)
2T
Here discuss the extension of some of our results the autoregressive case, although note that fully characterizing the network behavior in this setting will be an important direction for future work.
M.1. Exact solution of input-output connectivity modes
We apply the same approach in Appendix C get the exact solutions of the onput- connectivity modes in the autoregressive case, which is given by:
E. Ei b- t0, (0.. (ET,,
Et (t-
u((o) (E Ei 6- (..)) EE=1 Ei= (2(1-11) + 20 دللل //.' (Ei=, Σig (2((-0) (157)
34

---

Learning dynamics in linear recurrent neural networks
M.2. Local approximation of recurrent connectivity modes
Again, we can apply the same approach as in Appendix D. performing Taylor expansion of the learning dynamics of the recurrent mode. The equations in Appendix D remain identical except for the nith partial derivative of the energy function with respect to the recurrent mode b, which is given by the following in the autoregressive case:
(2t=n)/2
E'E dbs
(t (t-i) i) II -le-it i-)) st,1cab'
(t (-i) i)
II j=1
(2t 2i -5)
8 c2222212i-1
1=1
(158)
By substituting this form of d'E dbn
the approximation of the recurrent learning dynamics can be computed for the autoregres-
sive case.
M.3. Zero-loss solutions only exist for inverse-exponennial task dynamics
The autoregressive case has an energy function that is very similar to the single-output case. In this case, following the same logic as Appendix F, zero-loss solutions only exist for inverse-exponential task dynamics given by the form St,i = 5 -vt, i. This has global solution at X, ac
M.4. Existence of effective regularization term
By decomposing the energy function in the autoregressive case here, we can see that an effective regularization term appears as before, albeit in a different form:
T
t
T(1 b²) 63 b²T) (1 - 62)2
E
Σ>
2bt + 2r
a
(159)
2T
1=1 i=1
effective regularization term
data-driven term
M.5. Neural tangent kernel
Using the same approach as in Appendix I, the NTK is given by:
Vovec(Y:)
ΣΣΣ
x x WWW
(160)
1=
t-i-1 Σ (1 W W lt WaX, @ W, [T
(161)
0
IN, & X W W ((t W,X,)
(162)
N. Generalizing gradient flow equations to the eigenspace to capture rotations
In the main body of the paper, we study learning dynamics in the parameter singular value space, assuming that the network aligns itself with the input-output correlation matrices' singular vectors. This has the advantage of allowing for different input and output sizes and is also easier to analyze because singular values are real and nonnegative. However, one notable limitation of studying the learning dynamics in the singular value space is that restricts expressivity of the network. In particular, the assumption of constant left and right singular vectors, necessary simplify the gradient forms and arrive decoupled connectivity modes, restricts the RNN to unidirectional scaling and summation (i.e., integration) of input.
In this section, we derive our main gradient equations in the eigenspace to capture the other primary computation linear RNN can perform: rotations (oscillations). Notably, this derivation allows for complex eigenvalues and still has decoupled gradient dynamics across different dimensions. We make the same assumptions on the input as before, namely that inputs are whitened with mean, and we assume the gradient flow regime. The derivation remains the same as Appendix B until
35

---

Learning dynamics in linear recurrent neural networks
Equation (23), which copy here for convenience:
T
W
=
W
(T
-TT
- W.,,
(163)
dtg
==1
T-17-t-1
d
Wh
Σ
Wh Wy (EPX: wowk - " W. W (T
(164)
dtg
r=U
w dtg
d
Wo Wo E
-WW W wh W
(165)
Whereas before in Appendix B we assumed that the input-output correlation matrices yxX, singular vectors, here we instead assume that the input-output correlation matrices are normal and have constant eigenvectors across trajectory timesteps. This is a more general assumption than that of constant left and right singular vectors, because it allows for complex eigenvalues and thus captures rotational teachers.
X, have constant left and right
Proof. Data generated by a linear RNN teacher parameterized by diagonalizable matrices Wz. Wr. W. with the same eigenvectors P have constant eigenvectors across trajectory timesteps.
X
Up.T
T p.
(166)
Σ
W,, WWWW - W W. p.ppeppp W.WWT-Wi
(167)
p=1
WWWW
(168)
PD,P
(169)
-DD,DDDDpP'
(170)
PD,PI
(171)
where pl denotes the conjugate transpose of P.
Substituting the diagonalized form of the data-correlation matrix Y,X, into the gradient flow equations yields
T
T
dta
W
OT W WT
h
- W,,
(172)
1=1
T-
1-1--1-1
Wn
W W W -,,, "W,) W
((
(173)
dtg
r=
T
d dla
Wy,
Σ
-o, W.W, W. (T-TT
(174)
For real matrix A, A At.
Using this, we can rewrite the transposed parameter matrices above as
d
I
T
W
(t -w,,p W,W.. Ww,)
(175)
dtg
d
T-IT--1
WW W WiW 1i---
(176)
dtp
1=1
T=U
d
T
W,
Σ
(PD,Pl - W, wWW
(177)
dta
(178)
36

---

Learning dynamics in linear recurrent neural networks
As before, we assume the LRNN is data-aligned at initialization, but in the eigenspace, such that the network has the same eigenvectors the data correlation eigenvectors P. More specifically, W,(0) PW PW)( Wx(O) PW PW)( W (0) = PW, (0)P1, where Wa, W W,, y are diagonal matrices of eigenvalues. We substitute the di- agonalized form, use (PDPi)* = PDD*P (where D denotes the conjugate of D), and simplify to get:
d
T
W. 2 =
WH
- - W.W W.)
(179)
dtg
T-17-i-1
d
Wh h Σ
Wh W 3
(D; Wo W
W.) W. W (W
(180)
di
T dig
Wy, y - (D:-W
W.W
(181)
Let Gas Co:
Co: d be the ah diagonal entry of W., W W y, D; respectively. We can rewrite the above equations in terms of these connectivity modes that decouple along eigenvalue dimensions a.
T
d
aa
(T c2 (da, - (raaa
(182)
dtp
==1
T-1
T
dig
ba
(T - () - -caba 10- aaaa-aaaaaa (o((--i-))
(183)
i=1
d
Σ
(da.s caba aaaaaaaa
(184)
dig
i=1
Finally, we can formulate an energy function by writing the gradient flow equations of the connectivity modes as the Wirtinger partial derivatives the complex conjugates:
E
(d. cb" (a) cb)
(185)
2T
[d; cbT T aaa2
(186)
2T
i=1
d
OE
a
(187)
dlg
0a
d
h
(188)
dta
0b
d
0E
(189)
dlg
det
(190)
In the case where all terms are real, the energy function has an identical form to the SVD case considered in the main text. We make some initial investigations into the behavior of the connectivity modes and task dynamics in the complex plane in Appendix O, but note that this will be an important direction for future work.
O. Learning dynamics of rotations in the complex plane
In this section, study the learning dynamics of networks trained on rotational task dynamics (i.e., with complex eigenvalues) and show that our general framework extends naturally to this case. In particular, task dynamics are now represented by the data-correlation cigenvalues, which are potentially complex-valued but can still be decomposed into constant and dynamic components, such that d, = ff t). The task dynamics with 0-loss solutions (constant, inverse- exponential, and exponential) that we consider in the main text extend to this case and yield the same solutions as before, as shown in Figure A9.
37

---

Learning dynamics in linear recurrent neural networks
constant RA, t) 1
inverse-exponential flA t) = A
exponential fA, t) = =
trajectory timesteps (t)
Imaginary
Imaginary
i imaginary
0
task dynamics
-1
-1
-1
-1
i
real
real
real
recurrent mode (b)
1.0
1.0
recurrent mode (b)
1.0
A
recurrent mode
0.5
1/A
0.5
0.5
radius (R)
0.0
angle (Ø. radians)
0.0
0.0
-0.5
0
2000
4000
0
2000
4000
0
2000
4000
training steps (tg)
training steps (tg)
training steps (tg)
input-output mode (ac)
input-output mode (ac)
input-output mode (ac)
1.0
0.6
1
5M
0.4
0.5
0.2
-1
0.0
0.0
-2
2000
4000
4000 training steps (ta)
2000
0
2000 4000
training steps (ta)
training steps (tg)
Figure A9. Learning of different rotational task dynamics in the complex plane. The task dynamics with zero-loss solutions that we consider in the main text extend naturally to rotational dynamics in the complex plane- (left) constant f(A, t) = 1, (middle) inverse- exponential f(A,t) t) -²-,, and (right) exponential f(),t) X². (Top row) We plot examples of the task dynamics for the three cases the complex plane. We keep the rotational angle the same across the different dynamics (constant Φs x/6; anverse-xpoonentil, exponential
7/6) and vary the radius yield stable dynamics (constant RA 0.7, inverss--xponential R,, 0.9, exponential RA 1.05). We plot the learning dynamics of (middle row) the recurrent mode and (bottom row) the input-output mode ac in terms of their polar coordinates to show how the network learns (orange) the radius and (blue) angle components of the complex-valued data-correlation eigenvalues. In all cases, the different components of the connectivity modes converge to the predicted global optimums for the different task dynamics.
Eigenvalues in polar coordinates. More specifically, we rewrite the complex eigenvalues in terms of polar coordinates in the complex plane such that = Ra A Rx exi This means that inverse-exponentee task dynamics correspond p- (TT-) with the same global solutions as before at b = A Ry = 8 = Rae Similarly,
to f(),t) t) = R exponential task dynamics are given by () = R2 e with global solutions at b Rg (RresssTT Finally, constant task dynamics are given by ʃ(A 1, I) 1, with trivial solutions given by X ac Rge
RAeEAT ac = 5XT
Learning dynamics in polar coordinates. One interesting observation is that the learning dynamics of different modes differ significantly when studied in the complex plane. While the learning of the radius is consistent with the dynamics in non-rotational LRNNs, the dynamics of the rotational angle differ substantially (Figure A9), including across different connectivity modes and task dynamics. In particular, the angle component begins adapting immediately at the onset of training, especially compared to the radial component. We suspect this behavior may be similar to the phenomena of eigenvector alignment which happens early in training (Atanasov et al., 2022), although in this case the eigenvalue angle is aligning rather than the eigenvectors themselves.
38

---

Learning dynamics in linear recurrent neural networks
=
r//8
r//4
5m/4 (or -3m/4)
trajectory timestops (t)
1.0
1.0
1.0
1.0
imaginary
0.5
imaginary
imoginary
0.5
imaginary
0.0
0.0
0.0
0.0
-0.5
task dynamics
-0.5
-0.5-
-0.5
1,0
1.0
1.0
1
1
i
i
real
real
recurent mode (0)
1.00
recurent mode (b)
1.00
0.75
recurrent mode
0.75
recument mode
global salution
0.50
0.50
radius (R)
0.0
angle (p. radians)
0.00
0.00
0
2000
4000
2000
4000
0
2000
4000
2000
4000
training staps (ty)
training steps (fg)
training steps (fp)
training steps (fg)
1.00
inputoutguut mede (ac)
1.00
1.00
input autput mode (ac)
1.00
Input-output mode
0.75
mode
0.75
outp- mode
0.75
0.75
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
2000
4000
2000
4000
2000
4000
2000
4000
training steps (G))
training steps (ta)
training steps (fp)
training steps (ta)
Figure A10. Learning of inverse-exponential task dynamics with different rotational angles. We study how different rotational angles & impact task dynamics (top row) and learning dynamics ((middle row) recurrent mode, (bottom row) input-output mode). keeping the radius constant (R 1) for 6 = 7/8, /8, x 7/4,- -3T/4. In the two leftmost columns, by studying complex conjugates, we can see that positive imaginary parts (R sin) and that these also correspond to the direction of perturbation in the input- modes. In the two rightmost columns, we can see how increasing the rotational angle impacts the geometry of the task dynamics. We can also see that the mode trained on & 5 / converges the slowest because its optimum lies the furthest from the initialization at b(0) = 0.1.
0) produce clockwise rotations, and negative imaginary parts produce counterclockwise rotations,
Solution stability, extrapolation ability, and the ordering of learning. One consequence of the direct extension of the task dynamics we consider and their solutions to the complex plane is that we can show that our results of task dynamics determining solution stability and extrapolation ability still hold. Networks trained on exponential task dynamics still fail extrapolate because the global solution is dependent on trajectory length (a c* 5x7). Furthermore, -trlyyimportancee task dynamics lead to unstable solutions, yielding cigenvalues greater than 1. In the case of complex eigenvalues, early- importance task dynamics correspond to dynamics where the radius decreases over the trajectory, also known as spiral sink. Finally, the same principle of 'larger and later' eigenvalues being learned faster still applies in this setting. This is especially evident by comparing convergence speeds for different radii Rx, Rs (not shown). same as we do in Appendix E.
For simplicity, in the remainder of this section, we analyze the consequences of different À (setting 5 = 1) for inverse- exponential task dynamics (d,- = (A, I); (), -"= i.e., rotational LRNN teacher). Because only À is complex- valued in this case, we drop the subscripts on the radius and angle task dynamic parameters from hereon and simply refer to A as x= Re Note however that different values of d (= R3e still impact task dynamics and learning dynamics, although we do not study this here.
Impact of rotational angle. We illustrate some examples of how changing different components of À the angle (Figure A10) and the radius R (Figure A11) manifest in the task dynamics, the learning dynamics, and the network solutions. Recall that inverse-exponentie task dynamics are given by d, = 6XT-1 = (T The rotational angle is given by é. When the imaginary part of the solution (A) is positive such that s > O, the task dynamics rotate clockwise in the complex plane (Ø- = x /8 in Figure A10). Conversely, when the imaginary part is negative (n)))) < O), the task dynamics rotate counterelockwise in the complex plane (Ø = -n/ in Figure A10). In the case we study here, although the scaling component of the task dynamics 8 is real-valued, the input-output modes still adapt along a (small)
39

---

Learning dynamics in linear recurrent neural networks
R 0.4
R 0.9
R ==1
R= 1.05
trajectory (t)
Imaginary
Imaginay
Imaginary
task dynamics
i
1
D
1
1
real
real
recurrent mode (b)
1.00
recurrent mode (h)
1.00
recurent mode (b)
1.00
recurrent mode (b)
1.00
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
2000
4000
6000
2000
4000
6000
0
2000
4000
6000
2000
4000
6000
training steps (ty)
training stops (tp)
training stops (ty)
training steps (tp)
1.00
input- autp--utuuuuttt mode (ac)
1.00
out-outpuuttnno mode (ac)
1.00
input-ou output mode (ac)
1:00
0.75
0.75
0.75
0.75
ingut-output
0.50
0.50
0.50
0.50
radius
global solution
0.25
0.25
angle (g. radians)
0.00
0.00
0.00
0.00
2000
4000
6000
2000
4000
6000
0
2000
4000
6000
2000
4000
6000
training steps (tg)
training steps (te)
training steps (te)
training steps (to)
Figure All. Learning of inverss-exponential task dynamics with different radii. We study how different radii H impact task dynamics (top row) and learning dynamies (middle row) recurrent mode, (bottom row) input- mode), keeping the rotational angle constant (ő = x/6) for R- = 0.4, 0.9, 1, 1.05. First we can see that radii less than 1 (R = 0.4, 0.9) produce spiral source dynamics (or late- importance dynamics) where the radius increases through the trajectory of the task dynamics. Conversely, radii greater than 1 (R 1.05) produce the opposite- spiral sinks or -alyyimportaacee dynamics, which also lead to (middle and bottom) unstable solutions. Finally, radii equal to (R- = 1) produce a circular rotation/limit cycle the task dynamics. Finally, by comparing small radii (R = 0.4) and large radii (R= 1.05), we see that larger radii correspond to faster learning.
imaginary part before converging to final real-valued solution. The sign this imaginary part is determined by the sign of the imaginary part of the recurrent mode (i.e., both recurrent and input-output modes learn along either negative imaginary parts or both along positive imaginary parts). Furthermore, the magnitude of the imaginary deviation of the input-output modes is dependent on the magnitude of the angle (Ø = x//4 / in Figure A10).
Rotational angle affects learning speed. By studying the complex plane, we can also see that learning speed is impacted by the distance from initialization the global optimum and that the rotational angle influences this in way that differs from only considering the magnitude of the eigenvalue/radius being learned. In particular, we know that trajectories with 'larger and later' eigenvalues/radii are learned faster (Appendix E). However, learning speed is also impacted by the distance from initialization to the solution, and in the complex plane this can be independent from the magnitude of the radius, which especially clear when studying the effect of the rotational angle. For example, a set of modes all trained on inverse-exponential task dynamics with a radius of R = 1 will learn at different speeds depending on their rotational angle (0) relative to some fixed initialization. the recurrent modes are all initialized at. say, b(0) = 0.2, modes with their solution further from initialization (as given by their rotational angle; for instance at Φ 13T 4) will learn slower, and modes with solutions closer to initialization (for instance =o- = 4/4) will learn faster. On the other hand, if the network is initialized at b(0)= = -022, this effect is reversed and the modes trained on task dynamics with rotational angle & + /4 will learn faster and -Ó- x/4 will learn slower. This effect is subtly noticeable in Figure A10 comparing between & = T/8 versus 6 = - -xx /4. However, we note that this effect mostly emerges if weights have large initializations. When initializing the network with small random weights, this effect is quite small and learning speed is mostly determined by the magnitude of the eigenvalues.
40

---

Learning dynamics in linear recurrent neural networks
Impact of radius. The task dynamics we study in the main text correspond exactly the case when the rotational angle In fact, some of the same phenomena apply here and determined by the radius R. In the case of inverse-exponentin dynamics, radius less than I corresponds to what we call fate-importance" task dynamics (where the data-correlation eigenvalues increase across the trajectory [de < dd+11). which manifests as task dynamics with increasing radius across the trajectory in the complex plane such that R² < R2"'((+1), producing a spiral source (R = 0.4, 0.9 in Figure A 11). When the radius is I, we have a perfect rotation, or limit cycle (R = in Figure AlI). Finally, the radius is greater than I and we have "early-importance" task dynamics (data-correlation eigenvalues decrease across the trajectory |de] > |de+ D. the radius of the rotation RT-(> >> RT*-(t+)) (or the data-correlation eigenvalues) decreases across trajectory, producing spiral sink (R= 1.05 in Figure A11).
As we noted earlier, we can see that early-importance task dynamics (or task dynamics that produce spiral sinks), have global solutions at > 1, yielding unstable solutions (R = 1.05 in Figure AI1). Similarly to before, we also see that learning speed is dependent on the radius size such that larger radii are learned faster than smaller ones (R = 0.4 versus R 1.05 in Figure A All), 1), again recapitulating a similar result to that of the largest singular values being learned first in Saxe al. (2014).
P. Simulations
Code for all simulations can be found at http::/ggihhbbbcomm/ppccc.L..LLL ddynra dynamics
P.1. LRNN initialization
P.I.1. ALIGNED LRNN
To initialize aligned LRNNS, we reverse-engineer the weight matrices starting from the connectivity modes described in Appendix B and Appendix N. We specify the initialization of the connectivity modes (input, recurrent, output) in each dimension as hyperparameters, which are the diagonal matrices W.. a. Wh, Wy-
SVD
For the form based on SVD, we create orthogonal matrices R,,, R.y (R,)) and use the left and right singular matrices Uy: Va of the data correlation matrices (see below in Appendix P.4.1) form the weight matrices according
W- = R.W.VV
(191)
Wh. R.W,
(192)
W,, = U,W,
(193)
Eigendecomposition For the form based on eigendecompositionn the eigenvectors of the data correlation matrices to form the real-valued weight matrices according to
W₂ PW Pt
(194)
W₁ PW,Pi
(195)
W, PW pW
(196)
P.1.2. UNALIGNED LRNN
We create unaligned LRNNS by initializing the weights with Gaussian distribution of mean 0 and standard deviation a/ //V/i where a is a specified hyperparameter and N is the row-size of the corresponding weight matrix.
P.2. Training
Networks are trained using gradient descent on the mean squared error and automatic differentiation in order to validate our theoretical results. We modify our learning timescale to account for the additional scalars introduced by taking the mean over samples the and output dimension Ny (r PNy/n, where y is the learning rate) when comparing to simulation.

---

Learning dynamics in linear recurrent neural networks
P.3. Recovering connectivity modes
Recall that the learning dynamics are decoupled along singular value (or eigenvalue) dimensions (a) for networks with aligned weights at initialization.
SVD The matrices Ry. Rgs Uy. Va used for initializing LRNN weights will stay constant throughout training. To recover the connectivity modes, which drive learning in the network, at particular time in training, we simply perform the inverse of the operation done at initialization,
Wa (1o()) - R2 W, (l
(197)
W = Ry W (to) (tt) R
(198)
W (t(l)) U, (Lo()) R.y
(199)
When using networks that are not initialized with aligned weights, we just compute the singular values of each weight matrix.
Similarly, the cigenvectors P used for initializing LRNN weights will also stay constant throughout training. To recover the connectivity modes, we again perform the inverse of the operation done at initialization,
Eigendecomposition
Wa (Co))) = (to))
(200)
(n(te) = -P)
(201)
W (1o()) = -PO)
(202)
P.4. Tasks
P.4.1. STRUCTURED TASK DYNAMICS
SVD To create data with onput correlation matrices that have constant left and right singular vectors and temporally- structured singular value dynamics, we similarly reverse-engineer the equations Appendix B. We first generate random Gaussian input (X1.)) centered at O, which is then whitened.
The data singular values S1.7 are created by setting the singular values in each dimension and at each trajectory timestep I according to the specified task dynamics f (constant: f(>a, t) 1, exponential f()., t) X inverse-exponentiall f().,)) f) -t) and hyperparameters 8., Å. such that S d (fa t).
We generate the left and right singular vectors Un, Vz by taking the SVD of a random matrix. Finally, we create the output according to
T Yr Σ U,S.VV X.
(203)
Eigendecomposstiion We take similar approach, except with eigendecomposition. The eigenvalues D1.7 created by setting the (potentially complex-valued) eigenvalues in each dimension and at each trajectory timestep / according to the specified task dynamics f(A.,
To construct the data correlation matrices (EXX))) with constant eigenvectors (P) across time, we use a block- diagonal aproach (i.e., a real Schur form X, QHQ-1 where is upper or lower quasi-triangular and Q orthogonal). The eigenvalues are selected such that each complex eigenvalue has a complex conjugate in another dimension. For the eigenvalues at each trajectory timestep D,, construct block diagonal matrix H,, such that for each real eigenvalue there is X block and for each pair of complex conjugate eigenvalues p iv, there is a 2 x 2 block
p
42

---

Learning dynamics in linear recurrent neural networks
Next, construct the orthogonal matrix Q. For each real eigenvalue, add column one-hot vector with the value
in the dimension of the eigenvalue. For each complex eigenvalue block, create complex column vectors: [1. _ for + iv and [1, -i] for iv in their respective block dimensions and O elsewhere. Similar to the SVD case, we use the same constructed Q across time, only changing the eigenvalues according to the task dynamics, such that ΣYX; = QH,Q We create the output according to
T Yr
Σ QH,Q 1=1
(204)
P.4.2. MODIFIED TASK DYNAMICS
In Section 3.4, we study how changing the relationship between the correlations of input from the past (1 : T - 1) and the imput at the last timestep (I") influences the solutions Iearned by connectivity modes. To do this, we simply create the singular values from specified value.
as Appendix P.4.1, but change the the singular values at the last timestep T some
For the Dirac delta task, we simply set the singular values at the first (S1) and last (Sr) timesteps to some specified values, and set the rest of the singular values to 0.
P.4.3. SENSORY INTEGRATION TASKS
To create the censory-integralie tasks, we use the multisensory integration task from Neurogym (Molano-Maiee Mazón et al., 2022) to generate stimuli in four input dimensions (removing the fixation input). We do not modify the data further (i.e., whitened).
For the experiments extrapolation ability, to create the target, simply take the mean the sum of the inputs along the trajectory in each dimension.
For the experiments on stability, we create the target by either computing yr = 0.1 Ei tx, for late importance or yr Σt= 1 az for early importance.
43


Reviews:
[{"review_id": "UOYp9ezf4q", "reviewer": "Reviewer_CWVm", "summary": "This paper analyzes the learning dynamics encountered when training a linear recurrent neural network (i.e., a linear time invariant system)  using gradient descent. The paper derives a reduced form of the learning dynamics, and connects the stability of the model to the task being trained on.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback on the relevance and potential of the work for the neuroscience and machine learning community. These comments will enable substantial improvement to the manuscript.\n\nIn addition to addressing the limitations below, we would like to highlight additional results and developments we have made to the paper since the time of initial submission, which we hope will substantially improve our paper by making our theory more rigorous and general. In particular, our new results cover:\n* **Learnability**: We prove that task dynamics that are exponential-in-time (i.e., constant, exponential, and inverse-exponential) are the only task dynamics with 0-loss solutions in our model.\n* **Phase transition**: Using Landau theory, we show analytically that for Dirac delta task dynamics, there is a (first-order for $T>3$, second-order for $T=3$) phase transition, solely dependent on the ratio of the first and last singular value. We verify these results in simulation.\n* **Experimental results**: We extend our sensory-integration tasks to also illustrate how early-importance task dynamics lead to instability.\n* **Generalisation**: We perform additional experiments to show that our feature-learning result generalizes to RNNs learning tasks with rotational dynamics (non-constant singular vectors through time).\n* **Learning dynamics**: We extend our solution to the recurrent connectivity modes to an analytical approximation using the Faa di Bruno formula.\n\nWe also thank the reviewer for identifying the limitations of our paper, which we address below:\n\n> I think the paper would benefit from much clearer writing. There are too many distracting inline equations and symbols and the results are written in what is (in my opinion) an overly technical way\n\nWe thank the reviewer for the valuable feedback and apologize for the lack of clarity. This work was written as a theory paper, and as such we have tried to condense the most critical equations into the main text and left the remainder of the derivations to the appendix. However, we agree that theory is more useful when it is clear and can be widely understood. To enhance clarity, we will revise the writing to be clearer by describing concepts in simple terms and reducing the overuse of symbols in-text where possible. \n\n> The paper is of interest to neuroscience researchers, who have long been using recurrent neural networks as models of the brain. The paper is also of interest to people in deep learning working on SSMs, which are essentially (either time-invariant or time-varying) dynamical systems. This paper seems quite close in approach and spirit: https://arxiv.org/pdf/2407.07279\n> \n> Also, there does not appear to be any discussion of the recent popular work on State Space Models (SSMs), which are LTI systems: https://arxiv.org/abs/2008.07669\n\nWe thank the reviewer for pointing us to these important references, which are certainly related to our work (especially Smekal 2024). We did not discuss SSMs aside from citing a few references, but they are certainly very relevant to the study of RNNs and an interesting and important extension for future work. We will make sure to include the citations and more discussion around this topic, including the new results provided by our framework over and above those in these references (such as the results on stability and extrapolation, phase transition, rich and lazy learning, and validation in a sensory task).\n\n> What separates your approach from that of Saxe (2014, 2018)? Is it just the depth of the network you study?\n\nBecause the RNN receives sequential input that is related to the output in a time-dependent way, the energy function includes a summation over time which includes time-dependent singular values. In contrast, in Saxe et al. (2014, 2018), there is no time-dependence in the data (or sequence) and thus only a single set of singular values.\n\nMore explicitly, two aspects where the difference between our work and Saxe et al. (2014, 2018) can be seen clearly are in the data specification and in the energy function. Regarding the data, Saxe et al. (2014, 2018) considers the case where the data correlation has N singular values (one for each dimension), whereas we consider T data correlations, yielding T x N singular values in total. This helps us draw conclusions that are unique to the RNN setup (like the fact that later singular values are learned first), which would have been impossible to obtain by just extending the depth of Saxe et al.’s model. Regarding the energy function, Saxe et al. (2014, 2018) have an energy function given by $E=(s-ca)^2$, whereas our energy function is given by $E=\\sum_{i=1}^T (s_i – cb^{T-i}a)^2$. The sum and the power of $b^{T-i}$ also enable some RNN-specific conclusions (such as the phase transition governing recurrent and feedforward computations), which again would not be possible to study with deep feedforward networks.", "claims_and_evidence": "The paper is reasonably clear mathematically, although I found it rather difficult to follow what was being said much of the time in the main text.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I went through the derivation of equations 5-8 and I did not find any issues.", "experimental_designs_or_analyses": "Yes, the experiments corresponding to Figs 4 and 5. No issues encountered.", "relation_to_broader_scientific_literature": "The paper is of interest to neuroscience researchers, who have long been using recurrent neural networks as models of the brain. The paper is also of interest to people in deep learning working on SSMs, which are essentially linear (either time-invariant or time-varying) dynamical systems.", "essential_references_not_discussed": "This paper seems quite close in approach and spirit: https://arxiv.org/pdf/2407.07279\n\nAlso, there does not appear to be any discussion of the recent popular work on State Space Models (SSMs), which are LTI systems: https://arxiv.org/abs/2008.07669", "other_strengths_and_weaknesses": "I think the paper would benefit from much clearer writing. There are too many distracting inline equations and symbols and the results are written in what is (in my opinion) an overly technical way.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "What separates your approach from that of Saxe (2014, 2018)? Is it just the depth of the network you study?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tWb3VXzytN", "reviewer": "Reviewer_9zFA", "summary": "While there has been substantial progress in understanding the learning dynamics of feedforward networks, there is relatively less work on studying it in the context of recurrent networks, especially when considering the task dynamics as well. In this paper, the authors analyze the learning dynamics of a linear RNN and derive an analytical solution with certain assumptions on the data and model initialization, for which the motivate well. Using this solution, they show that task dynamics impact the stability of the solution and the network's ability to extrapolate in time. Specifically, they divide the task into time-independent and time-varying components and show how these different aspects influence the solution. They also show that there is a trade-off between computation being more recurrent or feedforward, and that there is a phase transition between these two modes in terms of the task dynamics. This specifically emerges in the case where the data is not perfectly learnable, such as when the last time step, for which the loss is computed, does not follow the task dynamics of the rest of the data trajectory. Moreover, the trade-off between these types of computations leads to low-rank solutions that are known to emerge during RNN training. This is due to an effective regularization term that pops out of the energy function. The authors also derive a neural tangent kernel (NTK) for finite-width linear RNNs. Using this NTK, they show that recurrence leads to rich learning. Finally, they use their theory to explain how linear RNNs learn to perform sensory integration tasks, which is a common paradigm in computational neuroscience.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and the clear summary of our work. In addition to addressing the limitations identified in this review, we would like to point the reviewer to the list of new developments we have made since the initial submission, available in our rebuttal to Reviewer CWVm ([direct link here](https://openreview.net/forum?id=KGOcrIWYnx&noteId=RuV35U4JcB)). These include further mathematical and simulation results, which we hope will make our theory more rigorous and general.\n\n> Overall, the paper did a good job supporting its claims with a thorough and extensive evaluation. One point of confusion for me is whether some of these results hinge on the fact that we only compute the loss on the final time step. This seems especially relevant for the phase transition between feedforward and recurrent computation. This does not make or break the study, but is an important point to address in the text when trying to understand these results.\n\nWe thank the reviewer for the positive feedback and for raising this important and nuanced issue. In the main text and results, we consider the case where only the output at the last timestep enters into the loss for simplicity and interpretability. We fully agree with the reviewer that it is important to consider more general cases and how our results generalize. In the supplementary material, we have made an effort to generalize our derivation of the energy function and gradient flow equations to the case where the loss is computed over the output at each timestep. Crucially, we were able to derive the functional form of the energy function for this (multi-output) case, and it only differs from the original (single-output) case by an additional summation over the singular values of the data correlations for different output timesteps.\n\nAn intuitive interpretation of this is that the multi-output energy function contains a summation of the single-output energy function for different outputs.  Although this can result in different solutions depending on the task dynamics as a result of optimizing for many different outputs, we would expect several of our main results to hold if task dynamics are consistent across different outputs (e.g., in the case of a teacher-student setup). We are currently working to generalize some of our other results to this setting, including our derivation of the NTK and variation of feedforward and recurrent computation (see response below). However, it is possible that there may be additional phenomena to account for when considering the relationship between output dynamics, which will be an important direction for future work.\n\n> A relevant paper that talks about rich and lazy learning in nonlinear RNNs that may be worth citing includes:\n> Payeur et al., \"Neural Manifolds and Learning Regimes in Neural-Interface Tasks,\" bioRxiv 2023.\n\nWe thank the reviewer for pointing us to this relevant reference and we will make sure to include in our discussion of related work.\n\n> I think the results in Section 3.4 could be better explained, as it took me awhile to understand what was going on. Figure 4, in particular, is hard to parse, and I need more hand-holding when going through it.\n\nWe thank the reviewer for the valuable feedback and we apologize for the lack of clarity on our part. We will revise Section 3.4 and modify Figure 4 to make the results easier to follow.\n\n> Does this trade-off between feed-forward and recurrent computation partially depend on the fact that the last time step is all that's included in your objective function? How would these results change if intermediate time steps are also penalized?\n\nWe thank the reviewer for the insightful question. Following your question, we’ve now simulated this in the Dirac delta task and observed that pruning of connectivity modes still exists based on the magnitudes of $s_1$ and $s_T$, similar to the single-output case, although we have not yet proved mathematically that it’s a proper phase transition. We will make sure to include this and more analyses for the case where the loss includes outputs over multiple timesteps in the manuscript.", "claims_and_evidence": "Overall, the paper did a good job supporting its claims with a thorough and extensive evaluation. One point of confusion for me is whether some of these results hinge on the fact that we only compute the loss on the final time step. This seems especially relevant for the phase transition between feedforward and recurrent computation. This does not make or break the study, but is an important point to address in the text when trying to understand these results.", "methods_and_evaluation_criteria": "The analyses performed on the linear RNN and assumptions on task dynamics make sense given the scope of the study. It is also interesting to see how well the theory matches simulations in the sensory integration task, which is of broader relevance to the computational neuroscience community. It was especially interesting to see how the choice of objective (mean- vs. sum-integration) impacted extrapolation performance.", "theoretical_claims": "I skimmed the proofs in Appendix A of the gradient flow equations and energy functions and Appendix J which extends their approach to multiple outputs over time. Nothing popped out to me as incorrect.", "experimental_designs_or_analyses": "I looked over the experimental design of the main results and found them to be sound.", "relation_to_broader_scientific_literature": "The authors do a great job situating their work in the broader related literature. Most relevant to their work is the body of literature on analyzing deep linear networks and neural tangent kernels, for which they cite important papers. Prior work which has looked at linear RNNs has not accounted for the impact of the task dynamics on learning dynamics and the solutions. The results in this paper also have bearing on the emergence of low-rank connectivity in RNNs, which is a common framework in computational neuroscience to account for the observation that neural activity tends to be low-dimensional in many brain regions. This work also bears on extrapolation to longer sequences, which is of primary concern in RNNs in general.", "essential_references_not_discussed": "A relevant paper that talks about rich and lazy learning in nonlinear RNNs that may be worth citing includes:\n- Payeur et al., \"Neural Manifolds and Learning Regimes in Neural-Interface Tasks,\" bioRxiv 2023.", "other_strengths_and_weaknesses": "The paper is well written and does a good job supporting  its claims. It is a novel approach to studying learning dynamics in linear RNNs, which is relevant to both deep learning theory and computational neuroscience.", "other_comments_or_suggestions": "I think the results in Section 3.4 could be better explained, as it took me awhile to understand what was going on. Figure 4, in particular, is hard to parse, and I need more hand-holding when going through it.", "questions_for_authors": "Does this trade-off between feed-forward and recurrent computation partially depend on the fact that the last time step is all that's included in your objective function? How would these results change if intermediate time steps are also penalized?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GCFINXrmmS", "reviewer": "Reviewer_zrFq", "summary": "The authors study the learning dynamics of linear recurrent neural networks (LRNNs). Using the approach of Saxe et al. (ICLR '14) to study deep linear feed-forward neural networks, the authors develop a similar theory for LRNNs. Under some assumptions on the weight matrices, the LRRN dynamics decouple into a set of uncorrelated modes that evolve independently. \n\nThe authors find that LRNNs learn with larger and later singular values first, akin to the faster learning of larger singular values by linear feed-forward networks (sec. 3.2). They consider a task generated by a matching linear RNN and are able to analyse the stability of training (i.e. the problem of vanishing / exploding gradients) and of extrapolation in this setting. Since their dynamical equations decompose into an equation for the recurrent mode, and for the input-output mode, they are also able to identify the emergence of low-rank structures in the weights (sec 3.4) They also find that the ''richness'' of the learning dynamics, as measured by the distance of the learnt network to its NTK at initialisation, increases as recurrent computations become more important. An numerical experiment with a sensory-integration experiment  where the assumptions are not exactly fulfilled shows good agreement with the theory regardless.\n\n\n## After the discussion\n\nI maintain my score.\nI want to note that while I am aware with some of the recent literature on the theory of RNNs, I am much more familiar with the literature on the learning dynamics in feed-forward neural networks, so there might be some related literature I am missing.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and the encouraging words. In addition to one clarification about this review below, we would like to point the reviewer to the list of new developments we have made since the initial submission, available in our rebuttal to Reviewer CWVm ([direct link here](https://openreview.net/forum?id=KGOcrIWYnx&noteId=RuV35U4JcB)). These include further mathematical and simulation results, which we hope will make our theory more rigorous and general.\n\n> The authors claim to introduce a new mathematical description of the learning dynamics of RNNs. While the assumption are somewhat restrictive (joint diagonalisability), they are stated clearly by the authors (kudos for that!) and they mirror the assumptions made by Saxe et al. and are therefore likely reasonable to make progress on this problem. In the feedforward case, the theory based on these assumptions has had quite an impact on subsequent research, so I think this is a good contribution (and long overdue!) to be done for linear RNNs, too. The wealth of phenomena that they analyse further suggests that their solution of linear RNNs is a useful toy model to understanding recurrent neural networks.\n\nWe complete agree with the reviewer that this is the beginning of a longer research direction. While the assumption of joint diagonalizability has been commonly used in the learning dynamics literature (Saxe et al., 2014; 2018) and there is evidence that the parameters of the network align in this way at the beginning of training (Atanasov et al., 2021) (see also our response to Reviewer q5bu), we agree that this assumption may be restrictive in certain cases. Luckily, even if our assumptions are somewhat restrictive, our empirical results seem to be a bit more general, such as in the case of our sensory integration tasks.", "claims_and_evidence": "The authors claim to introduce a new mathematical description of the learning dynamics of RNNs. While the assumption are somewhat restrictive (joint diagonalisability), they are stated clearly by the authors (kudos for that!) and they mirror the assumptions made by Saxe et al. and are therefore likely reasonable to make progress on this problem. In the feedforward case, the theory based on these assumptions has had quite an impact on subsequent research, so I think this is a good contribution (and long overdue!) to be done for linear RNNs, too. The wealth of phenomena that they analyse further suggests that their solution of linear RNNs is a useful toy model to understanding recurrent neural networks.", "methods_and_evaluation_criteria": "This is a theory paper, and its methods are appropriate.", "theoretical_claims": "I looked at the derivation of the dynamical equations, and it seemed all good to me. Assumptions for this analysis are clearly spelled out at the top of page 3.", "experimental_designs_or_analyses": "Not applicable.", "relation_to_broader_scientific_literature": "The authors do seem to acknowledge related literature appropriately, both in the introduction and the discussion at the end. however, while I am aware with some of the recent literature on the theory of RNNs, I am much more familiar with the literature on the learning dynamics in feed-forward neural networks, so there might be some related literature I am missing.", "essential_references_not_discussed": "None that I know of", "other_strengths_and_weaknesses": "This is a nice paper that provides a clean theoretical analysis of linear recurrent neural networks, which remain an important tool to study sequential data, and remain the most important neural network model for theoretical neuroscience. Given the huge impact of similar work on linear feed-forward neural networks, I think this framework will prove a valuable tool for theory, too, and I think it should be accepted at ICML.", "other_comments_or_suggestions": "See above", "questions_for_authors": "No further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KC0tckbMYX", "reviewer": "Reviewer_q5bu", "summary": "The authors study the learning dynamics of linear RNNs. This is similar to the  method in Saxe et al 2014, but extended to the recurrent setting. The extension leads to an energy function that has a sum of recurrent terms, and thus there is an interplay between recurrent and feedforward modes. The authors show the implication of the learning dynamics on what can and can’t be learned, and on the relative importance of late and early components of the task.\n\n## After rebuttal\n\nAfter reading all the rebuttals and discussion with all reviewers, I am keeping my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback. In addition to answering the question about our main derivation and related work, we would like to point the reviewer to the list of new developments we have made since the initial submission, available in our rebuttal to Reviewer CWVm ([direct link here](https://openreview.net/forum?id=KGOcrIWYnx&noteId=RuV35U4JcB)). These include further mathematical and simulation results, which we hope will make our theory more rigorous and general.\n\n> The main derivation is similar to that of Saxe 2014. It would be useful to stress why the cross-term does not appear in this case. Is this because of the aligned assumption?\n\nYes, the cross-term does not appear both because of the aligned assumption and because of the assumption that left and right singular vectors are constant across time, allowing us to diagonalize the weights such that each singular value dimension becomes decoupled. This can be thought of as rotating the weights so that they are aligned with the principal directions of the data at initialization. We note that other work (Braun et al., 2022) has alleviated the aligned assumption in feedforward networks but the derivation is quite involved and challenging to apply to recurrent networks as it is restricted to two-layer networks. In general, the aligned assumption appears to be reasonable when initializing a network with small random weights and has been shown to occur early in training in such cases (Atanasov et al., 2021). We will make sure to clarify this in-text.\n\n> The topic of lazy vs rich in recurrent networks is discussed in this paper: Schuessler, Friedrich, Francesca Mastrogiuseppe, Srdjan Ostojic, and Omri Barak. “Aligned and Oblique Dynamics in Recurrent Neural Networks.” Edited by Tatyana O Sharpee and Michael J Frank. eLife 13 (November 27, 2024): RP93060. https://doi.org/10.7554/eLife.93060.\n\nWe thank the reviewer for pointing us to this interesting and important reference. We are now running additional rich and lazy learning experiments varying the scale of input-output connectivity modes in LRNNs with rotational dynamics to see whether similar phenomena might occur in the linear case. \n\n> Line 298 “see” twice Line 815 “more” appears twice.\n\nWe thank the reviewer for pointing out the typo. We have fixed all typos in the new version of the manuscript.", "claims_and_evidence": "The claims are supported by both analytic derivations and numerical experiments.", "methods_and_evaluation_criteria": "There are no benchmarks, but none are needed. The simulations performed are adequate for the problem at hand.", "theoretical_claims": "I read all proofs, but did not verify every step of the math. The main derivation is similar to that of Saxe 2014. It would be useful to stress why the cross-term does not appear in this case. Is this because of the aligned assumption?", "experimental_designs_or_analyses": "No issues", "relation_to_broader_scientific_literature": "As explained in the paper, this work is related to learning dynamics in feedforward networks and to the evolution of kernels with learning.\nThe analysis of linear RNN with relation to data dynamics is novel.", "essential_references_not_discussed": "The topic of lazy vs rich in recurrent networks is discussed in this paper:\nSchuessler, Friedrich, Francesca Mastrogiuseppe, Srdjan Ostojic, and Omri Barak. “Aligned and Oblique Dynamics in Recurrent Neural Networks.” Edited by Tatyana O Sharpee and Michael J Frank. eLife 13 (November 27, 2024): RP93060. https://doi.org/10.7554/eLife.93060.", "other_strengths_and_weaknesses": "The extension of Saxe 2014 to RNNs provides new insights and explains phenomena that are unique to this setting.", "other_comments_or_suggestions": "Line 298 “see” twice\nLine 815 “more” appears twice.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper investigates the learning dynamics of linear recurrent neural network (LRNN) and present clear math results. All reviewers expressed excitement of the theoretical results and appreciate the identified learning dynamics of LRNNs. So I recommend acceptance of this paper.
Decision: Accept (oral)