From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks

## Abstract

Abstract Biological and artificial neural networks develop internal representations that enable them to perform complex tasks. In artificial networks, the effectiveness of these models relies on their ability to build task specific representation, a process influenced by interactions among datasets, architectures, initialization strategies, and optimization algorithms. Prior studies highlight that different initializations can place networks in either a lazy regime, where representations remain static, or a rich/feature learning regime, where representations evolve dynamically. Here, we examine how initialization influences learning dynamics in deep linear neural networks, deriving exact solutions for ‘ λ 𝜆 \lambda italic_λ -balanced’ initializations—defined by the relative scale of weights across layers. These solutions capture the evolution of representations and the Neural Tangent Kernel across the spectrum from the rich to the lazy regimes. Our findings deepen the theoretical understanding of the impact of weight initialization on learning regimes, with implications for continual learning, reversal learning, and transfer learning, relevant to both neuroscience and practical applications.

## 1 Introduction

Biological and artificial neural networks learn internal representations that enable complex tasks such as categorization, reasoning, and decision-making. Both systems often develop similar representations from comparable stimuli, suggesting shared information processing mechanisms Yamins et al. (). This similarity, though not fully understood, has drawn interest from neuroscience, AI, and cognitive science Haxby et al. (); Laakso & Cottrell (); Morcos et al. (); Kornblith et al. (); Moschella et al. (). The success of neural models relies on their ability to form these representations and extract relevant features from data to build internal representations, a complex process that in machine learning is defined by two regimes: *lazy* and *rich* Saxe et al. (); Pennington et al. (); Chizat et al. (); Bahri et al. ().

Lazy regime.
Extensive research has identified a fundamental phenomenon in overparameterized neural networks: during training, these networks frequently remain near their linearized form, undergoing minimal changes in the parameter space Chizat et al. (). Consequently, they adopt learning dynamics akin to kernel regression, characterized by the Neural Tangent Kernel (NTK) matrix and exhibiting exponential learning behavior Du et al. (); Jacot et al. (); Du et al. (); Allen-Zhu et al. (, ); Zou et al. ().
This behavior, known as the *lazy* or kernel regime, typically occurs in infinitely wide architectures and can be triggered by large variance initialization at the start of training Jacot et al. (); Chizat et al. ().
While the *lazy* regime offers valuable insights into how networks converge to a global minimum, it does not fully account for the generalization capabilities of neural networks trained with standard initializations. It is, therefore, widely believed that another regime, driven by small or vanishing initializations, underpins some of the successes of neural networks.

Rich regime. In contrast, the *rich* feature-learning regime is characterized by a NTK that evolves throughout training, accompanied by non-convex dynamics that navigate saddle points Baldi & Hornik (); Saxe et al. (, ); Jacot et al. (). This regime features sigmoidal learning curves and simplicity biases, such as low-rankness Li et al. () or sparsity Woodworth et al. ().
Numerous studies have shown that the *absolute scale* of initialization drives the *rich* regime, which typically emerges at small initialization scales Chizat et al. (); Geiger et al. ().
However, it’s also been shown that even at small initialization scales, differences in weight magnitudes between layers can induce the *lazy* learning regime Azulay et al. (); Kunin et al. (). This highlights the significance of both *absolute scale* (initialization variance) and *relative scale* (difference in weight magnitude between layers) in generating diverse learning dynamics.
Beyond *absolute scale* and *relative scale*, additional aspects of initialization can profoundly affect feature learning, including the effective rank of the weight matrices Liu et al. (), layer-specific initialization variances Yang & Hu (); Luo et al. (); Yang et al. (), and the use of large learning rates Lewkowycz et al. (); Ba et al. (); Zhu et al. (); Cui et al. ().
These findings illustrate the effect of initialization on inducing complex learning behavior through the resulting dynamics. Here we develop a solvable model which captures these diverse phenomena.

Rich and lazy regimes in the brain. The distinction between *rich* and *lazy* learning may also hold implications for neuroscience, where neural representations have been argued to have task-specific or task-agnostic characteristics in different settings Farrell et al. (); Ostojic & Fusi (); Tye et al. (). The *lazy* regime can be linked to the non-linear mixed selectivity of neurons, where task variables are represented in a high-dimensional space which mixes various potentially relevant variables Raposo et al. (); Tang et al. (); Rigotti et al. (); Bernardi et al. (). Conversely, the *rich* regime aligns with linear mixed selectivity Tye et al. () and the manifold learning regime, where the brain encodes tasks on a structured, low-dimensional, task-specific manifold, as observed in grid cells within the entorhinal cortex Chaudhuri et al. (); Bernardi et al. (); Flesch et al. ().

Figure: Figure 1: A minimal model of the *rich* and *lazy* regimes. A We examine a deep and wide linear network trained using gradient descent starting from an initialization characterized by a relative scale parameter $\lambda$— which characterizes the difference in the weight covariance between the first and second layers. B Network output for an example task over training time, starting from a range of relative scale values. The dynamics are influenced by the initialization. Solid lines represent simulations, while dotted lines indicate the analytical solutions derived in this work. C LeCun weight initialization LeCun et al. () in the limit matches a $\lambda$ balanced weights. Plot of ${\bf W}_{2}(0)^{T}{\bf W}_{2}(0)-{\bf W}_{1}(0){\bf W}_{1}(0)^{T}$ for a two layer network initialized with LeCun initialization
Refer to caption: extracted/5871880/figures/intro.png

Despite significant advances, these learning regimes and their characterization are not yet fully understood and would benefit from clearer theoretical predictions, particularly regarding the influence of prior knowledge (initialization) on the learning regime. In this work, we address this gap by deriving exact solutions for the learning dynamics in deep linear networks as a function of network initialization, providing one of the few analytical models of the *rich* and *lazy* regimes in wide and deep neural networks Xu & Ziyin (); Kunin et al. (); Tu et al. ().
To illustrate the dramatic effect of initialization and the kind of phenomenon we build a theory for, we consider a two-layer linear network parameterized by an encoding layer ${\bf W}_{1}$ and a decoding layer ${\bf W}_{2}$ (Fig.1A). This network can be initialized with different relative scalings, such that ${\bf W}_{1}{\bf W}_{1}^{T}\succ{\bf W}_{2}^{T}{\bf W}_{2}$, ${\bf W}_{1}{\bf W}_{1}^{T}\ \prec{\bf W}_{2}^{T}{\bf W}_{2}$, or ${\bf W}_{1}{\bf W}_{1}^{T}={\bf W}_{2}^{T}{\bf W}_{2}$, while maintaining the same absolute scale. As shown in Fig.1B, the choice of relative scaling can result in drastically different learning trajectories and representations and the theory we develop over the course of this paper describes these effects.
Through these solutions, we aim to gain insights into the *rich* and *lazy* regimes, as well as the transition between them during training, by examining the impact of relative scaling.
As shown in Fig.1C, initialization methods used in practice, such as LeCun initialization in wide networks, approximate the relative scaling initialization explored in this paper, making it relevant to machine learning community as further demonstrated in Kunin et al. ().
We consider applications relevant to machine learning and neuroscience, including continual learning Kirkpatrick et al. (); Zenke et al. (); Parisi et al. (), reversal learning Erdeniz & Atalay () and transfer learning Taylor & Stone (); Thrun & Pratt (); Lampinen & Ganguli (); Gerace et al. ().

Our contributions.

- •
We derive explicit solutions for the gradient flow, internal representational similarity, and finite-width NTK in unequal-input-output two-layer deep linear networks, under a broad range of lambda-balanced initialization conditions (Section 4).
- •
We model the full range of learning dynamics from *lazy* to *rich*, showing that this transition is influenced by a complex interaction of architecture, *relative scale*, and *absolute scale*, extending beyond just initialization *absolute scale* (Section 5).
- •
We present applications of these solutions relevant to both the neuroscience and machine learning field, providing exact solutions for continual learning dynamics, reversal learning dynamics, and transfer learning (Section 6).

## 2 Related Work

Linear networks. Our work builds upon a rich body of research on deep linear networks, which, despite their simplicity, have proven to be valuable models for understanding more complex neural networks Baldi & Hornik (); Fukumizu (); Saxe et al. (). Previous research has extensively analyzed convergence Arora et al. (); Du & Hu (), generalization properties Lampinen & Ganguli (); Poggio et al. (); Huh (), and the implicit bias of gradient descent Arora et al. (); Woodworth et al. (); Chizat & Bach (); Kunin et al. () in linear networks.
These studies have also revealed that deep linear networks have intricate fixed point structures and nonlinear learning dynamics in parameter and function space, reminiscent of phenomena observed in nonlinear networks Arora et al. (); Lampinen & Ganguli (). Seminal work by Saxe et al. () laid the groundwork by providing exact solutions to gradient flow dynamics under task-aligned initializations, demonstrating that the largest singular values are learned first during training. This analysis has been extended to deep linear networks Arora et al. (, ); Ziyin et al. () with more flexible initialization schemes
Gidel et al. (); Tarmoun et al. (); Gissin et al. (). This work directly builds on the matrix Riccati formulation proposed by Fukumizu (); Braun et al. () which extends these solutions to wide networks. We extend and refine these results to obtain the dynamics for lambda-balanced initialization dynamics of networks to more clearly demonstrate the impact of initialization on *rich* and *lazy* learning regimes also developed in Tu et al. () for a set of orthogonal initalizations. Our work extends previous analysis Xu & Ziyin (); Kunin et al. () of these regime to wide networks. Previous studies leveraged these solutions primarily to characterize convergence rates; however, our work goes beyond this by providing a comprehensive characterization of the complete dynamics of the system Tarmoun et al. ().

Infinite-width networks. Recent advances in understanding the *rich* regime have largely stemmed from examining how the initialization variance and layer-wise learning rates must scale in the infinite-width limit to maintain consistent behavior in activations, gradients, and outputs. Several studies have employed statistical mechanics tools to derive analytical solutions for the *rich* population dynamics of two-layer nonlinear neural networks initialized using the *mean field* parameterization Mei et al. (); Rotskoff & Vanden-Eijnden (); Chizat & Bach (); Sirignano & Spiliopoulos (); Rotskoff & Vanden-Eijnden (); Sirignano & Spiliopoulos ().
Other methods for analyzing deep network dynamics include the NTK limit, where the network effectively performs kernel regression without feature learning Jacot et al. (); Lee et al. (); Arora et al. ().
Our solution allows us to the study the evolution of the NTK and the influence of *absolute scale* and *relative scale* on the transition between *lazy* and *rich* learning in finite width networks Jacot et al. (); Xu & Ziyin (); Kunin et al. (); Chizat et al. (). Furthermore, these approaches typically require numerical integration or operate within a limited learning regime, and are unable to describe the learning dynamics of hidden representations. Instead, our work focuses on the impact of initialization on representation learning dynamics and derives explicit analytical solutions within tractable models.

## 3 Preliminaries

Consider a supervised learning task where input vectors $\mathbf{x}_{n}\in\mathbb{R}^{N_{i}}$, from a set of $P$ training pairs $\{(\mathbf{x}_{n},\mathbf{y}_{n})\}_{n=1}^{P}$, need to be mapped to their corresponding target output vectors $\mathbf{y}_{n}\in\mathbb{R}^{N_{o}}$.
We learn this task with a two-layer linear network model that produces the output prediction

$$ ${\bf\hat{y}}_{n}={\bf W}_{2}{\bf W}_{1}{\bf x}_{n}\text{,}$ (1) $$

with weight matrices ${\bf W}_{1}\in\mathbb{R}^{N_{h}\times N_{i}}$ and ${\bf W}_{2}\in\mathbb{R}^{N_{o}\times N_{h}}$, where $N_{h}$ is the number of hidden units. The network’s weights are optimized using full batch gradient descent with learning rate $\eta$ (or respectively time constant
$\tau=\frac{1}{\eta}$) on the mean squared error loss $\mathcal{L}({\bf\hat{y}},{\bf y})=\frac{1}{2}\left\langle||{\bf\hat{y}}-{\bf y
}||^{2}\right\rangle$, where $\langle\cdot\rangle$ denotes the average over the dataset.
The dynamics are completely determined by the input covariance and input-output correlation matrices of the dataset, defined as

$$ ${\bf\tilde{\Sigma}}^{xx}=\frac{1}{P}\sum_{n=1}^{P}{\bf x}_{n}{\bf x}_{n}^{T}\ \in\mathbb{R}^{N_{i}\times N_{i}}\quad\text{and}\quad{\bf\tilde{\Sigma}}^{yx}= \frac{1}{P}\sum_{n=1}^{P}{\bf y}_{n}{\bf x}_{n}^{T}\ \in\mathbb{R}^{N_{o} \times N_{i}}\text{,}$ (2) $$

and the initialization ${\bf W}_{2}(0)$,${\bf W}_{1}(0)$.
Our objective is to describe the entire dynamics of the network’s output and internal representations based on this initialization and the task statistics.
We consider an approach first introduced in the foundational work of Fukumizu Fukumizu () and extended in recent work by Braun et al. (), which rather than consider the dynamics of the parameters directly, we consider the dynamics of a matrix of the important statistics.
In particular, defining ${\bf Q}=\begin{bmatrix}{\bf W}_{1}&{\bf W}_{2}^{T}\end{bmatrix}^{T}\in\mathbb{
R}^{(N_{i}+N_{o})\times N_{h}}$, we consider the $(N_{i}+N_{o})\times(N_{i}+N_{o})$ matrix

$$ ${\bf Q}{\bf Q}^{T}(t)=\begin{bmatrix}{\color[rgb]{0,0.62109375,0.44921875}{\bf W }_{1}^{T}{\bf W}_{1}(t)}&{\color[rgb]{0.83984375,0.359375,0}{\bf W}_{1}^{T}{ \bf W}_{2}^{T}(t)}\\ {\color[rgb]{0.83984375,0.359375,0}{\bf W}_{2}{\bf W}_{1}(t)}&{\color[rgb]{ 0,0.44140625,0.69921875}{\bf W}_{2}{\bf W}_{2}^{T}(t)}\end{bmatrix},$ (3) $$

which is divided into four quadrants with interpretable meanings.
The approach monitors several key statistics collected in the matrix.
The off-diagonal blocks contain the network function $\hat{{\bf Y}}(t)={\color[rgb]{0.83984375,0.359375,0}{\bf W}_{2}{\bf W}_{1}(t)}
{\bf X}$, which can be used to evaluate the dynamics of the loss as shown in Fig. 2. The on-diagonal blocks capture the correlation structure of the weight matrices, allowing for the calculation of the temporal evolution of the network’s internal representations. This includes the representational similarity matrices (RSM) of the neural representations within the hidden layer, as first defined by Braun et al. (),

$$ $\operatorname{RSM}_{I}={\bf X}^{T}{\color[rgb]{0,0.62109375,0.44921875}{\bf W} _{1}^{T}{\bf W}_{1}(t)}{\bf X}\text{,}\quad\operatorname{RSM}_{O}={\bf Y}^{T}( {\color[rgb]{0,0.44140625,0.69921875}{\bf W}_{2}{\bf W}_{2}^{T}(t)})^{+}{\bf Y},$ (4) $$

where $+$ denotes the pseudoinverse; and the network’s finite-width NTK Jacot et al. (); Lee et al. (); Arora et al. ()

$$ $\operatorname{NTK}={\bf I}_{N_{o}}\otimes{\bf X}^{T}{\color[rgb]{ 0,0.62109375,0.44921875}{\bf W}_{1}^{T}{\bf W}_{1}(t)}{\bf X}+{\color[rgb]{ 0,0.44140625,0.69921875}{\bf W}_{2}{\bf W}_{2}^{T}(t)}\otimes{\bf X}^{T}{\bf X},$ (5) $$

where ${\bf I}$ is the identity matrix and $\otimes$ is the Kronecker product. Hence, the dynamics of ${\bf Q}{\bf Q}^{T}$ describes the important aspects of network behaviour.

## 4 Exact Learning Dynamics

In this section, we derive an exact solution for ${\bf Q}{\bf Q}^{T}$ providing a clean understanding of the learning dynamics, convergence behavior, and generalization properties of two-layer linear networks with prior knowledge.

Assumptions.
To derive these solutions we make the following assumptions:

- •
A1 (Whitened input).
The input data is whitened, that is ${\bf\tilde{\Sigma}}^{xx}={\bf I}$.
- •
A2 (Lambda-balanced).
The network’s weight matrices are lambda-balanced at the beginning of training, that is ${\bf W}_{2}(0)^{T}{\bf W}_{2}(0)-{\bf W}_{1}(0){\bf W}_{1}(0)^{T}=\lambda
\mathbf{I}$. If this condition holds at initialization, it will persist throughout training Saxe et al. (); Arora et al. (). For completeness, we prove this in Appendix A.
- •
A3 (Dimensions).
The hidden dimension of the network is defined as $N_{h}=\min(N_{i},N_{o})$, ensuring the network is neither bottlenecked ($N_{h}<\min(N_{i},N_{o})$) nor overparameterized ($N_{h}>\min(N_{i},N_{o})$).
- •
A4 (Full-rank).
The input-output correlation of the task and the initial state of the network function have full rank, that is $\operatorname{rank}({\bf\tilde{\Sigma}}^{xy})=\operatorname{rank}({\bf W}_{2}(
0){\bf W}_{1}(0))=\min(N_{i},N_{o})$.

These assumptions are strictly weaker than prior works by Fukumizu (); Braun et al. (); Kunin et al. (); Xu & Ziyin ().
The main distinction between our work and prior works is that both Fukumizu () and Braun et al. () assumed zero-balanced ${\bf W}_{1}(0){\bf W}_{1}(0)^{T}={\bf W}_{2}(0)^{T}{\bf W}_{2}(0)$, while we relax this assumption to $\lambda$-balanced.
The zero-balanced condition restricts the networks to a *rich* setting. We develop solutions to explore the continuum between the *rich* and the *lazy* regime.
While some works, such as Tarmoun et al. (), have considered removing this constraint, their solutions remain in an unstable and mixed form. Our work, in its form enable the understanding of different learning regimes by exploring initialization properties beyond just *absolute scale* and demonstrate that this transition can be accessed and controlled by adjusting a key parameter: the *relative scale*. Other studies, such as Xu & Ziyin () and Kunin et al. (), have similarly relaxed the balancedness assumption but were limited to single-output neuron settings.
See Appendix A.2 for a further discussion on each of these works assumptions and their relationship to ours.

Lemmas and definitions.
To derive exact solutions we start by presenting the main lemmas which we prove in the appendix.

###### Lemma 4.1 .

Under assumptions • ‣ 4 and • ‣ 4, the gradient flow dynamics of $\mathbf{Q}\mathbf{Q}^{T}(t)$, with initalization ${\bf Q}{\bf Q}^{T}(0)={\bf Q}(0){\bf Q}(0)^{T}$ can be written as a differential matrix Riccati equation

$$ $\tau\frac{d}{dt}(\mathbf{Q}\mathbf{Q}^{T})=\mathbf{F}\mathbf{Q}\mathbf{Q}^{T}+ \mathbf{Q}\mathbf{Q}^{T}\mathbf{F}-(\mathbf{Q}\mathbf{Q}^{T})^{2},\quad\text{ where }\bm{F}=\begin{pmatrix}-\frac{\lambda}{2}\mathbf{I}_{N_{i}}&(\tilde{\bm{ \Sigma}}^{yx})^{T}\\ \tilde{\bm{\Sigma}}^{yx}&\frac{\lambda}{2}\mathbf{I}_{N_{o}}\end{pmatrix}.$ (6) $$

As derived in Fukumizu () and extended in Braun et al. (), whenever $\bm{F}$ is symmetric and diagonalizable such that $\bm{F}=\bm{P}\bm{\Lambda}\bm{P}^{T}$, where $\bm{P}$ is an orthonormal matrix and $\bm{\Lambda}$ is a diagonal matrix, then the unique solution to this matrix Riccatti is given by,

$$ ${\bf Q}{\bf Q}^{T}(t)=e^{{\bf F}\frac{t}{\tau}}{\bf Q}(0)\left[{\bf I}+{\bf Q} (0)^{T}\bm{P}\left(\frac{e^{2\bm{\Lambda}\frac{t}{\tau}}-\mathbf{I}}{\bm{2 \Lambda}}\right)\bm{P}^{T}{\bf Q}(0)\right]^{-1}{\bf Q}(0)^{T}e^{{\bf F}\frac{ t}{\tau}}.$ (7) $$

In Appendix B.2
we prove that this equation is the unique solution to the initial value problem derived in Lemma 4.1 no matter the value of $\Lambda$.
However, as discussed in Braun et al. (), the solution in this form is not very useable or interpretable due to the matrix inverse mixing the blocks of ${\bf Q}{\bf Q}^{T}$.
Additionally, we need to diagonalize $\bm{F}$.
To do so we consider the compact singular value decomposition $\operatorname{SVD}({\bf\tilde{\Sigma}}^{yx})={\bf\tilde{U}}{\bf\tilde{S}}{\bf
\tilde{V}}^{T}\text{.}$
Here, $\mathbf{\tilde{U}}\in\mathbb{R}^{N_{o}\times N_{h}}$ denote the left singular vectors, $\mathbf{\tilde{S}}\in\mathbb{R}^{N_{h}\times N_{h}}$ the square matrix with ordered, non-zero eigenvalues on its diagonal, and $\mathbf{\tilde{V}}\in\mathbb{R}^{N_{i}\times N_{h}}$ the corresponding right singular vectors. For unequal input-output dimensions ($N_{i}\neq N_{o}$), the right and left singular vectors are not square. Accordingly, for the case $N_{i}>N_{h}=N_{o}$, we define $\mathbf{\tilde{U}}^{\perp}\in\mathbb{R}^{N_{o}\times|N_{o}-N_{i}|}$ as a matrix containing orthogonal column vectors that complete the basis for $\mathbf{\tilde{U}}$, i.e., make $\big{[}\mathbf{\tilde{U}}~{}\mathbf{\tilde{U}}^{\perp}\big{]}$ orthonormal, and $\mathbf{\tilde{V}}^{\perp}\in\mathbb{R}^{N_{i}\times|N_{o}-N_{i}|}$ as a matrix of zeros. Conversely, when $N_{i}=N_{h}<N_{o}$, then $\mathbf{\tilde{V}}^{\perp}$ is a matrix containing orthogonal column vectors that complete the basis for $\tilde{\bm{V}}$ and $\mathbf{\tilde{U}}^{\perp}$ is a matrix of zeros.
Using this SVD structure we can now describe the eigendecomposition of $\mathbf{F}$.

###### Lemma 4.2 .

Under assumptions • ‣ 4 and • ‣ 4, the eigendecomposition of $\mathbf{F}=\mathbf{P}\bm{\Lambda}\mathbf{P}^{T}$ is

$$ $\mathbf{P}=\frac{1}{\sqrt{2}}\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})&\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})&\sqrt{2}\tilde{\bm{V}}_{\perp}\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})&\sqrt{2}\tilde{\bm{U}}_{\perp} \end{pmatrix},\quad\bm{\Lambda}=\begin{pmatrix}\tilde{\bm{S}_{\lambda}}&0&0\\ 0&-\tilde{\bm{S}_{\lambda}}&0\\ 0&0&\bm{\lambda}_{\perp}\end{pmatrix},$ (8) $$

where the matrices $\tilde{\bm{S}_{\lambda}}$, $\bm{\lambda}_{\perp}$, $\tilde{\bm{H}}$, and $\tilde{\bm{G}}$ are diagonal matrices defined as:

$$ $\tilde{\bm{S}_{\lambda}}=\sqrt{\tilde{\bm{S}}^{2}+\frac{\lambda^{2}}{4}\mathbf {I}},\;\;\bm{\lambda}_{\perp}=\mathrm{sgn}(N_{o}-N_{i})\frac{\lambda}{2} \mathbf{I}_{|N_{o}-N_{i}|},\;\;\tilde{\bm{H}}=\mathrm{sgn}(\lambda)\sqrt{\frac {\tilde{\bm{S_{\lambda}}}-\tilde{\bm{S}}}{\tilde{\bm{S_{\lambda}}}+\tilde{\bm{ S}}}},\;\;\tilde{\bm{G}}=\frac{1}{\sqrt{\mathbf{I}+\tilde{\bm{H}}^{2}}}.$ (9) $$

Main theorem.
Thanks to the eigendecomposition of $\bm{F}$ we can separate the solution provided in equation 7 into four quadrants.
Following an approach used in Braun et al. (), we will find it useful to define the following variables of the initialization that will allow us to define the product $\bm{P}^{T}\bm{Q}(0)$ more succinctly,

$$ $\displaystyle{\bf B}$ $\displaystyle={\bf W}_{2}(0)^{T}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})+{\bf W}_{1}(0)\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})\in\mathbb{R}^{N_{h}\times N_{h}},$ (10) $\displaystyle{\bf C}$ $\displaystyle={\bf W}_{2}(0)^{T}\tilde{\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})-{\bf W}_{1}(0)\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})\in\mathbb{R}^{N_{h}\times N_{h}},$ (11) $\displaystyle\bm{D}$ $\displaystyle={\bf W}_{2}(0)^{T}{\bf\tilde{U}_{\perp}}+{\bf W}_{1}(0){\bf \tilde{V}_{\perp}}\in\mathbb{R}^{N_{h}\times|N_{o}-N_{i}|}.$ (12) $$

Using these variables of the initialization, this brings us to our main theorem:

###### Theorem 4.3 .

Under the assumptions of whitened inputs, • ‣ 4, lambda-balanced weights • ‣ 4, no bottleneck • ‣ 4, and
full rank • ‣ 4, the temporal dynamics of ${\bf Q}{\bf Q}^{T}$ are

$$ $\displaystyle{\bf Q}{\bf Q}^{T}(t)=\begin{pmatrix}{\color[rgb]{ 0,0.62109375,0.44921875}\bm{Z_{1}}(t)\bm{A}^{-1}(t)\bm{Z_{1}}^{T}(t)}&{\color[ rgb]{0.83984375,0.359375,0}\bm{Z_{1}}(t)\bm{A}^{-1}(t)\bm{Z_{2}}^{T}(t)}\\ {\color[rgb]{0.83984375,0.359375,0}\bm{Z_{2}}(t)\bm{A}^{-1}(t)\bm{Z_{1}}^{T}(t )}&{\color[rgb]{0,0.44140625,0.69921875}\bm{Z_{2}}(t)\bm{A}^{-1}(t)\bm{Z_{2}}^ {T}(t)}\end{pmatrix},$ $$

with the time-dependent variables $\bm{Z_{1}}(t)\in\mathbb{R}^{N_{i}\times N_{h}}$, $\bm{Z_{2}}(t)\in\mathbb{R}^{N_{o}\times N_{h}}$, and $\bm{A}(t)\in\mathbb{R}^{N_{h}\times N_{h}}$:

$$ $\displaystyle\bm{Z_{1}}(t)$ $\displaystyle=\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{ \bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}-\frac{1}{2}\tilde {\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{V}_{\perp}}e^{\bm{\lambda_{\perp }}\frac{t}{\tau}}\bm{D}^{T},$ (13) $\displaystyle\bm{Z_{2}}(t)$ $\displaystyle=\frac{1}{2}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{ \bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}+\frac{1}{2}\tilde {\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{U}_{\perp}}e^{\bm{\lambda_{\perp }}\frac{t}{\tau}}\bm{D}^{T},$ (14) $\displaystyle\bm{A}(t)$ $\displaystyle=\mathbf{I}+\bm{B}\left(\frac{e^{2\tilde{\bm{S}_{\lambda}}\frac{t }{\tau}}-\mathbf{I}}{4\tilde{\bm{S}_{\lambda}}}\right)\bm{B}^{T}-\bm{C}\left( \frac{e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}}{4\tilde{\bm{S}_ {\lambda}}}\right)\bm{C}^{T}+\bm{D}\left(\frac{e^{\bm{\lambda}_{\perp}\frac{t} {\tau}}-\mathbf{I}}{\bm{\lambda}_{\perp}}\right)\bm{D}^{T}.$ (15) $$

Figure: Figure 2: A The temporal dynamics of the numerical simulation (colored lines) of the loss, network function, correlation of input and output weights, and the NTK (row 1-5 respectively) are exactly matched by the analytical solution (black dotted lines) for $\lambda=-5$ . B $\lambda=0.001$ Large initial weight values. C $\lambda=5$ initial weight values initialized as described in E.7.
Refer to caption: extracted/5871880/figures/analytical_matches.png

The proof of Theorem 4.3 is in Appendix B.
With this solution we can calculate the exact temporal dynamics of the loss, network function, RSMs and NTK (Fig. 2A, C) over a range of lambda-balanced initializations.

Implementation and simulation. One issue with the expression we derived in Theorem 4.3 is that it can be numerically unstable when simulating it for long time $t\gg 0$ as it involves taking the inverse of terms that involve exponentials that are diverging with $t$. If we make the additional assumption that $\bm{B}$ is invertible, then we can rearaange this expression to only use exponentials with negative coefficients, which we derive in Appendix B.5.
In the next section we will discuss the significance of $\bm{B}$ being invertible at initialization on the convergence of the dynamics. Simulation details are in Appendix E.7.

###### Lemma 4.1 .

###### Lemma 4.2 .

###### Theorem 4.3 .

## 5 Rich and Lazy Learning

In this section we use these solutions to gain a deeper understanding of the transition between the *rich* and *lazy* regimes by examining the dynamics as a function of lambda – the *relative scale* - as it varies between positive and negative infinity.
We investigate four key indicators of the learning regimes: the dynamics of singular vectors, the structure and robustness of the representations, and the evolution of the NTK.

Dynamics of the singular values. Here we examine a *lambda-balanced* linear network initialized with *task-aligned* weights.
Previous research Saxe et al. () has demonstrated that initial weights that are aligned with the task remain aligned throughout training, restricting the learning dynamics to the singular values of the network. This setting offers a valuable opportunity to build intuition about the impact of $\lambda$ on the dynamics of learning regimes, extending beyond previous solutions Tarmoun et al. (); Varre et al. ().

###### Theorem 5.1 .

Under the assumptions of Theorem 4.3 and with a task-aligned initialization, as defined in Saxe et al. (), the network function is given by the expression $\bm{W}_{2}\bm{W}_{1}(t)=\tilde{\bm{U}}\bm{S}(t)\tilde{\bm{V}}^{T}$ where $\bm{S}(t)\in\mathbb{R}^{N_{h}\times N_{h}}$ is a diagonal matrix of singular values with elements $s_{\alpha}(t)$ that evolve according to the equation,

$$ $s_{\alpha}(t)=s_{\alpha}(0)+\gamma_{\alpha}(t;\lambda)\left(\tilde{s}_{\alpha} -s_{\alpha}(0)\right),$ (16) $$

where $\tilde{s}_{\alpha}$ is the $\alpha$ singular value of $\tilde{\bm{S}}$ and $\gamma_{\alpha}(t;\lambda)$ is a $\lambda$-dependent monotonic transition function for each singular value that increases from $\gamma_{\alpha}(0;\lambda)=0$ to $\lim_{t\to\infty}\gamma_{\alpha}(t;\lambda)=1$ defined explicitly in Appendix C.1.
We find that under different limits of $\lambda$, the transition function converges pointwise to the sigmoidal ($\lambda\to 0$) and exponential ($\lambda\to\pm\infty$) transition functions,

$$ $\lim_{\lambda\to 0}\gamma_{\alpha}(t;\lambda)\to\frac{e^{2\tilde{s}_{\alpha} \frac{t}{\tau}}-1}{e^{2\tilde{s}_{\alpha}\frac{t}{\tau}}-1+\frac{\tilde{s}_{ \alpha}}{s_{\alpha}(0)}},\qquad\lim_{\lambda\to\pm\infty}\gamma_{\alpha}(t; \lambda)\to 1-e^{-|\lambda|\frac{t}{\tau}}.$ (17) $$

The proof for Theorem 5.1 can be found in Appendix C.1.
As shown in Fig.3 B, as $\lambda$ approaches zero, the dynamics resemble sigmoidal learning curves that traverse between saddle points, characteristic of the *rich* regime Braun et al. ().
In this regime the network learns the most salient features first, which can be beneficial for generalization Lampinen & Ganguli ().
Conversely, as shown in Fig.3 A and C, as the magnitude of $\lambda$ increases, the dynamics become exponential, characteristic of the *lazy* regime.
In this regime, all features are treated equally and the network’s dynamics resemble that of a shallow network.
Notably, similar effects have been observed previously in the context of large *absolute scales* Saxe et al. () independently of the *relative scale*. Overall, our results highlight the critical influence
*relative scale* $\lambda$ has in shaping the learning dynamics, from sigmoidal to exponential, steering the network between the *rich* and *lazy* regimes.

Figure: Figure 3: Simulated and analytical dynamics of the singular values of the network function with *relative scale* lambda A $\lambda=-2$ B $\lambda=0$ C $\lambda=2$ initialized as described in E.7.
Refer to caption: extracted/5871880/figures/aligned.png

The dynamics of the representations.
We now consider how the representations of the individual parameters $\bm{W}_{1}$ and $\bm{W}_{2}$ change through training.
We note that under lambda-balanced initializations there is simple structure which persists throughout training that allows us to recover the dynamics of the parameters up to a time-dependent orthogonal transformation from the dynamics of ${\bf Q}{\bf Q}^{T}(t)$.

###### Theorem 5.2 .

Under assumptions • ‣ 4 and • ‣ 4, if the network function $\bm{W}_{2}\bm{W}_{1}(t)=\bm{U}(t)\bm{S}(t)\bm{V}^{T}(t)$, then we can recover the parameters $\bm{W}_{2}(t)=\bm{U}(t)\bm{S}_{2}(t)\bm{R}^{T}(t)$ and $\bm{W}_{1}(t)=\bm{R}(t)\bm{S}_{1}(t)\bm{V}^{T}(t)$ up to time-dependent orthogonal transformation $\bm{R}(t)\in\mathbb{R}^{N_{h}\times N_{h}}$, where $\bm{S_{\lambda}}(t)=\sqrt{\bm{S}^{2}(t)+\tfrac{\lambda^{2}}{4}\mathbf{I}}$ and

$$ $\bm{S}_{1}(t)=\begin{pmatrix}\left(\bm{S_{\lambda}}(t)-\frac{\lambda\mathbf{I} }{2}\right)^{\frac{1}{2}},0_{\max(0,N_{i}-N_{o})}\end{pmatrix},\;\bm{S}_{2}(t) =\begin{pmatrix}\left(\bm{S_{\lambda}}(t)+\frac{\lambda\mathbf{I}}{2}\right)^{ \frac{1}{2}};0_{\max(0,N_{o}-N_{i})}\end{pmatrix}.$ (18) $$

The effective singular values $\bm{S}_{\lambda}$ of the corresponding weights are either up-weighted or down-weighted depending on the magnitude and sign of $\lambda$, splitting the representation into two parts. This division is reflected in the network’s internal representations. With our solution, ${\bf Q}{\bf Q}^{T}(t)$, which captures the temporal dynamics of the similarity between hidden layer activations, we can analyze the network’s internal representations in relation to the task. This allows us to determine whether the network adopts a *rich* or *lazy* representation, depending on the value of $\lambda$. Assuming convergence to the global minimum, which is guaranteed when the matrix ${\bf B}$ is non-singular, the internal representation satisfies ${\bf W}_{1}^{T}{\bf W}_{1}={\bf\tilde{V}}{\bf\tilde{S}}_{1}^{2}{\bf\tilde{V}}^
{T}$ and ${\bf W}_{2}{\bf W}_{2}^{T}={\bf\tilde{U}}{\bf\tilde{S}}_{2}^{2}{\bf\tilde{U}}^
{T}$ with ${\bf W}_{2}{\bf W}_{1}={\bf\tilde{U}}{\bf\tilde{S}}{\bf\tilde{V}}^{T}$. Theorem C.3 in the Appendix provides a detailed proof of this limiting behavior.

To illustrate this, we consider a hierarchical semantic learning task(^1^11In this setting, the network has equal input and output dimensions), introduced in Saxe et al. (); Braun et al. (), where living organisms are organized according to their features (Fig. 4A). The representational similarity of the task’s inputs (${\bf\tilde{V}}{\bf\tilde{S}}{\bf\tilde{V}}^{T}$) reflects this hierarchical structure (Fig.4A).
Similarly, the representational similarity of the task’s target values (${\bf\tilde{U}}{\bf\tilde{S}}{\bf\tilde{U}}^{T}$) highlights the primary groupings of items. When training a two-layer network with *relative scale* $\lambda$ equal to zero and task-agnostic initialization Mishkin & Matas (), the input and output representational similarity matrices (Fig.4
B) match the task’s structure upon convergence. As derived in Theorem C.4 the network is guaranteed to find a *rich* solution regardless of the *absolute scale* , meaning ${\bf W}_{1}^{T}{\bf W}_{1}={\bf\tilde{V}}{\bf\tilde{S}}{\bf\tilde{V}}^{T}$ and ${\bf W}_{2}{\bf W}_{2}^{T}={\bf\tilde{U}}{\bf\tilde{S}}{\bf\tilde{U}}^{T}$, as shown in Fig. 4 C. Hence the network learns task-specific representations. We also show that as $\lambda$ approaches either positive or negative infinity, the network symmetrically transitions into the *lazy* regime.
As demonstrated in Theorem C.4 and illustrated in Fig. 4, the representations converge to an identity matrix for both large positive and large negative values of $\lambda$— emerging in the output representations for large positive $\lambda$ and input representations for large negative $\lambda$. This convergence indicates that the network adopts task-agnostic representations. Meanwhile, the other respective RSMs become negligible, with scales proportional to $1/\lambda$. Therefore, as shown in Theorem C.5, the NTK becomes static and equivalent to the identity matrix in the limit as $\lambda$ approaches infinity. However, the downscaled representations of the network remain structured and task-specific. This property could be beneficial if the weights are later rescaled, such as during fine-tuning, potentially enhancing generalization and transfer learning, as we will demonstrate in Section 6.
We compare this to the scenario where both weights are initialized with large Gaussian values, leading to *lazy* learning that maintains a fixed NTK but lacks any structural representation, as illustrated in Fig.4. Consequently, we propose a new *lazy* regime, which we refer to as the *semi-structured* *lazy* regime. We note that these existing regimes preserve only the input or output representation, resulting in a partial loss of structural information. All together, we find that initialization will determine which layer in the network the task specification features resides in: layers initialized with large values will be task-agnostic, while those initialized with small values will be task-specific.

Figure: Figure 4: A A semantic learning task with the SVD of the input-output correlation matrix of the task. (top) $U$ and $V$ represent the singular vectors, and $S$ contains the singular values. (bottom) The respective RSMs as $USU^{\top}$ for the input and $VSV^{\top}$ for the output task. B Simulation results and C Theoretical input and output representation matrices after training, showing convergence when initialized with values of $\lambda$ equal to $-2$, 0 0, and $2$, according to the initialization scheme described in E.7. D Final RSMs matrices after training converged when initialised from random large weights. E After convergence, the network’s sensitivity to input noise (top panel) is invariant to $\lambda$, but the sensitivity to parameter noise increases as $\lambda$ becomes smaller (or larger) than zero.
Refer to caption: extracted/5871880/figures/representations_example.png

Representation robustness and sensitivity to noise.
Here we examine the relationship between the learning regime and the robustness of the learned representations to added noise in the inputs and parameters.
The expected post-convergence loss with added noise to the inputs is determined by the norm of the network function Braun et al. (), which in our setting is independent of $\lambda$. Specifically, if we add zero-centered noise $\xi_{\bf X}$ with variance $\sigma^{2}_{\bf X}$ to the inputs, then the expected loss is$\left\langle\mathcal{L}\right\rangle_{\xi_{\bf X}}=\sigma^{2}_{\bf X}\sum_{i=1
}^{N_{h}}{\bf\tilde{S}}^{2}_{i}+c,$
where $c$ is a constant that depends solely on the statistics of the training data (Figure 4E, Appendix C.3). However, if instead noise is added to the parameters, the expected loss scales quadratically with the norm of the weight matrices Braun et al. (), which in our setting depend on $\lambda$. In particular, zero-centered parameter noise $\xi_{{\bf W}_{1}}$ and $\xi_{{\bf W}_{2}}$ with variance $\sigma^{2}_{\bf W}$ results in an expected loss of

$$ $\left\langle\mathcal{L}\right\rangle_{\xi_{{\bf W}_{1}},\xi_{{\bf W}_{2}}}= \frac{1}{2}N_{i}\sigma^{2}_{\bf W}||{\bf W}_{2}||_{F}^{2}+\frac{1}{2}N_{o} \sigma^{2}_{\bf W}||{\bf W}_{1}||_{F}^{2}+\frac{1}{2}N_{i}N_{h}N_{o}\sigma^{4} +c,$ (19) $$

with norms

$$ $||{\bf W}_{1}||_{F}^{2}=\frac{1}{2}\sum_{i=1}^{N_{h}}\left(\sqrt{4{\bf\tilde{S }}_{i}^{2}+\lambda^{2}}+\lambda\right)\quad\text{and}\quad||{\bf W}_{2}||_{F}^ {2}=\frac{1}{2}\sum_{i=1}^{N_{h}}\left(\sqrt{4{\bf\tilde{S}}_{i}^{2}+\lambda^{ 2}}-\lambda\right).$ (20) $$

This implies that, under the assumption of equal input-output dimensions, networks initialized with weights such that $\lambda=0$, corresponding to the rich regime, converge to solutions that are most robust to parameter noise (Figure 4E, Appendix C.3). In practice, parameter noise could be interpreted as the noise occurring within the neurons of a biological network. Hence, a rich solution may enable a more robust representation in such systems.

The impact of the architecture.
Thus far, we have found that the magnitude of the *relative scale* parameter $\lambda$ determines the extent or rich and lazy learning.
Here, we explore how a network’s learning regime is also shaped by the interaction of its architecture and the sign of the *relative scale*. We consider three types of network architectures, depicted in Fig. 5A: *funnel networks*, which narrow from input to output ($N_{i}>N_{h}=N_{o}$); *inverted-funnel networks*, which expand from input to output ($N_{i}=N_{h}<N_{o}$); and *square networks*, where input and output dimensions are equal ($N_{i}=N_{h}=N_{o}$). Our solution, ${\bf Q}{\bf Q}^{T}$, captures the dynamics of the NTK across these different network architectures. To examine the NTK’s evolution under varying $\lambda$ initializations, we compute the kernel distance from initialization, as defined in Fort et al. (). As shown in Fig. 5B, we observe that funnel networks consistently enter the *lazy* regime as $\lambda\rightarrow\infty$, while inverted-funnel networks do so as $\lambda\rightarrow-\infty$. The NTK remains static during the initial phase, rigorously confirming the rank argument first introduced by Kunin et al. () for the multi-output setting. In the opposite limits of $\lambda$, these networks transition from a *lazy* regime to a *rich* regime. During this second alignment phase, the NTK matrix undergoes changes, indicating an initial *lazy* phase followed by a *delayed rich* phase. We further investigate and quantify this *delayed rich* regime, showing the NTK movement over training in Fig. 5C. This behavior is also quantified in Theorem C.6, which describes the rate of learning in this network. For square networks with equal input and output dimensions, this behavior is discussed in Section 5. Across all architectures, as $\lambda\to 0$, the networks consistently transition into the *rich* regime. Altogether, we further characterize the *delayed rich* regime in wide networks.

Figure: Figure 5: A. Schematic representations of the network architectures considered, from left to right: funnel network, square network, and inverted-funnel network. B. The plot shows the NTK kernel distance from initialization, as defined in Fort et al. () across the three architecture depicted schematically. C. The NTK kernel distance away from initialization over training time.
Refer to caption: extracted/5871880/figures/not_aligned.png

###### Theorem 5.1 .

###### Theorem 5.2 .

## 6 Application

In this section, we apply the exact solutions for the learning dynamics in deep linear networks with lambda-balanced initialization (described in Section 4) to illustrate several specific phenomena relevant to machine learning and neuroscience.

Continual learning.
Continual learning, as thoroughly reviewed by Parisi et al. (), has long posed a significant challenge for neural network models in contrast to biological networks, particularly due to the issue of catastrophic forgetting McCloskey & Cohen (); Ratcliff (); French (). Similarly to the framework presented by Braun et al. (), our approach describes the exact solutions of the networks dynamics trained across a sequence of tasks. As detailed in Appendix D.1, we demonstrate that, regardless of the chosen value of lambda, training on subsequent tasks can result in the overwriting of previously acquired knowledge, leading to catastrophic forgetting McCloskey & Cohen (); Ratcliff (); French ().

Reversal learning. As demonstrated in Braun et al. (), reversal learning theoretically does not succeed in deep linear networks as the initalization aligns with the separatrix of a saddle point. While simulations show that the learning dynamics can escape the saddle point due to numerical imprecision, the process is catastrophically slowed in its vicinity. However, when $\lambda$ is non-zero, reversal learning dynamics consistently succeed, as they avoid passing through the saddle point due to the initialization scheme. This is both theoretically proven and numerically illustrated in Appendix D.2. We also present a spectrum of reversal learning behaviors controlled by the *relative scale* $\lambda$, ranging from *rich* to *lazy* learning regimes. This spectrum has the potential to explain the diverse dynamics observed in animal behavior, offering insights into the learning regimes relevant to various neuroscience experiments.

Transfer learning.
We consider how different $\lambda$ initializations influence generalization to a new feature after being trained on an initial task. As detailed in Appendix D.3 we first train each network on the hierarchical semantic learning task described in Fig. 4. After, we add a new feature to the dataset for example ‘eats worms’
We train it specifically on the corresponding item, in this case, the goldfish, while keeping the rest of the network parameters unchanged. Afterwards, we evaluate the generalization to the other items. We observe in Appendix figure 8 that the hierarchical structure of the data is effectively transferred to the new feature when the representation is task-specific and $\lambda$ is zero. Conversely, when the output feature representation is *lazy*, meaning the hidden representation lacks adaptation, no hierarchical generalization is observed.
Strikingly, when $\lambda$ is positive, the hierarchical structure in the input weights remains small but structured, while the output weights exhibit a *lazy* representation and the network generalizes hierarchically. This indicates that the *lazy* regime structure can be beneficial for transfer learning.

## 7 Discussion

We derive exact solutions to the learning dynamics within a tractable model class: deep linear networks. While our findings extend the range of analytically describable two-layer linear network problems, they are still limited by a set of assumptions. In particular, relaxing the assumptions that input covariance must be white and that initialization must be *$\lambda$-balanced* could bring the analysis closer to practical applications in machine learning and neuroscience. Moving towards the nonlinear setting would also make the findings more applicable in real-world scenarios. Additionally, removing the requirement for networks to be full rank would enable us to describe the dynamics of linear autoencoders, including settings important in practice, such as LoRA Hu et al. ().
Despite these limitations, our solutions provide valuable insights into network behaviour. We examine the transition between the *rich* and *lazy* regimes by analyzing the dynamics as a function of $\lambda$—the *relative scale*—across its full range from positive to negative infinity.
Our analysis demonstrates that the *relative scale*, $\lambda$, is pivotal in managing the transition between *rich* and *lazy* regimes. Notably, we identify a structured *lazy* regime that promotes transfer learning. Building on previous work Kunin et al. () that shows these findings extend to basic nonlinear settings and practical scenarios, our theory suggests that further exploration of unbalanced initialization could optimize efficient feature learning. We leave for future work, the extension of this initialization to deep networks. Future work will focus on extending the application of the solution to the dynamics of fine-tuning and linear autoencoders.

#### Acknowledgments

This research was funded in whole, or in part, by the Wellcome Trust [216386/Z/19/Z]. For the purpose of Open Access, the author has applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.
C.D. and A.S. were supported by the Gatsby Charitable Foundation (GAT3755). Further, A.S. was supported by the Sainsbury Wellcome Centre Core Grant (219627/Z/19/Z) and A.S. is a CIFAR Azrieli Global Scholar in the Learning in Machines & Brains program.
A.M.P. was supported by the Imperial College London President’s PhD Scholarship. L.B. was supported by the Woodward Scholarship awarded by Wadham College, Oxford. and the Medical Research Council [MR/N013468/1]. D.K. thanks the Open Philanthropy AI Fellowship for support.
This research was funded in whole, or in part, by the Wellcome Trust [216386/Z/19/Z].

#### Author Contributions

Clementine and Nicolas led the work on the equal input-output solution to the exact dynamics, with support from Daniel. Clementine extended this work to the unequal input-output case, examining the impact of the architecture. Clementine and Nicolas also spearheaded the development of the minimal model for *rich* and *lazy* learning, emphasizing exact solutions and representation convergence.
Daniel, Nicolas, and Clementine collaborated on analyzing the dynamics of singular values, while Lukas led the investigation into representation robustness and sensitivity to noise. Clementine and Nicolas primarily conducted the analysis of the continual learning framework, while Alexandra and Clementine focused on transfer learning in the *rich* and *lazy* settings. Clementine took the lead on the reversal learning analysis. She was primarily responsible for leading the manuscript writing, with assistance from Daniel. All authors contributed to writing the appendix and refining the final manuscript.

## Appendix A Preliminaries

### A.1 Appendix: Balanced Condition

###### Definition A.1 (Definition of $\lambda$ -balanced property ( Saxe et al. ( 2013 ) , Marcotte et al. ( 2023 ) )) .

The weights $\bm{W_{1}},\bm{W_{2}}$ are *$\lambda$-balanced* if and only if there exists a Balanced Coefficient $\lambda\in\mathbb{R}$ such that:

$$ $B(\bm{W_{1}},\bm{W_{2}})=\bm{W_{2}}^{T}\bm{W_{2}}-\bm{W_{1}}\bm{W_{1}}^{T}= \lambda\mathbf{I}$ (21) $$

where $B$ is called the Balanced Computation.
For $\lambda=0$ we have Zero-Balanced given as
A5 ().
${\bf W}_{1}(0){\bf W}_{1}(0)^{T}={\bf W}_{2}(0)^{T}{\bf W}_{2}(0)$.

###### Theorem A.2 .

Balanced Condition Persists Through Training

Suppose at initialization

$$ $\bm{W_{2}(0)}^{T}\bm{W_{2}(0)}-\bm{W_{1}(0)}\bm{W_{1}(0)}^{T}=\lambda\mathbf{I}$ (22) $$

Then for all $t\geq 0$

$$ $\bm{W_{2}(t)}^{T}\bm{W_{2}(t)}-\bm{W_{1}(t)}\bm{W_{1}(t)}^{T}=\lambda\mathbf{I}$ (23) $$

###### Proof.

Consider:

$$ $\displaystyle\tau\frac{d}{dt}\left[\bm{W}_{2}(t)\bm{W}_{2}(t)^{T}-\bm{W}_{1}(t )\bm{W}_{1}(t)^{T}\right]$ $\displaystyle=\left(\tau\frac{d}{dt}\bm{W}_{2}(t)\right)^{T}\bm{W}_{2}(t)+\bm{ W}_{2}(t)^{T}\left(\tau\frac{d}{dt}\bm{W}_{2}(t)\right)$ $\displaystyle\quad-\left(\tau\frac{d}{dt}\bm{W}_{1}(t)\right)\bm{W}_{1}(t)^{T} -\bm{W}_{1}(t)\left(\tau\frac{d}{dt}\bm{W}_{1}(t)\right)^{T}$ $\displaystyle=\bm{W}_{1}(t)\left(\tilde{\bm{\Sigma}}^{yx}-\bm{W}_{2}(t)\bm{W}_ {1}(t)\tilde{\bm{\Sigma}}^{xx}\right)^{T}\bm{W}_{2}(t)$ $\displaystyle\quad+\bm{W}_{2}(t)^{T}\left(\tilde{\bm{\Sigma}}^{yx}-\bm{W}_{2}( t)\bm{W}_{1}(t)\tilde{\bm{\Sigma}}^{xx}\right)\bm{W}_{1}(t)$ $\displaystyle\quad-\bm{W}_{2}(t)^{T}\left(\tilde{\bm{\Sigma}}^{yx}-\bm{W}_{2}( t)\bm{W}_{1}(t)\tilde{\bm{\Sigma}}^{xx}\right)\bm{W}_{1}(t)$ $\displaystyle\quad-\bm{W}_{1}(t)\left(\tilde{\bm{\Sigma}}^{yx}-\bm{W}_{2}(t) \bm{W}_{1}(t)\tilde{\bm{\Sigma}}^{xx}\right)\bm{W}_{2}(t)$ $\displaystyle=\bm{0}$ $$

Note that $\bm{W_{2}(t)}^{T}\bm{W_{2}(t)}-\bm{W_{1}(t)}\bm{W_{1}(t)}^{T}$ is conserved for any initial value $\lambda$.
∎

###### Definition A.1 (Definition of λ 𝜆 \lambda italic_λ -balanced property ( Saxe et al. ( 2013 ) , Marcotte et al. ( 2023 ) )) .

###### Theorem A.2 .

###### Proof.

### A.2 Discussion Assumptions

##### Whittened Inputs.

Although the whitened input assumption is quite strong, it is commonly used in analytical work to obtain exact solutions, and much of the existing literature relies on these solutions Fukumizu (); Braun et al. (); Kunin et al. () . Kunin et al. () goes further by exploring the implicit bias of the trajectory without relying on exact solutions. When $X^{\intercal}X$ is low-rank, they can only predict the trajectories in the limit as $\lambda\to\pm\infty$. If the interpolating manifold is one-dimensional, the solution can be solved exactly in terms of $\lambda$ (black dots).

##### Dimension.

Fukumizu assumed equal input and output dimensions $N_{i}=N_{o}$, but allowed for a bottleneck in the hidden dimension of the network $N_{h}\leq N_{i}=N_{o}$.
The work by Braun et al. () extended Fukumizu () solutions to cases with unequal input and output dimensions $N_{i}\neq N_{o}$, but to so did not allow a bottleneck $N_{h}=\min\{N_{i},N_{o}\}$ and added an assumption on the invertibility of a statistic of the singular vector overlap between the model and the input-output statistics.
In our work we allow for unequal input and output $N_{i}\neq N_{o}$ and do not introduce an additional invertibility assumption.

##### Balancedness.

The main distinction between our work and prior works is that both Fukumizu () and Braun et al. () assumed zero-balanced ${\bf W}_{1}(0){\bf W}_{1}(0)^{T}={\bf W}_{2}(0)^{T}{\bf W}_{2}(0)$, while we relax this assumption to $\lambda$-balanced.
The zero-balanced condition restricts the networks to a *rich* setting. We develop solutions to explore the continuum between the *rich* and the *lazy* regime.
While some works, such as Tarmoun et al. (), have considered removing this constraint, their solutions remain in an unstable and mixed form. Our work, in its form enable the understanding of different learning regimes by exploring initialization properties beyond just *absolute scale* and demonstrate that this transition can be accessed and controlled by adjusting a key parameter: the *relative scale*. Other studies, such as Kunin et al. () and Xu & Zheng (), have similarly relaxed the balancedness assumption but were limited to single-output neuron settings.

## Appendix B Appendix: Exact learning dynamics with prior knowledge

### B.1 Appendix: Fukumizu Approach

###### Lemma B.1 .

We introduce the variables

$$ $\mathbf{Q}=\begin{bmatrix}\mathbf{W}_{1}^{T}\\ \mathbf{W}_{2}\end{bmatrix}\quad\text{and}\quad\mathbf{Q}\mathbf{Q}^{T}=\begin {bmatrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T} \\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{bmatrix}.$ (24) $$

Defining

$$ $\mathbf{F}=\begin{bmatrix}-\frac{\lambda}{2}I&(\tilde{\Sigma}^{yx})^{T}\\ \tilde{\Sigma}^{yx}&\frac{\lambda}{2}I\end{bmatrix},$ (25) $$

the gradient flow dynamics of $\mathbf{Q}\mathbf{Q}^{T}(t)$ can be written as a differential matrix Riccati equation

$$ $\tau\frac{d}{dt}(\mathbf{Q}\mathbf{Q}^{T})=\mathbf{F}\mathbf{Q}\mathbf{Q}^{T}+ \mathbf{Q}\mathbf{Q}^{T}\mathbf{F}-(\mathbf{Q}\mathbf{Q}^{T})^{2}.$ (26) $$

###### Proof.

We introduce the variables

$$ $\mathbf{Q}=\begin{bmatrix}\mathbf{W}_{1}^{T}\\ \mathbf{W}_{2}\end{bmatrix}\quad\text{and}\quad\mathbf{Q}\mathbf{Q}^{T}=\begin {bmatrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T} \\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{bmatrix}.$ (27) $$

We compute the time derivative

$$ $\tau\frac{d}{dt}(\mathbf{Q}\mathbf{Q}^{T})=\tau\begin{bmatrix}\frac{d\mathbf{W }_{1}^{T}}{dt}\mathbf{W}_{1}+\mathbf{W}_{1}^{T}\frac{d\mathbf{W}_{1}}{dt}& \frac{d\mathbf{W}_{1}^{T}}{dt}\mathbf{W}_{2}+\mathbf{W}_{1}^{T}\frac{d\mathbf{ W}_{2}}{dt}\\ \frac{d\mathbf{W}_{2}}{dt}\mathbf{W}_{1}+\mathbf{W}_{2}\frac{d\mathbf{W}_{1}}{ dt}&\frac{d\mathbf{W}_{2}^{T}}{dt}\mathbf{W}_{2}+\mathbf{W}_{2}^{T}\frac{d \mathbf{W}_{2}}{dt}\end{bmatrix}.$ (28) $$

Using equations 18 and 19, we compute the four quadrants separately giving

$$ $\tau\left(\frac{d\mathbf{W}_{1}^{T}}{dt}\mathbf{W}_{1}+\mathbf{W}_{1}^{T}\frac {d\mathbf{W}_{1}}{dt}\right)=$ (29) $$

$$ $=(\Sigma^{yx}-\mathbf{W}_{2}\mathbf{W}_{1})^{T}\mathbf{W}_{2}\mathbf{W}_{1}+ \mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T}(\Sigma^{yx}-\mathbf{W}_{2}\mathbf{W}_{1})$ (30) $$

$$ $=(\Sigma^{yx})^{T}\mathbf{W}_{2}\mathbf{W}_{1}+\mathbf{W}_{1}^{T}\mathbf{W}_{2 }^{T}\Sigma^{yx}-\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T}\mathbf{W}_{2}\mathbf{W}_ {1}-(\mathbf{W}_{2}\mathbf{W}_{1})^{T}\mathbf{W}_{2}\mathbf{W}_{1}$ (31) $$

$$ $=(\Sigma^{yx})^{T}\mathbf{W}_{2}\mathbf{W}_{1}+\mathbf{W}_{1}^{T}\mathbf{W}_{2 }^{T}\Sigma^{yx}-\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T}\mathbf{W}_{2}\mathbf{W}_ {1}-\mathbf{W}_{1}^{T}\mathbf{W}_{1}\mathbf{W}_{1}^{T}\mathbf{W}_{1}-\lambda \mathbf{W}_{1}^{T}\mathbf{W}_{1},$ (32) $$

$$ $\tau\left(\frac{d\mathbf{W}_{1}^{T}}{dt}\mathbf{W}_{2}^{T}+\mathbf{W}_{1}^{T} \frac{d\mathbf{W}_{2}^{T}}{dt}\right)=$ (33) $$

$$ $=(\Sigma^{yx}-\mathbf{W}_{2}\mathbf{W}_{1})^{T}\mathbf{W}_{2}\mathbf{W}_{2}^{T }+\mathbf{W}_{1}^{T}\mathbf{W}_{1}(\Sigma^{yx}-\mathbf{W}_{2}\mathbf{W}_{1})^{T}$ (34) $$

$$ $=(\Sigma^{yx})^{T}\mathbf{W}_{2}\mathbf{W}_{2}^{T}+\mathbf{W}_{1}^{T}\mathbf{W }_{1}(\Sigma^{yx})^{T}-\mathbf{W}_{1}^{T}\mathbf{W}_{1}\mathbf{W}_{1}^{T} \mathbf{W}_{2}^{T}-\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T}\mathbf{W}_{2}\mathbf{W }_{2}^{T},$ (35) $$

$$ $\tau\left(\frac{d\mathbf{W}_{2}}{dt}\mathbf{W}_{1}+\mathbf{W}_{2}\frac{d \mathbf{W}_{1}}{dt}\right)=$ (36) $$

$$ $=(\Sigma^{yx}-\mathbf{W}_{2}\mathbf{W}_{1})\mathbf{W}_{1}^{T}\mathbf{W}_{1}+ \mathbf{W}_{2}\mathbf{W}_{2}^{T}(\Sigma^{yx}-\mathbf{W}_{2}\mathbf{W}_{1})$ (37) $$

$$ $=\Sigma^{yx}\mathbf{W}_{1}^{T}\mathbf{W}_{1}+\mathbf{W}_{2}\mathbf{W}_{2}^{T} \Sigma^{yx}-\mathbf{W}_{2}\mathbf{W}_{2}^{T}\mathbf{W}_{2}\mathbf{W}_{1}- \mathbf{W}_{2}\mathbf{W}_{1}\mathbf{W}_{1}^{T}\mathbf{W}_{1},$ (38) $$

$$ $\tau\left(\frac{d\mathbf{W}_{2}}{dt}\mathbf{W}_{2}^{T}+\mathbf{W}_{2}\frac{d \mathbf{W}_{2}^{T}}{dt}\right)=$ (39) $$

$$ $(\tilde{\Sigma}^{yx}-\mathbf{W}_{2}\mathbf{W}_{1})\mathbf{W}_{1}^{T}\mathbf{W} _{2}^{T}+\mathbf{W}_{2}\mathbf{W}_{1}(\tilde{\Sigma}^{yx}-\mathbf{W}_{2} \mathbf{W}_{1})^{T}$ (40) $$

$$ $=\tilde{\Sigma}^{yx}\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T}+\mathbf{W}_{2}\mathbf {W}_{1}(\tilde{\Sigma}^{yx})^{T}-\mathbf{W}_{2}\mathbf{W}_{1}\mathbf{W}_{1}^{T }\mathbf{W}_{2}^{T}-\mathbf{W}_{2}\mathbf{W}_{1}(\mathbf{W}_{2}\mathbf{W}_{1}) ^{T}$ (41) $$

$$ $=\tilde{\Sigma}^{yx}\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T}+\mathbf{W}_{2}\mathbf {W}_{1}(\tilde{\Sigma}^{yx})^{T}-\mathbf{W}_{2}\mathbf{W}_{1}\mathbf{W}_{1}^{T }\mathbf{W}_{2}^{T}-\mathbf{W}_{2}\mathbf{W}_{1}\mathbf{W}_{1}^{T}\mathbf{W}_{ 2}^{T}$ (42) $$

$$ $=\tilde{\Sigma}^{yx}\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T}+\mathbf{W}_{2}\mathbf {W}_{1}(\tilde{\Sigma}^{yx})^{T}-\mathbf{W}_{2}\mathbf{W}_{1}\mathbf{W}_{1}^{T }\mathbf{W}_{2}^{T}-\mathbf{W}_{2}\mathbf{W}_{2}^{T}\mathbf{W}_{2}\mathbf{W}_{ 2}^{T}+\lambda\mathbf{W}_{2}\mathbf{W}_{2}^{T}.$ (43) $$

Defining

$$ $\mathbf{F}=\begin{bmatrix}-\frac{\lambda}{2}I&(\tilde{\Sigma}^{yx})^{T}\\ \tilde{\Sigma}^{yx}&\frac{\lambda}{2}I\end{bmatrix},$ (44) $$

the gradient flow dynamics of $\mathbf{Q}\mathbf{Q}^{T}(t)$ can be written as a differential matrix Riccati equation

$$ $\tau\frac{d}{dt}(\mathbf{Q}\mathbf{Q}^{T})=\mathbf{F}\mathbf{Q}\mathbf{Q}^{T}+ \mathbf{Q}\mathbf{Q}^{T}\mathbf{F}-(\mathbf{Q}\mathbf{Q}^{T})^{2}.$ (45) $$

We write $\tau\frac{d}{dt}(\mathbf{Q}\mathbf{Q}^{T})$ for completeness

$$ $\displaystyle\tau\frac{d}{dt}(\mathbf{Q}\mathbf{Q}^{T})=$ $\displaystyle\left[\begin{matrix}-\frac{\lambda}{2}&(\tilde{\Sigma}^{yx})^{T} \\ \tilde{\Sigma}^{yx}&\frac{\lambda}{2}\end{matrix}\right]\left[\begin{matrix} \mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_{1}^{T}\mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix} \right]+\left[\begin{matrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_{1}^{T }\mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix} \right]^{T}\left[\begin{matrix}-\frac{\lambda}{2}&(\tilde{\Sigma}^{yx})^{T}\\ \tilde{\Sigma}^{yx}&\frac{\lambda}{2}\end{matrix}\right]$ (46) $\displaystyle-\left[\begin{matrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_ {1}^{T}\mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix} \right]^{2}$ $$

$$ $\displaystyle=$ $\displaystyle\left[\begin{matrix}-\frac{\lambda}{2}&(\tilde{\Sigma}^{yx})^{T} \\ \tilde{\Sigma}^{yx}&\frac{\lambda}{2}\end{matrix}\right]\left[\begin{matrix} \mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_{1}^{T}\mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix} \right]+\left[\begin{matrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_{1}^{T }\mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix} \right]^{T}\left[\begin{matrix}-\frac{\lambda}{2}&(\tilde{\Sigma}^{yx})^{T}\\ \tilde{\Sigma}^{yx}&\frac{\lambda}{2}\end{matrix}\right]$ (47) $\displaystyle-\left[\begin{matrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_ {1}^{T}\mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix} \right]\left[\begin{matrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_{1}^{T} \mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix}\right]$ $$

$$ $\displaystyle=\left[\begin{matrix}-\frac{\lambda}{2}\mathbf{W}_{1}^{T}\mathbf{ W}_{1}+(\tilde{\Sigma}^{yx})^{T}\mathbf{W}_{2}\mathbf{W}_{1}&-\frac{\lambda}{2 }\mathbf{W}_{1}^{T}\mathbf{W}_{2}+(\tilde{\Sigma}^{yx})^{T}\mathbf{W}_{2} \mathbf{W}_{2}^{T}\\ \tilde{\Sigma}^{yx}\mathbf{W}_{1}^{T}\mathbf{W}_{1}+\frac{\lambda}{2}\mathbf{W }_{2}\mathbf{W}_{1}&\tilde{\Sigma}^{yx}\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T}+ \frac{\lambda}{2}\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix}\right]$ (48) $\displaystyle+\left[\begin{matrix}-\frac{\lambda}{2}\mathbf{W}_{1}^{T}\mathbf{ W}_{1}+\mathbf{W}_{1}^{T}\mathbf{W}_{1}(\tilde{\Sigma}^{yx})^{T}&\frac{\lambda }{2}\mathbf{W}_{1}^{T}\mathbf{W}_{2}+\mathbf{W}_{1}^{T}\mathbf{W}_{2}(\tilde{ \Sigma}^{yx})^{T}\\ -\frac{\lambda}{2}\mathbf{W}_{2}^{T}\mathbf{W}_{1}+\mathbf{W}_{2}\mathbf{W}_{1 }(\tilde{\Sigma}^{yx})^{T}&\frac{\lambda}{2}\mathbf{W}_{2}\mathbf{W}_{2}^{T}+ \mathbf{W}_{2}\mathbf{W}_{2}^{T}(\tilde{\Sigma}^{yx})^{T}\end{matrix}\right]$ $\displaystyle-\left[\begin{matrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_ {1}^{T}\mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix} \right]\left[\begin{matrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}&\mathbf{W}_{1}^{T} \mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix}\right]$ $$

$$ $\displaystyle=\left[\begin{matrix}-\frac{\lambda}{2}\mathbf{W}_{1}^{T}\mathbf{ W}_{1}+(\tilde{\Sigma}^{yx})^{T}\mathbf{W}_{2}\mathbf{W}_{1}&-\frac{\lambda}{2 }\mathbf{W}_{1}^{T}\mathbf{W}_{2}+(\tilde{\Sigma}^{yx})^{T}\mathbf{W}_{2} \mathbf{W}_{2}^{T}\\ \tilde{\Sigma}^{yx}\mathbf{W}_{1}^{T}\mathbf{W}_{1}+\frac{\lambda}{2}\mathbf{W }_{2}\mathbf{W}_{1}&\tilde{\Sigma}^{yx}\mathbf{W}_{1}^{T}\mathbf{W}_{2}^{T}+ \frac{\lambda}{2}\mathbf{W}_{2}\mathbf{W}_{2}^{T}\end{matrix}\right]$ (45) $\displaystyle+\left[\begin{matrix}-\frac{\lambda}{2}\mathbf{W}_{1}^{T}\mathbf{ W}_{1}+\mathbf{W}_{1}^{T}\mathbf{W}_{1}(\tilde{\Sigma}^{yx})^{T}&\frac{\lambda }{2}\mathbf{W}_{1}^{T}\mathbf{W}_{2}+\mathbf{W}_{1}^{T}\mathbf{W}_{2}(\tilde{ \Sigma}^{yx})^{T}\\ -\frac{\lambda}{2}\mathbf{W}_{2}^{T}\mathbf{W}_{1}+\mathbf{W}_{2}\mathbf{W}_{1 }(\tilde{\Sigma}^{yx})^{T}&\frac{\lambda}{2}\mathbf{W}_{2}\mathbf{W}_{2}^{T}+ \mathbf{W}_{2}\mathbf{W}_{2}^{T}(\tilde{\Sigma}^{yx})^{T}\end{matrix}\right]$ $\displaystyle-\left[\begin{matrix}\mathbf{W}_{1}^{T}\mathbf{W}_{1}\mathbf{W}_{ 1}^{T}\mathbf{W}_{1}+\mathbf{W}_{1}^{T}\mathbf{W}_{2}\mathbf{W}_{2}^{T}\mathbf {W}_{1}&\mathbf{W}_{1}^{T}\mathbf{W}_{1}\mathbf{W}_{1}^{T}\mathbf{W}_{2}+ \mathbf{W}_{1}^{T}\mathbf{W}_{2}\mathbf{W}_{2}^{T}\mathbf{W}_{2}\\ \mathbf{W}_{2}\mathbf{W}_{1}\mathbf{W}_{1}^{T}\mathbf{W}_{1}+\mathbf{W}_{2} \mathbf{W}_{2}^{T}\mathbf{W}_{2}\mathbf{W}_{1}&\mathbf{W}_{2}\mathbf{W}_{1} \mathbf{W}_{1}^{T}\mathbf{W}_{2}+\mathbf{W}_{2}\mathbf{W}_{2}^{T}\mathbf{W}_{2 }\mathbf{W}_{2}^{T}\end{matrix}\right]$ $$

∎

The four quadrants of 28 are equivalent to equations 32, 35, 38, and 43 respectively.

###### Lemma B.1 .

###### Proof.

### B.2 𝐐𝐐 T superscript 𝐐𝐐 𝑇 {\bf Q}{\bf Q}^{T} bold_QQ start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT Diagonalisation

###### Lemma B.2 .

If $\bm{F}=\bm{P}\bm{\Lambda}\bm{P}^{T}$ is symmetric and diagonalizable, then the matrix Riccati differential equation $\frac{d}{dt}(\mathbf{Q}\mathbf{Q}^{T})=\mathbf{F}\mathbf{Q}\mathbf{Q}^{T}+
\mathbf{Q}\mathbf{Q}^{T}\mathbf{F}-(\mathbf{Q}\mathbf{Q}^{T})^{2}$ with initialization ${\bf Q}{\bf Q}^{T}(0)={\bf Q}(0){\bf Q}(0)^{T}$ has a unique solution for all $t\geq 0$, and the solution is given by

$$ ${\bf Q}{\bf Q}^{T}(t)=e^{{\bf F}\frac{t}{\tau}}{\bf Q}(0)\left[{\bf I}+{\bf Q} (0)^{T}\bm{P}\left(\frac{e^{2\bm{\Lambda}\frac{t}{\tau}}-\mathbf{I}}{\bm{2 \Lambda}}\right)\bm{P}^{T}{\bf Q}(0)\right]^{-1}{\bf Q}(0)^{T}e^{{\bf F}\frac{ t}{\tau}}.$ (49) $$

This is true even when there exists $\bm{\Lambda}_{i}=0$.

###### Proof.

First we show that there exists a unique solution to the initial value problem stated. This is true by Picard-Lindelöf theorem.
Now we show that the provided solution satisfies the ODE.
Let $\bm{L}=e^{{\bf F}\frac{t}{\tau}}{\bf Q}(0)$ and $\bm{C}={\bf I}+{\bf Q}(0)^{T}\bm{P}\left(\frac{e^{2\bm{\Lambda}\frac{t}{\tau}}
-\mathbf{I}}{2\bm{\Lambda}}\right)\bm{P}^{T}{\bf Q}(0)$ such that solution ${\bf Q}{\bf Q}^{T}(t)=\bm{L}\bm{C}^{-1}\bm{L}^{T}$.
The time derivative of ${\bf Q}{\bf Q}^{T}$ is then given by

$$ $\frac{d}{dt}({\bf Q}{\bf Q}^{T})=\frac{d}{dt}(\bm{L})\bm{C}^{-1}\bm{L}^{T}+\bm {L}\frac{d}{dt}(\bm{C}^{-1})\bm{L}^{T}+\bm{L}\bm{C}^{-1}\frac{d}{dt}(\bm{L}^{T})$ (50) $$

Solving for these derivatives individually, we find

$$ $\displaystyle\frac{d}{dt}(\bm{L})$ $\displaystyle=\frac{d}{dt}e^{{\bf F}\frac{t}{\tau}}{\bf Q}(0)=\bm{F}e^{{\bf F} \frac{t}{\tau}}{\bf Q}(0)=\bm{F}\bm{L}$ (51) $\displaystyle\frac{d}{dt}(\bm{C}^{-1})$ $\displaystyle=-\bm{C}^{-1}\frac{d}{dt}(\bm{C})\bm{C}^{-1}=-\bm{C}^{-1}{\bf Q}( 0)^{T}\bm{P}\frac{d}{dt}\left(\frac{e^{2\bm{\Lambda}\frac{t}{\tau}}-\mathbf{I} }{2\bm{\Lambda}}\right)\bm{P}^{T}{\bf Q}(0)\bm{C}^{-1}$ (52) $$

We consider the derivative of the fraction serpately,

$$ $\frac{d}{dt}\left(\frac{e^{2\bm{\Lambda}\frac{t}{\tau}}-\mathbf{I}}{2\bm{ \Lambda}}\right)=e^{2\bm{\Lambda}\frac{t}{\tau}}$ (53) $$

this is true even in the limit as $\lambda_{i}\to 0$. Plugging these derivatives back in we see that the solution satisfies the ODE.
Lastly, let $t=0$, we see that the the solution satisfies the initial conditions.
∎

###### Lemma B.2 .

###### Proof.

### B.3 𝑭 𝑭 \bm{F}\quad bold_italic_F Diagonalization

###### Lemma B.3 .

Under assumptions of full-rank • ‣ 4, the eigendecomposition of $\mathbf{F}=\mathbf{P}\bm{\Lambda}\mathbf{P}^{T}$ where

$$ $\mathbf{P}=\frac{1}{\sqrt{2}}\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})&\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})&\sqrt{2}\tilde{\bm{V}}_{\perp}\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})&\sqrt{2}\tilde{\bm{U}}_{\perp} \end{pmatrix},\quad\bm{\Lambda}=\begin{pmatrix}\tilde{\bm{S}_{\lambda}}&0&0\\ 0&-\tilde{\bm{S}_{\lambda}}&0\\ 0&0&\bm{\lambda}_{\perp}\end{pmatrix}$ (54) $$

and the matrices $\tilde{\bm{S}_{\lambda}}$, $\bm{\lambda}_{\perp}$, $\tilde{\bm{H}}$, and $\tilde{\bm{G}}$ are the diagonal matrices defined as:

$$ $\tilde{\bm{S}_{\lambda}}=\sqrt{\tilde{\bm{S}}^{2}+\frac{\lambda^{2}}{4}\mathbf {I}},\;\;\bm{\lambda}_{\perp}=\mathrm{sgn}(N_{o}-N_{i})\frac{\lambda}{2} \mathbf{I},\;\;\tilde{\bm{H}}=\mathrm{sgn}(\lambda)\sqrt{\frac{\tilde{\bm{S_{ \lambda}}}-\tilde{\bm{S}}}{\tilde{\bm{S_{\lambda}}}+\tilde{\bm{S}}}},\;\; \tilde{\bm{G}}=\frac{1}{\sqrt{\mathbf{I}+\tilde{\bm{H}}^{2}}}.$ (55) $$

Beyond the invertibility of $F$, notice from the equation (Fukumizu solution) we need to understand the relationship between $F$ and $Q(0)$.
To do this the following lemma relates the structure between the SVD of the model with the SVD structure of the individual parameters.

###### Proof.

We leave for the reader by computing

$$ $\bm{F}=\bm{P}\bm{\Lambda}\bm{P}^{T}$ (56) $$

∎

###### Lemma B.3 .

###### Proof.

### B.4 Solution Unequal-Input-Output

###### Theorem B.4 .

Under the assumptions of whitened inputs, • ‣ 4, lambda-balanced weights • ‣ 4, no bottleneck • ‣ 4, and
full rank • ‣ 4, the temporal dynamics of ${\bf Q}{\bf Q}^{T}$ are

$$ $\displaystyle{\bf Q}{\bf Q}^{T}(t)=\begin{pmatrix}{\color[rgb]{ 0,0.62109375,0.44921875}\bm{Z_{1}}\bm{A}^{-1}\bm{Z_{1}}^{T}}&{\color[rgb]{ 0.83984375,0.359375,0}\bm{Z_{1}}\bm{A}^{-1}\bm{Z_{2}}^{T}}\\ {\color[rgb]{0.83984375,0.359375,0}\bm{Z_{2}}\bm{A}^{-1}\bm{Z_{1}}^{T}}&{ \color[rgb]{0,0.44140625,0.69921875}\bm{Z_{2}}\bm{A}^{-1}\bm{Z_{2}}^{T}}\end{ pmatrix},$ $$

where the variables $\bm{Z_{1}}\in\mathbb{R}^{N_{i}\times N_{h}}$, $\bm{Z_{2}}\in\mathbb{R}^{N_{o}\times N_{h}}$, and $\bm{A}\in\mathbb{R}^{N_{h}\times N_{h}}$ are defined as

$$ $\displaystyle\bm{Z_{1}}(t)=$ $\displaystyle\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm {G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}-\frac{1}{2}\tilde{ \bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda }}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{V}_{\perp}}e^{\bm{\lambda_{\perp}}\frac {t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}$ (57) $\displaystyle\bm{Z_{2}}(t)=$ $\displaystyle\frac{1}{2}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm {G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}+\frac{1}{2}\tilde{ \bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda }}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{U}_{\perp}}e^{\bm{\lambda_{\perp}}\frac {t}{\tau}}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)$ (58) $\displaystyle\bm{A}(t)=$ $\displaystyle\mathbf{I}+\bm{B}\left(\frac{e^{2\tilde{\bm{S}_{\lambda}}\frac{t} {\tau}}-\mathbf{I}}{4\tilde{\bm{S}_{\lambda}}}\right)\bm{B}^{T}-\bm{C}\left( \frac{e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}}{4\tilde{\bm{S}_ {\lambda}}}\right)\bm{C}^{T}+{\bf W}_{2}(0)^{T}{\bf\tilde{U}_{\perp}}\left( \frac{e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}-\mathbf{I}}{\bm{\lambda}_{\perp}} \right){\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)$ $\displaystyle+{\bf W}_{1}(0){\bf\tilde{V}_{\perp}}\left(\frac{e^{\bm{\lambda}_ {\perp}\frac{t}{\tau}}-\mathbf{I}}{\bm{\lambda}_{\perp}}\right){\bf\tilde{V}_{ \perp}}^{T}{\bf W}_{1}(0)^{T}$ (59) $$

###### Proof.

We start and use the diagonalization of ${\bf F}$ to rewrite the matrix exponential of $\bm{F}$ and $\bm{F}$.
Note that ${\bf P}^{T}{\bf P}={\bf P}{\bf P}^{T}={\bf I}$ and therefore ${\bf P}^{T}={\bf P}^{-1}$.

$$ $\displaystyle e^{{\bf F}\frac{t}{\tau}}$ $\displaystyle={\bf P}e^{\bf\Gamma}{\bf P}^{T}$ $\displaystyle=\frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})&{\bf\tilde{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})&\sqrt{2}{\bf V_{\perp}}\\ {\bf\tilde{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-{\bf\tilde{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})&\sqrt{2}{\bf U_{\perp}}\end{ bmatrix}\begin{bmatrix}e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}&0&0\\ 0&e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}&0\\ 0&0&e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}\\ \end{bmatrix}\frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})&{\bf\tilde{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})&\sqrt{2}{\bf V_{\perp}}\\ {\bf\tilde{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-{\bf\tilde{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})&\sqrt{2}{\bf U_{\perp}}\end{ bmatrix}^{T}$ $\displaystyle=\frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})&{\bf\tilde{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})\\ {\bf\tilde{U}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})&-{\bf\tilde{U}}( \tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})\end{bmatrix}\begin{bmatrix}e^{ \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}&0\\ 0&e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\end{bmatrix}\frac{1}{\sqrt{2}} \begin{bmatrix}{\bf\tilde{V}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})&{ \bf\tilde{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})\\ {\bf\tilde{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-{\bf\tilde{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\end{bmatrix}^{T}+2\frac{1}{\sqrt{ 2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\end{bmatrix}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}\frac {1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\end{bmatrix}^{T}$ $\displaystyle={\bf O}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O}+2{\bf M}e^{\bm{ \lambda}_{\perp}\frac{t}{\tau}}{\bf M}^{T}\text{.}$ (60) $$

$$ $\displaystyle e^{{\bf F}\frac{t}{\tau}}{\bf F}^{-1}e^{{\bf F}\frac{t}{\tau}}-{ \bf F}^{-1}$ $\displaystyle={\bf O}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O}^{T}{\bf O}{\bf \Lambda}^{-1}{\bf O}^{T}{\bf O}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O}^{T}-{\bf O }{\bf\Lambda}^{-1}{\bf O}^{T}+{\bf M}(e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}- \mathbf{I})(\bm{\lambda}_{\perp})^{-1}{\bf M}^{T}.$ (61) $\displaystyle{{\bf F}}$ $\displaystyle={\bf O}\Lambda{\bf O}^{T}+2{\bf M}\bm{\lambda}_{\perp}{\bf M}^{T}$ (62) $$

Where $\bm{M}=\frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}\\
{\bf\tilde{U}_{\perp}}\end{bmatrix}^{T}$. Placing these expressions into equation 49 gives

$$ $\displaystyle{\bf Q}{\bf Q}^{T}(t)=$ $\displaystyle\left[{\bf O}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O}^{T}+2{\bf M}e^ {\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf M}^{T}\right]{\bf Q}(0)$ $\displaystyle\left[{\bf I}+\frac{1}{2}{\bf Q}(0)^{T}\left({\bf O}\left(e^{2{ \bf\Lambda}\frac{t}{\tau}}-{\bf I}\right){\bf\Lambda}^{-1}{\bf O}^{T}+{\bf M}( e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}-{\bf I})\bm{\lambda}_{\perp}^{-1}{\bf M }^{T}\right){\bf Q}(0)\right]^{-1}$ (63) $\displaystyle\ {\bf Q}(0)^{T}\left[{\bf O}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O }^{T}+2{\bf M}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf M}^{T}\right]^{T}$ $$

$$ $\bm{O}^{T}\bm{Q}(0)=\frac{1}{\sqrt{2}}\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm {G}}-\tilde{\bm{H}}\tilde{\bm{G}})&\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H} }\tilde{\bm{G}})\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\end{pmatrix}^{T}\begin{pmatrix} \bm{W}_{1}^{T}(0)\\ \bm{W}_{2}(0)\end{pmatrix}$ $$

$$ $=\frac{1}{\sqrt{2}}\begin{pmatrix}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}} )\tilde{\bm{V}}^{T}\bm{W}_{1}^{T}(0)+(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{ G}})\tilde{\bm{U}}^{T}\bm{W}_{2}(0)\\ (\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})\tilde{\bm{V}}^{T}\bm{W}_{1}^{T}( 0)-(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\tilde{\bm{U}}^{T}\bm{W}_{2}(0 )\end{pmatrix}$ $$

$$ $=\frac{1}{\sqrt{2}}\begin{pmatrix}\bm{B}^{T}\\ -\bm{C}^{T}\end{pmatrix}$ (64) $$

where

$$ $\displaystyle{\bf B}$ $\displaystyle={\bf W}_{2}(0)^{T}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})+{\bf W}_{1}(0)\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})\in\mathbb{R}^{N_{h}\times N_{h}}$ (65) $\displaystyle{\bf C}$ $\displaystyle={\bf W}_{2}(0)^{T}\tilde{\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})-{\bf W}_{1}(0)\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})\in\mathbb{R}^{N_{h}\times N_{h}}$ (66) $$

$$ $\bm{O}e^{\bm{\Lambda}t/\tau}=\frac{1}{\sqrt{2}}\begin{pmatrix}\tilde{\bm{V}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})&\tilde{\bm{V}}(\tilde{\bm{G}}+ \tilde{\bm{H}}\tilde{\bm{G}})\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\end{pmatrix}\begin{pmatrix}e^{ \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}&0\\ 0&e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\end{pmatrix}$ $$

$$ $=\frac{1}{\sqrt{2}}\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}&\tilde{\bm{V}}( \tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{ t}{\tau}}\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}&-\tilde{\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{ \bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\end{pmatrix}$ (67) $$

$$ $\bm{O}e^{\bm{\Lambda}t/\tau}\bm{O}^{T}\bm{Q}(0)=\frac{1}{2}\begin{pmatrix} \tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}&\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{ \bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}&-\tilde{\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{ \bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\end{pmatrix}\begin{pmatrix }\bm{B}^{T}\\ -\bm{C}^{T}\end{pmatrix}$ $$

$$ $=\frac{1}{2}\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{ \bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}-\tilde{\bm{V}}( \tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{ t}{\tau}}\bm{C}^{T}\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{B}^{T}+\tilde{\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H} }\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}\end{pmatrix}$ (68) $$

$$ $\displaystyle 2{\bf M}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf M}^{T}{\bf Q} (0)$ $\displaystyle=2\frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\\ \end{bmatrix}\begin{bmatrix}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}&0\\ 0&e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}\\ \end{bmatrix}\frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\\ \end{bmatrix}^{T}\begin{bmatrix}{\bf W}_{1}(0)^{T}\\ \ {\bf W}_{2}(0)\\ \end{bmatrix}$ $\displaystyle=\begin{bmatrix}{\bf\tilde{V}_{\perp}}e^{\bm{\lambda}_{\perp} \frac{t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}&0\\ 0&{\bf\tilde{U}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf\tilde{U}_{ \perp}}^{T}\\ \end{bmatrix}\begin{bmatrix}{\bf W}_{1}(0)^{T}\\ \ {\bf W}_{2}(0)\\ \end{bmatrix}$ $\displaystyle=\begin{bmatrix}{\bf\tilde{V}_{\perp}}e^{\bm{\lambda}_{\perp} \frac{t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\\ {\bf\tilde{U}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf\tilde{U}_{ \perp}}^{T}{\bf W}_{2}(0)\\ \end{bmatrix}$ $$

Putting it together we get the expressions for $\bm{Z_{1}}(t)$ and $\bm{Z_{2}}(t)$

$$ $\displaystyle\left[{\bf O}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O}^{T}+2{\bf M}e^ {\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf M}^{T}\right]{\bf Q}(0)=$ $\displaystyle=\frac{1}{2}\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{ \bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}- \tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{C}^{T}\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{B}^{T}+\tilde{\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H} }\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}\end{ pmatrix}+\begin{bmatrix}{\bf\tilde{V}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{ \tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\\ {\bf\tilde{U}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf\tilde{U}_{ \perp}}^{T}{\bf W}_{2}(0)\\ \end{bmatrix}$ (70) $$

$$ $\displaystyle\bm{Z_{1}}(t)=$ $\displaystyle\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm {G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}-\frac{1}{2}\tilde{ \bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda }}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{V}_{\perp}}e^{\bm{\lambda_{\perp}}\frac {t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}$ (71) $\displaystyle\bm{Z_{2}}(t)=$ $\displaystyle\frac{1}{2}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm {G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}+\frac{1}{2}\tilde{ \bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda }}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{U}_{\perp}}e^{\bm{\lambda_{\perp}}\frac {t}{\tau}}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)$ (72) $$

We now compute the terms inside the inverse

$$ $\displaystyle{\bf Q}(0)^{T}{\bf M}(e^{\bm{\lambda}_{\perp}\frac{t}{\tau}})\bm{ \lambda}_{\perp}^{-1}{\bf M}^{T}{\bf Q}(0)$ $\displaystyle=\begin{bmatrix}{\bf W}_{1}(0)&{\bf W}_{2}(0)^{T}\end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\\ \end{bmatrix}\begin{bmatrix}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}&0\\ 0&e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}\\ \end{bmatrix}\begin{bmatrix}\bm{\lambda}_{\perp}&0\\ 0&{\lambda_{\perp}}\\ \end{bmatrix}^{-1}\frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\\ \end{bmatrix}^{T}\begin{bmatrix}{\bf W}_{1}(0)^{T}\\ {\bf W}_{2}(0)\\ \end{bmatrix}$ $\displaystyle=\begin{bmatrix}{\bf W}_{1}(0)&{\bf W}_{2}(0)^{T}\end{bmatrix} \begin{bmatrix}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}\bm{\lambda}_{\perp}^{-1} {\bf\tilde{V}_{\perp}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\\ e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}\bm{\lambda}_{\perp}^{-1}{\bf\tilde{U}_{ \perp}}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)\\ \end{bmatrix}$ $\displaystyle=\begin{bmatrix}\left({\bf W}_{1}(0){\bf\tilde{V}_{\perp}}e^{\bm{ \lambda}_{\perp}\frac{t}{\tau}}\bm{\lambda}_{\perp}^{-1}{\bf\tilde{V}_{\perp}} ^{T}{\bf W}_{1}(0)^{T}+{\bf W}_{2}(0)^{T}{\bf\tilde{U}_{\perp}}e^{\bm{\lambda} _{\perp}\frac{t}{\tau}}\bm{\lambda}_{\perp}^{-1}{\bf\tilde{U}_{\perp}}^{T}{\bf W }_{2}(0)\right)\end{bmatrix}$ (73) $$

$$ $\displaystyle{\bf Q}(0)^{T}{\bf M}\bm{\lambda}_{\perp}^{-1}{\bf M}^{T}{\bf Q}(0)$ $\displaystyle=2\begin{bmatrix}{\bf W}_{1}(0)&{\bf W}_{2}(0)^{T}\end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\\ \end{bmatrix}\begin{bmatrix}{\lambda_{\perp}}&0\\ 0&{\lambda_{\perp}}\\ \end{bmatrix}^{-1}\frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\\ \end{bmatrix}^{T}\begin{bmatrix}{\bf W}_{1}(0)^{T}\\ \ {\bf W}_{2}(0)\\ \end{bmatrix}$ $\displaystyle=$ $\displaystyle\begin{bmatrix}{\bf W}_{1}(0)&{\bf W}_{2}(0)^{T}\end{bmatrix} \begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\\ \end{bmatrix}\begin{bmatrix}\bm{\lambda}_{\perp}^{-1}{\bf\tilde{V}_{\perp}}{ \bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\\ \bm{\lambda}_{\perp}^{-1}{\bf\tilde{U}_{\perp}}{\bf\tilde{U}_{\perp}}^{T}{\bf W }_{2}(0)\\ \end{bmatrix}$ $\displaystyle=\begin{bmatrix}{\bf W}_{1}(0){\bf\tilde{V}_{\perp}}\bm{\lambda}_ {\perp}^{-1}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}+{\bf W}_{2}(0)^{T}{ \bf\tilde{U}_{\perp}}\bm{\lambda}_{\perp}^{-1}{\bf\tilde{U}_{\perp}}^{T}{\bf W }_{2}(0)\end{bmatrix}$ (74) $$

Now

$$ $\displaystyle\frac{1}{2}{\bf Q}(0)^{T}{\bf O}\left(e^{2{\bf\Lambda}\frac{t}{ \tau}}-{\bf I}\right){\bf\Lambda}^{-1}{\bf O}^{T}$ $\displaystyle=\frac{1}{4}\left[\bm{B}-\bm{C}\right]\left(e^{\bm{\Lambda}\frac{ t}{\tau}}-\mathbf{I}\right)\bm{\Lambda}^{-1}\begin{pmatrix}\bm{B}^{T}\\ -\bm{C}^{T}\end{pmatrix}$ $\displaystyle=\frac{1}{4}\left(\bm{B}\left(e^{2\tilde{\bm{S}_{\lambda}}\frac{t }{\tau}}-\mathbf{I}\right)(\tilde{\bm{S}_{\lambda}})^{-1}\bm{B}^{T}-\bm{C} \left(e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}\right)(\tilde{ \bm{S}_{\lambda}})^{-1}\bm{C}^{T}\right)$ (75) $$

Putting it all together

$$ $\displaystyle\bm{A}(t)=$ $\displaystyle\mathbf{I}+\bm{B}\left(\frac{e^{2\tilde{\bm{S}_{\lambda}}\frac{t} {\tau}}-\mathbf{I}}{4\tilde{\bm{S}_{\lambda}}}\right)\bm{B}^{T}-\bm{C}\left( \frac{e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}}{4\tilde{\bm{S}_ {\lambda}}}\right)\bm{C}^{T}+{\bf W}_{2}(0)^{T}{\bf\tilde{U}_{\perp}}\left( \frac{e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}-\mathbf{I}}{\bm{\lambda}_{\perp}} \right){\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)$ $\displaystyle+{\bf W}_{1}(0){\bf\tilde{V}_{\perp}}\left(\frac{e^{\bm{\lambda}_ {\perp}\frac{t}{\tau}}-\mathbf{I}}{\bm{\lambda}_{\perp}}\right){\bf\tilde{V}_{ \perp}}^{T}{\bf W}_{1}(0)^{T}$ (76) $$

So, final form:

$$ $\displaystyle{\bf Q}{\bf Q}^{T}(t)=$ $\displaystyle\left[\begin{pmatrix}\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^ {T}-\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{- \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{V}_{\perp}}e^{\bm {\lambda}_{\perp}\frac{t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\\ \frac{1}{2}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{ \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}+\frac{1}{2}\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{ t}{\tau}}\bm{C}^{T}+{\bf\tilde{U}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{\tau }}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)\end{pmatrix}\right]$ $\displaystyle\left[\mathbf{I}+\frac{1}{4}\left(\bm{B}\left(\frac{e^{2\tilde{ \bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}}{\tilde{\bm{S}_{\lambda}}}\right) \bm{B}^{T}-\bm{C}\left(\frac{e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}- \mathbf{I}}{\tilde{\bm{S}_{\lambda}}}\right)\bm{C}^{T}\right)\right.$ $\displaystyle+$ $\displaystyle{\bf W}_{2}(0)^{T}{\bf\tilde{U}_{\perp}}\left(\frac{e^{\bm{ \lambda}_{\perp}\frac{t}{\tau}}-\mathbf{I}}{\bm{\lambda}_{\perp}}\right){\bf \tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)\left.+{\bf W}_{1}(0){\bf\tilde{V}_{\perp} }\left(\frac{e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}-\mathbf{I}}{\bm{\lambda}_{ \perp}}\right){\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\right]^{-1}$ $\displaystyle\left[\begin{pmatrix}\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^ {T}-\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{- \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{V}_{\perp}}e^{\bm {\lambda}_{\perp}\frac{t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\\ \frac{1}{2}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{ \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}+\frac{1}{2}\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{ t}{\tau}}\bm{C}^{T}+{\bf\tilde{U}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{\tau }}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)\end{pmatrix}\right]^{T}$ $$

∎

###### Theorem B.4 .

###### Proof.

### B.5 Stable solution Unequal-Input-Output

###### Theorem B.5 .

Given the assumptions of Theorem 4.3 further assuming that $\bm{B}$ is invertible and defining $e^{{\lambda_{\perp}}\frac{t}{\tau}}=\mathrm{sgn}(N_{o}-N_{i})\frac{\lambda}{2}$, the temporal evolution of ${\bf Q}{\bf Q}^{T}$ is described as follows:

$$ $\displaystyle{\bf Q}{\bf Q}^{T}(t)=\bm{Z}\left[e^{-\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}}\bm{B}^{-1}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau }}\right.$ (78) $\displaystyle+\left(\frac{\mathbf{I}-e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{ \tau}}}{4\tilde{\bm{S}_{\lambda}}}\right)-e^{-\tilde{\bm{S}_{\lambda}}\frac{t} {\tau}}\bm{B}^{-1}\bm{C}\left(\frac{e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau} }-\mathbf{I}}{4\tilde{\bm{S}_{\lambda}}}\right)\bm{C}^{T}\bm{B}^{-T}e^{-\tilde {\bm{S}_{\lambda}}\frac{t}{\tau}}$ $\displaystyle-e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}{\bf B}^{-1}{\bf W}_{ 2}(0)^{T}{\bf\tilde{U}_{\perp}}\bm{\lambda}_{\perp}^{-1}{\bf\tilde{U}_{\perp}} ^{T}{\bf W}_{2}(0){\bf B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}$ $\displaystyle e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}e^{\frac{\lambda_{ \perp}}{2}\frac{t}{\tau}}{\bf B}^{-1}{\bf W}_{2}(0)^{T}{\bf\tilde{U}_{\perp}} \bm{\lambda}_{\perp}^{-1}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0){\bf B}^{-T}e ^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}$ $\displaystyle+e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}e^{\frac{\lambda}{2} \frac{t}{\tau}}{\bf B}^{-1}{\bf W}_{1}(0){\bf\tilde{V}_{\perp}}\bm{\lambda}_{ \perp}^{-1}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}{\bf B}^{-T}e^{-\tilde{ \bm{S}_{\lambda}}\frac{t}{\tau}}$ $\displaystyle-$ $\displaystyle\left.e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}{\bf B}^{-1}{\bf W }_{1}(0){\bf\tilde{V}_{\perp}}\bm{\lambda}_{\perp}^{-1}{\bf\tilde{V}_{\perp}}^ {T}{\bf W}_{1}(0)^{T}{\bf B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}} \right]^{-1}\bm{Z}^{T}$ $$

$$ $\displaystyle\bm{Z}=\begin{pmatrix}\frac{1}{2}\tilde{\bm{V}}\left[(\tilde{\bm{ G}}-\tilde{\bm{H}}\tilde{\bm{G}})-(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}} )e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}\bm{B}^{-T}e^{-\tilde{ \bm{S}_{\lambda}}\frac{t}{\tau}}\right]+{\bf\tilde{V}_{\perp}}{\bf\tilde{V}_{ \perp}}^{T}{\bf W}_{1}(0)\bm{B}^{-T}e^{{\lambda_{\perp}}\frac{t}{\tau}}e^{- \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\\ \frac{1}{2}\tilde{\bm{U}}\left[(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})+( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{ t}{\tau}}\bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}} \right]+{\bf\tilde{U}_{\perp}}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)^{T}\bm{ B}^{-T}e^{{\lambda_{\perp}}\frac{t}{\tau}}e^{-\tilde{\bm{S}_{\lambda}}\frac{t} {\tau}}\end{pmatrix}$ $$

###### Proof.

We start from

$$ $\displaystyle{\bf Q}{\bf Q}^{T}(t)=$ $\displaystyle\left[\begin{pmatrix}\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^ {T}-\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{- \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{V}_{\perp}}e^{\bm {\lambda}_{\perp}\frac{t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\\ \frac{1}{2}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{ \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}+\frac{1}{2}\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{ t}{\tau}}\bm{C}^{T}+{\bf\tilde{U}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{\tau }}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)\end{pmatrix}\right]$ $\displaystyle\left[\mathbf{I}+\frac{1}{4}\left(\bm{B}\left(\frac{e^{2\tilde{ \bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}}{\tilde{\bm{S}_{\lambda}}}\right) \bm{B}^{T}-\bm{C}\left(\frac{e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}- \mathbf{I}}{\tilde{\bm{S}_{\lambda}}}\right)\bm{C}^{T}\right)\right.$ $\displaystyle+$ $\displaystyle{\bf W}_{2}(0)^{T}{\bf\tilde{U}_{\perp}}\left(\frac{e^{\bm{ \lambda}_{\perp}\frac{t}{\tau}}-\mathbf{I}}{\bm{\lambda}_{\perp}}\right){\bf \tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)\left.+{\bf W}_{1}(0){\bf\tilde{V}_{\perp} }\left(\frac{e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}-\mathbf{I}}{\bm{\lambda}_{ \perp}}\right){\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\right]^{-1}$ $\displaystyle\left[\begin{pmatrix}\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^ {T}-\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{- \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}+{\bf\tilde{V}_{\perp}}e^{\bm {\lambda}_{\perp}\frac{t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}\\ \frac{1}{2}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{ \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{T}+\frac{1}{2}\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{ t}{\tau}}\bm{C}^{T}+{\bf\tilde{U}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{\tau }}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)\end{pmatrix}\right]^{T}$ $$

We extract $\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}$ from all terms as exemplified bellow

$$ $\bm{O}e^{\bm{\Lambda}t/\tau}\bm{O}^{T}\bm{Q}(0)=\frac{1}{2}\begin{pmatrix} \tilde{\bm{V}}\left[(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})-(\tilde{\bm{ G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}} \bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\right]\\ \tilde{\bm{U}}\left[(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})+(\tilde{\bm{ G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}} \bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\right]\end{ pmatrix}\bm{B}^{T}e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}$ (81) $$

and rewrite the dynamis as

$$ $\displaystyle{\bf Q}{\bf Q}^{T}(t)=$ $\displaystyle\left[\begin{pmatrix}\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})-\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{ \bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}\bm {B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}+{\bf\tilde{V}_{\perp}}e^{ \bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T }\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\\ \frac{1}{2}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})+\frac{1 }{2}\tilde{\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{ S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}}+{\bf\tilde{U}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{ \bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}}\end{pmatrix}\right]$ $\displaystyle\left[e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{-1}\bm{B }^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}+\frac{1}{4}\left(\left(\frac {\mathbf{I}-e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}}{\tilde{\bm{S}_{ \lambda}}}\right)-e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{-1}\bm{C} \left(\frac{e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}}{\tilde{ \bm{S}_{\lambda}}}\right)\bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}}\right)\right.$ $\displaystyle+$ $\displaystyle e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{-1}{\bf W}_{2 }(0)^{T}{\bf\tilde{U}_{\perp}}\left(\frac{e^{\bm{\lambda}_{\perp}\frac{t}{\tau }}-\mathbf{I}}{\bm{\lambda}_{\perp}}\right){\bf\tilde{U}_{\perp}}^{T}{\bf W}_{ 2}(0)\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}$ $\displaystyle\left.+e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{-1}{\bf W }_{1}(0){\bf\tilde{V}_{\perp}}\left(\frac{e^{\bm{\lambda}_{\perp}\frac{t}{\tau }}-\mathbf{I}}{\bm{\lambda}_{\perp}}\right){\bf\tilde{V}_{\perp}}^{T}{\bf W}_{ 1}(0)^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\right]^{-1}$ $\displaystyle\left[\begin{pmatrix}\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})-\frac{1}{2}\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{ \bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}\bm {B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}+{\bf\tilde{V}_{\perp}}e^{ \bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T }\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\\ \frac{1}{2}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})+\frac{1 }{2}\tilde{\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{ S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}}+{\bf\tilde{U}_{\perp}}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{ \bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}}\end{pmatrix}\right]^{T}$ $$

$$ $\displaystyle{\bf Q}{\bf Q}^{T}(t)=$ $\displaystyle\begin{pmatrix}\frac{1}{2}\tilde{\bm{V}}\left[(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})-(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{ -\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S} _{\lambda}}\frac{t}{\tau}}\right]+{\bf\tilde{V}_{\perp}}{\bf\tilde{V}_{\perp}} ^{T}{\bf W}_{1}(0)\bm{B}^{-T}e^{{\lambda_{\perp}}\frac{t}{\tau}}e^{-\tilde{\bm {S}_{\lambda}}\frac{t}{\tau}}\\ \frac{1}{2}\tilde{\bm{U}}\left[(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})+( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{ t}{\tau}}\bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}} \right]+{\bf\tilde{U}_{\perp}}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)^{T}\bm{ B}^{-T}e^{{\lambda_{\perp}}\frac{t}{\tau}}e^{-\tilde{\bm{S}_{\lambda}}\frac{t} {\tau}}\end{pmatrix}$ $\displaystyle\left[e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{B}^{-1}\bm{B }^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\right.$ $\displaystyle+\left(\frac{\mathbf{I}-e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{ \tau}}}{4\tilde{\bm{S}_{\lambda}}}\right)-e^{-\tilde{\bm{S}_{\lambda}}\frac{t} {\tau}}\bm{B}^{-1}\bm{C}\left(\frac{e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau} }-\mathbf{I}}{4\tilde{\bm{S}_{\lambda}}}\right)\bm{C}^{T}\bm{B}^{-T}e^{-\tilde {\bm{S}_{\lambda}}\frac{t}{\tau}}$ $\displaystyle-e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}{\bf B}^{-1}{\bf W}_{ 2}(0)^{T}{\bf\tilde{U}_{\perp}}\bm{\lambda}_{\perp}^{-1}{\bf\tilde{U}_{\perp}} ^{T}{\bf W}_{2}(0){\bf B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}$ $\displaystyle e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}e^{\frac{\lambda_{ \perp}}{2}\frac{t}{\tau}}{\bf B}^{-1}{\bf W}_{2}(0)^{T}{\bf\tilde{U}_{\perp}} \bm{\lambda}_{\perp}^{-1}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0){\bf B}^{-T}e ^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}$ $\displaystyle+e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}e^{\frac{\lambda}{2} \frac{t}{\tau}}{\bf B}^{-1}{\bf W}_{1}(0){\bf\tilde{V}_{\perp}}\bm{\lambda}_{ \perp}^{-1}{\bf\tilde{V}_{\perp}}^{T}{\bf W}_{1}(0)^{T}{\bf B}^{-T}e^{-\tilde{ \bm{S}_{\lambda}}\frac{t}{\tau}}$ $\displaystyle-\left.e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}{\bf B}^{-1}{ \bf W}_{1}(0){\bf\tilde{V}_{\perp}}\bm{\lambda}_{\perp}^{-1}{\bf\tilde{V}_{ \perp}}^{T}{\bf W}_{1}(0)^{T}{\bf B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{ \tau}}\right]^{-1}$ $\displaystyle\begin{pmatrix}\tilde{\bm{V}}\left[(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})-(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S} _{\lambda}}\frac{t}{\tau}}\bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}}\right]+{\bf\tilde{V}_{\perp}}{\bf\tilde{V}_{\perp}}^{T}{\bf W} _{1}(0)\bm{B}^{-T}e^{{\lambda_{\perp}}\frac{t}{\tau}}e^{-\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\\ \tilde{\bm{U}}\left[(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})+(\tilde{\bm{ G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}} \bm{C}^{T}\bm{B}^{-T}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\right]+{\bf \tilde{U}_{\perp}}{\bf\tilde{U}_{\perp}}^{T}{\bf W}_{2}(0)^{T}\bm{B}^{-T}e^{{ \lambda_{\perp}}\frac{t}{\tau}}e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\end {pmatrix}^{T}$ $$

where $e^{{\lambda_{\perp}}\frac{t}{\tau}}=\mathrm{sgn}(N_{o}-N_{i})\frac{\lambda}{2}$ is a scalar
∎

###### Theorem B.5 .

###### Proof.

#### B.5.1 Proof Exact learning dynamics with prior knowledge unequal dimension

We follow a similar derivation presented in Braun et al. () and start with the following equation

$$ $\displaystyle{\bf Q}{\bf Q}^{T}(t)=$ $\displaystyle\underbrace{\left[{\bf O}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O}^{T }+2{\bf M}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf M}^{T}\right]{\bf Q}(0)}_ {{\bf L}}$ $\displaystyle\underbrace{\left[{\bf I}+\frac{1}{2}{\bf Q}(0)^{T}\left({\bf O} \left(e^{2{\bf\Lambda}\frac{t}{\tau}}-{\bf I}\right){\bf\Lambda}^{-1}{\bf O}^{ T}+{\bf M}(e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}-{\bf I})\bm{\lambda}_{\perp} ^{-1}{\bf M}^{T}\right){\bf Q}(0)\right]^{-1}}_{{\bf C}^{-1}}$ (84) $\displaystyle\ \underbrace{{\bf Q}(0)^{T}\left[{\bf O}e^{{\bf\Lambda}\frac{t}{ \tau}}{\bf O}^{T}+2{\bf M}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf M}^{T} \right]}_{{\bf R}}$ $\displaystyle=$ $\displaystyle{\bf L}{\bf C}^{-1}{\bf R},$ (85) $$

Substituting our solution into the matrix Riccati equation then yields

$$ $\displaystyle\tau\frac{d}{dt}{\bf Q}{\bf Q}^{T}$ $\displaystyle={\bf F}{\bf Q}{\bf Q}^{T}+{\bf Q}{\bf Q}^{T}{\bf F}-({\bf Q}{\bf Q }^{T})^{2}$ (86) $\displaystyle\Rightarrow\tau\frac{d}{dt}{\bf L}{\bf C}^{-1}{\bf R}$ $\displaystyle\stackrel{{\scriptstyle?}}{{=}}{\bf F}{\bf L}{\bf C}^{-1}{\bf R}+ {\bf L}{\bf C}^{-1}{\bf R}{\bf F}-{\bf L}{\bf C}^{-1}{\bf R}{\bf L}{\bf C}^{-1 }{\bf R}\text{.}$ (87) $$

Using the chain rule $\partial({\bf A}{\bf B})=(\partial{\bf A}){\bf B}+{\bf A}(\partial{\bf B})$ and the identities

$$ $\frac{d}{dt}({\bf A}^{-1})={\bf A}^{-1}(\frac{d}{dt}{\bf A}){\bf A}^{-1}\qquad \text{and}\qquad\frac{d}{dt}(e^{t{\bf A}})={\bf A}e^{t{\bf A}}=e^{t{\bf A}}{ \bf A}$ (88) $$

$$ $\displaystyle\tau\frac{d}{dt}{\bf Q}{\bf Q}^{T}$ $\displaystyle=\tau\frac{d}{dt}{\bf L}{\bf C}^{-1}{\bf R}$ (89) $\displaystyle=\tau\left(\frac{d}{dt}{\bf L}\right){\bf C}^{-1}{\bf R}+\tau{\bf L }\left(\frac{d}{dt}C^{-1}{\bf R}\right)$ (90) $\displaystyle=\tau\left(\frac{d}{dt}{\bf L}\right){\bf C}^{-1}{\bf R}+\tau{\bf L }{\bf C}^{-1}\left(\frac{d}{dt}{\bf R}\right)+\tau{\bf L}\left(\frac{d}{dt}{ \bf C}^{-1}\right){\bf R}\text{,}$ (91) $$

Next, we note that

$$ $\mathbf{O}=\frac{1}{\sqrt{2}}\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})&\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\end{pmatrix}^{T}$ (92) $$

$$ $\displaystyle{\bf O}^{T}{\bf O}$ $\displaystyle=\frac{1}{\sqrt{2}}\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})&\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\end{pmatrix}^{T}\frac{1}{\sqrt{2} }\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})& \tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\end{pmatrix}$ (93) $\displaystyle=\mathbf{I}$ (94) $$

$$ $\displaystyle{\bf O}^{T}{\bf M}$ $\displaystyle=\frac{1}{\sqrt{2}}\begin{bmatrix}\tilde{\bm{V}}(\tilde{\bm{G}}- \tilde{\bm{H}}\tilde{\bm{G}})&\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\end{bmatrix}\frac{1}{\sqrt{2}} \begin{bmatrix}{\bf\tilde{V}_{\perp}}\\ {\bf\tilde{U}_{\perp}}\end{bmatrix}$ (95) $\displaystyle=\frac{1}{2}\begin{bmatrix}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{ \bm{G}})^{T}{\bf\tilde{V}}^{T}{\bf\tilde{V}_{\perp}}+(\tilde{\bm{G}}+\tilde{ \bm{H}}\tilde{\bm{G}})^{T}{\bf\tilde{U}}^{T}{\bf\tilde{U}_{\perp}}\\ (\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})^{T}{\bf\tilde{V}}^{T}{\bf\tilde{ V}_{\perp}}-(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})^{T}{\bf\tilde{U}}^{T }{\bf\tilde{U}_{\perp}}\end{bmatrix}$ (96) $\displaystyle={\bf 0}$ (97) $$

and

$$ $\displaystyle{\bf M}^{T}{\bf O}$ $\displaystyle=\frac{1}{\sqrt{2}}\begin{bmatrix}{\bf\tilde{V}_{\perp}}^{T}&{\bf \tilde{U}_{\perp}}^{T}\end{bmatrix}\begin{pmatrix}\tilde{\bm{V}}(\tilde{\bm{G} }-\tilde{\bm{H}}\tilde{\bm{G}})&\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})\\ \tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})&-\tilde{\bm{U}}( \tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\end{pmatrix}$ (98) $\displaystyle=\frac{1}{2}\begin{bmatrix}{\bf\tilde{V}_{\perp}}^{T}{\bf\tilde{V }}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})+{\bf\tilde{U}_{\perp}}^{T}{\bf \tilde{U}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})\\ {\bf\tilde{V}_{\perp}}^{T}{\bf\tilde{V}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{ \bm{G}})-{\bf\tilde{U}_{\perp}}^{T}{\bf\tilde{U}}(\tilde{\bm{G}}-\tilde{\bm{H} }\tilde{\bm{G}})\end{bmatrix}$ (99) $\displaystyle={\bf 0}.$ (100) $$

we get

$$ $\displaystyle\tau\frac{d}{dt}{\bf Q}{\bf Q}^{T}$ $\displaystyle=\tau\frac{d}{dt}\left({\bf L}{\bf C}^{-1}{\bf R}\right)$ (101) $\displaystyle=\tau\left(\frac{d}{dt}{\bf L}\right){\bf C}^{-1}{\bf R}+\tau{\bf L }\left(\frac{d}{dt}C^{-1}{\bf R}\right)$ (102) $\displaystyle=\tau\left(\frac{d}{dt}{\bf L}\right){\bf C}^{-1}{\bf R}+\tau{\bf L }{\bf C}^{-1}\left(\frac{d}{dt}{\bf R}\right)+\tau{\bf L}\left(\frac{d}{dt}{ \bf C}^{-1}\right){\bf R}\text{,}$ (103) $$

with

$$ $\displaystyle\tau\left(\frac{d}{dt}{\bf L}\right){\bf C}^{-1}{\bf R}$ $\displaystyle=\tau\left({\bf O}\frac{1}{\tau}{\bf\Lambda}e^{{\bf\Lambda}\frac{ t}{\tau}}{\bf O}^{T}+2{\bf M}\frac{\lambda_{\perp}\mathbf{I}}{2\tau}e^{\bm{ \lambda}_{\perp}\frac{t}{\tau}}{\bf M}^{T}\right){\bf Q}(0){\bf C}^{-1}{\bf R}$ (104) $\displaystyle=\left({\bf O}{\bf\Lambda}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O}^{ T}+{\bf M}\lambda_{\perp}\mathbf{I}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf M }^{T}\right){\bf Q}(0){\bf C}^{-1}{\bf R}$ (105) $\displaystyle=({\bf O}\lambda_{\perp}{\bf O}^{T}+2{\bf M}\bm{\lambda}_{\perp}{ \bf M}^{T})\left({\bf O}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O}^{T}+2{\bf M}e^{ \bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf M}^{T}\right){\bf Q}(0){\bf C}^{-1}{ \bf R}$ (106) $\displaystyle={\bf F}{\bf L}{\bf C}^{-1}{\bf R},$ (107) $$

$$ $\displaystyle\tau{\bf L}{\bf C}^{-1}\left(\frac{d}{dt}{\bf R}\right)$ $\displaystyle=\tau{\bf L}{\bf C}^{-1}{\bf Q}(0)^{T}\left({\bf O}\frac{1}{\tau} e^{{\bf\Lambda}\frac{t}{\tau}}{\bf\Lambda}{\bf O}^{T}+2{\bf M}e^{\bm{\lambda}_ {\perp}\frac{t}{\tau}}\frac{\lambda_{\perp}\mathbf{I}}{2\tau}{\bf M}^{T}\right)$ (108) $\displaystyle={\bf L}{\bf C}^{-1}{\bf Q}(0)^{T}\left({\bf O}\frac{1}{\tau}e^{{ \bf\Lambda}\frac{t}{\tau}}{\bf\Lambda}{\bf O}^{T}+2{\bf M}e^{\bm{\lambda}_{ \perp}\frac{t}{\tau}}\frac{\lambda_{\perp}\mathbf{I}}{2\tau}{\bf M}^{T}\right)$ (109) $\displaystyle={\bf L}{\bf C}^{-1}{\bf R}{\bf F}$ (110) $$

and

$$ $\displaystyle\tau{\bf L}\left(\frac{d}{dt}{\bf C}^{-1}\right){\bf R}$ $\displaystyle=-\tau{\bf L}{\bf C}^{-1}\left(\frac{d}{dt}{\bf C}\right){\bf C}^ {-1}{\bf R}$ (111) $\displaystyle=-{\bf L}{\bf C}^{-1}\bigg{[}\tau\frac{1}{2}{\bf Q}(0)^{T}{\bf O} 2\frac{1}{\tau}e^{2{\bf\Lambda}\frac{t}{\tau}}{\bf\Lambda}{\bf\Lambda}^{-1}{ \bf O}^{T}{\bf Q}(0)$ (112) $\displaystyle\qquad\qquad\qquad+\tau\frac{1}{2}{\bf Q}(0)^{T}4\frac{1}{\tau}{ \bf M}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}\bm{\lambda}_{\perp}\left(\bm{ \lambda}_{\perp}\right)^{-1}{\bf M}^{T}{\bf Q}(0)\bigg{]}{\bf C}^{-1}{\bf R}$ $\displaystyle=-{\bf L}{\bf C}^{-1}\bigg{[}{\bf Q}(0)^{T}{\bf O}e^{2{\bf\Lambda }\frac{t}{\tau}}{\bf O}^{T}{\bf Q}(0)+2{\bf Q}(0)^{T}{\bf M}e^{\bm{\lambda}_{ \perp}\frac{t}{\tau}}{\bf M}^{T}{\bf Q}(0)\bigg{]}{\bf C}^{-1}{\bf R}$ (113) $\displaystyle=-{\bf L}{\bf C}^{-1}\bigg{[}{\bf Q}(0)^{T}{\bf O}e^{{\bf\Lambda} \frac{t}{\tau}}{\bf O}^{T}{\bf O}e^{{\bf\Lambda}\frac{t}{\tau}}{\bf O}^{T}{\bf Q }(0)$ $\displaystyle\qquad\qquad\qquad+2{\bf Q}(0)^{T}{\bf O}e^{{\bf\Lambda}\frac{t}{ \tau}}\underbrace{{\bf O}^{T}{\bf M}}_{\bf 0}e^{\bm{\lambda}_{\perp}\frac{t}{ \tau}}{\bf M}^{T}{\bf Q}(0)$ (114) $\displaystyle\qquad\qquad\qquad+2{\bf Q}(0)^{T}{\bf M}e^{\bm{\lambda}_{\perp} \frac{t}{\tau}}\underbrace{{\bf M}^{T}{\bf O}}_{\bf 0}e^{{\bf\Lambda}\frac{t}{ \tau}}{\bf O}^{T}{\bf Q}(0)$ $\displaystyle\qquad\qquad\qquad+4{\bf Q}(0)^{T}{\bf M}e^{\bm{\lambda}_{\perp} \frac{t}{\tau}}{\bf M}^{T}{\bf M}e^{\bm{\lambda}_{\perp}\frac{t}{\tau}}{\bf M} ^{T}{\bf Q}(0)\bigg{]}{\bf C}^{-1}{\bf R}$ $\displaystyle=-{\bf L}{\bf C}^{-1}{\bf R}{\bf L}{\bf C}^{-1}{\bf R}.$ (115) $$

Finally, substituting equations 104, 108 and 111 into the left hand side of equation 87 proves equality.
$\hfill\square$

## Appendix C Rich-Lazy

### C.1 Dynamics of the Singular Values

###### Theorem C.1 .

Under the assumptions of Theorem 4.3 and with a task-aligned initialization given by $\bm{W}_{1}(0)=\bm{R}\bm{S}_{1}\tilde{\bm{V}}^{T}$ and $\bm{W}_{2}(0)=\tilde{\bm{U}}\bm{S}_{2}\bm{R}^{T}$, where $\bm{R}\in\mathbb{R}^{N_{h}\times N_{h}}$ is an orthonormal matrix, then the network function is given by the expression $\bm{W}_{2}\bm{W}_{1}(t)=\tilde{\bm{U}}\bm{S}(t)\tilde{\bm{V}}^{T}$ where $\bm{S}(t)\in\mathbb{R}^{N_{h}\times N_{h}}$ is a diagonal matrix of singular values with elements $s_{\alpha}(t)$ that evolve according to the equation,

$$ $s_{\alpha}(t)=s_{\alpha}(0)+\gamma_{\alpha}(t;\lambda)\left(\tilde{s}_{\alpha} -s_{\alpha}(0)\right),$ (116) $$

where $\tilde{s}_{\alpha}$ is the $\alpha$ singular value of $\tilde{\bm{S}}$ and $\gamma_{\alpha}(t;\lambda)$ is a $\lambda$-dependent monotonic transition function for each singular value that increases from $\gamma_{\alpha}(0;\lambda)=0$ to $\lim_{t\to\infty}\gamma_{\alpha}(t;\lambda)=1$ defined as

$$ $\gamma_{\alpha}(t;\lambda)=\frac{\tilde{s}_{\lambda,\alpha}s_{\lambda,\alpha} \sinh\left(2\tilde{s}_{\lambda,\alpha}\frac{t}{\tau}\right)+\left(\tilde{s}_{ \alpha}s_{\alpha}+\frac{\lambda^{2}}{4}\right)\cosh\left(2\tilde{s}_{\lambda, \alpha}\frac{t}{\tau}\right)-\left(\tilde{s}_{\alpha}s_{\alpha}+\frac{\lambda^ {2}}{4}\right)}{\tilde{s}_{\lambda,\alpha}s_{\lambda,\alpha}\sinh\left(2\tilde {s}_{\lambda,\alpha}\frac{t}{\tau}\right)+\left(\tilde{s}_{\alpha}s_{\alpha}+ \frac{\lambda^{2}}{4}\right)\cosh\left(2\tilde{s}_{\lambda,\alpha}\frac{t}{ \tau}\right)+\tilde{s}_{\alpha}\left(\tilde{s}_{\alpha}-s_{\alpha}\right)},$ (117) $$

where $\tilde{s}_{\lambda,\alpha}=\sqrt{\tilde{s}_{\alpha}^{2}+\frac{\lambda^{2}}{4}}$, $s_{\lambda,\alpha}=\sqrt{s_{\alpha}(0)^{2}+\frac{\lambda^{2}}{4}}$, and $s_{\alpha}=s_{\alpha}(0)$.
We find that under different limits of $\lambda$, the transition function converges pointwise to the sigmoidal ($\lambda\to 0$) and exponential ($\lambda\to\pm\infty$) transition functions,

$$ $\gamma_{\alpha}(t;\lambda)\to\begin{cases}\frac{e^{2\tilde{s}_{\alpha}\frac{t} {\tau}}-1}{e^{2\tilde{s}_{\alpha}\frac{t}{\tau}}-1+\frac{\tilde{s}_{\alpha}}{s _{\alpha}(0)}}&\text{as }\lambda\to 0,\\ 1-e^{-|\lambda|\frac{t}{\tau}}&\text{as }\lambda\to\pm\infty\\ \end{cases}.$ (118) $$

###### Proof.

According to Theorem 4.3, the network function is given by the equation

$$ $\bm{W}_{2}\bm{W}_{1}(t)=\bm{Z}_{2}(t)\bm{A}^{-1}(t)\bm{Z}_{1}^{T}(t),$ (119) $$

which depends on the variables of the initialization ${\bf B}$ and ${\bf C}$.
Plugging the expressions for a task-aligned initialization $\bm{W}_{1}(0)$ and $\bm{W}_{2}(0)$ into these variables we get the following simplified expressions,

$$ $\displaystyle{\bf B}$ $\displaystyle=\bm{R}\underbrace{\left(\bm{S}_{2}(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})+\bm{S}_{1}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\right) }_{\bm{D}_{B}},$ (120) $\displaystyle{\bf C}$ $\displaystyle=\bm{R}\underbrace{\left(\bm{S}_{2}(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})-\bm{S}_{1}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})\right) }_{\bm{D}_{C}},$ (121) $$

where we define the diagonal matrices $\bm{D}_{B}$ and $\bm{D}_{C}$ for ease of notation.
Using these expressions, we now get the following time-dependent expressions for $\bm{Z}_{2}(t)$, $\bm{A}^{-1}(t)$, and $\bm{Z}_{1}(t)$,

$$ $\displaystyle\bm{Z}_{1}(t)$ $\displaystyle=\frac{1}{2}\tilde{\bm{V}}\left((\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{B}-(\tilde{ \bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau }}\bm{D}_{C}\right)\bm{R}^{T}$ (122) $\displaystyle\bm{Z}_{2}(t)$ $\displaystyle=\frac{1}{2}\tilde{\bm{U}}\left((\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{B}+(\tilde{ \bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau }}\bm{D}_{C}\right)\bm{R}^{T}$ (123) $\displaystyle\bm{A}(t)$ $\displaystyle=\bm{R}\left(\mathbf{I}+\left(\frac{e^{2\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}}-\mathbf{I}}{4\tilde{\bm{S}_{\lambda}}}\right)\bm{D}_{B}^{2}- \left(\frac{e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}}{4\tilde{ \bm{S}_{\lambda}}}\right)\bm{D}_{C}^{2}\right)\bm{R}^{T}$ (124) $$

Plugging these expressions into the expression for the network function, notice that the $\bm{R}$ terms cancel each other resulting in following equation

$$ $\bm{W}_{2}\bm{W}_{1}(t)=\tilde{\bm{U}}\underbrace{\left(\frac{\left((\tilde{ \bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau} }\bm{D}_{B}-(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{D}_{C}\right)\left((\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})e^{\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{B}+(\tilde{ \bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau }}\bm{D}_{C}\right)}{4\mathbf{I}+\left(\frac{e^{2\tilde{\bm{S}_{\lambda}}\frac {t}{\tau}}-\mathbf{I}}{\tilde{\bm{S}_{\lambda}}}\right)\bm{D}_{B}^{2}-\left( \frac{e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}}{\tilde{\bm{S}_{ \lambda}}}\right)\bm{D}_{C}^{2}}\right)}_{\bm{S}(t)}\tilde{\bm{V}}^{T},$ (125) $$

Notice that the middle term is simply a product of diagonal matrices.
We can factor the numerator of this expressions as,

$$ $(\tilde{\bm{G}}^{2}-\tilde{\bm{H}}^{2}\tilde{\bm{G}}^{2})e^{2\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{D}_{B}^{2}+\left((\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})^{2}-(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})^{2}\right) \bm{D}_{B}\bm{D}_{C}-(\tilde{\bm{G}}^{2}-\tilde{\bm{H}}^{2}\tilde{\bm{G}}^{2}) e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{C}^{2}$ (126) $$

We can further factor this expression as,

$$ $\tilde{\bm{G}}^{2}(\mathbf{I}-\tilde{\bm{H}}^{2})\left(e^{2\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{D}_{B}^{2}-e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{ \tau}}\bm{D}_{C}^{2}\right)-4\tilde{\bm{G}}^{2}\tilde{\bm{H}}\bm{D}_{B}\bm{D}_ {C}.$ (127) $$

Putting it all together we find that $\bm{S}(t)$ can be expressed as,

$$ $\bm{S}(t)=\frac{\tilde{\bm{G}}^{2}(\mathbf{I}-\tilde{\bm{H}}^{2})\left(e^{2 \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{B}^{2}-e^{-2\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{D}_{C}^{2}\right)-4\tilde{\bm{G}}^{2}\tilde{\bm{H} }\bm{D}_{B}\bm{D}_{C}}{4\mathbf{I}+\left(\frac{e^{2\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}}-\mathbf{I}}{\tilde{\bm{S}_{\lambda}}}\right)\bm{D}_{B}^{2}- \left(\frac{e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}-\mathbf{I}}{\tilde{ \bm{S}_{\lambda}}}\right)\bm{D}_{C}^{2}}.$ (128) $$

Now using the relationship between $\tilde{\bm{H}}$ and $\tilde{\bm{G}}$ we use the following two identities:

$$ $\tilde{\bm{G}}^{2}(\mathbf{I}-\tilde{\bm{H}}^{2})=\frac{\tilde{\bm{S}}}{\tilde {\bm{S}_{\lambda}}},\qquad 4\tilde{\bm{G}}^{2}\tilde{\bm{H}}=\frac{\lambda}{ \tilde{\bm{S}_{\lambda}}}$ (129) $$

Plugging these identities into the previous expression and multiplying the numerator and denominator by $\tilde{\bm{S}_{\lambda}}$ gives,

$$ $\bm{S}(t)=\frac{\tilde{\bm{S}}\left(e^{2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau} }\bm{D}_{B}^{2}-e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{C}^{2} \right)-\lambda\bm{D}_{B}\bm{D}_{C}}{4\tilde{\bm{S}_{\lambda}}+e^{2\tilde{\bm{ S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{B}^{2}-e^{-2\tilde{\bm{S}_{\lambda}}\frac{ t}{\tau}}\bm{D}_{C}^{2}+\bm{D}_{C}^{2}-\bm{D}_{B}^{2}}.$ (130) $$

Add and subtract $\tilde{\bm{S}}\left(4\tilde{\bm{S}_{\lambda}}+\bm{D}_{C}^{2}-\bm{D}_{B}^{2}\right)$ from the numerator such that

$$ $\bm{S}(t)=\tilde{\bm{S}}-\frac{\tilde{\bm{S}}\left(4\tilde{\bm{S}_{\lambda}}+ \bm{D}_{C}^{2}-\bm{D}_{B}^{2}\right)+\lambda\bm{D}_{B}\bm{D}_{C}}{4\tilde{\bm{ S}_{\lambda}}+e^{2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{B}^{2}-e^{-2 \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{C}^{2}+\bm{D}_{C}^{2}-\bm{D}_{B }^{2}}.$ (131) $$

Using the form of $\bm{D}_{B}$ and $\bm{D}_{C}$ notice the following two identities:

$$ $\bm{D}_{B}\bm{D}_{C}=\frac{\lambda}{\tilde{\bm{S}_{\lambda}}}\left(\tilde{\bm{ S}}-\bm{S}_{2}\bm{S}_{1}\right),\qquad\bm{D}_{C}^{2}-\bm{D}_{B}^{2}=-\frac{4}{ \tilde{\bm{S}_{\lambda}}}\left(\tilde{\bm{S}}\bm{S}_{2}\bm{S}_{1}+\frac{ \lambda^{2}}{4}\mathbf{I}\right)$ (132) $$

From the second identity we can derive a third identity,

$$ $4\tilde{\bm{S}_{\lambda}}+\bm{D}_{C}^{2}-\bm{D}_{B}^{2}=4\frac{\tilde{\bm{S}}} {\tilde{\bm{S}_{\lambda}}}\left(\tilde{\bm{S}}-\bm{S}_{2}\bm{S}_{1}\right)$ (133) $$

Plugging the first and third identities into the numerator for the previous expression gives,

$$ $\bm{S}(t)=\tilde{\bm{S}}-\frac{\frac{\left(4\tilde{\bm{S}}^{2}+\lambda^{2} \mathbf{I}\right)}{\tilde{\bm{S}_{\lambda}}}\left(\tilde{\bm{S}}-\bm{S}_{2}\bm {S}_{1}\right)}{4\tilde{\bm{S}_{\lambda}}+e^{2\tilde{\bm{S}_{\lambda}}\frac{t} {\tau}}\bm{D}_{B}^{2}-e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{C}^{ 2}+\bm{D}_{C}^{2}-\bm{D}_{B}^{2}}.$ (134) $$

Multiply numerator and denominator by $\frac{\tilde{\bm{S}_{\lambda}}}{4}$ and simplify terms gives the expression,

$$ $\bm{S}(t)=\tilde{\bm{S}}-\frac{\tilde{\bm{S}_{\lambda}}^{2}}{\tilde{\bm{S}_{ \lambda}}^{2}+\frac{\tilde{\bm{S}_{\lambda}}}{4}\left(e^{2\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{D}_{B}^{2}-e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{ \tau}}\bm{D}_{C}^{2}\right)-\frac{\tilde{\bm{S}_{\lambda}}}{4}\left(\bm{D}_{B} ^{2}-\bm{D}_{C}^{2}\right)}\left(\tilde{\bm{S}}-\bm{S}_{2}\bm{S}_{1}\right).$ (135) $$

Thus we have found the transition function,

$$ $\gamma(t;\lambda)=\frac{\frac{\tilde{\bm{S}_{\lambda}}}{4}\left(e^{2\tilde{\bm {S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{B}^{2}-e^{-2\tilde{\bm{S}_{\lambda}}\frac {t}{\tau}}\bm{D}_{C}^{2}\right)+\frac{\tilde{\bm{S}_{\lambda}}}{4}\left(\bm{D} _{C}^{2}-\bm{D}_{B}^{2}\right)}{\frac{\tilde{\bm{S}_{\lambda}}}{4}\left(e^{2 \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{B}^{2}-e^{-2\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}}\bm{D}_{C}^{2}\right)+\frac{\tilde{\bm{S}_{\lambda}}}{ 4}\left(4\tilde{\bm{S}_{\lambda}}+\bm{D}_{C}^{2}-\bm{D}_{B}^{2}\right)}.$ (136) $$

We will use our previous identities and the definitions of $\bm{D}_{B}^{2}$ and $\bm{D}_{C}^{2}$ to simplify this expression.
Notice the following identity,

$$ $\frac{\tilde{\bm{S}_{\lambda}}}{4}\left(e^{2\tilde{\bm{S}_{\lambda}}\frac{t}{ \tau}}\bm{D}_{B}^{2}-e^{-2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}\bm{D}_{C}^{2 }\right)=\tilde{\bm{S}_{\lambda}}\bm{S}_{\lambda}\sinh\left(2\tilde{\bm{S}_{ \lambda}}\frac{t}{\tau}\right)+\left(\tilde{\bm{S}}\bm{S}(0)+\frac{\lambda^{2} }{4}\mathbf{I}\right)\cosh\left(2\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}\right)$ (137) $$

Putting it all together we get

$$ $\gamma(t;\lambda)=\frac{\tilde{\bm{S}_{\lambda}}\bm{S}_{\lambda}\sinh\left(2 \tilde{\bm{S}_{\lambda}}\frac{t}{\tau}\right)+\left(\tilde{\bm{S}}\bm{S}(0)+ \frac{\lambda^{2}}{4}\mathbf{I}\right)\cosh\left(2\tilde{\bm{S}_{\lambda}} \frac{t}{\tau}\right)-\left(\tilde{\bm{S}}\bm{S}(0)+\frac{\lambda^{2}}{4} \mathbf{I}\right)}{\tilde{\bm{S}_{\lambda}}\bm{S}_{\lambda}\sinh\left(2\tilde{ \bm{S}_{\lambda}}\frac{t}{\tau}\right)+\left(\tilde{\bm{S}}\bm{S}(0)+\frac{ \lambda^{2}}{4}\mathbf{I}\right)\cosh\left(2\tilde{\bm{S}_{\lambda}}\frac{t}{ \tau}\right)+\tilde{\bm{S}}\left(\tilde{\bm{S}}-\bm{S}(0)\right)}$ (138) $$

We will now show why under certain limits of $\lambda$ this expression simplifies to the sigmoidal and exponential dynamics discussed in the previous section.

Sigmoidal dynamics. When $\lambda=0$, then $\tilde{\bm{S}_{\lambda}}=\tilde{\bm{S}}$ and $\bm{S}_{\lambda}=\bm{S}(0)$.
Notice, that the coefficients for the hyperbolic functions all simplify to $\tilde{\bm{S}}\bm{S}(0)$.
Using the hyperbolic identity $\sinh(x)+\cosh(x)=e^{x}$, we can simplify the expression for the transition function to

$$ $\gamma(t;\lambda)=\frac{\tilde{\bm{S}}\bm{S}(0)e^{2\tilde{\bm{S}}\frac{t}{\tau }}-\tilde{\bm{S}}\bm{S}(0)}{\tilde{\bm{S}}\bm{S}(0)e^{2\tilde{\bm{S}}\frac{t}{ \tau}}-\tilde{\bm{S}}\bm{S}(0)+\tilde{\bm{S}}^{2}}.$ (139) $$

Dividing the numerator and denominator by $\tilde{\bm{S}}\bm{S}(0)$ gives the final expression.

Exponential dynamics. In the limit as $\lambda\to\pm\infty$ the expressions $\tilde{\bm{S}_{\lambda}}\to\frac{|\lambda|}{2}$ and $\bm{S}_{\lambda}\to\frac{|\lambda|}{2}$.
Additionally, in these limits because $\frac{\lambda^{2}}{4}\mathbf{I}\gg\tilde{\bm{S}}\bm{S}(0)$ then $\left(\tilde{\bm{S}}\bm{S}(0)+\frac{\lambda^{2}}{4}\mathbf{I}\right)\to\frac{
\lambda^{2}}{4}\mathbf{I}$.
As a result of these simplifications the coefficients for the hyperbolic functions all simplify to $\frac{\lambda^{2}}{4}\mathbf{I}$.
As a result we can again use the hyperbolic identity $\sinh(x)+\cosh(x)=e^{x}$ to simplify the expression as

$$ $\gamma(t;\lambda)=\frac{\frac{\lambda^{2}}{4}e^{|\lambda|\frac{t}{\tau}}-\frac {\lambda^{2}}{4}\mathbf{I}}{\frac{\lambda^{2}}{4}e^{|\lambda|\frac{t}{\tau}}+ \tilde{\bm{S}}\left(\tilde{\bm{S}}-\bm{S}(0)\right)}.$ (140) $$

Dividing the numerator and denominator by $\frac{\lambda^{2}}{4}$ results in all terms without a coefficient proportional to $\lambda^{2}$ vanishing, which simplifying further gives the final expression.
∎

###### Theorem C.1 .

###### Proof.

### C.2 Dynamics of the representation from the Lazy to the Rich Regime

The *lazy* and *rich* regimes are defined by the dynamics of the NTK of the network. *Lazy* learning occurs when the NTK is constant, *rich* learning occurs when it is not. (Farrell et al. ())
The NTK intuitively measures the movement of the network representations through training. As shown in (Braun et al. ()), in specific experimental setup, we can calculate the NTK of the network in terms of the internal representations in a straightforward way:

$$ $\mathrm{NTK}=\mathbf{I}_{N_{o}}\otimes\mathbf{X}^{T}\mathbf{W}_{1}^{T}\mathbf{ W}_{1}(t)\mathbf{X}+\mathbf{W}_{2}\mathbf{W}_{2}^{T}(t)\otimes\mathbf{X}^{T} \mathbf{X}$ (141) $$

In order to better understand the effect of $\lambda$ on NTK dynamics, we first prove some theorems involving the Singular Values of the *$\lambda$-balanced* weights, and the representations of a *$\lambda$-balanced* network.

#### C.2.1 Lambda-balanced singular value

###### Theorem C.2 .

Under a $\lambda$-Balanced initialization • ‣ 4, if the network function $\bm{W}_{2}\bm{W}_{1}(t)=\bm{U}(t)\bm{S}(t)\bm{V}^{T}(t)$ is full-rank • ‣ 4 and we define $\textstyle\bm{S_{\lambda}}(t)=\sqrt{\bm{S}^{2}(t)+\frac{\lambda^{2}}{4}\mathbf
{I}}$. , then we can recover the parameters $\bm{W}_{2}(t)=\bm{U}(t)\bm{S}_{2}(t)\bm{R}^{T}(t)$, $\bm{W}_{1}(t)=\bm{R}(t)\bm{S}_{1}(t)\bm{V}^{T}(t)$ up to time-dependent orthogonal transformation $\bm{R}$(t) of size $N_{h}\times N_{h}$, where

$$ $\bm{S}_{1}(t)=\begin{pmatrix}\left(\bm{S_{\lambda}}(t)-\frac{\lambda\mathbf{I} }{2}\right)^{\frac{1}{2}}&0_{\max(0,N_{i}-N_{o})}\end{pmatrix}\qquad\bm{S}_{2} (t)=\ \begin{pmatrix}\left(\bm{S_{\lambda}}(t)+\frac{\lambda\mathbf{I}}{2} \right)^{\frac{1}{2}}0_{\max(0,N_{o}-N_{i})}\end{pmatrix}$ (142) $$

###### Proof.

We prove the case $N_{i}\leq N_{o}$ and $N_{h}=min(N_{i},N_{o})$. The proof for $N_{o}\leq N_{i}$ follows the same structure. Let $\bm{U}\bm{S}\bm{V}^{T}=\bm{W}_{2}(t)\bm{W}_{1}(t)$ be the Singular Value Decomposition of the product of the weights at training step $t$. We will use $\bm{W}_{2}=\bm{W}_{2}(t),\bm{W}_{1}=\bm{W}_{1}(t)$ as a shorthand.
By properties of Singular Value Decomposition, we can write $\bm{W}_{2}=\bm{U}\bm{S}_{2}\bm{R}^{T},\bm{W}_{1}=\bm{R}\bm{S}_{1}\bm{V}^{T}$, where $\bm{R}$ is an orthonormal matrix and $\bm{S}_{2},\bm{S}_{1}$ are diagonal (possibly rectangular) matrices.
The Balanced property states that $\bm{W}_{2}^{T}\bm{W}_{2}-\bm{W}_{1}\bm{W}_{1}^{T}=\lambda\mathbf{I}$. We know this holds for any $t$ since this is a conserved quantity in linear networks.
Hence

$$ $\bm{R}\bm{S}_{2}^{T}\bm{S}_{2}\bm{R}^{T}-\bm{R}\bm{S}_{1}\bm{S}_{1}\bm{R}^{T}= \lambda\mathbf{I}$ (143) $$

$$ $\bm{S}_{2}^{T}\bm{S}_{2}-\bm{S}_{1}\bm{S}_{1}=\lambda\mathbf{I}$ (144) $$

The matrices $\bm{S}_{1},\bm{S}_{2},$ have shapes $(N_{h},N_{i})$, $(N_{o},N_{h})$ respectively. We introduce the diagonal matrices $\bm{\hat{S}}_{1}$ of shape $(N_{h},N_{i})$, $\bm{\hat{S}}_{2}$ of shape $(N_{i},N_{h})$ such that the zero matrix has size $(N_{o}-N_{i},N_{h})$ :

$$ $\bm{S}_{1}=\begin{pmatrix}\bm{\hat{S}}_{1}\end{pmatrix},\quad\bm{S}_{2}=\begin {pmatrix}\bm{\hat{S}}_{2}\\ 0\end{pmatrix}$ (145) $$

Hence

$$ $\bm{S}_{2}^{T}\bm{S}_{2}-\bm{S}_{1}\bm{S}_{1}=\lambda\mathbf{I}$ (146) $$

From the equation above and the fact that $\bm{\hat{S}}_{1}\bm{\hat{S}}_{2}=\bm{S}$ we derive that:

$$ $\bm{\hat{S}}_{2}=\left(\frac{\sqrt{\lambda^{2}\mathbf{I}+4\bm{S}^{2}}+\lambda \mathbf{I}}{2}\right)^{\frac{1}{2}},\quad\bm{\hat{S}}_{1}=\left(\frac{\sqrt{ \lambda^{2}\mathbf{I}+4\bm{S}^{2}}-\lambda\mathbf{I}}{2}\right)^{\frac{1}{2}},$ (147) $$

Hence

$$ $\bm{W}_{2}=\bm{U}\begin{pmatrix}\left(\frac{\sqrt{\lambda^{2}\mathbf{I}+4\bm{S }^{2}}+\lambda\mathbf{I}}{2}\right)^{\frac{1}{2}}\\ 0_{\max(0,N_{o}-N_{i})}\end{pmatrix},\bm{R}^{T},\quad\bm{W}_{1}=\bm{R}\begin{ pmatrix}\left(\frac{\sqrt{\lambda^{2}\mathbf{I}+4\bm{S}^{2}}-\lambda\mathbf{I} }{2}\right)^{\frac{1}{2}}&0_{\max(0,N_{i}-N_{o})}\\ \end{pmatrix}\bm{V}^{T}$ (148) $$

∎

###### Theorem C.2 .

###### Proof.

#### C.2.2 Convergence proof

With our solution, ${\bf Q}{\bf Q}^{T}(t)$, which captures the temporal dynamics of the similarity between hidden layer activations, we can analyze the network’s internal representations in relation to the task. This allows us to determine whether the network adopts a *rich* or *lazy* representation, depending on the value of $\lambda$. Consider a $\lambda$-Balanced network training on data $\bm{\Sigma}^{yx}=\tilde{\bm{U}}\tilde{\bm{S}}\tilde{\bm{V}}^{T}$. We assume that the convergence is toward global minima and B is invertible

###### Theorem C.3 .

Under the assumptions of Theorem B.5, the network function converges to ${\bf\tilde{U}}{\bf\tilde{S}}{\bf\tilde{V}}^{T}$ and acquires the internal representation, that is ${\bf W}_{1}^{T}{\bf W}_{1}={\bf\tilde{V}}{\bf\tilde{S}}_{1}^{2}{\bf\tilde{V}}^
{T}$ and ${\bf W}_{2}{\bf W}_{2}^{T}={\bf\tilde{U}}{\bf\tilde{S}}_{2}^{2}{\bf\tilde{U}}^
{T}$

###### Proof.

As training time increases, all terms including a matrix exponential with negative exponent in Equation 78 vanish to zero, as $\bm{S_{\lambda}}=\tilde{\bm{S}_{\lambda}}$ is a diagonal matrix with entries larger zero

As training time increases, all terms in the equations vanish to zero.
Terms in Equation 78 decay as

$$ $\lim_{t\to\infty}e^{-\sqrt{\tilde{\bm{S}}^{2}+\frac{\lambda^{2}\mathbf{I}}{4}} \frac{t}{\tau}}=\textbf{0}\text{.}$ (149) $$

and

$$ $\lim_{t\to\infty}e^{{\lambda_{\perp}}\frac{t}{\tau}}e^{-\sqrt{\tilde{\bm{S}}^{ 2}+\frac{\lambda^{2}}{4}\mathbf{I}}\frac{t}{\tau}}=\textbf{0}\text{.}$ (150) $$

where
$\tilde{\bm{S_{\lambda}}}=\tilde{\bm{S}_{\lambda}}$ is a diagonal matrix with entries larger zero

Therefore, in the temporal limit, eq. 78 reduces to

$$ $\displaystyle\lim_{t\to\infty}{\bf Q}{\bf Q}^{T}(t)$ $\displaystyle=\lim_{t\to\infty}\begin{bmatrix}{\bf W}_{1}^{T}{\bf W}_{1}(t)&{ \bf W}_{1}^{T}{\bf W}_{2}^{T}(t)\\ {\bf W}_{2}{\bf W}_{1}(t)&{\bf W}_{2}^{T}{\bf W}_{2}(t)\end{bmatrix}$ (151) $\displaystyle=\begin{bmatrix}{\bf\tilde{V}}(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})\\ {\bf\tilde{U}}(\tilde{\bm{H}}\tilde{\bm{G}}+\tilde{\bm{G}})\end{bmatrix}\begin {bmatrix}\tilde{\bm{S}_{\lambda}}^{-1}\end{bmatrix}^{-1}\begin{bmatrix}({\bf \tilde{V}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}}))^{T}&({\bf\tilde{U}}( \tilde{\bm{H}}\tilde{\bm{G}}+\tilde{\bm{G}}))^{T}\end{bmatrix}$ (152) $\displaystyle=\begin{bmatrix}{\bf\tilde{V}}(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})\tilde{\bm{S}_{\lambda}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{ \bm{G}})^{T}{\bf\tilde{V}}^{T}&{\bf\tilde{V}}(\tilde{\bm{G}}-\tilde{\bm{H}} \tilde{\bm{G}})\tilde{\bm{S}_{\lambda}}(\tilde{\bm{H}}\tilde{\bm{G}}+\tilde{ \bm{G}})^{T}{\bf\tilde{U}}^{T}\\ {\bf\tilde{U}}(\tilde{\bm{H}}\tilde{\bm{G}}+\tilde{\bm{G}})\tilde{\bm{S}_{ \lambda}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})^{T}{\bf\tilde{V}}^{T}&{ \bf\tilde{U}}(\tilde{\bm{H}}\tilde{\bm{G}}+\tilde{\bm{G}})\tilde{\bm{S}_{ \lambda}}(\tilde{\bm{H}}\tilde{\bm{G}}+\tilde{\bm{G}})^{T}{\bf\tilde{U}}^{T} \end{bmatrix}.$ (153) $$

$$ $\displaystyle(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})\tilde{\bm{S_{ \lambda}}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{G}})$ $\displaystyle=\frac{\bm{S_{\lambda}}(1-\tilde{\bm{H}}^{2})}{1+\tilde{\bm{H}}^{ 2}}=\tilde{\bm{S}}$ (154) $$

$$ $\displaystyle\tilde{\bm{S}_{\lambda}}(\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{ G}})^{2}$ $\displaystyle=\frac{\tilde{\bm{S}_{\lambda}}(1+\tilde{\bm{H}}^{2})}{1+\tilde{ \bm{H}}^{2}}-\frac{\tilde{\bm{S}_{\lambda}}(2\tilde{\bm{H}})}{1+\tilde{\bm{H}} ^{2}}=\frac{\sqrt{4\tilde{\bm{S}}^{2}+\lambda^{2}\mathbf{I}}-\lambda\mathbf{I} }{2}$ (155) $$

$$ $\displaystyle\tilde{\bm{S}_{\lambda}}(\tilde{\bm{G}}+\tilde{\bm{H}}\tilde{\bm{ G}})^{2}$ $\displaystyle=\frac{\tilde{\bm{S}_{\lambda}}(1+\tilde{\bm{H}}^{2})}{1+\tilde{ \bm{H}}^{2}}+\frac{\tilde{\bm{S}_{\lambda}}(2\tilde{\bm{H}})}{1+\tilde{\bm{H}} ^{2}}=\frac{\sqrt{4\tilde{\bm{S}}^{2}+\lambda^{2}\mathbf{I}}+\lambda\mathbf{I} }{2}$ (156) $$

$$ $\displaystyle\lim_{t\to\infty}{\bf Q}{\bf Q}^{T}(t)$ $\displaystyle=\lim_{t\to\infty}\begin{bmatrix}{\bf W}_{1}^{T}{\bf W}_{1}(t)&{ \bf W}_{1}^{T}{\bf W}_{2}^{T}(t)\\ {\bf W}_{2}{\bf W}_{1}(t)&{\bf W}_{2}^{T}{\bf W}_{2}(t)\end{bmatrix}$ (157) $\displaystyle=\begin{bmatrix}{\bf\tilde{V}}\bm{S}_{1}^{2}{\bf\tilde{V}}^{T}&{ \bf\tilde{V}}\tilde{\bm{S}}{\bf\tilde{U}}^{T}\\ {\bf\tilde{U}}\tilde{\bm{S}}{\bf\tilde{V}}^{T}&{\bf\tilde{U}}\bm{S}_{2}^{2}{ \bf\tilde{U}}^{T}\end{bmatrix}.$ (158) $$

∎

###### Theorem C.3 .

###### Proof.

#### C.2.3 Representation in the limit

###### Theorem C.4 .

Under the assumptions of Theorem B.5, training on data $\bm{\Sigma}^{yx}=\tilde{\bm{U}}\tilde{\bm{S}}\tilde{\bm{V}}^{T}$, as $\lambda\to\infty$ the representation tends to

$$ $\bm{W}_{2}\bm{W}_{2}^{T}=\tilde{\bm{U}}\begin{pmatrix}\lambda\mathbf{I}&0_{ \max(0,N_{o}-N_{i})}\\ 0_{\max(0,N_{o}-N_{i})}&0\end{pmatrix}\tilde{\bm{U}}^{T}\quad\bm{W}_{1}^{T}\bm {W}_{1}=\frac{1}{\lambda}\tilde{\bm{V}}\begin{pmatrix}\tilde{\bm{S}}^{2}&0_{ max(0,N_{i}-N_{o})}\\ 0_{\max(0,N_{i}-N_{o})}&0\end{pmatrix}\tilde{\bm{V}}^{T}$ $$

As $\lambda\to-\infty$

$$ $\bm{W}_{2}\bm{W}_{2}^{T}=-\frac{1}{\lambda}\tilde{\bm{U}}\begin{pmatrix}\tilde {\bm{S}}^{2}&0_{\max(0,N_{o}-N_{i})}\\ 0_{\max(0,N_{o}-N_{i})}&0\end{pmatrix}\tilde{\bm{U}}^{T},\quad\bm{W}_{1}^{T} \bm{W}_{1}=\tilde{\bm{V}}\begin{pmatrix}-\lambda\mathbf{I}&0_{\max(0,N_{i}-N_{ o})}\\ 0_{\max(0,N_{i}-N_{o})}&0\end{pmatrix}\tilde{\bm{V}^{T}}$ $$

As $\lambda\to-\infty$

$$ $\bm{W}_{2}\bm{W}_{2}^{T}=-\frac{1}{\lambda}\tilde{\bm{U}}\begin{pmatrix}\tilde {\bm{S}}^{2}&0_{\max(0,N_{o}-N_{i})}\\ 0_{\max(0,N_{o}-N_{i})}&0\end{pmatrix}\tilde{\bm{U}}^{T},\quad\bm{W}_{1}^{T} \bm{W}_{1}=\tilde{\bm{V}}\begin{pmatrix}-\lambda\mathbf{I}&0_{\max(0,N_{i}-N_{ o})}\\ 0_{\max(0,N_{i}-N_{o})}&0\end{pmatrix}\tilde{\bm{V}^{T}}$ $$

###### Proof.

We start from the representation derived in C.3 and using the Taylor expansion of $f(x)=\sqrt{1+x^{2}}$, we compute

$$ $\frac{\sqrt{\lambda^{2}\mathbf{I}+4\tilde{\bm{S}}^{2}}+\lambda\mathbf{I}}{2}= \frac{|\lambda|\sqrt{1+\left(\frac{2\tilde{\bm{S}}}{\lambda}\right)^{2}}+ \lambda\mathbf{I}}{2}$ (159) $$

$$ $\frac{|\lambda|\left(1+\left(\frac{2\tilde{\bm{S}}}{\lambda}\right)^{2}+O( \lambda^{-4})\right)+\lambda\mathbf{I}}{2}=\frac{|\lambda|+\lambda}{2}+\frac{ \tilde{\bm{S}}^{2}}{|\lambda|}+O(\lambda^{-3})$ (160) $$

Hence

$$ $\lim_{\lambda\to\infty}\frac{\sqrt{\lambda^{2}\mathbf{I}+4\tilde{\bm{S}}^{2}}+ \lambda\mathbf{I}}{2}=\lambda\mathbf{I},\quad\lim_{\lambda\to-\infty}\frac{ \sqrt{\lambda^{2}\mathbf{I}+4\tilde{\bm{S}}^{2}}+\lambda\mathbf{I}}{2}=\frac{ \tilde{\bm{S}}^{2}}{|\lambda|}=-\frac{\tilde{\bm{S}}^{2}}{\lambda}$ (161) $$

Similarly,

$$ $\frac{\sqrt{\lambda^{2}\mathbf{I}+4\tilde{\bm{S}}^{2}}-\lambda\mathbf{I}}{2}= \frac{|\lambda|-\lambda}{2}+\frac{\tilde{\bm{S}}^{2}}{|\lambda|}+O(\lambda^{-3})$ (162) $$

$$ $\lim_{\lambda\to\infty}\frac{\sqrt{\lambda^{2}\mathbf{I}+4\tilde{\bm{S}}^{2}}- \lambda\mathbf{I}}{2}=\frac{\tilde{\bm{S}}^{2}}{\lambda},\quad\lim_{\lambda\to -\infty}\frac{\sqrt{\lambda^{2}\mathbf{I}+4\tilde{\bm{S}}^{2}}-\lambda\mathbf{ I}}{2}=\frac{\tilde{\bm{S}}^{2}}{|\lambda|}=-\lambda\mathbf{I}$ (163) $$

Since $\tilde{\bm{U}},\tilde{\bm{V}}$ are independent of $\lambda$:

$$ $\lim_{\lambda\to\pm\infty}\bm{W}_{2}\bm{W}_{2}^{T}=\tilde{\bm{U}}\left(\lim_{ \lambda\to\pm\infty}\bm{S}_{2}\right)\tilde{\bm{U}}^{T}$ (164) $$

$$ $\lim_{\lambda\to\pm\infty}\bm{W}_{1}^{T}\bm{W}_{1}=\tilde{\bm{V}}\left(\lim_{ \lambda\to\pm\infty}\bm{S}_{1}\right)\tilde{\bm{V}}^{T}$ (165) $$

∎

As $|\lambda|\to\infty$, one of the network representations approaches a scaled identity matrix, while the other tends toward zero. Intuitively, this suggests that the representations shift less and less as $|\lambda|$ increases. Next, we demonstrate that the NTK becomes progressively less variable as $|\lambda|$ grows and ultimately converges to zero.

###### Theorem C.4 .

###### Proof.

#### C.2.4 NTK movement

Relationship between $\lambda$ and the NTK of the network

###### Theorem C.5 .

Under the assumptions of Theorem B.5, consider a linear network training on data $\bm{\Sigma}^{yx}=\tilde{\bm{U}}\tilde{\bm{S}}\tilde{\bm{V}}^{T}$. At any arbitrary training time $t\geq 0$, let $\bm{W}_{2}(t)\bm{W}_{1}(t)=\bm{U}^{*}\bm{S}^{*}\bm{V}^{*T}$. Then,

- 1.
For any $\lambda\in\mathbf{R}$:
$\displaystyle\text{NTK}(0)$
$\displaystyle=\mathbf{I}_{N_{o}}\otimes\bm{X}^{T}\bm{V}\begin{pmatrix}\frac{
\sqrt{\lambda^{2}\mathbf{I}+4\bm{S}^{*2}}-\lambda\mathbf{I}}{2}&0\\
0&0\end{pmatrix}\bm{V}^{T}\bm{X}$
(166)
$\displaystyle\quad+\bm{U}\begin{pmatrix}\frac{\sqrt{\lambda^{2}\mathbf{I}+4\bm
{S}^{*2}}+\lambda\mathbf{I}}{2}&0\\
0&0\end{pmatrix}\bm{U}^{T}\otimes\bm{X}^{T}\bm{X}$
$\displaystyle\text{NTK}(t)$
$\displaystyle=\mathbf{I}_{N_{o}}\otimes\bm{X}^{T}\bm{V}^{*}\begin{pmatrix}
\frac{\sqrt{\lambda^{2}\mathbf{I}+4\bm{S}^{*2}}-\lambda\mathbf{I}}{2}&0\\
0&0\end{pmatrix}\bm{V}^{*T}$
(167)
$\displaystyle\quad+\bm{U}^{*}\begin{pmatrix}\frac{\sqrt{\lambda^{2}\mathbf{I}+
4\bm{S}^{*2}}+\lambda\mathbf{I}}{2}&0\\
0&0\end{pmatrix}\bm{U}^{*T}\otimes\bm{X}^{T}\bm{X}$
- 2.
As $\lambda\to\infty$:
$\displaystyle\text{NTK}(t)-\text{NTK}(0)$
$\displaystyle\to\frac{1}{\lambda}\left(\mathbf{I}_{N_{o}}\otimes\bm{X}^{T}\bm{
V}^{*}\tilde{\bm{S}}^{*2}\bm{V}^{*T}\bm{X}-\mathbf{I}_{N_{o}}\otimes\bm{X}^{T}
\bm{V}\tilde{\bm{S}}^{2}\bm{V}^{T}\bm{X}\right)\to 0$
(168)
- 3.
As $\lambda\to-\infty$:
$\displaystyle\text{NTK}(t)-\text{NTK}(0)$
$\displaystyle\to\frac{1}{\lambda}\left(\bm{U}\tilde{\bm{S}}^{2}\bm{U}^{T}
\otimes\bm{X}^{T}\bm{X}-\bm{U}^{*}\tilde{\bm{S}}^{*2}\bm{U}^{*T}\otimes\bm{X}^
{T}\bm{X}\right)\to 0$
(169)

###### Proof .

Follows by substituting the expressions for the network representations in terms of $\lambda$ from (Braun et al. ())’s expression for the NTK of a linear network.
Similarly, follows from substituting the limit expressions for the network representations and the fact that the Kronecker product is linear in both arguments.
∎

The theorem above demonstrates that as $|\lambda|\to\infty$, the NTK of a $\lambda$-Balanced network remains constant. This indicates that the network operates in the *lazy* regime throughout all training steps. This finding is significant as it highlights the impact of weight initialization on learning regimes.

###### Theorem C.5 .

###### Proof .

### C.3 Representation robustness and sensitivity to noise

As derived in (Braun et al., ), the expected mean squared error under additive, independent and identically distributed input noise with mean $\mu=0$ and variance $\sigma_{\bf x}^{2}$ is

$$ $\displaystyle\left\langle\frac{1}{2P}\sum_{i=1}^{P}||{\bf W}_{2}{\bf W}_{1} \left({\bf x}_{\bf x}+\xi_{i}\right)-{\bf y}_{i}||_{2}^{2}\right\rangle_{\xi_{ \bf x}}=\sigma_{\bf x}^{2}||{\bf W}_{2}{\bf W}_{1}||_{F}^{2}+c,$ (170) $$

where $c=\frac{1}{2}\operatorname{Tr}({\bf\tilde{\Sigma}}^{yy})-\frac{1}{2}
\operatorname{Tr}({\bf\tilde{\Sigma}}^{yx}{\bf\tilde{\Sigma}}^{yxT})$ is a noise independent constant that only depends on the statistics of the training data. In Theorem C.3 we show that the network function converges to ${\bf\tilde{U}}{\bf\tilde{S}}{\bf\tilde{V}}^{T}$ and therefore

$$ $\displaystyle\sigma_{\bf x}^{2}||{\bf W}_{2}{\bf W}_{1}||_{F}^{2}$ $\displaystyle=\sigma_{\bf x}^{2}||{\bf\tilde{U}}{\bf\tilde{S}}{\bf\tilde{V}}^{ T}||_{F}^{2}$ (171) $\displaystyle=\sigma_{\bf x}^{2}||{\bf\tilde{S}}||_{F}^{2}$ $\displaystyle=\sigma_{\bf x}^{2}\sum_{i=1}^{N_{h}}{\bf\tilde{S}}_{i}^{2}$ $$

As derived in (Braun et al., ), under the assumption of whitened inputs (Assumption • ‣ 4), in the case of additive parameter noise with $\mu=0$ and variance $\sigma_{\bf W}^{2}$, the expected mean squared error is

$$ $\displaystyle\left\langle\frac{1}{2P}\sum_{i=1}^{P}||\left({\bf W}_{2}+\xi_{{ \bf W}_{2}}\right)\left({\bf W}_{1}+\xi_{{\bf W}_{1}}\right){\bf x}_{i}-{\bf y }_{i}||_{2}^{2}\right\rangle_{\xi_{{\bf W}_{1}},\xi_{{\bf W}_{2}}}$ (172) $\displaystyle=\frac{1}{2}N_{i}\sigma^{2}_{\bf W}||{\bf W}_{2}||_{F}^{2}+\frac{ 1}{2}N_{o}\sigma^{2}_{\bf W}||{\bf W}_{1}||_{F}^{2}+\frac{1}{2}N_{i}N_{h}N_{o} \sigma^{4}+c.$ $$

Using Theorem C.3, we have

$$ $\displaystyle||{\bf W}_{1}||_{F}^{2}$ $\displaystyle=\operatorname{Tr}({\bf W}_{1}^{T}{\bf W}_{1})$ (173) $\displaystyle=\operatorname{Tr}\left(\frac{\sqrt{\lambda^{2}{\bf I}+4{\bf \tilde{S}}^{2}}+\lambda{\bf I}}{2}\right)$ $\displaystyle=\frac{1}{2}\left(\sum_{i=1}^{N_{h}}\sqrt{\lambda^{2}+4{\bf\tilde {S}}_{i}^{2}}+\lambda\right)$ $$

and

$$ $\displaystyle||{\bf W}_{2}||_{F}^{2}$ $\displaystyle=\operatorname{Tr}({\bf W}_{2}{\bf W}_{2}^{T})$ (174) $\displaystyle=\operatorname{Tr}\left(\frac{\sqrt{\lambda^{2}{\bf I}+4{\bf \tilde{S}}^{2}}-\lambda{\bf I}}{2}\right)$ $\displaystyle=\frac{1}{2}\left(\sum_{i=1}^{N_{h}}\sqrt{\lambda^{2}+4{\bf\tilde {S}}_{i}^{2}}-\lambda\right).$ $$

To find the $\lambda$ that minimises the expected loss, we substitute the equations for the norms, take the partial derivative with respect to $\lambda$ and set it to zero

$$ $\displaystyle\frac{\partial\left\langle\mathcal{L}\right\rangle_{\xi_{{\bf W}_ {1}},\xi_{{\bf W}_{2}}}}{\partial\lambda}\stackrel{{\scriptstyle!}}{{=}}0$ (175) $\displaystyle\Leftrightarrow$ $\displaystyle\frac{1}{4}N_{i}\sigma^{2}_{\bf W}\frac{\partial}{\partial\lambda }\Big{(}\sum_{i=1}^{N_{h}}\sqrt{\lambda^{2}+4{\bf\tilde{S}}_{i}^{2}}-\lambda \Big{)}+\frac{1}{4}N_{o}\sigma^{2}_{\bf W}\frac{\partial}{\partial\lambda} \left(\sum_{i=1}^{N_{h}}\sqrt{\lambda^{2}+4{\bf\tilde{S}}_{i}^{2}}+\lambda \right)=0$ $\displaystyle\Leftrightarrow$ $\displaystyle N_{i}\sum_{i=1}^{N_{h}}\frac{\lambda}{\sqrt{\lambda^{2}+4{\bf \tilde{S}}_{i}^{2}}}-N_{i}N_{h}+N_{o}\sum_{i=1}^{N_{h}}\frac{\lambda}{\sqrt{ \lambda^{2}+4{\bf\tilde{S}}_{i}^{2}}}+N_{o}N_{h}=0$ $\displaystyle\Leftrightarrow$ $\displaystyle\sum_{i=1}^{N_{h}}\frac{\lambda}{\sqrt{\lambda^{2}+4{\bf\tilde{S} }_{i}^{2}}}=N_{h}\frac{N_{i}-N_{o}}{N_{i}+N_{o}}.$ $$

It follows, that under the assumption that $N_{i}=N_{o}$, the equation reduces to

$$ $\sum_{i=1}^{N_{h}}\frac{\lambda}{\sqrt{\lambda^{2}+4{\bf\tilde{S}}_{i}^{2}}}=0.$ (176) $$

We note, that the denominator is always positive and therefore, that the left-hand side of the equation is always larger zero for any $\lambda>0$, and smaller than zero for any $\lambda<0$. The euqation is therefore only solved for $\lambda=0$.

### C.4 Effect of the architecture from the lazy to the Rich Regime

###### Theorem C.6 .

Under the conditions of Theorem B.5, when $\lambda_{\perp}>0$, the network enters a regime referred to as the delayed-rich phase. In this phase, the learning rate is determined by two competing exponential factors:

$$ $e^{\lambda_{\perp}\frac{t}{\tau}}e^{-\sqrt{\tilde{\bm{S}}^{2}+\frac{\lambda^{2 }}{4}\mathbf{I}}\frac{t}{\tau}}$ $$

and

$$ $e^{-\sqrt{\tilde{\bm{S}}^{2}+\frac{\lambda^{2}}{4}\mathbf{I}}\frac{t}{\tau}}.$ $$

As $\lambda$ increases, different parts of the network exhibit distinct learning behaviors: some components adapt quickly and converge exponentially with lambda, while others are constrained by the singular values of the network, resulting in slower adaptation.

###### Proof.

The solution to Theorem B.5 is governed by two time-dependent terms:

$$ $e^{-\sqrt{\tilde{\bm{S}}^{2}+\frac{\lambda^{2}\mathbf{I}}{4}}\frac{t}{\tau}} \quad\text{and}\quad e^{\lambda_{\perp}\frac{t}{\tau}}e^{-\sqrt{\tilde{\bm{S}} ^{2}+\frac{\lambda^{2}}{4}\mathbf{I}}\frac{t}{\tau}}.$ $$

The first term exhibits exponential decay with rate $\lambda$, approaching zero as time progresses:

$$ $\lim_{t\to\infty}e^{-\sqrt{\tilde{\bm{S}}^{2}+\frac{\lambda^{2}\mathbf{I}}{4}} \frac{t}{\tau}}=\mathbf{0}.$ $$

The second term also decays, but at a rate governed by the singular values $\tilde{\bm{S}}$, as $\lambda$ tends to infinity:

$$ $\lim_{t\to\infty}e^{\lambda_{\perp}\frac{t}{\tau}}e^{-\sqrt{\tilde{\bm{S}}^{2} +\frac{\lambda^{2}}{4}\mathbf{I}}\frac{t}{\tau}}=\mathbf{0}.$ $$

Since

$$ $\lambda_{\perp}-\sqrt{\tilde{\bm{S}}^{2}+\frac{\lambda^{2}}{4}}\mathbf{I}>0,$ $$

we have

$$ $\lim_{\lambda\to\infty}\left(\lambda_{\perp}-\sqrt{\tilde{\bm{S}}^{2}+\frac{ \lambda^{2}}{4}}\mathbf{I}\right)=\tilde{\bm{S}}.$ $$

Thus, as $\lambda$ increases, the convergence rate slows for certain parts of the network (those governed by larger singular values), while other components continue to learn more quickly. This explains the delay observed in the delayed-rich regime.
∎

###### Theorem C.6 .

###### Proof.

## Appendix D Appendix: Application

### D.1 Appendix: Continual Learning

We build upon the derivation presented in Braun et al. () to incorporate the dynamics of continual learning throughout the entire learning trajectory.
Utilizing the assumption of whitened inputs, the entire batch loss for the $i$th task is

$$ $\displaystyle\mathcal{L}_{i}\left(\mathcal{T}_{j}\right)$ $\displaystyle=\frac{1}{2P}\left\|\mathbf{W}_{2}\mathbf{W}_{1}\mathbf{X}_{i}- \mathbf{Y}_{i}\right\|_{F}^{2}$ $\displaystyle=\frac{1}{2P}\operatorname{Tr}\left(\left(\mathbf{W}_{2}\mathbf{W }_{1}\mathbf{X}_{i}-\mathbf{Y}_{i}\mid\right)\left(\mathbf{W}_{2}\mathbf{W}_{1 }\mathbf{X}_{i}-\mathbf{Y}_{i}\mid\right)^{T}\right)$ $\displaystyle=\frac{1}{2P}\operatorname{Tr}\left(\mathbf{W}_{2}\mathbf{W}_{1} \mathbf{X}_{i}\mathbf{X}_{i}^{T}(\mathbf{W}_{2}\mathbf{W}_{1})^{T}\right)- \frac{1}{P}\operatorname{Tr}\left(\mathbf{W}_{2}\mathbf{W}_{1}\mathbf{X}_{i} \mathbf{Y}_{i}^{T}\right)+\frac{1}{2P}\operatorname{Tr}\left(\mathbf{Y}_{i} \mathbf{Y}_{i}^{T}\right)$ $\displaystyle=\frac{1}{2}\operatorname{Tr}\left(\mathbf{W}_{2}\mathbf{W}_{1}( \mathbf{W}_{2}\mathbf{W}_{1})^{T}\right)-\operatorname{Tr}\left(\mathbf{W}_{2} \mathbf{W}_{1}\tilde{\bm{\Sigma}}_{i}^{yx^{T}}\right)+\frac{1}{2}\operatorname {Tr}\left(\tilde{\bm{\Sigma}}_{i}^{yy}\right)$ $\displaystyle=\frac{1}{2}\operatorname{Tr}\left(\left(\mathbf{W}_{2}\mathbf{W} _{1}-\tilde{\bm{\Sigma}}_{i}^{yx}\right)\left(\mathbf{W}_{2}\mathbf{W}_{1}- \tilde{\bm{\Sigma}}_{i}^{yx}\right)^{T}-\tilde{\bm{\Sigma}}_{i}^{yx}\tilde{\bm {\Sigma}}_{i}^{yx^{T}}\right)+\frac{1}{2}\left(\tilde{\bm{\Sigma}}_{i}^{yy}\right)$ $\displaystyle=\frac{1}{2}\left\|\mathbf{W}_{2}\mathbf{W}_{1}-\tilde{\bm{\Sigma }}_{i}^{yx}\right\|_{F}^{2}\underbrace{-\frac{1}{2}\operatorname{Tr}\left( \tilde{\bm{\Sigma}}_{i}^{yx}\tilde{\bm{\Sigma}}_{i}^{yx^{T}}\right)+\frac{1}{2 }\left(\tilde{\bm{\Sigma}}_{i}^{yy}\right)}_{c}.$ $$

Hence, the extent of forgetting, denoted as $\mathcal{F}$ for task $\mathcal{T}_{i}$ during training on task $\mathcal{T}_{k}$ subsequent to training the network on task $\mathcal{T}_{j}$, specifically, the relative change in loss, is entirely dictated by the similarity structure among tasks.

$$ $\displaystyle\mathcal{F}_{i}\left(\mathcal{T}_{j},\mathcal{T}_{k}\right)$ $\displaystyle=\mathcal{L}_{i}\left(\mathcal{T}_{k}\right)-\mathcal{L}_{i}\left (\mathcal{T}_{j}\right)$ $\displaystyle=\frac{1}{2}\left\|\tilde{\bm{\Sigma}}_{k}^{yx}-\tilde{\bm{\Sigma }}_{i}^{yx}\right\|_{F}^{2}+c-\frac{1}{2}\left\|\mathbf{W}_{2}\mathbf{W}_{1}- \tilde{\bm{\Sigma}}_{i}^{yx}\right\|_{F}^{2}-c$ $\displaystyle=\frac{1}{2}\left(\left\|\tilde{\bm{\Sigma}}_{k}^{yx}-\tilde{\bm{ \Sigma}}_{i}^{yx}\right\|_{F}^{2}-\left\|\mathbf{W}_{2}\mathbf{W}_{1}-\tilde{ \bm{\Sigma}}_{i}^{yx}\right\|_{F}^{2}\right).$ $$

It is important to note that the amount of forgetting is a function of the weight trajectories. Therefore, we have analytical solutions for trajectories of forgetting as well.

Figure: Figure 6: Continual learning. A Top: Network training from small zero-balanced weights across a sequence of tasks (colored lines represent simulations, and black dotted lines represent analytical results). Bottom: Evaluation loss for the tasks in the sequence (dotted lines) while training on the current task (solid lines). As the network optimizes its function on the current task, the loss on previously learned tasks increases.
Refer to caption: extracted/5871880/figures/continual_learning.png

Figure. 6 panel was generated by training a linear network with $N_{i}=5$, $N_{h}=10$, $N_{o}=6$ subsequently on four different random regression tasks with $N=25$. The learning rate was $\eta=0.05$ and the initial weights were small ($\sigma=0.0001$).

### D.2 Appendix: Reversal Learning

As first introduced in Braun et al. (), in the following discussion, we assume that the input and output dimensions are equal. We denote the $i$-th columns of the left and right singular vectors as ${\bf u}_{i}$, ${\bf\tilde{u}}_{i}$, and ${\bf v}_{i}$, ${\bf\tilde{v}}_{i}$, respectively.

Reversal learning occurs when both the task and the initial network function share the same left and right singular vectors, i.e., ${\bf U}={\bf\tilde{U}}$ and ${\bf V}={\bf\tilde{V}}$, with the exception of one or more columns of the left singular vectors, where the direction is reversed: $-{\bf u}_{i}={\bf\tilde{u}}_{i}\text{.}$

It is important to note that if a reversal occurs in the right singular vectors, such that $-{\bf v}_{i}={\bf\tilde{v}}_{i}$, this can be equivalently represented as a reversal in the left singular vectors, as the signs of the right and left singular vectors are interchangeable.

In the reversal learning setting, both $\bm{B}=\bm{S}_{2}\tilde{\bm{U}}^{T}\tilde{\bm{U}}(\tilde{\bm{G}}+\tilde{\bm{H}
}\tilde{\bm{G}})+\bm{S}_{1}\bm{V}^{T}\tilde{\bm{V}}(\tilde{\bm{G}}-\tilde{\bm{
H}}\tilde{\bm{G}})$ and $\bm{C}=\bm{S}_{2}\tilde{\bm{U}}^{T}\tilde{\bm{U}}(\tilde{\bm{G}}-\tilde{\bm{H}
}\tilde{\bm{G}})-\bm{S}_{1}\bm{V}^{T}\tilde{\bm{V}}(\tilde{\bm{G}}+\tilde{\bm{
H}}\tilde{\bm{G}})$ are diagonal matrices.

In the case where lambda is zero, the same argument given in Braun et al. () follows, the diagonal entries of ${\bf C}$ are zero if the singular vectors are aligned and non zero if they are reversed. Similarly, diagonal entries of ${\bf B}$ are non-zero if the singular vectors are aligned and zero if they are reversed. Therefore, in the case of reversal learning, ${\bf B}$ is a diagonal matrix with 0 0 values and thus is not invertible. As a consequence, the learning dynamics cannot be described by Equation 57. However, as ${\bf B}$ and ${\bf C}$ are diagonal matrices, the learning dynamics simplify. Let ${\bf b}_{i}$, ${\bf c}_{i}$, ${\bf s}_{i}$ and ${\bf\tilde{s}}_{i}$ denote the $i$-th diagonal entry of ${\bf B}$, ${\bf C}$, ${\bf S}$ and ${\bf\tilde{S}}$ respectively, then the network dynamics can be rewritten as

$$ $\displaystyle{\bf W}_{2}{\bf W}_{1}(t)=$ $\displaystyle\ \frac{1}{2}{\bf\tilde{U}}\left[(\tilde{\bm{G}}+\tilde{\bm{H}} \tilde{\bm{G}})e^{\tilde{\bm{S}}_{\lambda}\frac{t}{\tau}}{\bf B}^{T}+(\tilde{ \bm{G}}-\tilde{\bm{H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}}_{\lambda}\frac{t}{\tau }}{\bf C}^{T}\right)$ $\displaystyle\quad\Big{[}\bm{S}_{\lambda}^{-1}+\frac{1}{4}{\bf B}\left(e^{2 \tilde{\bm{S}}_{\lambda}\frac{t}{\tau}}-{\bf I}\right){\bf\tilde{S}_{\lambda}} ^{-1}{\bf B}^{T}-\frac{1}{4}{\bf C}\left(e^{-2\tilde{\bm{S}}_{\lambda}\frac{t} {\tau}}-{\bf I}\right){\bf\tilde{S}_{\lambda}}^{-1}{\bf C}^{T}\Big{]}^{-1}$ (177) $\displaystyle\quad\frac{1}{2}\left((\tilde{\bm{G}}-\tilde{\bm{H}}\tilde{\bm{G} })e^{\tilde{\bm{S}}_{\lambda}\frac{t}{\tau}}{\bf B}-(\tilde{\bm{G}}+\tilde{\bm {H}}\tilde{\bm{G}})e^{-\tilde{\bm{S}_{\lambda}}\frac{t}{\tau}}{\bf C}\right){ \bf\tilde{V}}^{T}$ $\displaystyle=$ $\displaystyle\ \sum_{i=1}^{N_{i}}\frac{{\bf b}_{i}^{2}e^{2{\bf\tilde{s}}_{ \lambda i}\frac{t}{\tau}}-{\bf c}_{i}^{2}e^{-2{\bf\tilde{s}}_{\lambda i}\frac{ t}{\tau}}}{4{\bf s_{\lambda}}_{i}^{-1}+{\bf b}_{i}^{2}e^{2{\bf\tilde{s}}_{ \lambda i}\frac{t}{\tau}}{\bf\tilde{s}_{\lambda i}}^{-1}-{\bf b}_{i}^{2}{\bf \tilde{s}_{\lambda i}}^{-1}-{\bf c}_{i}^{2}e^{-2{\bf\tilde{s}}_{\lambda i} \frac{t}{\tau}}{\bf\tilde{s}_{\lambda i}}^{-1}+{\bf c}_{i}^{2}{\bf\tilde{s}_{ \lambda i}}^{-1}}\ {\bf\tilde{u}}_{i}{\bf\tilde{v}}_{i}^{T}$ (178) $\displaystyle=$ $\displaystyle\ \sum_{i=1}^{N_{i}}\frac{{\bf s_{\lambda}}_{i}{\bf b}_{i}^{2}{ \bf\tilde{s}_{\lambda i}}-{\bf s_{\lambda}}_{i}{\bf c}_{i}^{2}{\bf\tilde{s}}_{ i}e^{-4{\bf\tilde{s}}_{i}\frac{t}{\tau}}}{4{\bf\tilde{s}_{\lambda i}}e^{-2{\bf \tilde{s}}_{i}\frac{t}{\tau}}+{\bf s_{\lambda}}_{i}{\bf b}_{i}^{2}\left(1-e^{- 2{\bf\tilde{s}}_{\lambda i}\frac{t}{\tau}}\right)+{\bf s_{\lambda}}_{i}{\bf c} _{i}^{2}\left(e^{-2{\bf\tilde{s}}_{\lambda i}\frac{t}{\tau}}-e^{-4{\bf\tilde{s }}_{\lambda i}\frac{t}{\tau}}\right)}{\bf\tilde{u}}_{i}{\bf\tilde{v}}_{i}^{T}$ (179) $$

It follows, that in the reversal learning case, i.e. ${\bf b}=0$, for each reversed singular vector, the dynamics vanish to zero

$$ $\lim_{t\to\infty}\frac{-{\bf s_{\lambda}}_{i}{\bf c}_{i}^{2}{\bf\tilde{s}}_{i} e^{-4{\bf\tilde{s}}_{\lambda i}\frac{t}{\tau}}}{4{\bf\tilde{s}_{\lambda,i}}e^{ -2{\bf\tilde{s}}_{\lambda i}\frac{t}{\tau}}+{\bf s}_{i}{\bf c}_{i}^{2}\left(e^ {-2{\bf\tilde{s}}_{\lambda i}\frac{t}{\tau}}-e^{-4{\bf\tilde{s}}_{\lambda i} \frac{t}{\tau}}\right)}{\bf\tilde{u}}_{i}{\bf\tilde{v}}_{i}^{T}=0\text{.}$ (180) $$

Analytically, the learning dynamics are initialized on and remain along the separatrix of a saddle point until the corresponding singular value of the network function decreases to zero and stays there, indicating convergence to the saddle point. In numerical simulations, however, the learning dynamics can escape the saddle points due to the imprecision of floating-point arithmetic. Despite this, numerical optimization still experiences significant delays, as escaping the saddle point is time-consuming Lee et al. (). In contrast, when the singular vectors are aligned (${\bf c}=0$), the equation governing temporal dynamics, as described in Saxe et al. (), is recovered. Under these conditions, training succeeds, with the singular value of the network function converging to its target value.

$$ $\displaystyle\lim_{t\to\infty}\sum_{i=1}^{N_{i}}\frac{{\bf s_{\lambda i}}{\bf b }_{i}^{2}{\bf\tilde{s}_{\lambda i}}}{4{\bf\tilde{s}_{\lambda i}}e^{-2{\bf \tilde{s}}_{\lambda i}\frac{t}{\tau}}+{\bf s_{\lambda i}}{\bf b}_{i}^{2}\left( 1-e^{-2{\bf\tilde{s}}_{\lambda i}\frac{t}{\tau}}\right)}{\bf\tilde{u}}_{i}{\bf \tilde{v}}_{i}^{T}$ $\displaystyle=\frac{{\bf s_{\lambda i}}{\bf b}_{i}^{2}{\bf\tilde{s}_{\lambda i }}}{{\bf s_{\lambda i}}{\bf b}_{i}^{2}}{\bf\tilde{u}}_{i}{\bf\tilde{v}}_{i}^{T}$ (181) $\displaystyle={\bf\tilde{s}_{\lambda i}}{\bf\tilde{u}}_{i}{\bf\tilde{v}}_{i}^{ T}\text{.}$ (182) $$

In summary, in the case of aligned singular vectors, the learning dynamics can be described by the convergence of singular values. However in the case of reversal learning, analytically, training does not succeed. In simulations, the learning dynamics escape the saddle point due to numerical imprecision, but the learning dynamics are catastrophically slowed in the vicinity of the saddle point as shown in figure 7 .

In the case where $\lambda$ is non-zero, the diagonal of ${\bf C}$ are also non-zero; this is true regardless of whether they are reversed or aligned. Similarly, the diagonal entries of ${\bf B}$ remain non-zero whether the singular vectors are aligned or reversed. Therefore, in the case of reversal learning, ${\bf B}$ is a diagonal matrix with elements that are zero.
In figure 7

Figure: Figure 7: Plot showing the steps to convergence for two tasks: (1) the reversal learning task and (2) a randomly sampled continual learning task across a range of $\lambda$ values. The reversal learning task exhibits catastrophic slowing at $\lambda=0$.
Refer to caption: extracted/5871880/figures/reversal.png

### D.3 Appendix: Generalization and structured learning

We study how the representations learned for different $\lambda$ initializations impact generalization of properties of the data. To do this, we consider the case where a new feature is associated to a learned item in a dataset and how this new feature may then be related to other items based on prior knowledge. In particular, we first train each network (for different values of $-10\leq\lambda\leq 10$) on the hierarchical semantic learning task in Section 5 and then add a new feature (e.g., ‘eats worms’) to a single item (e.g., the goldfish) (Fig. 8A), correspondingly increasing the output dimension to represent the novel feature. In order to learn the new feature without affecting prior knowledge, we append a randomly initialized row to ${\bf W}_{2}$ and train it on the single item with the new feature, while keeping the rest of the network frozen. Thus, we only change the weights from the hidden layer to the new feature which may produce different behavior depending on how the hidden layer representations vary based on $\lambda$. After training on the new feature-item association, we query the network with the rest of the data to observe how the new feature is associated with the other items. We find that as $\lambda$ increases positively, the network better transfers the hierarchy such that it projects the feature onto items based on their distance to the trained item (Fig. 8B,C). For example, after learning that a goldfish eats worms, the network can extrapolate the hierarchy to infer that another fish, or birds, may also eat worms; instead, plants are not likely to eat worms. Alternatively, as $\lambda$ becomes more negative, the network ceases to infer any hierarchical structure and only learns to map the new feature to the single item trained on. In this case, after learning that a goldfish eats worms, the network does not infer that other fish, birds, or plants may also eat worms.

Interestingly, this setting highlights how asymmetries in the representations yielded by different $\lambda$ can actually benefit transfer and generalization. This can be shown by observing that the learning of a new feature association only depends on the first layer ${\bf W}_{1}$. Let $\hat{\bm{y}}_{f}$ denote the vector of the representation of the new feature $f$ across items $i$ in the dataset. Additionally, let $\bm{w}_{2}^{(f)T}$ be the new row of weights appended to ${\bf W}_{2}$ which map the hidden layer to the new feature. Following Saxe et al. (), if $\bm{w}_{2}^{(f)T}$ is initialized with small random weights and trained on item $\tilde{\bm{H}}_{i}$, it will converge to

$$ $\displaystyle\bm{w}_{2}^{(f)T}$ $\displaystyle=\tilde{\bm{H}}_{i}^{T}{\bf W}_{1}^{T}/\|{\bf W}_{1}\tilde{\bm{H} }_{i}\|_{2}^{2}$ (183) $\displaystyle\hat{\bm{y}}_{f}$ $\displaystyle=(\tilde{\bm{H}}_{i}^{T}{\bf W}_{1}^{T}{\bf W}_{1}\tilde{\bm{H}}) /\|{\bf W}_{1}\tilde{\bm{H}}_{i}\|_{2}^{2}$ (184) $$

From this we can see that differences in the representations of the new feature across items $\hat{\bm{y}}_{f}$ across $\lambda$ are only influenced by ${\bf W}_{1}$.

In the case of the rich learning regime where $\lambda=0$, the semantic relationship between features and items is distributed across both layers. Instead, when $\lambda>0$, the second layer ${\bf W}_{2}$ exhibits lazy learning, yielding an output representation ${\bf W}_{2}{\bf W}_{2}^{T}$ of a weighted identity matrix. However, the first layer ${\bf W}_{1}$ still learns a rich representation of the hierarchy, albeit at a smaller scaling. Furthermore, rather than distributing this learning across both layers, in the $\lambda>0$ case, all learning of the hierarchy occurs in the first layer, allowing it to more readily transfer this structure to the learning of a new feature (which only depends on the first layer). Thus, in this case, the ‘shallowing’ of the network into the first layer is actually beneficial. Finally, we can also observe the opposite case when $\lambda<0$. Here, rich learning happens in the second layer, while the first layer is lazy and learns to represent a weighted identity matrix. As such, these networks do not learn to transfer the hierarchy of different items to the new feature.

Figure: Figure 8: Transfer learning for different $\lambda$. A A new feature (such as ‘eats worms’) is introduced to the dataset after training on the hierarchical semantic learning task (Section 5). A randomly initialized row is added to ${\bf W}_{2}$ and trained on a single item with the new feature (for example, the goldfish), with the rest of the network frozen. The network is then tested on the transfer of the new feature to other items, such that items closer to the goldfish in the hierarchy are more likely to have the same feature. B The generalization loss on the untrained items with the new feature decreases as $\lambda$ increases. C As $\lambda$ increases positively, networks better transfer the hierarchical structure of the data to the representation of the new feature.
Refer to caption: extracted/5871880/figures/transfer_learning.png

## Appendix E Implementation and Simulations

The details of the simulation studies are described as follows. Specifically, $N_{i}$, $N_{h}$, and $N_{o}$ represent the dimensions of the input, hidden layer, and output (target), respectively. The total number of training samples is denoted by $N$, and the learning rate is defined as $\eta=\frac{1}{\tau}$.

### E.1 Lambda-balanced weight initialization

In practice, to initialize the network with lambda-balanced weights, we use Algorithm 1. In this algorithm, $\alpha$ serves as a scaling factor that controls the variance of the weights, allowing for adjustments between smaller and larger weight initializations.

Figure: Algorithm 1 Get *$\lambda$-balanced*

### E.2 Tasks

In the following, we describe the different tasks that are used throughout the simulation studies.

#### E.2.1 Random regression task

In the random regression task, the inputs ${\bf X}\in\mathbb{R}^{N_{i}\times N}$ are generated from a standard normal distribution, ${\bf X}\sim\mathcal{N}(\mu=0,\sigma=1)$. The input data ${\bf X}$ is then whitened to satisfy $\frac{1}{N}{\bf X}{\bf X}^{T}={\bf I}$. The target values ${\bf Y}\in\mathbb{R}^{N_{o}\times N}$ are independently sampled from a normal distribution with variance scaled according to the number of output nodes, ${\bf Y}\sim\mathcal{N}(\mu=0,\alpha=\frac{1}{\sqrt{N_{o}}})$. Consequently, the network inputs and target values are uncorrelated Gaussian noise, implying that a linear solution may not always exist.

#### E.2.2 Semantic hierarchy

We use the same task as in Braun et al. () and modify it to match the theoretical dynamics. The modification ensures that the inputs are whitened. In the semantic hierarchy task, input items are represented as one-hot vectors, i.e., ${\bf X}=\frac{{\bf I}}{8}$. The corresponding target vectors, ${\bf y}_{i}$, encode the item’s position within the hierarchical tree. Specifically, a value of $1$ indicates that the item is a left child of a node, $-1$ denotes a right child, and 0 0 indicates that the item is not a child of that node. For example, consider the blue fish: it is a blue fish, a left child of the root node, a left child of the animal node, not part of the plant branch, a right child of the fish node, and not part of the bird, algae, or flower branches, resulting in the label $[1,1,1,0,-1,0,0,0]$. The labels for all objects in the semantic tree, as shown in Figure 4
A, are given by:

$$ $\displaystyle{\bf Y}=8*\begin{bmatrix}1&1&1&1&1&1&1&1\\ 1&1&1&1&-1&-1&-1&-1\\ 1&1&-1&-1&0&0&0&0\\ 0&0&0&0&1&1&-1&-1\\ 1&-1&0&0&0&0&0&0\\ 0&0&1&-1&0&0&0&0\\ 0&0&0&0&1&-1&0&0\\ 0&0&0&0&0&0&1&-1\\ \end{bmatrix}.$ (185) $$

The singular value decomposition (SVD) of the corresponding correlation matrix, ${\bf\tilde{\Sigma}}^{yx}$, is not unique due to identical singular values: the first two, the third and fourth, and the last four values are the same. To align the numerical and analytical solutions, this permutation invariance is addressed by adding a small perturbation to each column ${\bf y}_{i}$, for $i\in{1,...,N}$, of the labels:

$$ ${\bf y}_{i}={\bf y}_{i}\cdot\left(1+\frac{0.1}{i}\right),$ (186) $$

resulting in singular values that are nearly, but not exactly, identical.

### E.3 Figure 1

Panels B illustrates three simulations conducted on the same task with varying initial $\lambda$-balanced weights respectively $\lambda=-2$, $\lambda=0$, $\lambda=2$. The regression task parameters were set with $(\sigma=\sqrt{10})$. The network architecture consisted of $N_{i}=3$, $N_{h}=2$, $N_{o}=2$,with a learning rate of $\eta=0.0002$. The batch size is $N=10$. The zero-balanced weights are initialized with variance $\sigma=0.00001$. The lambda-balanced network are initialized with $sigmaxy=\sqrt{1}$ of a random regression task with same architecture.

On Panel C , we plot the ballancedness ${\bf W}_{2}(0)^{T}{\bf W}_{2}(0)-{\bf W}_{1}(0){\bf W}_{1}(0)^{T}$ for a two layer network initialised with Lecun initialization with dimension $N_{i}$ = 40 ,$N_{h}$= 120 ,$N_{o}$=250

### E.4 Figure 2

Panel A, B, C illustrates three simulations conducted on the same task with varying initial $\lambda$-balanced weights respectively $\lambda=-2$, $\lambda=0$, $\lambda=2$ according to the initialization scheme described in E.7. The regression task parameters were set with $(\sigma=\sqrt{10})$. The network architecture consisted of $N_{i}=3$, $N_{h}=2$, $N_{o}=2$ with a learning rate of $\eta=0.0002$. The batch size is $N=10$. The zero-balanced weights are initialized with variance $\sigma=0.00001$. The lambda-balanced network are initialized with $sigmaxy=\sqrt{1}$ of a random regression task with same architecture.

### E.5 Figure 3

Panel A, B, C illustrates three simulations conducted on the same task with varying initial $\lambda$-balanced weights respectively $\lambda=-2$, $\lambda=0$, $\lambda=2$ according to the initialization scheme described in E.7. The regression task parameters were set with $(\sigma=\sqrt{12})$. The network architecture consisted of $N_{i}=3$, $N_{h}=3$, $N_{o}=3$ with a learning rate of $\eta=0.0002$. The batch size is $N=5$. The zero-balanced weights are initialized with variance $\sigma=0.0009$. The lambda-balanced network are initialized with $sigmaxy=\sqrt{12}$ of a random regression task with same architecture.

### E.6 Figure 4

In Panel A presents a semantic learning task with the SVD of the input-output correlation matrix of the task. $U$ and $V$ represent the singular vectors, and $S$ contains the singular values. This decomposition allows us to compute the respective RSMs as $USU^{\top}$ for the input and $VSV^{\top}$ for the output task. The rows and columns in the SVD and RSMs are ordered identically to the items in the hierarchical tree.

The results in Panel B display simulation outcomes, while Panel C presents theoretical input and output representation matrices at convergence for a network trained on the semantic task described in Braun et al. (); Saxe et al. (),. These matrices are generated using varying initial $\lambda$-balanced weights set at $\lambda=-2$, $\lambda=0$, and $\lambda=2$, following the initialization scheme outlined in E.7. The network architecture includes $N_{i}=8$, $N_{h}=8$, and $N_{o}=8$ with a learning rate of $\eta=0.001$ and a batch size of $N=8$. Zero-balanced weights are initialized with a variance of $\sigma=0.00001$, while $\lambda$-balanced networks are initialized with $\sigma_{xy}=\sqrt{1}$ based on a random regression task with the same architecture.

Panel D illustrates results from running the same task and network configuration but initialized with randomly large weights having a variance of $\sigma=1$.

In panel E, we trained a two-layer linear network with $N_{i}=N_{h}=N_{o}=4$ on a random regression task for $\lambda\in[-5,-4,-3,-2,-1,0,1,2,3,4,5]$ to convergence. Subsequently, we added Gaussian noise with $\mu=0,\sigma\in[0,0.5,1]$ to the inputs (top panel) or synaptic weights (bottom panel) and calculated the expected mean squared error.

### E.7 Figure 5

Panel A illustrates schematic representations of the network architectures considered: from left to right, a funnel network ($N_{i}=4$, $N_{h}=2$, $N_{o}=2$), a square network ($N_{i}=4$, $N_{h}=4$, $N_{o}=4$), and an inverted-funnel network ($N_{i}=2$, $N_{h}=2$, $N_{o}=4$).

Panel B shows the Neural Tangent Kernel (NTK) distance from initialization, as defined in Fort et al. (), across the three architectures shown schematically. The kernel distance is calculated as:

$$ $S(t)=1-\frac{\langle K_{0},K_{t}\rangle}{\|K_{0}\|_{F}\|K_{t}\|_{F}}.$ $$

The simulations conducted on the same task with eleven varying initial $\lambda$-balanced weights in $[-9,9]$. The regression task parameters were set with $(\sigma=\sqrt{3})$. The task has batch size $N=10$. The network has with a learning rate of $\eta=0.01$. The lambda-balanced network are initialized with $\sigma xy=\sqrt{1}$ of a random regression task.

Panel C shows the Neural Tangent Kernel (NTK) distance from initialization for the funnel architectures shown schematically with dimensions $N_{i}=3$, $N_{h}=2$, and $N_{o}=2$.
The simulations conducted on the same task with twenty one varying initial $\lambda$-balanced weights in $[-9,9]$. The regression task parameters were set with $(\sigma=\sqrt{3})$. The task has batch size $N=30$. The network has with a learning rate of $\eta=0.002$. The lambda-balanced network are initialized with $\sigma xy=\sqrt{1}$ of a random regression task.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: a. Could the authors comment on how they would expect the results to generalize to more complex tasks?

b. Although it is mentioned some choice of $\lambda$ may be beneficial to transfer learning due to induced lazy learning in the shallow layer (D3), it would be at least better to show a performance increase.
soundness: 4
strengths: Originality: The paper built on top of existing work by extending from zero-balanced condition into $\lambda$-balanced scenarios, leading to a continuum from lazy to rich regime in terms of weight structure of two consecutive layers. 

Quality: Solid theoretical derivations backed with rich simulation results. 

Clarity: The paper is very well-written. 

Significance: The $\lambda$-balanced discussed in the paper covers from architecture shapes to initialization schemes and relate these structural properties to the learning regimes, which will be of interest to the neuroscience community as well as continual learning and beyond communities.
summary: The paper theoretically derived the exact solutions for the learning dynamic of $\lambda$-balanced initializations in two-layer linear networks and discussed the implications in continual learning etc.
weaknesses: The only weakness of the paper is on the weak demonstration of applications.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1.	A minor point: over-parameterization is commonly referring to the relation between the amount of parameters in the network (capacity of the network) vs. the amount of parameters needed to achieve zero training loss. I believe this may be different from ‘N_h > min(N_i, N_o)’?
2.	The assumption that the input data is whitened is relatively strong and not very applicable to realistic settings. Do you have any empirical evidence demonstrating that your observation at least qualitatively still holds for benchmark datasets without necessarily whitening the data?
3.	This may be just a typo. In Fig.2 b, in the caption it says lambda=0.001 but in the figure lambda=0.
4.	In previous works concerning lazy and rich dynamics, especially in infinitely wide networks, there’s usually the assumption that N goes to infinity and depending on how initialization scales with N the network resides in either the lazy or the rich regime. Here there doesn’t seem to be any assumptions about large N or the scaling of the initialization. Specifically since Nh=min(Ni, No), it seems that N is finite. However, it seems that the authors are able to get fixed NTK in the lazy regime without the assumption of large N in section 5. Is this specific to the linear network? Can you comment on how your results may scale to larger dimensional networks, and on the relation of your results and the lazy and rich regimes in the infinite width limit? 
5.	In Fig.4 b the simulation w1w1 and w2w2 labels seem to be reversed compared to the theory. 
6.	In Fig.5, how does the ratio between Nh and Ni (or Nh and No) play a role here?
soundness: 3
strengths: 1.	The paper is very well structured and the theoretical results are clearly written. 
2.	It is a novel and interesting idea to model the range of dynamics from lazy to rich with the balance parameter \lambda. 
3.	The authors analyzed their theoretical results on interesting semantic task examples, and showed multiple implications of their theory on other learning paradigms.
summary: This work extends previous results on the exact dynamics of deep linear networks to include the lambda-balanced initialization conditions, where lambda is a tunable parameter that allows for exploration of a full range of dynamics from lazy to rich learning. By detailed analysis of the result, the authors showed that the transition from the lazy regime to the rich regime depends on the complex interaction of multiple factors, including the balance parameter lambda and the network architecture. This paper then also discusses potential application of the results on more complicated learning paradigms such as continual learning, reversal learning and transfer learning.
weaknesses: 1.	The technical and theoretical novelty of this work is limited, the method used in this paper has mostly been established in Braun et. al. 2022, this work extends to the \lambda-balanced case, which is more general. However, the extension is achieved by enforcing a stricter assumption that the input is whitened, limiting the applicability of the results. 
2.	This paper makes several very strong assumptions such as task-aligned initialization, whitened inputs, linear networks, etc. It would be nice if the authors can show even some empirical evidence that these assumptions may be somewhat relaxed while the main results still qualitatively hold. 
3.	The applications are only briefly discussed. I think the paper would benefit from making the theoretical results in section 4 more concise and moving some of the application results (transfer learning for example I think is very interesting) to main text.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: •	It would be interesting to understand more how this relative scale interact with other knobs that impact learning regimes. For instance, could you provide more discussion on how the relative scale and absolute scale of the weights interact? See the weakness point above. 

•	Could you offer more intuition on how \(\lambda \to \infty\) results in an identity representation (i.e., a task-agnostic representation)?
soundness: 4
strengths: •	This work is novel and timely.

•	Strong theoretical foundation: the approach extends a previous theoretical framework (Braun et al., 2022) to derive exact solutions for NTK, representation similarity, gradient flow, and loss. This allows for precise determination of how initialization (specifically, the $\lambda$ parameter) influences representation and learning dynamics under the given assumptions. However, I did not have the bandwidth to verify the proofs in the appendix, which affects my confidence score.

•	Insightful experiments demonstrating how $\lambda$ interpolates between learning regimes, with applications to both neuroscience and machine learning.

•	The manuscript is well-written, with a comprehensive literature review.
summary: This study derives exact solutions (Theorem 4.3) to examine how initialization --- particularly the relative scale of weights across layers --- influences rich versus lazy learning in linear feedforward neural networks. The study relaxes several assumptions from prior work, notably the balanced assumption in Braun et al., which is a special case in here when $\lambda=0$. Extending this, the authors show in Theorem C.4 that the network transitions into a rich regime as $\lambda$ approaches 0, and into a lazy regime as $\lambda$ approaches infinity. Overall, this work significantly contributes to the ongoing investigation into how various network parameters influence rich versus lazy learning regimes, with demonstrated relevance to both neuroscience and deep learning.
weaknesses: •	This work relies on a list of assumptions and focuses on a simple two-layer linear feedforward network, which deviates from real-world settings. However, this didn’t significantly impact my score, as the assumptions are already more relaxed compared to previous works, and the authors have adequately addressed these limitations in the discussion.

•	I wish there were more intuitive explanations of how \(\lambda\) interpolates between learning regimes. The authors provide some results in this direction, such as Theorem 5.1, which shows that under different limits of \(\lambda\), the transition function converges to sigmoidal (\(\lambda \to 0\)) and exponential (\(\lambda \to \pm \infty\)). However, it would be helpful to have more insight into how the limit of \(\lambda \to \pm \infty\) leads to a task-agnostic identity representation. 

•	I’m not sure if I have a good understanding of how the relative and absolute scales interact. While the authors demonstrated that in the extreme case where \(\lambda \to 0\), the rich regime occurs regardless of the absolute scale, I’m less certain about the case where \(\lambda \to \infty\), as the weight norm of one of the layers would presumably become very large. Large initialization can lead to lazy learning, so I’m unsure if it’s possible to decouple the contributions of \(\lambda\)-balanced initialization from large weight initialization. I believe some control experiments with large \(\lambda\) values using random (unbalanced) initialization but with the same \(W_1\) and \(W_2\) norms as the balanced case would help distinguish the two.

Meta-review: This paper analyzes the learning dynamics of two-layer linear networks and derives explicit solutions for gradient flow. In particular, it focuses on how $\lambda$-balancedness of initialization affects the learning regime (including lazy and rich regimes). The results have implications for continual learning, reversal learning, and transfer learning. The reviewers all acknowledge the novel contributions of this paper and unanimously lean toward acceptance.

Decision: Accept (Poster)