The Importance of Being Lazy: Scaling Limits of Continual Learning
Authors: Jacopo Graldi
Alessandro Breccia
Giulia Lanzillotta
Thomas Hofmann
Lorenzo Noci
Venue: ICML 2025 poster (2025)
Keywords: feature learning
continual learning
deep learning
forgetting
ntk
muP
lazy
dmft
Primary area: deep_learning
TL;DR: We investigate how network size and feature learning regimes affect catastrophic forgetting in continual learning.

Abstract: Despite recent efforts, neural networks still struggle to learn in non-stationary environments, and our understanding of catastrophic forgetting (CF) is far from complete.
In this work, we perform a systematic study on the impact of model scale and the degree of feature learning in continual learning. We reconcile existing contradictory observations on scale in the literature, by differentiating between *lazy* and *rich* training regimes through a variable parameterization of the architecture. We show that increasing model width is only beneficial when it reduces the amount of *feature learning*, yielding more laziness. Using the framework of dynamical mean field theory, we then study the infinite width dynamics of the model in the feature learning regime and characterize CF, extending prior theoretical results limited to the lazy regime. We study the intricate relationship between feature learning, task non-stationarity, and forgetting, finding that high feature learning is only beneficial with highly similar tasks. We identify a transition modulated by task similarity where the model exits an effectively lazy regime with low forgetting to enter a rich regime with significant forgetting. Finally, our findings reveal that neural networks achieve optimal performance at a critical level of feature learning, which depends on task non-stationarity and *transfers across model scales*. This work provides a unified perspective on the role of scale and feature learning in continual learning.

The Importance of Being Lazy: Scaling Limits of Continual Learning
Jacopo Graldi" 1 Alessandro Breccia
Giulia Lanzillotta 34
Thomas Hofmann" Lorenzo Noci
Abstract
1. Introduction
Despite recent efforts, neural networks still strug- gle to learn in non-stationary environments, and our understanding of catastrophic forgetting (CF) is far from complete. In this work, we perform a systematic study on the impact of model scale and the degree of feature learning in continual learn- ing. We reconcile existing contradictory observa- tions on scale in the literature, by diffcrentiating between lazy and rich training regimes through a variable parameterization of the architecture. We show that increasing model width is only benefi- cial when it reduces the amount of feature learn- ing. yielding more laziness. Using the framework of dynamical mean field theory, we then study the infinite width dynamics of the model the feature learning regime and characterize CF, extending prior theoretical results limited to the lazy regime. We study the intricate relationship between fea- ture learning, task non-stationarity, and forgetting, finding that high feature learning is only bene- ficial with highly similar tasks. We identify a transition modulated by task similarity where the model exits an effectively lazy regime with low forgetting to enter a rich regime with significant forgetting. Finally, our findings reveal that neural networks achieve optimal performance at a crit- ical level of feature learning, which depends on task non-stationarity and transfers across model scales. This work provides a unified perspective on the role of scale and feature learning in contin- ual learning.
Modern neural networks (NNs) have achieved impressive re- sults on many hard benchmarks, recently beating all expecta- tions with the introduction of large autoregressive language models (OpenAI, 2023). However, the learning algorithms in use are only applicable to stationary data distributions. In particular, these algorithms fail to retain knowledge when learning new skills, a phenomenon termed Catastrophic Forgetting (CF) (McCloskey & Cohen, 1989) in the litera- ture. Arguably, the design of algorithms which can adapt to changes in the environment is crucial step towards the development of more widely applicable, trustworthy, and scalable Al systems (Wang et al., 2024). Continual learning (CL) (e.g. Ring. 1994; Thrun & Mitchell, 1995; Silver et al., 2013; Parisi et al., 2019; Hadsell et al., 2020; Lesort et al., 2020) directly aims at devising algorithms which allow the model to learn under distribution shifts. This entails two requirements: knowledge adaptation also called plasticity and knowledge retention stability. For NNs these ob- jectives inherently conflicting, creating unavoidable tradeoff, often called the stability-llaticii dilemma. Vari- ous studies have investigated the causes of CF in artificial neural networks. However, thcoretical models of CF have only been established in simplified settings, e.g., by assum- ing fixed features during training (Bennani al., 2020; Doan et al., 2021; Evron et al., 2022; Goldfarb & Hand, 2023; Lin al., 2023; Goldfarb et al., 2024). At the same time, empirical studies have significantly contributed to the under- standing of CF in modern deep networks used in practice (Mirzadeh et al., 2020; 2021; 2022a; Ramasesh et al., 2020; 2022). Among these, multiple recent works have focused on the role of scale and overparamettrrization in continual learning, producing different if not contradictory answers (Ramasesh et al., 2022; Mirzadeh et al., 2022a;b; Goldfarb & Hand, 2023; Lin et al., 2023; Wenger et al., 2023) as elaborated below in Section 2. In particular, the question of whether scaling alone can ameliorate forgetting is still open.
'Equal contribution 'Dept. Information Technology and Elec- trical Engineering, ETH Zurich, Switzerland 'Depi. of Physics and Astronomy, University of Padua, Italy 'Dept. of Computer Science, ETH Zurich 'ETH Al Center. Correspondence to: <ja- copograldi @ gmail.com>.
A separate thread of literature has studied the so-called scaling limits of neural networks, where the network di- mensions (i.e., width and depth) are taken to infinity (Neal, 1996). Depending on the parameterization used - see formal definition in Sec. 3 - the network exhibits fundamentally different training dynamics. More specifically, in one ex-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
1.0
0.5
0.5
1050
0.
2018
6.8
0.4
0.4
1874
0.3
0.3
Witth
Averugee
512
0.4
0.2
0.2
0.3
0.2
0.3
0.1
128
0.0 10 10 10
0.0 10
20 10 30 0.0
10
10-2 10"1 107
T
700
(a)
(b)
(c)
Figure 1. (a) Test error of stationary training (joint CIFARIO). (b) Final average error and (c) Catastrophic Forgetting rate (CFr) of non-stationary sequential training (Split-CIFARIO). The factor nIu interpolates between lazy (?u -> 0) and rich (Nu = 1) regimes. (a) In stationary training. more feature learning and width scaling are strictly beneficial.
(b) In non-stationary training the optimal performance is achieved at a critical level of feature learning 76 R 0.1 above which (c) forgetting explodes worsening the final performance, and the increased scale of the network wasted without any benefits. The region of to which forgetting transitions from low high defines a change of training regime of the network from heing effectively lazy to having rich dynamics. All figures report the average over 5 random seeds.
treme the Neural Tangent Parameterization (NTP) the dynamics are lazy (Chizat et al., 2019), meaning that the network activations (a. features) evolve with vanishingly small magnitude during training, effectively keeping the network close to its initialization (Jacot et al., 2018; Arora et al., 2019; Lee et al., 2019); while in the other extreme the Maximal Update Parameterization (nP) - the dynamics are rich, and feature learning is maintained at any scale (Yang & Hu, 2020; Bordelon & Pehlevan, 2022).
effectively lazy to being rich and uncover its intertwined nature with the non-stationarity of data.
Contributions and Paper Structure
In Section 4, we demonstrate that the effect of width scaling on catastrophic forgetting depends on the net- work parameterization: while NTP leads to reduced forgetting with scale, jaP does not. This indicates the entangled relationship between scale and training regime. We then extend the Mean Field formalism to task-sequential training, modeling infinite- widhh rich training dynamics. This model aligns with finite-size network predictions, making it a useful framework for studying NNs under task non-stationarities.
Harnessing the power of exactly characterizing the network training dynamics, in this work we study the scaling limits of catastrophic forgetting. aiming at a comprehensive un- derstanding of the role of scale in continual learning. In particular, we interpolate between the lazy and rich training regimes, varying parameter to of the network parameteri- zation that measures the degree of feature learning. In the analysis of scale, " reveals to be the key to explaining the aforementioned inconsistencies in the literature.
In Section 5, we find that there is a non-lineas rela- tionship between the degree of feature learning "o and forgetting characterized by sharp low-to-high for- getting transition. We also characterize the % that optimizes the stability-plasticity tradeoff, showing that it transfers across model scales.
We find that in non-stationary training the effect of scal- ing is essentially modulated by the degree of feature learn- ing of the model, finding that past certain degree of feature learning scaling the network does not benefit per- formance (Fig. 1b, center). Curiously, we find that the optimal plasticity-stablliiy tradeoff is achieved at a fixed data- dependent degree of feature learning no < 1, which transfers across widths. Thus, in practice, scaling the net- work beyond this threshold effectively wastes the network capacity. In order to appreciate the subtlety of our results, it is important consider the standard setting of stationary training: increasing the degree of feature learning the model scale (in particular width) decreases the error (see Fig. la, left). On the other hand, an optimal n0 < 1 in the non-stationary case underlies the destructive nature of high feature learning on knowledge retention. We find that large To sharply increases catastrophic forgetting (Fig. Ic, right), reflecting transition of the network dynamics from being
In Section 6 we study the role of the degree of non- stationarity by controlling the task similarity in both synthetic and natural tasks. We show that lower task similarity causes a shift of the low-to-high forgetting transition towards larger values of Yo, which translates to larger 70- This suggests that laziness is key in highly non-stationary scenarios.
2. Related Work
Catastrophic Forgetting and the Question of Scaling Catastrophic Forgetting is a known phenomenon in the deep learning literature since the early nineties (McCloskey & Cohen, 1989; French, 1999), and to date it has been con- sistently observed in distinct benchmarks and network ar- chitectures (Wang et al., 2024; Mai et al., 2022; De Lange
2

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
al., 2021; Ke & Liu, 2022; Khetarpal al., 2022). Even large language models (LLM) are vulnerable to CF when trained on highly non-stationary data streams (Luo et al., 2023; Wu et al., 2024).
pared to their work, here we focus on the gradient descent training dynamics in the fixed data, large width limit.
3. Methodology
Goldfarb & Hand (2023); Lin et al. (2023) study the effect of overparamettrizztioo in linear regression models, proving that in this setting scaling reduces CF. For neural networks used in practice, the existing empirical evidence depicts a more nuanced picture, and due to the complexity of the question - there is currently no theory on it. Ramasesh et al. (2022) find that scaling benefits CL only for pretrained mod- els, and not for models trained from scratch. By contrast, Mirzadeh et al. (2022a;b) observe that increasing the width of a network- - but not the depth reduces CF even when training from scratch. In response, Wenger et al. (2023) show that the observation of Mirzadeh et al. (2022a) is de- pendent on the training time: increasing the training time eliminates the positive effect of scaling on CF. Our work contributes to this line of research, offering a solid hypothe- sis - based on theoretical and empirical grounds - as to why and when scaling helps CF.
Let us introduce the notation for residual networks of fixed width N and with L residual layers (i.e. the depth), where the ε-th layer's parameters are initialized as W (0,,, oz). For an input x E IR,, the preactivations of the first block are defined as h² (x) = 3o W The N- dimensional preactivations of the ε-th block have a residual branch scaled by Bg, and the outputs f(x) E R are addition- ally inversely scaled by y:
h (x) = oh ah'(x) + W (o'((((()))
f(x) = BL wh (x)())).
(I)
y
h4+1
where we use the scalar a E {0. 1) to turn on and off skip connections. The choice of how scale the factors 3g and 7, the weights initialization variance o7. and the (possibly time-varying) learning rate 17(t) as the width increases, dif- ferentiates between the two parameterizations considered in this work: the Neural Tangent Parameterization (NTP) and the Maximal Update Parameterization (μP). We summarize the scaling of these parameters in Tab. 1. Importantly, by varying the "o parameter it possible to smoothly interpo- late between lazy (%o -> 0) and rich (%0 1) regimes. To
Scaling Limits and Continual Learning The research on scaling limits aims at theoretically characterizing the behav- ior of neural networks in the limit of infinite width and/or depth. Early works describe the network function at initial- ization (Lee et al., 2017; Yang, 2020; Hayou et al., 2021; Noci et al., 2021), which has been useful to prescribe opti- mal initialization conditions for stable training (Schoenholz et al., 2016; Hanin, 2018: Hanin & Rolnick, 2018: Martens et al., 2021). Later, the field advanced to study the network while training, either in the so-called rich lazy regime (Yang & Hu, 2020; Bordelon & Pehlevan, 2022). Although rich regimes are often credited as superior in performance (Fort et al., 2020), this premise has been partially reconsid- ered: Petrini et al. (2022) demonstrate that rich mean-field training can drive fully connected networks overfit to sparse features, reducing generalization. These results have been derived under the assumption of stationary input distribution. In this work, we extend the theory on scaling limits to the multi-task setting.
Table 1. Branch and output scales, learning rate, and weight vari- ance in the three parameterizations: NTP (PyTorch default) (Yang & Hu, 2020) and Mean Field jP (Bordelon Pehlevan, 2022).
NTP
Mean Field / pP
N
1/2 1/2
1/2
f 8>0
0
N N-1/2
E> 0
Branch Scale 3e
f= 0
D D-1/22 e= 0
N/A21/2
Output Scale ?
LR Schedule 1y(t)
(t)
mo(t))o
Weight Variance of
We are not the first to apply scaling limits to the study of continual learning. Doan et al. (2021); Bennani et al. (2020) provide theory of catastrophic forgetting in the lazy regime. Crucially, their results do not extend to the rich feature learn- ing regime. Our results show that the presence of feature learning is a key factor in the behavior of CF with scale, marking the relevance of our contribution to the existing lit- crature. Also, recent work has applied tools from statistical mechanies to continual learning (Ingrosso et al., 2024; Mori et al., 2024; Shan et al., 2024). In Shan et al. (2024), the authors study the large data, width asymptotics of continual learning by analyzing the posterior Gibbs ensemble. Com-
have comparable starting points of the scaling behavior, we normalize the parameterizations of the jPP to be equivalent to the NTP at a base width of N 64. Further details are in App. A.3.
3.1. Experimental Setup
In most of our experiments, we utilize ResNet architec- ture (with base width N = 64 and depth L, = 6), trained with Stochastic Gradient Descent (SGD) and evaluated on MNIST, CIFAR1O, and TinyImagene and their suitable adaptions to the continual learning setting. We also evalu- ate a Convolutional Neural Network (by setting a = 0 in
3

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
Eq. 1) in App. B.7 finding no significant deviations from the ResNet case. For the experiments involving the infinite- width simulations of the network dynamics, instead, we will use simpler non-linear two-layer perceptron (MLP henceforth), on a small subset of MNIST with 30 samples, suitably modified as a 2-tasks CL benchmark. The choice of this simplified setting is imposed by the numerical complex- ity (cubic in both the number of samples and the number of tasks) of the wiithh simulations. Further details the experimental setup are reported in App. A.4.
et al., 2023), consider two different parameterizations of the architecture when scaling: NTP and J.P.
In Fig. 2 we show the CFr for varying width and param- eterization. We identify two separate behaviors in width, depending on the parameterization. Thus the parameteriza- tion shapes the relationship between width and forgetting, where only NTP endows a diminishing forgetting with scale, When the network is parameterized following the JaP, width does not offer reductions in forgetting. Moreover, the for- getting curves of wider jP models almost coincide, a width consistency reminiscent of the observations of Vyas et al. (2024) in the stationary online setting.
3.1.1. METRICS
For a benchmark with T tasks, we define the test accuracy on the task T, after training on Tj, with E {L.... T). as az The learning accuracy (LA) is a measure of plasticity, and it defined simply as the average (ac.s); over the task index (Mirzadeh al., 2022a). We will also consider the learning error (LE), namely the complement of the LA (1 - LA). The average accuracy (AA) measures the plasticity-stabiliy tradeoff, and is the average (ar,a)s- We will also use its complement - the average error (AE), (1-AA AA).
AP
10"
0.5
10'
0.4
CF, CFr (les)
10
5 0.3
10a
0.2
10'
0.1
10-
Catastrophic Forgetting rate (CFr) To evaluate the mod- els' capacity to retain the knowledge of the past tasks, prior work (e.g. Mirzadeh et al. (2022a)) defines catastrophic for- getling (CF) (Def. A.3), as the average drop in accuracy after training on all later tasks. However, it implicitly depends on the raw learning accuracy, making it challenging to fairly such
NIP.
A.P.A...... 8
0.0
64
128
256
512 1024 2048 4096 Width
64
128
258
512 Width
1026
2018 4096
(a)
(b)
Figure 2. The effect of width scaling on forgetting with models
parameterized with the NTP and muP (%o 1). (a) Split Citit-CIFRRIO with ResNet model; (b) Permuted-MNIST with MLP and infinite- width asymptotic behavior.
compare models with different performance as in this work-and has led to misleading conclusions in the literature (as argued by Wenger al. (2023)). For this reason, we introduce novel metric- the Catastrophic For- getting rate (CFr)- the average relative drop in accuracy after training on all other tasks (Def. A.4).
Width Scaling Dilemma Our finding provides a new key interpret the observations of (Mirzadeh et al., 2022a; 2020; Wenger et al., 2023; Ramasesh et al., 2020) regarding the effect of scale on CF. In particular, we postulate that the conclusion that width-scaling reduces CF depends on lazy training dynamics (recall that the SP is equivalent to our NTP). On the other hand, Wenger et al. (2023)'s longer training time implicitly induces rich training dynamics. To confirm this, revisit Wenger et al. (2023)'s experiment in App. B.I. We show that longer training time and reasonably large learning rates induce more evolution of the NTK and thus more deviations from the lazy regime (as shown in (Fort et al., 2020)).
All experiments involving the MLP and the infinite-width limit are evaluated in terms of training loss performance since the proposed theory (introduced in Sec. 4. will pre- dict those dynamics. Concretely, if we define the training loss the task T; after training on Tj. with i, E [T....T). as Lj,,, then can substitute with C in the definitions of the LE, AE and CFr above, obtaining the Learning Loss, Average Loss, and CFr (loss). Formal definitions of these metrics are found in App. A.I.
4. The Effect of Width Scaling Depends on the Parameterization
In summary, this first result highlights at least two new observations. First, scaling per se neither positive nor negative, as its effect is intrinsically modulated by the net- work parameterization. Second, the difference between rich and lazy training regimes, i.e. feature learning, is a crucial factor in CF. To further study the large-width behavior of forgetting, we proceed to characterize the infinite-widie dy- namics of a model trained under non non-stationarities in the feature learning regime.
We begin our investigation of the role of width scaling in non-stationary training on the CL benchmark Split- CIFARIO and with ResNet model (more details in App. A). We reckon that this setup is similar to previous work on the role of width scaling in CL. However- crucially in con- trast to previous studies (Mirzadeh et al., 2022a; Wenger
4

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
4.1. Infinite Width Dynamics in Non-Stationary Training
which indicates that the NTK is fully determined by the kernels Φ aa (t); G 8, (t)- This results in the following Caus, finite width dynamics, where the scaling factor y and its dependency to "oo and N crucially differentiates NTP from nPP
We derive the dynamics under mean field scaling (see Tab. 1) of continually learned two-layer non-linear network in the infinite-width limit using the framework of dynamical mean field theory (DMFT) of Bordelon & Pehlevan (2022). DMFT has been successfully employed to describe the net- work dynamics both infinite-width (Bordelon& & Pehlevan, 2022; Bordelon et al., 2024) and -depth limits (Bordelon al., 2023), but never in the non-stationary setting. Due to space limitations, we defer more details, including the proofs, to Appendix C.
M (1) - (1)
+ VN I Σū ú,(t) Es,, (s)g,, (5) jiaa
j=1
z (t) (t)
2 VN I Σ Ú,(م) (h) (t))
We are interested in tracking the evolution of the model dy- namics in function space under gradient descent and Mean Squared Error (MSE) loss. As will clearer later, an im- portant quantity that governs the dynamics and catastrophic forgetting is the Neural Tangent Kernel (NTK) across tasks, defined as:
In Proposition 4.1, we show that in the infinite width limit for uP there exists a set of variables including (h),, Dey h { ( G Ga that forms close system of self-consistent (i.e. implicit) equations that fully describe the output f in the large N limit:
Ka,8,(1) (1) Σ {=1
(W,lhtt1Xa 1), (w ()),
Proposition 4.1 (Informal). Let be the output of neural network with single hidden layer in Eq. with 0 (i.e. without skip connections) trained for T tasks Tı...., Tr for a finite number of steps on each task, and for j., As N + 00, the pre-activations and pre pre-gradient vari- ables (h),, can be described as i.idd draws from marginal distributions described by the following stochastic processes:
(2) where a; E Ti and 83 E Tj and (---) is the standard inner product in Euclidean space. In the presence of training with multiple tasks T....,; Tr, the equations governing the dynamics at task T are:
d = Σū₁( Σ dt
1x 1=1 a,ET,
Kuya (1).
(3)
X
T
We have introduced Ü,(1) = U(L- - l), where Ut-tr--1) - is the Hcaviside step function, which we use to model the transitions from the (j - 1)-th to the j-th task. An, oJ -xx3) is the negative residual and K Kma, (t) is the NTK across tasks. We also introduce gra- L+1 dient neurons (l) E V defined as g (4) = / N ah as well as the forward and backward kernels (also called feature and gradient kernels):
+
70
Ú,(5) Σ (s)gh, ds
j=1
ag
(8)
20
=
T
Σ ú,(5) (s) (o))ds
+
j=1
where [XH. X0,] (O,, k. K3,1) and NNO,, I) the Gaussian Processes characterizing the initialization. Furthermore, the dynamics of the output f. as well as the kernels $) G¹ concentrate around their expectation:
1 N (0hh,, (1)) (h') (0))
$ (0,),(1)
(4)
Go,,, (t)
N E (t), 3 (t))
(5)
lim 2,, (1) = E (0hh (t)) (t)
(9)
N
Intuitively, the feature (resp. gradient) kernel controls the effect of the weights on the geometry of the feature space in the forward (resp. backward) pass. Often, will be more convenient manipulate the pre- pre-gradient variables zf. 1 W e++1 Then we have that E (t) = o(h o z N
lim G Ga (t) = E (ob, (t)gh, (t)
(10)
N-000
where the expectation is taken over the marginals for hp..
These quantities allow us to re-write the NTK across tasks as:
Remarks. Notice that the preactivations are one- dimensional variables, and not vectors, as every unit can be seen as an i.i.d. draw from this marginal, which corre- sponds to the single site process in the infinite width limit.
Ka, (t) = Σ s (t)Gk, (t),
(6)
Fm1
5

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
Finally, the term in "o in Equation (8) is the feature learn- ing correction characterizing the time evolution from the initial process. If 70 + 0, we recover the lazy regime (Jacot et al., 2018; Chizat et al., 2019), where it is clear from the equations that there is no time evolution of the hidden layer.
0.3
0.5
Peatures Evolution
01- -Ck A)
014
0.3
0.2
0.3
0.1
Proposition 4.1 provides an exact description of the network dynamics at infinite width. The self-consistent system of equations can be simulated reproduce the features and function evolution during training. Here, we apply this result to characterize the dynamics of CF. For simplicity, take the definition of CF to be the following':
0.0
10
17
0.0
10
10-
70
(a)
(b)
0.5
10'
N
10'
0.4
CRr (loss)
10"
0.3
CF(t): = Σ (t)
| A (4) 2
10"
0.2
(11)
10-
0.1
10
10
10
10-
0.0
0.2 Features Evolation -KA)
where t >> t1 and {Xa,: Vea Jos is the data of the first task. In the case of two tasks, it can be shown that the dynamics of forgetting are determined by the errors A and the NTK across tasks:
(c)
(d)
Figure 3. Entanglement between ")o, feature evolution, and forget-
ting. (a) Non-linear relationship between nu and feature evolution
d CF(()= E (t,())()122( dt
(12)
(1 CKA) in ResNet model (N = 4096, Split- Split-CIFARI0;; transi- tion happens at tto R % IRT (b) nto and CFr (ResNet, N 4096).
11002
(c) no and CFr at the indinite-width limit (MIP, permuted-MNIST
This equation is very general, and it is valid for all degrees of feature learning 70 and all input distributions. In Ap- pendix E, we use perturbation theory to isolate the effect of small to corrections to the lazy limit, up to second order.
(d) The amount of feature evolution correlates with forgetting
across various widths and values of To (ResNet, (pFit-CIFARIO),
We first measure the amount by which the features evolve during training by monitoring the internal representation of a task's data. More specifically, after training on a task, we look at the activation vectors at every residual block of the model of the same data while training on the remaining tasks. We compare activations in a permutation eerrutation-ivarriant invariant manner by measuring the cosine-alignment of the respective kernels, a measure known as Centered Kernel Alignment (CKA) (Kornblith et al., 2019). We plot the average over training and all tasks.
In Fig. 2b, we report the asymptotic behavior of catastrophic forgetting the loss under μP and NTP, at both finite and infinitewwidths. For the infinite-width limit, we use the dy- namics of Proposition 4.1 with = 1 for μP and 7o = 0 for NTP. As expected, as the width is increased the CFr at finite sizes approaches the theoretical limit. More impor- tantly, we again observe two separate behaviors depending on the parameterization, substantiating the claim that the effect of scaling on CF depends the training regime.
We uncover a surprisingly non-linear relationship between "o and the evolution of features (Fig. 3a). In particular, we identify a critical Lazy-Rich Transition (LRT) region "/o R 76 of that non-linearly separates two different behav- iors: for 70 < of the relationship is mostly flat and the features change by very small amounts, signifying that the network is in an effectively lazy regime. Conversely, for "lo > 70 LRT the features' evolution sharply increases with "00 thus the network is in a rich regime. We hypothesize that this transition could be related to our choice of LR scaling (Tab. 1, the LR scales quadratically with 70). Indeed, the concurrent work of Atanasov et al. (2024) observes that this scaling is optimal for the lazy regime, but not for the rich and ultra-rich regimes (their Fig. lb), where instead sub- quadratic LR scaling is optimal. Hence, as we increase Yoo our LR shifts from the optimal LR towards a larger-thare optimal LR, potentially triggering the sharp rise we observe.
5. Feature Learning and Catastrophic Forgetting in Non-Stationary Training
We now investigate the relationship between feature learn- ing (in particular its presence or absence) and catastrophic forgetting. In particular, we ask whether the presence of feature learning during training is conducive to CF.
5.1. Lizh-Rich and Low-High Forgetting Transitions
We leverage the "o parameter in the juP to smoothly interpo- late between the two lazy (%o + 0) and rich (%u = 1) train- ing regimes. This methodology has already been employed to study phenomena tied to network training dynamics (Ku- mar et al., 2024; Atanasov et al., 2024).
'This quantity is also known as backward transfer.
6

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
We leave for future study the exploration of this interesting hypothesis.
its literature (Yang et al., 2022a; 2023; Bordelon et al., 2023; Noci al., 2024). The anfinitewwidth simulation confirms our empirical findings with striking precision. Furthermore, we notice also that for Yo > % the increased scale of the network is wasted as the larger width does not bring any benefits in performance.
The relationship between "o and CFr is strikingly similar (Fig 3b): for "o < 7 CFr is very low regardless of To while it grows significantly for "/o > TO This result is confirmed consistently across different finite widths (Fig. 1b), as well as at the infinite-width limit (Fig. 3c). By analyzing the relationship between features evolution and CFr across scales and "Yo recover consistently that feature learning negatively impacts CF (Fig. 3d), with strong (p-value < 10 10-30) positive correlation between the amount of evolution in the features and CF. Taken together, these results convincingly point to the fact that the presence of feature learning is indeed conducive to CF, and higher degrees of feature learning lead to higher levels of forgetting in non-stationary learning.
3196
no
Learnnng Erver
0.4
2048
Average Loss
1024
0.3
512
0.2
266
0.
0.1-
0.
125
0.8
0.000
0.2 CFr
0.4
64
(a)
(b)
Remark on the Lacy-Rich Transition Curiously, this
Figure 4. (a) Average Loss for varying To finite and infinite widths (MLP, permuted- permuted-MNIST, 30 samples). The optimal n transfers all finite and infinite widths. (b) The plasticity-stabiie tradeoff is controlled by the amount of feature learning (ResNet, Split- SPFRI-CIFARIO.
transition also appears to mark shift in the behavior of scatt--caaing in forgetting. As visible in Fig. Ic, whereby for no < 7 L.IT width scaling improves CFr, while for to > .LTT it does not. This appears to happen not only on 70
Split-CIFARIO and the ResNet, but also on the MLP setting of the infinite-widde simulations (Fig. 11). Nevertheless, this inversion is not observed in the feature evolution but only in forgetting (cf. Fig. 13a). Our last remark hints at a fundamental shift in the interaction between width, 7o. and training dynamics happening at the lazy-rich transition. This is topic that deserves further investigation and it lies beyond the scope of this work. However, in Appendix B.5 we present preliminary results, looking the changes in the loss landscape geometry as we vary to and width jointly.
Finally, we disentangle the effect of plasticity and stability from the compound metric of the average performance, by measuring them through the Learning Error and CFr, respec- tively. In Fig. 4b we observe that varying "o allows us to navigate the stability-plasticin tradeoff, which generates a Pareto front. In particular, we see that at small 7o we have low CFr, but also a high LE. Increasing "oo the LE first undergoes rapid decrease without significantly impacting the CFr, striking the optimal tradeoff for 10 = 0.1. If we further increase "Yo, the CFr starts to sharply increase in return for a diminishing benefit of LE. We reproduce this tradeoff also with the MLP and the infinite-width simulation in Fig. 12. All in all, this experiment recovers the stability- plasticity dilemma and highlights that neither scaling nor feature learning allow the circumvention of the fundamental constraints imposed by this tradeoff
5.2. Intermediate Feature Learning Optimizes the Stability Tradcoff
When considering the overall final performance, trade-off between LA and CF naturally arises, commonly referred to as the plasticity-stabilii dilemma. Consolidated evidence in Deep Learning proves that higher degrees of feature learning help fit a task better (Fig, la), thus lifting the LA. However, our results so far have shown that increased feature learning also causes increased forgetting. Therefore we ask, which degree of feature learning achieves the optimal trade-off between plasticity and stability?
5.3. Discussion on Depth Scaling
Mirzadeh et al. (2022a;b) show experimentally that depth, unlike width, worsens the performance of the model in CL. We can interpret this result in light of our findings on the relationship of feature learning and CF. In fact, Bordelon & Pehlevan (2022) proved that in NTP and jP depth seal- ing induces evolution of the NTK, therefore increasing the amount of feature learning. According to the results pre- sented in this Section, increased feature learning comes at the cost of higher forgetting. Bordelon et al. (2023) and Yang et al. (2023) recently introduced a modified μP specif- ically for depth scaling, which allows scaling the network depth boundlessly while maintaining a tethhinndepeneent level of feature learning. In App. B.2, we perform scaling
In Figures 1b and 4a we plot the average performance as a function of Yo in our Split-CIFAR (ResNet) and Permuted- MNIST (MLP with finite and infinite widths) experiments, respectively. The optimal performance is achieved at inter- mediate and relatively low "oovelu %0 Ru 0.1 for both benchmarks. Surprisingly. vú transfers across widths, which means that one can determine % by tuning at low widths to then scale up without additional costs. A similar transfer of other hyperparameters has been observed in the scaling lim-
7

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
experiments in depth with this modified parameterization. Our evidence, although of preliminary nature, indicates that once again scaling in this rich regime does not reduce (nor increase) CF. Nevertheless, we leave a thorough char- acterization of this aspect for future study.
of non-stationarity directly impacts the amount of feature evolution: higher task similarity induces lower feature evo- lution across all "oo values?. ?. In other words, the lazy-rich transition happens at a higher Yo for more stationary data streams, and thus the degree of feature learning % which trades off optimally LA and CF moves closer to 1 (Fig. 5c), Thus in the stationary limit, the maximal performance reached for %0 1.
Features Evolution
0.15
0.8
0.7-
-CKA)
0.5
0.10
0.5
0.05
0.3
0.4
Fonturees Exulution
2122
0.6
3/31
202
0.2
(1 CCK
0.15
0.5
5/98
2000
0.1-
0.4
0.00
10
10
0.0
10
0.10
0.3
0,85-
0.2
(a)
(b)
1.0
0,00
0.0
10
10
0.7
70
Average Error
0.6
0.8
(a)
(b)
0.5
0.41
1.0
Other Tacks Evolation
0.25
52
0.3
0.9
0.2
0.8
(1.00
0.1
0.2
0.7
Coon.
0.15
0.0
0.0
20.5
0.10
10"
0.3
5/2
To
0.3
[00G
(c)
0.2
6.1
1/01
(.00)
11.5
0.000
0.8
Figure 5. Results on the synthetic dataset PeRMUted-MNIST with varying levels of task similarity; a ResNet with a width of 4096 is used. (a) The evolution features is modulated by the task simi- larity for fixed yo: LLT shifts towards for higher task similarity. (b) Following the shift of the lazy-rich transition, forgetting sharply increases a later "u for higher task similarity levels. (c) The higher the task similarity, the more the optimal 7 shifts towards I.
0.0
10
10
10
Tiak Evolution (1 Cos. Stm.)
(c)
(d)
Figure 6. Results on Split-Tiny Tinylmagenet with varying number of tasks and classes per task (e.g. 5/2 signifies 5 tasks of 2 classes each); ResNet with width of 1024 is used. (a) The task similar- ity shifts the lazy-rich transition. (b) CFr for varying Yo: for many classes per task (5/40, 20/10), the CFr follows a non-monotonic relationship with yo. (c) The optimal level of feature learning nō shifts towards I with the task similarity, controlled by the num- ber of classes per task. (d) Features evolution during the first, and later tasks with varying number classes per task, on the Split-Tiny Tinylmagenet dataset. When the tasks have many classes, the evolution of features on later tasks is constant in "/o suggesting features reuse and pretraining effect.
6. Interpolating Between Stationary and Non-Stationary Data
In the previous sections, we have shown that what holds true for the stationary setting (that feature learning is beneficial for performance) does not hold true for the non-stationary setting. In this section we consider these two separate set- tings as two extremes of a continuous spectrum, directly linking our findings with consolidated knowledge. In par- ticular, interpolate between stationary and (fully) non- stationary scenarios by manipulating the data distribution.
6.2. The Pretraining Effect
Secondly, we control the non-stationarity in the TinyIma- genet dataset by varying the number of classes per task (and the number of tasks) in the benchmark. We use the notation t/c for setting with t tasks of e classes each. Intuitively, increasing the number of classes per task enhances data di- versity within each task, leading to greater overlap between task distributions and thus higher similarity between tasks.
6.1. Task Similarity, Lazy-Rich Transition and %
First, we control the non-stationarity in Permuted MNIST by varying the permutation size (see App. A.4). We measure the task similarity P E [0, 1] as proxy for stationarity: the lower the greater the non-stationarity.
Consistently with the evidence from Permuted MNIST, we observe higher feature evolution when the number of classes
Analyzing the levels of features' evolution for varying de- grees of task similarity (Fig. 5a) we observe that the amount
Ass consequence, the assumption of lazy training dynamics in CL. does not hold equally for all datasets and benchmarks.
8

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
per task is low and so is the task similarity (Fig. 6a). This confirms that varying the number of classes per task controls the amount of non-stationarity, validating our methodology. Similarly to before. the increased task similarity allows a 76 R 1 to be optimal.
this context, our results can be inserted in the recent line of work of zero-shot hyperparameter transfer (Yang et al., 2022a; 2023; Bordelon et al., 2023; Lingle, 2024; Bjorck et al., 2024). Future work might also investigate mitigation methods (like experience replay (Chaudhry et al., 2019) and EWC (Kirkpatrick et al., 20)7 and their scaling proper- ties. this setting, devising parameterizations that achieve well-defined feature learning limit can lead NNs that better navigate the plasticity-stability frontier, with the end goal of achieving maximal feature learning without forget- ting at scale.
Intriguingly, in this setting alone we observe a non- monotonic relationship between the degree of feature learn- ing and CF the higher end of the range of non-stationarity tested (Fig. 6b). In fact, for these high-similarity tasks, in- creasing feature learning beyond a certain threshold appears to decrease the CFr. We hypothesize that this effect occurs when the features learned on the first task are useful for later tasks as well, thereby attenuating the amount of feature evolution in later tasks. We call this effect the pretraining effect, implying that the first task training acts similarly to pretraining, by finding features that transfer to later tasks.
Acknowledgements
AB would like to thank Marco Baiesi for helpful feedback and comments. LN would like to acknowledge the support of Google PhD research fellowship. GL would like to acknowledge the support of the AI Center PhD fellowship.
To verify this hypothesis we compare the evolution of fea- tures before and after training on a certain task (with cosine similarity). We differentiate the evolution during the first task, from that of later tasks (Fig. 6d). We observe that the feature trajectories overlap at lower values of "Yu for the different combinations of classes per task. However, they separate for higher values of "/o in the 5/2 curve the evolu- tion during the first task is proportional to that of later tasks, whereas at the other extreme of the task similarity, the later tasks exhibit no further feature learning beyond a certain "r- In other words, the network switches to an effectively lazy regime after the first task. Additionally, we note that width is strictly beneficial both in terms of forgetting and final performance when there is the pretraining effect (App. B.6), reproducing the results of Ramasesh et al. (2022) on the entangled relationship between scale and pretraining.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang, R. On exact computation with an infinitely wide neural net. Advances in neural information process- ing systems, 32, 2019.
7. Conclusions
Atanasov, A., Meterez, A., Simon, J. B., and Pehlevan, C. The optimization landscape of sgd across the fea- ture learning strength. arXiv preprint 27X00:2410.04662. 2024.
Our work takes a decisive step towards understanding the roles of feature learning and scale in continual learning, as well as their interplay with task similarity. We show that there is no intrinsic benefit of scale and that the degree of feature learning is ultimately responsible for forgetting. We observe a non-linear transition in the relationship between feature learning and the parameterization factor Yo- We extend DMFT in infinitely wide NNs to the treatment of multiple tasks. Our theoretical simulations confirm that the phenomena observed at finite widths are still valid at infinite widths. Finally, our findings add to the evolving perspective on feature learning in modern NNs, challenging the common "more feature learning is better" narrative, and underscoring the importance of modeling non non-stationarity.
Bennani, M. A., Doan, T., and Sugiyama, M. Generali- sation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006.1 1194, 2020.
Bjorck, J., Benhaim, A., Chaudhary, V., Wei, F., and Song, X. Scaling optimal Ir across token horizons. arXiv preprint (tti:22009.99111. 2024.
Bordelon, B. and Pehlevan, C. celf-consistent dynamical field theory of kernel evolution in wide neural networks. Advances in Neural Information Processing Systems, 35: 32240-32256, 2022.
We foresee that our findings could be of practical guidance: the optimal degree of feature learning 70 can be runed only for small models and then transferred to large scales. In
Bordelon, B., Noci, L.,, Li, M. B., Hanin, B., and Pehle- van, C. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, In The Twelfth

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
International Conference on Learning Representations, 2023.
Goldfarb, D. and Hand, P. Analysis of catastrophic for- getting for random orthogonal transformation tasks in the overparameterized regime. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Leurning Reseurch, pp. 2975-2993, PMLR, 25-27 Apr 2023. URL httptttt/poocceeeeeeeee v206 http:/oolfaabbaaaaaaoomm htmml.
Bordelon, B., Chaudhry, H. T., and Pehlevan, C. Infinite limits of multi-hean transformer dynamics. arXiv preprint arXiv:2405. 157/2, 2024.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K., Torr, P. H., and Ranzato, M. On tiny episodic memories in continual learning, arXiv preprint arXiv: 10486 2019.
mlr press/
Goldfarb, D., Evron, I., Weinberger, N., Soudry, D., and HAnd, P. The joint effect of task similarity and over- parameterization on catastrophic forgetting an an- alytical model. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / /ppenreview. net Forum?iiee forum?id=33dh128700
Chizat, L., Oyallon, E., and Bach, F. On lazy training in differentiable programming. Advances in neural informa- tion processing systems, 32, 2019.
Cohen, J., Kaur, S., Li, Y., Kolter, Z., and Talwalkar, A. Gradient descent on neural networks typically oc- curs the edge of stability. In International Confer- on Learning Representations, 2021. https: //openreview www.vviieenn.tteteeeee wwwn/et//oouue ?id= wwteeniidddddddeeeeeee hhhh-rTtkkGe...
Gong, D., Yan, Q., Liu, Y., Hengel, A. d., and Shi, J. Q. Learning bayesian sparse networks with full ex- perience replay for continual learning. arXiv preprint UrXiv:2202.10200 10203, 2022.
Damian, A., Nichani, E., and Lee, J. D. Self- seff-stabilization: The implicit bias of gradient descent at the edge of stabil- ity. arXiv preprint arXiv:2209. 15594, 2022.
Hadsell, R., Rao, D., Rusu, A. A., and Pascanu, R. Em- bracing change: Continual learning in deep neural net- works. Trends in Cogmitive Sciences, 24(12):1028-104)
2020. doi: 10. /tt.////.icc..22.......
URL https: ///oi. org. 10. 10w11111611.. httpg///...1///.iiicee.. ticisstiii..... 2020 09. 004.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars. T. A continual learning survey: Defying forgetting in classifi- cation tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366 3005,33355222 2021.
Hanin, B. Which neural net architectures give rise to ex- ploding and vanishing gradients? Advances in neural information processing systems, 31, 2018.
Hanin, B. and Nica, M. Finite depth and width cor- rections to the neural tangent kernel. arXiv preprint 2019.
Doan, T., Bennani, M. A., Mazoure, B., Rabusseau, G., and Alquier, A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference Artificial Intelligence and Statistics, pp. 1072-1080. PMLR, 2021.
Hanin, B. and Rolnick, D. How to start training: The effect of initialization and architecture. Advances in neural information processing systems, 31, 2018.
Evron, I., Moroshko, E., Ward, R., Srebro, N., and Soudry, D. How catastrophic can catastrophic forgetting be in linear regression? In Conference on Learning Theory, pp. 20784079. PMLR, 2022.
Hayou, S., Clerico, E., He, B., Deligiannidis, G., Doucet, A., and Rousseau, J. Stable resnet. In International Conference on Artificial Intelligence and Statistics, pp. 1332- PMLR, 2021.
Fort, S., Dziugaite, G. K., Paul, M., Kharaghani, S., Roy, D. M., and Ganguli, S. Deep learning versus kernel learning: an empirical study of loss landscape geome- try and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33: 5850-5861, 2020.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Ingrosso, A., Pacelli, R., Rotondo, P., and Gerace, F. Sta- tistical mechanics of transfer learning in fully-connected networks in the proportional limit. arXiv preprint arXiv:2407.07168, 2024.
French, R. M.
Catastrophic forgetting in connec- tionist networks. Trends in Cognitive Sciences,
3(4):128-135. 1999. ISSN 13646613. doi:
10.1016/S1364 10110066S3344-6663000992942
URL https:
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. 31, 2018. URL https: https::/troootteee http////ooooii/eeeee/c ngs. neurips.
10

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
cclpappe__flll//ppper/////////
Lin, S., Ju, P., Liang, Y., and Shroff, N. Theory on for- getting and generalization of continual learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learn- ing, volume 202 of Proceedings of Machine Learn- ing Research, pp. 21078-21100, PMLR, 23-29 2023. URL https: http://pocceeeeieeeeeeeee/eessee mllr. v202/1in232 html.
5a4bel fa3.ee2bb8a pdf.
Ke, Z. and Liu, B. Continual learning of natural lan- guage processing tasks: A survey. arXiv preprint 1rXiv:2211.12701, 2022.
Khetarpal, K., Riemer, M., Rish, I,, and Precup, D. To- wards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, 2022.
/
mar.sssssss.p.sss////.. press/
Lingle, L. A large-scale exploration of p-transfer. arXiv preprint arXiv:2404.0577 2024.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J.,
Desjardins, G., Rusu, A. A., Milan, K., Quan, J., A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcom- ing catastrophic forgetting in neural networks. Pro- ceedings of the National Academy of Sciences, 114 (13):3521-3526, March 2017a. ISSN 1091-6490. doi: 10. 00.1077/pnas. http:/1077/paaa.111111...
Luo, Y., Yang. Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint arXiv:2308.08743 2023.
Ramalho, T.,
Mai, Z., Li, R., Jeong, J., Quispe, D., Kim, H., and Sanner, S. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28-51, 2022.
URL http: / dx.ddii.....oooooooooo..dddd..dd
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3351 3526, 2017b.
Martens, New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1-76, 2020.
Martens, Ballard, A., Desjardins, G., Swirszez, G., Dal- ibard, V.
J., and Schoenholz, S. S. Rapid training of deep neural networks without skip connections or normalization layers using deep kernel shaping. arXiv preprint arXiv:2 10.00166.. 2021.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Sim- ilarity of neural network representations revisited. In International conference machine learning, pp. 3519- 3529. PMLR. 2019.
Martin, P. C., Siggia, E. D., and Rose, H. A. Sta- tistical dynamics classical systems. Phys. Rev. A, 8:423 -37,,437--0 Jul 1973. doi: 10.1103/PhysRevA. 8.423. URL http://///iikka...oo.oo....
Kumar, T., Bordelon, B., Gershman, S. J., and Pehlevan, C. Grokking the transition from lazy to rich training dynamics. In The Twelfth International Conference Learning Representations, 2024.
1
Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Penning- ton, J., and Sohl-Dickstein, J. Deep neural networks as gaussian processes. arXiv preprint arXiv: /71 17111.00000.5 1.00 2017.
McCloskey, M. and Cohen, N. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pp. 109-165. Academic Press, 1989. doi: https//bbioog/lo..o.ssss777//2/2 URL https://www.cciennceeeeee ccckeee//rrtaaee////////77221222
Lee, J., Xiao, I.,, Schoenholz, S., Bahri, Y., Novak, R., Sohl- Dickstein, J., and Pennington, J. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems. 32, 2019.
Mirzadeh. S. I., Farajtabar, M., Pascanu, R., and
Ghasemzadeh, H. Understanding the role of training regimes in continual learning. Advances in Neural Infor- mation Processing Systems, 33:7308-7320, 2020.
Lesort, T. Lomonaco, V., Stoian, A., Maltoni, D., Filliat, D., and Díaz-Rodríguez, N. Continual learning for robotics: Definition, framework, learn- ing strategies, opportunities and challenges. Infor- mation Fusion, 58:52 - 68, 2020. ISSN 1566- 2535. doi: http::/doo.ogg//0..01j/j/n.tu.tu
Mirzadeh, S. I., Farajtabar, M., Gorur, D., Pascanu, R., and Ghasemzadeh, H. Linear mode connectivity in multi- task and continual learning. In The Ninth International Conference on Learning Representations (ICLR 2022), 2021.
004. uttttp::://wwwcceeeeedeeeettt.. hrtaace/arrcceee///////662133199

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
Mirzadeh, S. I., Chaudhry, A., Yin, D., Hu, H., Pascanu, R., Gorur, D., and Farajtabar, M. Wide neural networks forget less catastrophically. In International conference on machine learning, pp. 15699-15711
Sagun, L.,, Evci, U., Guney, V. U., Dauphin, Y., and Bottou, L. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv: 7706, 706.04454, 2017.
PMLR, 2022a.
Schoenholz, S. S., Gilmer, J., Ganguli, S., and Sohl- Dickstein, J. Deep information propagation. arXiv preprint 161102221 2016.
Mirzadeh, S. I., Chaudhry, A., Yin, D., Nguyen, T., Pascanu, R., Gorur, D., and Farajtabar, M. Architecture matters in continual learning. arXiv preprint arXiv:2202.00225. 2022b.
Shan, H., Li, Q., and Sompolinsky, H. Order parameters and phase transitions of continual learning in deep neural networks. arXiv preprint ar 2rXiv:2407.1033
Mori, F., Mannelli, S. S., and Mignacco, Optimal pro- tocols for continual learning via statistical physics and control theory. arXiv preprint UrXiv:2409.18066 2024.
2024.
Silver, D. L.., Yang, Q. and Li, L.. Lifelong machine learning systems: Beyond learning algorithms. In 2013 AAAI spring symposium series, 2013.
Neal, R. M. Bayesian Learning for Neural Networks. Lec- ture Notes in Statistics. Springer New York, New York, NY, 1996.
Thrun, S. and Mitchell, T. M. Lifelong robot leam- ing. In The biology and technology of intelligent autonomous agents, pp. 165-196. Springer. 1995. URL ist psu. psu. eduvviewdoc/
Noci, L.,, Bachmann, G., Roth, K., Nowozin, S., and Hof- mann, T. Precise characterization of the prior predictive distribution of deep relu networks. Advances in Neural Information Processing Systems, 34:20851-20866 2021.
http::/citesse download?dei=100 1.1. 1. 11..1..1.......... type=pdf.
Vyas, N., Atanasov, A., Bordelon, B., Morwani, D., Sainathan, S., and Pehlevan, C. Feature-leamiing net- works are consistent across widths at realistic scales. Ad- vances in Neural Information Processing Systems, 36, 2024.
Noci, L., Meterez, A., Hofmann, T. and Orvieto, A. Super consistency of neural network landscapes and learning rate transfer. In The Tiiht Annual Conference on Neural Information Processing Systems, 2024. URL
n net/forum? forum?
id=rgwhJ7INt2.
Wang. L... Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
OpenAL. Gpt-4 technical report. arXiv:2303.08774, 2023.
arXiv preprint
Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. Continual lifelong learning with neural networks: A review. Neural networks, 1 113:54-71, 2019.
Wenger, J., Dangel, F., and Kristiadi, A. On the disconnect between theory and practice of overparametrized neural networks. arXiv preprint arXiv:2310.00113,, 2023.
Petrini. L., Cagnetta, F., Vanden-Eijnden E., and Wyart, M. Learning sparse features can lead to overfitting in neural networks. Advances in Neural Information Processing Systems, 35:9403 1594003-9416,, 2022.
Wu, T., Luo, L., Li, Y.-F., Pan, S., Vu, T.-T., and Haffari, G. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01.: 01364, 2024.
Yang. G. Tensor programs ii: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006. /4548, 2020.
Ramasesh, V. V., Dyer, E., and Raghu, M. Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint uxXiv22007.07400, 2020.
Yang, G. and Hu, E. J. Feature learning in infinite-width neural networks. arXiv preprint trXiv:2011.14522 2020.
Ramasesh, V. V. Lewkowycz, A., and Dyer, Effect of scale catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022. URL https:
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X.. Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.084. 03466,, 2022a.
id=Ghvsa_yFeEa.
Ring, M. Continual learning in reinforcement environ- ments. PhD thesis, USA, 1994.
Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint A0l::2003003466 2022b.
Roberts, D. A., Yaida, S., and Hanin, B. The principles of deep learning theory. volume 46. Cambridge University Press Cambridge, MA, USA, 2022.
12

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.02244. 2023.
Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. Py- hessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference big data (Big data), pp. 581-590. IEEE, 2020.
Znnke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70, pp. 3987-3995. PMLR, 2017.
13

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
A. Experimental Details
A.1. Evaluation Metrics
We report here the formal definitions of the evaluation metrics introduced in Sec. 3.1.1.
For benchmark with tasks, define the test accuracy on the task Tj after training on T,. with i,j E {I...., a,,s We the Learning Accuracy as measure of plasticity.
Definition A.1 (Learning Accuracy (LA)). The Learning Accuracy (LA) is the average over all tasks of the accuracy the task it has just been trained on:
T
LA
ai
(13)
1=1
Similarly, the complement of the LA is the Learning Error.
Definition A.2 (Learning Error (LE)), The Learning Error (LE) is the complement of the LA:
LE 1 - LA
(14)
Forgetting is traditionally measured as the average drop in accuracy. Definition A.3 (Catastrophic Forgetting (CF)). The Catastrophic Forgetting is the average drop in accuracy after training on all other tasks:
CF
max (ar.)) (6....... T-11
ar,
(15)
T
The limitations of the CF are more easily comprehended with simple toy example. Let consider two models trained benchmark with only two tasks: the first model reaches 100% accuracy on the first task, and after training on the second task drops to 80%, i.e. it has a CF of 20%. The second model, instead, reaches 40% accuracy on the first task, and after training on the second task it drops to 30%, i.c. it has a CF 10%. Looking only at the CF, the second model would be favored but, in reality, the relative drop accuracy higher for the second model. Therefore, we introduce a novel metric that measures the Catastrophic Forgetting rate.
Definition A.4 (Catastrophic Forgetting Rate (CFr)). The Catastrophic Forgetting Rate is the average relative drop in accuracy after training all other tasks:
1
max (i.... (at,s) ar,
CFr
(16)
T
max T (ar,s)
If we consider the CFr for the two models in the example above, we would see that the first model has CFr of 20%, while the second model has CFr of 25%, rightly favoring the first model.
Definition A.5 (Average Accuracy (AA)). The Average Accuracy is the average accuracy all tasks after all of them have been trained sequentially:
AA
(17)
T
1=0
Definition A.6 (Average Error (AE)). The Average Error is the complement of the AA.
AE = 1 - AM AA
(18)
We define also the same metrics in terms of the training loss, needed for the experiments with the infinite-width simulations. Concretely, for benchmark with T tasks, we define the training loss on the task T, after training on Ty. with i, E {1,..., T). as Lz
14

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
Definition A.7 (Learning Loss (LL)). The Learning Loss (LL) is the average over all tasks of the loss the task it has just been trained on:
LL
C.
(19)
Definition A.8 (Catastrophic Forgetting (CF) (loss)). The Catastrophic Forgetting in terms of loss is the average drop in loss after training on all other tasks:
T-1
CF (loss) =
max (Lr,) LT,
(20)
T (f.....T-1))
Definition A.9 (Catastrophic Forgetting Rate (CFr) (loss)). The Catastrophic Forgetting Rate in terms of loss is the average relative drop in loss after training on all other tasks:
(i... (Cr,,) LT. max -11 (Cr.i)
CFr (loss)
(21)
Definition A.10 (Average Loss (AL)). The Average Loss is the average loss on all tasks after all of them have been trained sequentially:
AL
CT
(22)
T Y=0
A.2. Model Architecture
We use an architecture of the ResNet family (He et al., 2016). particular, we first apply a convolutional layer with kernel size 7x7 and N filters with stride followed by Batch Normalization (BN), ReLU, and max pooling. Then, repeat times the following configuration: T./3 residual blocks (each with 1 convolutional layer with N channels, BN, and ReLUU,, apply a global average pooling, flatten the output, and apply linear
with the first block having stride of 2. Finally, layer. We will simply call N the width, and L the depth of the model. The base configuration uses N = 64 and = 6. For some of the CL datasets considered, separate classification head is used for each task (see Sec. A.4).
Experiments involving the widttite-width simulation are executed on 2-layer non-linear (ReLU) perceptron.
A.3. Parameterization Details
Note that many equivalent parameterizations can achieve the same functional behavior (Yang & Hu, 2020; Bordelon & Pehlevan, 2022; Yang et al., 2023; Bordelon et al., 2023). In Tab. I we report the notation of (Bordelon et al., 2023, Tab. 1), but with the NTP from (Yang & Hu, 2020, Tab. 1) for implementational simplicity. Also, note that the standard parameterization (SP) of PyTorch (i.e. the SP column in (Bordelon et al., 2023, Tab. 1)) is equivalent the NTP used here.
A.4. Datasets and Training Details
ResNet Experiments In all experiments if not otherwise specified we use Stochastic Gradient Descent (SGD) optimization without momentum nor weight decay. To find the learning rate (0), we do hyperparameter search on the model at base width and depth (i.e. where all parameterizations are equivalent) the full (i.e. non-CL) dataset taking the optimal test accuracy as metric. We use cosine learning rate schedule without warmup, restarting the learning rate at the beginning of each task. For all datasets, we use batch size of 128.
Infinite-Widde Experiments on 2-layer non-linear MLP In the infinite width limit simulations and finite-width compar- isons, due to limited computational resources, was used a small set of 30 samples of the MNIST dataset (see below), three per class. This is because simulations involving the theoretical limit scale with P2T2 in terms of memory and of p3T3 terms of time (Bordelon & Pehlevan, 2022), since kernels are of order p2T2 and undergo a natrix-vector product in
15

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
dimension P and integration over T time steps. The simulations rely on drawing the random initial conditions of the kernels, namely the random initial fields, from a sampled distribution of size S 3000 of the relative Gaussian Process describing their infinite width limit. From those, we implemented an Euler-based, discrete-time ODE solver obtain the dynamics of all the fundamental quantities describing the network, including kernels and residuals. The MLP is optimized with full batch Gradient Descent on MSE loss with a LR tuned with a grid search on the real network on the non-CI. data. To avoid the explosion of the initialization output at low values of To. we initialize the last layer of the MLP to to have a well-defined output of at / = 0. This practice is advised (Yang et al., 2022b, App. D.2).
All training runs and experiments are executed either on single NVIDIA GeForce RTX 4090 or on an NVIDIA RTX A6000.
A.4.1. SPLIT-CIFARIO
The Split-CIFARIO (Zenke et al., 2017) (TIL type of benchmark) dataset has 5 tasks of 2 classes each (i.e. the 10 classes of CIFARIO are split into 5 tasks with non-overlapping classes), and as common for TIL benchmarks, the model uses a separate head for each task. We train for 5 epochs on each task with a learning rate of (0) = 30.0. Note that the LR is larger than usual values; this is due to the particular choice of parameterization. If not otherwise specified, all experiments on Split-IFFRR1 10 are repeated 5 times with different random seeds, and the average is reported.
A.4.2. PERMUTED MNIST
Inspired by previous works (Goldfarb et al., 2024; Kirkpatrick et al., 2017b), we use the permuted input MNIST dataset with 5 tasks to investigate the impact of task similarity on CF. Specifically, each task of this benchmark consists of the MNIST dataset with a random but fixed permutation of the pixels.
In partieular, we consider task similarity as the fraction of pixels that are not permuted between tasks, and the inner square of the image is permuted first. As an example, task similarity of 1.0 corresponds to the original MNIST dataset; task similarity of 0.0 corresponds corresponds tasks where the middle square containing 50% of the pixels is permuted. These tasks are synthetic, and except for the untouched pixels, the tasks are not related to each other: this allows us to artificially cause CF by design.
dataset where each task permutes all pixels of the images; task similarity of 0.5
ResNet Experiments Each task is trained for 5 epochs with (0) = 2.0. We do a slight modification of the architecture presented above, by omitting batch normalization layers (following the observation of Mirzadeh et al. (2022b)). This is a DIL type of benchmark, where the model uses a single head for all tasks.
Infinite-Widhe Experiments on 2-layer non-linear MLP For this simplified setup, instead, we use a fixed task similarity of p = (all pixels permuted) and 2 tasks. We optimize utilizing small subset of training data comprising 30 samples (3 samples per label). Each task is optimized for 1000 epochs and fixed LR of njo 0.25.
A.4.3. SPLITTINYIMAGGNET
Similar to Split-CIFAR10, the classes of Tinylmagenel are split into tasks with non-overlapping classes (Gong et al., 2022) (i.e. is TIL benchmark and we use separate classification heads for each task). Throughout the experiments, we consider varying the number of tasks and classes-per-task: 5 tasks of 2 classes each (we denote it 5/2), 5/10, 5/40, as well as 20 tasks of 2 classes each (20/2), and 20/10. We optimize each task for 10 epochs with (0) = 15.0.
16

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
B. Additional Experiments
B.1. Duration of Training and Feature Learning
Motivated by the observations of Wenger et al. (2023), in Fig. 7 we investigate the effect of the training duration on feature evolution, forgetting, and performance. As shown by (Fort et al., 2020), we observe that longer training time increases the deviation from the lazy regime features evolve more (Fig. 7a). As thoroughly presented above, more feature evolution reflects also on increased forgetting (Fig. 7b). As observed in Wenger al. (2023), the benefit of width-scaling and thereof-induced laziness is reduced for longer training when the deviations from the NTK regime are significant. As done by To in the jP, the training duration also impacts the balance between plasticity and stability (Fig, 7c), inducing a critical and width-dependent training-duration that optimizes the Average Error (Fig. 7d).
4096
4086
0.4
0.4
2048
Features Evolution
2048
[1 CCK A)
0.3
1024
0.3
1024
512
0.2
512
0.2
256
256
0.1
0.1
128
128
0.0
64
0.0
10a
10
10a
10
Epochs
Epochs
(a)
4096
0.4
4096
0.2
Epachs
2048
2048
Leaming Error
Average Error
0.3
1024
1024
Width
0.1
0.2
512
256
256
0.1
Epochs 50
128
128
0.0
64
0.0
61
0.1
0.2
0.3
0.4
10a
10
CFr
Epochs
(c)
(d)
Figure 7. The effect of width and training duration with the NTP on Split-IIFRROO. The longer the training, the higher the feature evolution (a). Consequently, forgetting increases (b). Due to the plasticity-stability tradeoff (c), the Average Error shows a width-dependdnn optimal training duration (c).
B.2. Depth Scaling
The role of depth was analyzed in (Mirzadeh et al., 2022a;b), where was found that - in their setting - depth would not provide any benefit in terms of CF. With the precepts of scaling limits, and the observations done so far, we can interpret the reasons for this result: not suitably parameterized, the depth augments feature learning by stimulating the evolution of the NTK (Bordelon & Pehlevan, 2022).
This effect of depth has been recently and concurrently addressed by Bordelon al. (2023) and Yang et al. (2023); for notational simplicity, we will hereby consider the 1/11/ parameterization introduced in (Bordelon et al., 2023). This parameterization guarantees that the feature learning magnitude is decoupled from both width and depth. We have therefore a tool to disentangle CFr and the impact of depth, namely without the confounding effect of varying degrees of feature learning and laziness. Concretely, and following the notation in Tab. 1, pP+1/vII differs from uP in the branch scale 3,, which depends not only on the width N, but also on the depth I. as follows:
17

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
N N-1/2 1/2
1= (LN)-1/2
0<|<b.
(23)
D-1/2 1=0
We empirically verify this hypothesis, fixing the width at 64 and varying the depth of the model in the three parameterizations. Here we use "Yo = 1.0 for μP and μP+1/vL. Firstly we can see in Fig. 8 that the jdP and NTP are equivalent since they are at base-width and share the same depth-scaling properties. We observe that initially the CFr increases, and then the stability of the network decays, with the learning error increasing until random performance. This yields an average error that significantly degrades with depth the yP and NTP. Instead, the behavior of pP+1/VL is different: the CFr is stable with depth (similarly width for the μP), and the learning error is constant good performance. This yields average error that is stable across depths.
0.5
1.0
eP
AP
NTP
0.9
NIP
0.4
PP4I/VE.
Leaming Ervor
0.8-
PP41/V/
0.7-
0.3
0.6
0.5
0.2
0.4
0.3
0.1
0.2
0.1
0.0
0.0
6
12
24
48
96
192
384
12
24
48
96
192
384
Depth
Depth
(a)
(b)
1.0
0,9
X
Average Error
0.8
PP41/vE
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
48
192
384
Depth
Figure 8. Performance the three parameterizations with varying depth and fixed width 64, The pP and NTP are equivalent since the experiment is at basc-width and they share the same parameterization w.r.t. the depth; 7o 1. (a) CFr; (b) learning error; (c) average error.
Finally, repeat the experiment varying the degree of feature learning at different depths; we fix the width at 512 and vary the depth of the model for various "Y values. Firstly, we notice that feature learning negatively impacts CFr for all depths in both yP and /P+1/v/ /T. (Fig. 9 and Fig. 10, respectively). Furthermore, the curves for juP highly vary for different depths, namely we find a depth- deeeth-dependentttt 7 (Fig. 9). The curves for nP+1/vIL LL, instead, are extremely stable for depths, and in particular, the % is constant across depths (Fig. 10): the optimal 7 = 0.1 transfers not only across widths (as found in Sec. 5) but +P++1/ S//L.. also across depths (as shown by previous works for non-CL hyperparameters (Yang et al., 2023; Bordelon al., 2023)).
Curiously, on the CIFA-II considered here, depth does not provide any plasticity improvement (i.e. learning error). We hypothesize that this is due to the simplicity of the CIFAR10 dataset, where the model can easily learn the task at hand with shallow architecture, and where the additional depth does not provide any benefits. Evaluations on more challenging datasets might provide further insights. However, a thorough investigation of the role of depth out of the scope of this work, and we leave further experiments future work.
18

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
0.5
384
0.9
384
0.8
192
192
0.4
Leaming Error
0.7
96
0.6
96
0.3
Deptli
0.5
48
48
0.4
0.2
24
0.3
24
0.1
-12
0.2
12
0.1
0.0
0.0
10
10
10"
10
10-2
10
P00
(a)
0.8
384
0.7
192
Average Erro
0.6
96
0.5
0.4
48
0.3
24
0.2
0.1
0.0
10
10
10"
(c)
Figure 9. Depth scaling of the μPP, with a fixed width of 512 and various degrees of feature learning. (a) CFr; (b) learning error; (c) average сттоr.
19

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
0.5
384
0.5
384
192
192
0.4
Learning Frror
0.4
96
96
CF:
0.3
Dep
0.3
48
48
0.2
0.2
24
24
0.1
12
0.1
0.0
10
1
10
10"
0.0
10
10
10
10'
70
70
(a)
(b)
0.5
384
192
0.4
Average Error
96
0.3
0.2
24
0.1
12
0.0
10
10"
710
(c)
Figure Depth scaling of the /P41/ LVT, with a fixed width 512 and various degrees feature learning. (a) CFr; (b) learning error; (c) average error.
20

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
B.3. Additional Experiments with the InfiniteWWidth Limit
Here report additional experiments on the MLP with the infinite-width limit simulation.
In Fig. 11 we report the curves of CFr with varying "o for finite and infinite width. We recover also with this setup, the characteristic relationship observed in Fig. 1c and analyzed in Sec. 5.1. Furthermore, we note that in this case too, the lazy-rich transition seems to differentiate two different width-scaling regimes, where for To < %0 LLt' the infinite-width limit is approached from above and thus width scaling reduces forgetting (Fig. l1b), while for To > ~00 the infinite-width limit is approached from below and thus width scaling increases forgetting (Fig. 11c). We leave the characterization the finite-width width convergence the infinite limit for future work.
4096
10"
N + oz
10
CFr (loss)
10a
1024
512
10
10
256
10
128
10
10
10
10
10
64
%0
(a)
10P
N -
4006
N + 00
1096
2048
10a
20488
CFr (loss)
1024
CFr (loss)
1024
512
512
256
256
128
128
10
10
10
64
10
10-1
10a
64
70
%
(b)
(c)
Figure 11. (a) Finite infinite-widiie relationship between Too and CFr (loss). Zoomed in versions for (b) low To and (c) high 7o-
In Fig. 12 we recover even for the MLP and infinite-width simulation the plasticity-stability tradeoff (Fig. 4b). This further suggests that even at the infinite-widee limit NNs (trained with SGD) suffer from the plasticity-stabiliy tradeoff.
B.4. Features Evolution at finite widths
In Fig. 13 we report the counterpart of Fig 3a for varying finite widths.
B.5. Loss Landscape Analysis
Motivated by the theoretical and empirical verifications of Mirzadeh et al. (2020), uncovering the crucial role of the geometrical properties of the landscape for forgetting, we investigate its peculiarities across parameterizations and scaling the width. This approach has also proven to be important tool uncover networks' behavior in multiple studies (Sagun et al., 2017; Martens, 2020). In particular, Noci et al. (2024) has recently uncovered crucial differences in the loss landscape of the NTP and μ.P.
The sharpness of the landscape, i.e. the maximum eigenvalue of the Hessian, is a crucial property of the loss landscape. Recently, Cohen et al. (2021) have observed an intriguing mechanism when optimizing deep networks with (full-batch) gradient descent: the models exhibit rapid increase in sharpness towards critical point. This increase is termed progressive
21

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
4096
2,5
2048
0.001
2-
1024
Learning
1.5
256
0.5
70 =
0.2
128
10
10
10
10a
10a
10r
10a
CFr (loss)
(a)
Figure 12, Plasticity (Learning Loss) stability (CFR (loss)) tradeoff with finite and infinite-wide width simulation.
Features Evolution (1 CKA
4096
0.6
2048
0.5
1024
0.4
Width
512
0.3
0.2
255
0.1
128
0.0
64
10-2
10-1
10
(a)
Figure 13. The lazy-rich transition: above a certain threshold of 70 the evolution sharply increases consistently for all widths (ResNet model on Split-CIFAR10).
22

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
sharpening, and the critical point is the Edge of Stability (EoS). Interestingly, the dynamics of the model do not diverge even beyond the EoS, thanks mechanism of seff-Stabilization (Damian et al., 2022). These observations partly apply also to our case of mini-batch stochastic optimization (Cohen et al., 2021; Noci et al., 2024).
We define formally the sharpness as follows:
Definition B.1 (Sharpness). The sharpness of the loss landscape at a point W is defined as the maximum eigenvalue of the Hessian of the loss function at W:
Sharpness W) = Amaax (V² (V2L(W)) = Amax (H()).
(24)
We will abbreviate the sharpness as Amar-
The EoS threshold arises from the convergence requirement for the gradient descent optimization with learning rate nj of a quadratic function, namely that the sharpness of the Hessian should not exceed 2/m)
As in yP the learning rate 77 scales with the width, the standard sharpness of Def. B.1 is hardly representative of the actual model behavior. In particular, as the LR increases - by the EoS the sharpness of the landscape would decrease proportionally. However, from a practical perspective and from the point of view of optimization, the lower sharpness and the proportionally higher learning rate would compensate for each other. Intuitively and qualitatively, the valley of the optimization becomes flatter, but the velocity of the optimization increases, yielding a constant effective sharpness. To compensate for this effect, we introduce the effective sharpness as a scaled sharpness.
Definition B.2 (effective Sharpness). The effective sharpness for model parameterized with the μPP in Tab. 1, is defined as
effective Sharpness( W) = )max(?²² -H(H()))))
(25)
We will abbreviate the effective sharpness as Åmaz. For μP, we will often refer to the effective sharpness as the sharpness for simplicity.
Crucially for our analysis, Noci et al. (2024) have shown that the NTP and μP have different (effective) sharpness properties when scaling the width: μP exhibits consistent effective sharpness at all widths, while for the NTP the sharpness dynamies are width-dependent, and are characterized by the inversely proportional relationship between sharpness and width (for short enough training). With these resulting differences, we study the landscape in the context of CL for the two parameterizations.
The trace of the Hessian provides another meaningful perspective on the curvature of the loss landscape. However, comparing the trace across models of varying dimensions (e.g., as we scale their widths) is inherently challenging, as it represents a sum over an increasing number of terms. One possible approach is to normalize the trace by the number of model parameters. However, this metric can become unreliable when the number of zero eigenvalues in the Hessian scales with the parameter number, causing the normalized trace to decrease regardless of the behavior of the non-zere eigenvalues. To date, the literature on experimental analysis of loss landscapes has not resolved these challenges, and our results should therefore be considered preliminary.
Definition B.3 (Normalized Trace and effective Normalized Trace). For W E RRII, the normalized trace of the Hessian is defined as
1
1 W
Normalized Trace(W) [W tr())
tr (r().)).
(26)
The effective normalized trace for model in the μP (Tab. 1) is defined as
effective Normalized Trace W) = [W H(W)).
(27)
We analyze multiple compound metrics of the landscape based on these fundamental quantities on Split-CIFARIO. Firstly, we observe the average effective sharpness and normalized trace at convergence, i.e. the average of these quantities at the end of training on each task and with the data of that same task. Lastly, we want to gain an insight into the multi-task loss landscape, i.e. of the compound loss of all the tasks. To do so, we measure the average effective sharpness and trace at the end of training on each task, but using data from all previous tasks.
The sharpness and normalized trace of the Hessian are calculated using one batch of training data with the PyHessian library (Yao al., 2020).
23

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
B.5.1. To MODULATES THE LANDSCAPE
Firstly, we observe that the average sharpness of the converged models is modulated by "o but, in contrast to the findings of Noci et al. (2024), the width has larger impact on the converged sharpness hinting at the complex influence that learning chain of tasks has the optimization properties (Fig. 14a). A change of regime is clearly distinguishable at To = 0.1. Similar behaviors are also observed for the average sharpness of the multi-task optimization (Fig. 14b), shedding an important insight for our CF problem: higher curvature of the landscape is observed for higher feature learning, meaning that the multi-task performance is more sensitive to changes in the network parameters.
4096
4096
2048
2048
Convergee Amar
0.3
Multi-Task Amas
0.3
1024
1024
Widtt
Multi-1 Task
0.2
512
0.2
512
256
256
128
128
64
64
10
10-2
10
10
10
10
10
10"
70
70
(a)
(h)
Figure 14. Average effective sharpness in the μP for varying %o and width. (a) Converged; (b) multi-task.
second insight gained by analyzing the effective normalized trace of the Hessian, finding that is modulated by both to and width in non-trivial fashion (Fig. 15) and with clear phase transition at % 0.1. This is particularly pronounced for the multi-tasks trace: for low " the trace is positive and small (signalizing a convex and wide valley), and "/o As 0.2 breakdown occurs, with the trace becoming negative and large the wider the model. This means that the landscape becomes non-convex and with high curvature, i.e. for the multi-task objective the optimization reaches a non-minimum point in the landscape (a saddle point or even a maximum). This is yet another hint at the ill-conditioning of the landscape for higher feature learning in CL and a potential cause for the increased forgetting observed for higher "n- Furthermore, this seems to indicate that higher widths worsen the bad conditioning of the landscape, possibly hinting a reason for the negative effect of width on CFr observed in Sec. 5.1 (Fig. Ic) for % 70 L.T
Convergod Normalized (i(H))
10-5
Multi- Task Normalized (r(H))
10
1096
10-6
2048
10
2048
10
10-7
1024
Width
10-8
1024
10
512
10-
512
256
-10-
10-10
128
-10-8
128
10-11
-10-7
10-3
10
10
10
10-8
10-2
10
10P
70
770
(a)
(b)
Figure 15. Average effective normalized trace in the μP for varying to and width. (a) Converged: (b) multi-task.
Lastly, we want to stress the notable consistency of the dependency between To and the landscape properties across widths, meaning that the landscape properties vary similarly with To for all widths. Since this super-consistency is at the root of the LR transfer properties of μP (Noci et al., 2024), we hypothesize that this might also be a reason for the transfer of the optimal % across widths and depths observed above. Nevertheless, we leave thorough investigation the transfer properties 7 for future work.
24

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
B.5.2. FORGETTING CORRELATES WITH THE LANDSCAPE SHARPNESS
Mirzadeh et al. (2020) has first hypothesized the crucial role of the curvature of the loss landscape for the CF problem. Relying on simple second-order Taylor expansion of the loss function at the minimum of the optimization, they derived bound on the CF proportional to the curvature of the loss landscape. Although this bound relies on assumptions the well- behavedness of the loss function, it was validated empirically in variety of settings, showing clear correlation between the curvature of the loss landscape and CF. Furthermore, Mirzadeh et al. (2021) have further analyzed this correlation in the context of linear mode connectivity, finding a causal relationship between the curvature of the loss landscape and the CF within a region where the approximation holds. However, their bound is vacuous if there is arbitrary displacement of the parameters, namely when the second-order local approximation is not valid. This is indeed the case for the juP, and thus it priori unclear if their hypothesis regarding the curvature of the landscape holds for the y.P. We empirically investigate how the landscape curvature correlates with the CF in the μP and we find clear correlation between landscape curvature and forgetting (Fig. 16). We stress that it is noteworthy that we find this correlation even in the case of varying scales, and training regimes, thus without any assumptions on the locality of the optimization or the linear connectivity of the landscape.
0.5
4096
0.5
10a
2048
0.4
0.4
1024
10-1
0.3
0.3
512
0.2
0.2-
256
10-2
0.1
0.1
0,0
64
10
0.00
0.05
0.10
0.15
0.20
0.25
0.00
0.05
0.10
0.15
0.20
0.25
Multi Task Aunia
Multi Task Amaz
(a)
(b)
Figure 16. Multi Task sharpness and its correlation with CFr JPP..
B.6. Width Dependency on Split-Tiny Imagentt
In this section, report the width-deppendncc of the Split-Tinylmagenes results presented in Fig. 6. In particular, in Fig. 17 we observe the width dependency for the 5/2 (5 tasks of 2 classes each), finding similar results to the ones Split-CIFAR1O, width does not mitigate forgetting, and the optimal n0 for the AE is at an intermediate level which transfers across widths. In Fig. 18, instead, we see the effect of width on the 5/40 case where the pretraining effect occurs (Sec. 6.2). In particular, all widths exhibit the non-monotonic relationship of CFr with "o (characteristic of the pretraining effect), and an optimal to = 1 for the AE. Interestingly, we also find that the width is beneficial not only in terms of Average Error but also in terms of forgetting. This recovers a property of pretraining already observed by Ramasesh al. (2022).
B.7. CNN Model
Here, were report additional experiments executed without skip connections of the ResNet model, namely including a simpler Convolutional Neural Network (CNN) Model. Comparing these results (Fig. 19) with those on the ResNet model (Fig, 1) we observe that the results are on-par.
25

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
4096
0.6
0.4
2048
0.5
2048
Average Error
0.3
1024
0.4
1024
CPt
Width
0.3
0.2
256
0.2
256
0.1
128
0.1
128
0.0
10
10-2
10-1
10"
10
10-2
10
10'
(a)
(b)
Figure 17. The role of width and Split-Tinylmagent with 5 tasks of 2 classes each. (a) CFr; (b) average emoor.
4096
4096
1.0
1.0
0.9
2048
0.9
2048
0.8
0.8
0.7
1024
0.7
1024
0.6
Width
0.6
Width
0.5
512
Average
0.5
512
0.4
0.4
256
256
0.3
0.3
0.2
0.2
0.1
0.1
0.0
64
0.0
61
10-2
10
10
10
Ya
700
(a)
(b)
Figure 18. The role width and yu on with tasks of 40 classes each. (a) CFr; (b) average error.
0.5
4096
0.5
4096
2048
2048
0.4
0.4
Average Error
1024
1024
0.3
Width
0.3
Width
0.2
0.2
0.1
128
0.1
128
0.0
64
0.0
64
10
10
10"
10
10
10P
710
(a)
(b)
Figure 19. Results CNN model (i.c. architecture App. A.2 but without skip connections): (a) CFr, (b) AE.
26

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
C. Network's 's Function Evolution in Non-Stationary Learning Scenarios
C.1. Non-Stationary Learning Under pPP Fields and Dynamics
In order to simplify the notation and the theoretical derivations, consider two different tasks T1. T2 represented by the datasets (X,, Y1), (X2, Y2), each with respectively Pi, P2 number of samples of dimension D. this scenario, we train sequentially T1 first and T2 after. While training the latter, we have no access the previous task. This be easily scaled up in terms of the number of tasks, training multiple tasks sequentially, and having access only the one As discussed in the main paper, consider the following architecture, with weights represented by 0 = (ee ..... w and forward pass defined as:
training on.
fP- -hatt 111
het1
1
ww (h
VN
(28)
h h+1
W ((hh))
n1. vT)
- w x
1
The loss is the Mean Squared Error (MSE) - - p. model performances on tasks Ti when we train the task Tz. This involves obtaining the dynamics of the network outputs fu that are derived assuming gradient flow dynamics, i.c. on task Tí. which can be considered as an inference step on that task after each step of the training process on the current training task T2. as following:
lp. = =E 2 (8k, - s..) Key quantities worth tracking are the
de
= - ae We write the evolution of the network's output
de
1 1h
Ot
7 Ot
01 00 T
00 OL
Lh1+1 m1
OL T
7
Σ
7 02
d0
00
(29)
"
Σ
0h L+1
0 0f
ac
d0
00
Ofaz
02
ah 1++1
0h
L+1
7
p1
7
d0
00
A2
where we made use of the gradient flow dynamics on task To and then made explicit the dependence of the Loss on the network output. In this way, calling Aay
Oc
(Yag Saz), equation above can be written as:
8fa2
05m Σ 0t 72
Kazz K
dh dh 1++1
02
L+1
T
dhL+1 11
dh
I++1
dhL+1
(30)
d0
d0
d0
d0
with Karaz the Neural Tangent Kernel across tasks Tj and Tz. Since at initialization we have Karas is of order O(1), we choose the setting y O()2) to have order (1)) evolution.
We express all quantities involved in terms of features h,,, hp and gradients INr 9p2 defined as:
ThL+1 S/1 = vN
= (h/1.)
0 0 z
1
W T e+1
9
Oh n:
VN
(31)
5 = à (n4 o Tt
where Zns, are the pre-gradients of task T. This allows to explicit the gradient updates to the parameters in the backward pass as a function of the fields h,.. 9p.:
3hL4 L+1
N
Min
e+1
(32)
OW
Dh
27

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
enabling write the NTK across tasks as:
ah
Kpra0=
00
00
1
Oh L+1
Dh I+1
#1
ow aw
(33)
t=
(1
9p
gotl
Φ
h
(hos))
g
K
K
N
N
N
N
2100
introducing the forward and backward across tasks kernels 4 dimension:
Ppiag Guraz written as the inner product along the hidden
D M1ag N Φ ( Φ (1io))
1 0
(n o (naz)
= N
(34)
G p100
N g
Juz T
N Iu: 9
together with the input covariance matrix across tasks K ('(1) Ve E [0, M VI > 0, in terms of the aforementioned forward and backward fields:
a We can express the weights dynamics, namely
ow (t)
ac
Σ
"
Öt
Ow
Pi
0h
1++1 14
(35)
OW
+ N [ ds (n) +
(s)0 (5)) (h (3)
14
where T; is the task we are training on. As a first step, we derive the evolutions of the forward and backward fields of the training, making use of the recursive relation for h, and z; respectively in equations 28 and 31:
h (t) X ++1 (t) + 7 S ds Σ
Au.get e+1 (s) (s
VN
(36)
7 N / ds E (s) (n (s) (s,
- - (1)
where we have introduced the stochastic fields 1+1 (t)( E/,, (1), arising from the randomness of initialisation conditions:
(+1 (t) 1
w (0))(h'(()) (1))
VN
(37)
1
Sh (t) = VN
(0)
Let us now consider non-stationary learning setting, with tasks T and Tz, trained sequentially on a single hidden layer network (I==1) in this order. Once have trained the first tasks, we interested in tracking the evolution of features and gradients, in order to compute the NTK across the two tasks. To obtain the fields evolution equation, we again utilize the recursive relations definitions of h and 91, where this time the evolution of the weights is due to the training in tasks Tz:
1
W (t) = =x,, +70 S (' Ann (
h (t)
VN
(38)
x² (t) = w (t) = g¹ + 1.
Aay (s))ds
ag
28

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
with Xh, and g stochastic fields. In the general case of T tasks, these equations can be adapted, making use of the Heaviside function (t), to split the Gradient Descent evolution the total set of tasks:
h1 Ai (t) w (t) = =Xp, +70
/
Σ Ú,(s)
T
Σ Da (s)g
(s) ds
VN
(39)
z² (t) w (t) E¹ + 70 / Σ U.(s)
Dan
Dap (5) (s))ds
with (t) = Utt t - t). splitting the integral in multiple ones with the correct time extrema, inside which task T, is under training. Once we have the fields (h" g²} we can construct the primitive kernels +1 NTK across tasks. The latter is then used to compute the residual evolution as:
Miogg G R1022 and then the
8 A 0r
(1) (1) = ( pi (1)
G #1 ouz (1) eord A
(40)
we thus can predict the evolution of residuals of different tasks with respect to the one we are training. Moreover, we have the equations governing the internal representation evolution during other tasks.
C.2. DMFT for One Hidden Layer NN in CL Scenario (Proof of Proposition 4.1)
The derivation is carried out using the Martin-Siggia Rose-Janssen-DD N Dominicis formalism (Martin et al., 1973), which allows us to prove that at N + do the kernels concentrate, making the evolution of residuals deterministic. Since kernels depend on the stochastic fields X4. Xau- we are interested in characterizing the distributions of them at infinite width under the non-stationary scenario. Specifically, we aim to obtain the expectation of the inner product (Xay Xa,) (in this section write (-) the expectation B[:)), since it represents the initial condition for the forward kernel $1 a, Furthermore, determinist, since limit. To do that, we look at the Moment Generating Function (MGF) of fields {x1r (p.e(e))) X, hrae[p.] and &. defined as follows:
to prove that in the infinite limit the evolutions of residuals of tasks Ti while training task Tz
kernels 41 41 #1aos G1 1lag concentrate in the N
Z [6i JmEEP.)) (m.JuiE(P)):
exp
Esm Xms + PA1
Ei Xuz P2
+
v E
(41)
As next step, we enforce the definition of the random fields through Dirac deltas, making explicit the source of randomness (weights initialization), introducing new integration variables, and thus enabling us compute the average.
/
dxnr dxn (2T)
1
exp
iEpm
VD
W W(0)z (0)zpM
pz
dx d2us exp (2T)
iXpa (
Xwaa"
W(0)=Hp
(42)
VD
/
dEdE
exp (iÊ (E
(6- (()))))
(2T)
We then insert these definitions in the MGF:
Z [1 IuLE(P)); (I (m
// II
Xur Rug dEdE
exp( Σ jm Xum +
Xuz + v.
(2T) (2.T((2))))) (2
Majuz
1
(43)
+ +ix (X,m
W ((0)=pp)) i8nm (Xpa
W ((0))pp))
VD
DD
E (E w())))
Now we can compute the average over the random noise, namely 9g since we know that Wt wy ~ ~N 1). Having the Gaussian terms plus a linear term deriving from the explicit definition of fields, the integral be computed is essentially a
29

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
Fourier transform of the original Gaussian, which leads to another Gaussian (inverse of the Huubard--Stratooviin transform):
d' dRe, dXur dx dEdÉ (2T) (2 )N (2TT)*
Z [(in hnre(p)): (5 Inie(s)))
II M1
(Eim + (Rm)) Xm, +
exp
Ei + +1x)e) Xu2
1
Σ
Rus La,
K Hian
+
Σ Russ" Ray K m1a2
(44)
2
.01
11,02
+ Σ Ruz2"
-Ra K
+
Σ
Ruz
Raz K (2zoo)
1 + (v +v+ iε) -6)
2
142,011
#2,02
Now we introduce the definitions of the order parameters, namely kernels
1 @ Panna (t, 8) = N p(h (t)) ((h (s)), Gusmalt, 8) N 9 (t) (s)(s)
through Dirac delta definitions, as we did for the random initial fields:
dG pnnn (1, s)dG un (1,8) exp (Gri (t, pran 9r1 (t) 9u2 (5))
1
2T
(45)
d2 mine (t, (edddwwna (t, exp (panz(l, 8) (N manal @b (1)) +5 (2 (
2wiN-1
that put together in the above formula for Z lead to:
/
II
d / (1,88) s)d
Mlwall, 8) 8)dG h (1,
&,G,
(46)
1LP2ts
where S is the action and is given by:
510 1np Garna Grave]=
Σ
f dsdl 1 mnnn (t, pr (1,8) + Grrrall ())
In (Zlj,, v/))
dxu, di dx dx dEde
Zlj, u]
II
(2nr) NV (2TT) N (2TT)
exp
ΣGin Cis + ixux) -ux + ELiv + iixua) Xuz
2 Rus No. K Kuian
+
Σ Xus Rog Kiia Σ
Ruz Roz K + Σ Rus &ay KT K2z02)
10,002
$2.00
1
2 (u + +() - 1 s) (1) ((h)z())) - Gminz (1) (t) gwa(s)
(47)
We then take the saddle point solution for the integral defining Z, which essentially means finding the values of the order parameters maximizing S. This leads to the following equations:
8S
85
N
84p u1 (t, s) = (t, = O,
= $ (t,
(6 (hap (t)) Φ (1ipg (s))); =
8Φ
(t,
N
(48)
65
8s
N
Gyre (1, 0.
(p.ps(t.8)
(gps (t) (8)), 0
8G DGpnngg (t,
8G &Gmnu (t,
N
where the average ()v is defined as:
(0 2 Us] /
II dxn. dxa dx
exp exp (-H
[Xxm &es Xuai Rast b. Sr. 2z 06x.2.518
30

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
where H. is the logarithm of the Eiii, ve]. Once we plug in the saddle point solutions we can rewrite the remaining terms defining the matrix and vectors:
A
K
K
pran
K Kpaz
K p22z
y
Xxi X12_
Xmi X12
j
Jpm Jo2]
(49)
so that we can rewrite Z in the following way:
2|5,0]= - / (2Tr) N (2x) (2T) exp( x Ax + + iay
dedydEd
22 2
1
+ (v + +1E) -())-
(50)
dydg
I' Cy jy - + uE) "リ -
1
(2x) N (2x) (2x) exp(
where we have defined the matrix C A-1 At this point, we keep integrating and obtain:
Z e] =] expl 1 5" Aj + x2
(51)
2
which is the MGF of the fields Xar XM2 If are interested in the value of (Xas -uud we can easily get it from Z by:
(Xm Xpa)= =
Z
j.==
0j n2 0jm
0
expl 2 (5
K Iup + Thh K Kinaa p2 + Iy K Kuan + S K Rangirs + 1
Öj 1z D5
K2pp2
(52)
31

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
D. Comparison of Infinite Width Simulations and Experimental Results
In order to verify the goodness of the theoretical framework developed for the non-stationary learning case, we compared the results from single hidden layer MLP of width 4096, ReLU activation function, μ& parameterized, to 1 and learning rate 7/0 0.25, with the predictions from the infinite width limit version. The dataset is the same used in the previous experiments, namely the PermutedMNIST restricted to 30 samples, per label. The number of epochs 500 and the total number of tasks is 4, each generated from different complete permutation of the images pixels.
The interesting quantities to observe are the ones involving the internal representation of the samples, gradients and the losses evolution, in order to test the accuracy of the theory from both low and high level observables. We show below the distributions of the pre- areaaiiivations {he (1,ii((11,,) (1,2,3,4), of gradients {9s lig (1,2,3,4) for all 30 samples:
10
10'
10-1
p(()
10
10-5.
10
10-1
Mateeek dr
2.5
2.5
5.0
2.5
2.5
5.0
0
h
30
10
10'
10a
p(b)
p(g)
30-3
10-1
10
10-3
2.5
2.5
2.5
0
(a) Pre-activations distributions for all 30 samples of the different tasks. The shaded lines are the distributions at 450, the full lines the distributions 550.
(b) Gradients distributions for all 30 samples of the different tasks. The shaded lines are the distributions at 450, the full lines are the distrihutions at 550.
Figure 20.
is clear in above figure the effect of training on the pre-activations, and consequently gradients, distributions: there is shift from the initial Gaussian curves towards heavily tailed, non-Gaussian distributions. Having the pre-activations and gradients we can easily obtain the primitive forward and backward kernels TTT,, and GT.T,- We show below the kernels KTTT at time steps = 450 and I 550, thus 50 epochs before and after the switch
TTTT, Kn and from task Ti to task T2:
Thirry 450
Fep. 450
Thwey 550
Eep. 550
G
KTT
Figure 21. All primitive kernels and NTK across tasks Ti and Tz. The theoretical one are obtained through simulations of the infinite width model, the experimental ones comes from the trained network aforementioned
32

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
Theoey
Thoory
Exp,
4
Figure 22. All primitive kernels and NTK across tasks T3 and T4. The theoretical one are obtained through simulations of the infinite
width model, the experimental ones comes from the trained network aforementioned
Last, from equation 30, can compute the loss evolution of all tasks' losses during all tasks' training:
14
Exp. 7.
Th.
12
Exp.
Th.
10
Th.
Loss
Exp.
Th.
4-
250
750
1000
1250
1500
1750
2000
Epochs
Figure Tasks' losses a four task scenario mentioned above. Vertical dot-dashed represent epochs which the shift between tasks training happens.
In order to gain deeper insight into the learning dynamics, useful measure is the alignment of the last-layer forward Kernel f along the task-relevant subspace YY as defined in (Atanasov et al., 2024, App. P) and termed Kernel-Targe Alignment (KTA). We denote the KTA as A(Φ, YYT). This measure provides qualitative picture of how and when features develop structures able to cluster samples according to respective labels. As shown in Fig. 24 the KTA increases only during training of the respective task, and the alignment decreases during other tasks' training.
We can see that the infinite width model matches also this measure, serving as a further validation of the closely matching characteristics of the experimental and theoretical setup.
33

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
0.5
0.4
460. VYY
0.3
Theory
Exp.
0.2
Exp.
Exp.
0.1
250
500
750
1000
1250
1500
1750
2000
Epochs
Figure 24. Theoretical and experimental tasks' alignment A(d, YY²) for all four tasks during all training processes. Vertical dot-dashed lines represent the epochs at which shift between tasks training happens.
34

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
E. Catastrophic Forgetting Under Perturbation Theory in Yo
E.1. Effect of Feature Learning on CF the Infinite Width Limit
The dependence on "Y in Equation (12) is implicit and recursively applied over the training steps. To get a better sense of the influence of feature learning on forgetting, we apply perturbation theory, which assumes the following expansion for forgetting:
CF(() - =C
(53)
The aim is compute the coefficients (I, CF ..... identifying the respective terms in front of the relative To expansion deriving from the other set of variables composing the self consistent system of equations. Notice that here superscripts index the coefficients and not powers (e.g. (F (t) is the second order coefficient of the expansion). By design, the expansion coefficients {CFO] do not depend on no- For two-layer linear network, up to order nd we find the following:
Proposition E.1. Let f be the output of a neural network Eq. with identity activation function, a single hidden layer, and a 0 (i.e without skip connections) trained for T = tasks Tı, T2 for finite number of steps. Let i be the number of training steps Ti and t > t1. In the limit N - oc, the catastrophic forgetting defined in Eq. 11 admits the expansion of Eq. 53. The first three coefficients are:
(F) = L/ 4
(0) (8) (0 (0)
TiT,
CF =0
(F( - S/' ( 10) (s) ())
T172
(0) (0) A (s)
00) (s) + 4 (s)
T1T A 2 (s) ds.
T.T2 To
where (0) k(O), K())A (2) are the perturbative coefficients for the residual A and the NTK K. The full set of equations, which includes the expansion of the kernels & and G, can be found in Appendix E.
Remarks. The zeroth order term corresponds the catastrophic forgetting incurred in the NTK/lazy regime, where "To 0. This term was analyzed (Doan al., 2021; Bennani al., 2020) for slightly different definition forgetting. Here, the zero-order term of the NTK K.O, KTTT, (which corresponds to the NTK in the lazy regime) is fixed to initialization, thus exclude the explicit time dependency. The first order term is zero, found in Bordelon & Pehlevan (2022) for the features and backward kernels. Compared to them, here we also compute leading order perturbations for A, as necessary for our analysis. Also note that this characterization of forgetting is only valid for small values of "ro, where the higher order terms (01)) can be ignored. Our theory characterizes the effect of increasing the degree of feature learning on CF, starting from the lazy training setting - for which CF is already known. Perturbative methods have been applied in other contexts of deep learning theory, particularly in computing finite size corrections to the infinite width limit (Roberts al., 2022; Hanin Nica, 2019).
Our analysis offers general blueprint for caleulating the effect of small feature learning perturbations CF However, in practice, the expansion coefficients can only be solved in closed form by considering specific learning instances.
E. DERIVATION
Following previous works by (Bordelon & Pehlevan, 2022), we inspected the expansion of the fundamental quantities governing the evolution of residuals and the residuals themselves around to 0. This is done using a perturbation theory
35

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
approach, thus expanding the fields and the residuals in power series of "oo so that:
hag (t) = XM1 + Σ
10
( (t)
(54)
n=1
20
2p1 (t) =
n (t)
(55)
Ar, (t)
70 A
(n)
(56)
#1
1=
In this study, we will stop the expansion at the second order term, due to practical reasons, but also because the parameter "n is typically the range [0, 1] and the lazy-rich transition observed in experiments is typically of the order 0.1. We begin
by considering the update equations for the fields and residuals, and plug on the left side the equation the expansions in
oo
S Σ Aoz (s) ds
(00) Mi
70 m1 + 700h +... Xu1 + J:
(57)
21) + 702 + x10 +... - = 70 /'
"'
Aazpl11nn
ds
(58)
c1
A (1) + 7o 20 (1) + = Y ( (s)
(59)
The next step is identify on the right side of the equations above the corresponding terms in the left-hand side expansions, which is long but trivial. We finally obtain:
((0)
Xun
(60)
ag aap ds + 1
(1))
(0) E
Σ
(0) (0)
g K
ds
(61)
P1
o2 azan
11 02
02
122
K
ds
/")
Σ
(0) kr Kagu ds
(62)
m1
11
(0)
(63)
(1)
A az
Xa, ds
(64)
02
(2)
Σ
(0) h ds
(65)
(66)
36

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
while for the residuals we identify differential equations involving the evolution of higher order terms in the To expansion. From the fields obtain the expansion of kernels pyoag Gurazi
Xaz)
(67)
(
=0
(68)
A10g
min K Kazss I[" dt'
't'A An (t') S/ de (t")
q (12)
Σ
K
(69)
11002
KT Minz Kagls [" di' di' (1') (" di" Ai (f") + sym
Σ
(70)
P2
[f d'''' (6))
[ do'Ao, (6')
K mnnn Kag8
(71)
21
[% di' An (1)) [%/0 do' do (6))
Σ
KJ m102 K Ko282
(72)
P232
(())
(EE)
(73)
G(2)
(74)
11102
Σ Kang [ di' Al' (1') ( Ai (1")
((2)
(75)
0181
[' dt' (t)) 1. dt
Σ
K
(76)
0282
['ar dt' (()) [ do' do' (0)
Σ
K
(77)
trtd
a1
Sr dt' Das (t)) [ do' Das (6))
(78)
p2a2
02/2
(79)
and the residuals' expansion reads as:
0 (0)
U(h -1) Σ
h A + Ut - 41) Σ
K (0)
(80)
Øt
N1o.
1100
02
11
02
0 (1)
U(t1 -t) -t)
K A
+ U(t - t1) Σ
k
(1) m2
(81)
Øt
Dj
Miag
ai
Q2
(4 Øt
Utt -t) (
K A + (2) (0)
+ U(t - -1)
Σ
K A + K(2)
(82)
totd
M102 A
(83)
37

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
that with the initial condition Ar, (0) = T7 (0) we obtain 4// A {
Y and A (1) = (1) vi e (1,
and for all times Giving these "Y expansion coefficients, can compute the CF expansion through 11, obtaining:
dCF()) dt
Σ
(A
+ +
+
+...)(kpp)
Minz + k M102
(4 7 + +...)
(84)
Σ
A
K
A
%0
Σ
A 10) K Mala
(5)
(2))
A ( (1)
(s) (s)
11/22
P2
12
M122
11
(85)
dCF(O)
22F(2)
+ 10
+ 0(0)
(86)
dt
(87)
The zeroth order term represents the NTK limit term and can be solved quite casily in the non-stationary case, keeping in mind the equations above for the residuals' evolutions, that follow exponential decays.
E.2. CF Perturbation %o with Non-Stationarity
E.2.1. MODELING SIMPLE SON-NAATIOS
Calling the input dimension and the number of data points per task P, we adopted the following setup:
The inputs covariance matrices across same tasks are identity matrices, K = I, with E {1....,T) T}. This is equivalent to having orthonormal samples.
The inputs covariance matrices across different tasks are proportional to identity matrices, so K = pl, with j i (L...., T) and [0, 1]. In this way, we smoothly control the similarity across tasks, with the totally orthogonal setting of p 0 and the same task setting of 1. Different tasks' samples correspond to rotations of other tasks' samples with a certain angle of which cosine is p. To clarify, imagine the samples' matrix X7, E R DxP as:
0
0
1
0
0
0
(88)
0
0
with P columns and D rows, and the identity matrix that lies in the upper part of it. The second task matrix XTa can be constructed as:
P
p
0
:
P
(89)
0.P-1
6p41
2P+1
bp
The last step is building an orthonormal base in the lower part of the matrix, which is always possible if we have enough degrees of freedom for the coefficients { D................ (z:):=++t......)) i.e. D 2 P. Thus the condition is the one mentioned in the main text D 2 2P. This corresponds to a rotation in a D-dimensional space of P orthogonal vectors.
38

---

The Importance of Being Lazy: Scaling Limits of Continual Learning The data modeling process above thus be summarized as:
a == } =
otherwise
E.2.2. INFINITE- TIMEI LIMIT OF A LINEAR MLP
In order to simplify the previous expansions, and with the goal of reaching a simple and interpretable formulation of forgetting w.r.t. the parameter "oo we embark on further exploration of the CF perturbation in "o for infinitely long training + bo) of a linear I-hidden-layer MLP. To reach a final formulation of forgetting, we need to formulate the expansions of Ars and 4u at the end of training the second task. To get these expressions we first need derive the expression of the same quantities the end of training the first task (for infinitely long time).
We recall that we model the non-stationarity level KT; KTTT I and the across tasks kemnll K T. pll.
First, we write down the backward and forward fields following perturbation theory up to the second order:
hys (f) = Xus + +70 S ds >
Ke Hiay Za, (s)
-mm + +00 I' ds
(02) (s) (0
K (5+
( +1440 - (s))
= + 70 [ ds
FA (s) + 10 / ds Σ K an 2 A()
K pra
[' ds Σ
(1 (8) + 2 I ds Σ
inon (1) 1' ds' (5)
Xmi +
Ke mian
K2
a1
z(1) + 70
ds Σ hag Aan (s)
= E 70
ds
((a + Toh4
+
(s) + {
+ 10 I ds Σ 1
= + 70
ds
(8) Xan ci
(10) (s)
a1
+ +10 J ds Σ ()) / ds ds'Ka (5)
= 70
ds
Xa1
39

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
This allows to rewrite equations as:
Rym (t) Xu1 + Σ PoKiiaa {Aa + Σ yoo Xus Ba, (t)
-11 + Σ 205A, (1) + Σ
Xa Bay (1)
2
=(1) = = Σ
T0Xmr Aai (1) + Σ
10s mian Bay (1)
Ke
+
YoXun Am (t) + Σ Br (t)
(t) Xp2 + Σ Yo M1az AAa
Σ
Ko Miag Xu. Ba,
Q1
=xpa +
Σ
PoppAa (t) Σ
J0XX,s Bay (t)
01
where we have defined the function An. (t) = So ds (0) (s) and (t) ()) So ds' A (s'). Now, having all we need to compute kernels, these latter are obtained through expectations of the inner product between fields, leading to:
Happy (t) = (has (t)h (t)
p + rop Aa, (t) + 2Ba (t)
Happa (t) (han (t)h (t)
+ 70 Ao, (1)² + 2Ba (t)
G(t) = (=(t)²(t) T)
= 1 + 70 An. (t)² + + +2Bmn (t)
composing the NTK expansion in Y
Kaypy (t) Hagm (t) + G(t) agm = + +500(A (Aa, Aa, (1)2 2B,,())
(8) 20,
=0(1+18
+30 (Aa, (A
(f)2 + (t) +Ba (1))
2p
+18 2p Am, (1)2 + +2B., (1)
(())
Kage1
Kigg
Now we can expand also the residuals powers of "oo resulting in a of differential equations:
0.4p;
Σ Kyran Aa:
(A44 + A (1)
Σ
(K(0)) (() pra:) [A (0) AnA!!!) -
Mian
40

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
that can be rearranged
0,44 (0) Eaj
K An1
(0)
4. (0) you
0,4 (1) Ea, Ko
a Ag)
(I)
(()()) (0) 0
0,4/1 11
(2) = - Eny an (K
An1
+
Kjja 1
K(
-0
0,4 (0) Eaj K Ani
Anj (0) yai
42
0,A (1) = Eai K (0)) All)
Am =0
M2
0,4/3
(2) = Ea, p2gy !
+ Kg A 19),
A 2 (0) =0
that allows us conclude that the first order term of both tasks' residuals is namely (!) (t) Alg 0. We can, then, solve the zeroth order differential equation and obtain the lazy regime solutions:
(0) (t) = ye ye- 2t
A 50) (t) y(1 p)(1 6 22)
and from these equations we can obtain the analytical form of Aa, (1) and Ban (t), that allows us to compute the second order differential equation, obtaining:
A (1) = (e
7 + + 4t
1
- 61-
4
(t) = py 21
1
4 +e
-
4
This allows inspect the infinite-time limit of task 2 residuals 4ng (f) 1 (t) A3 + after training on task
1, that gives us the following results:
lim A A (t) = y(1 - p)
t-000
lim
A (2) (t) 0
t-y00
lim App (1) y(1 - p)
1-000
so that the final loss value expansion till the second order reads as:
lim 1ッx 5 ( (t)
マ p1=
(90)
From this equation, we can conclude that the nime limit of task 2 loss after infinitely long training on task 1, is independent of 70. and this is validated empirically in Fig. 25.
0.5
0,001
0.06
0.4-
C.4
Las
0.2
X Thear
0.7
0.0
0.3
0.2
0.4
0.8
1.0
Figure 25. Last value of the loss on Task Tz after (long) training on Task Tı on a single hidden layer NN, linear activation, and Gaussian data in order to tune task similarity and obtain the theoretical setup. Black crosses represent the theoretical values different levels of similarity, following equation 90. At low p the theoretical constraints on the setup become weaker due to the intrinsic noise level of Gaussian data.

---

The Importance of Being Lazy: Scaling Limits of Continual Learning
Now, reach the expression for forgetting, would need to express the residuals after training on the second task. However, such lengthy formulation is out of scope for this work, and is thus left for future development. Nevertheless, what we can easily achieve from the formulation so far, is the infinite-time limit expression for the lazy limit ("0 0, i.e. the expression for CF
In order to do so, we write down the dynamics for residuals of both tasks during first task training, which read as:
A j1 ye 22
A j2 y(1 [1] E22
that will converge to the following limiting behaviors:
lim Am 0
lim Ap2 1111 p)
This represents the "new initial conditions" for the second task differential equations the lazy regime, which are:
0,4 K(0) A 1110
Σ
A1
0,4 P2
K P2n2
(0)
Solving the above ODE obtain the following equations for residuals, where simplify the notation by writing time t as t, considering the effective training time from the beginning of the second task training. This allows to eliminate the factors that vanish when plugging in the assumption of an infinite training time for the first task (h1 oc). Thus we obtain:
A (0) yp(1 (1 - p)(E 1)
(0) y(1 - -)e--2
and computing the limiting values of both losses is now straightforward since Ca., 4
y²
lim (1)
2 0² p)²
t0000
lim 4/ (1) 0 (1000
Since CF in terms of losses is defined as CF L(t) L(h), ), under these simplified conditions, lime-soo L(t)) 0, the infinite-time behavior of Catastrophic Forgetting is captured by:
y² p
lim CF (t)
-000002-110 (1 p)² p)
with quadratic dependence on similarity level p. maximized at exactly p 0.5, as well known for linear models (Goldfarb et al., 2024).
42


Reviews:
[{"review_id": "ZTpetkCQ2S", "reviewer": "Reviewer_hpnB", "summary": "The authors present a theoretical and experimental analysis of the effect of the neural network parametrisation on Catastrophic Forgetting.\nThe study extends previous works which focused on the lazy regime only.\nThe authors identify a spectrum of training regimes from the lazy regime to the feature learning regime. This spectrum is parametrised with a single parameter gamma.\nThe authors show that depending on the parametrisation, the width scaling laws is different for Catastrophic Forgetting.\nAlso, the authors identify an optimal tradeoff between accuracy and forgetting, as a function of the parametrisation. \nThe tradeoff occurs because forgetting increases the more the feature rich the training regime, while the accuracy increases.\nFinally, the authors define and study the EoL, which varies as a function of the stationarity of the tasks.\nThe more stationary the tasks, the higher the EoL.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We sincerely thank you for taking the time to review our paper and for the thoughtful feedback. We are glad to hear that you found the work really interesting and stimulating. We are also grateful to hear that you appreciate our contribution to the literature.*\n\n---\n\n## The Pretraining Effect\n\nWe are glad you found the pretraining effect particularly interesting and intriguing; we share your enthusiasm for this phenomenon. We agree that these observations offer an important perspective on our results, in particular as the setting approaches training scenarios closer to the real world. In this regard, we think you might be interested in inspecting also the interplay between the pretraining effect and width scaling, as reported in Figures 17-19 of the appendix.\n\nBefore answering your specific questions, we would like to clarify that the design choices for our experiments targeted the main goals of the paper. However, we agree with you that looking into the pre-training effect more thoroughly is certainly very interesting.\n\n### Split-CIFAR\n> Is there a specific reason not to consider the CIFAR100 task?\n\nWe find no specific reason for not considering CIFAR100 in our experiments. We have chosen CIFAR10 as a representative dataset for our experiments for computational convenience, and then directly extended our analysis to Split-TinyImagenet, aiming to provide stronger empirical evidence on a more challenging dataset. Nevertheless, we agree that CIFAR100 might be an interesting in-between dataset and we thank you for pointing it out.\n\n> Wouldn't the pretraining effect be expected for it to occur in CIFAR10 as well?\n\nIn Split-TinyImageNet, we observe the pretraining effect happening for a larger number of classes per task, namely when the tasks have enough data diversity, and thus a greater overlap between task distributions. Our intuition is therefore that the reason why this behavior is not observed in CIFAR10, is that the number of classes is too small to allow for a significant overlap between task distributions.\n\n### Other Questions on the Pretraining Effect\n> Is it sensible to measure the pretraining effect with the forward transfer metric?\n\nWe haven't considered measuring the pretraining effect with the forward transfer metric, instead, we have focused on disentangling the feature evolution of the first and later tasks. However, we agree that the forward transfer metric could be a valuable addition to our analysis and we will look into integrating it.\n\n> How do you control the hierarchy of class similarities in Split-TinyImageNet?\n\nWe naively split sequentially the classes of TinyImageNet, without accounting for the semantics of the classes. However, when we vary the number of classes per task, we ensure to keep fixed the classes-to-task assignment, to have a fair comparison across runs with different nr. of classes per task.\n\nExploring the effect of the semantical hierarchy is, however, a very interesting avenue. For example, one interesting aspect one could explore in future work is whether one can reproduce our \"pretraining effect\" even with a fixed number of classes per task, and only by modifying the variety of the data from a semantical perspective.\n\n---\n\n## Other Comments\n- **Tractability of forgetting expression at $t \\to \\infty$**:\n\n    The short answer is yes, albeit the formula is not present in the current version of the paper. Indeed, we were recently able to extend our perturbation theory approach for the infinite-time solution in the two-layer linear network setting, where we have a non-trivial relation between $\\gamma_0$ and similarity $\\rho$.\n\n- **Cubic complexity of infinite-width simulations**:\n\n    The fields h and z are composed of sums over P data points, and discretized integrals over T time steps, i.e., h and g are $\\mathcal{O}(PT)$. The NTK is a sum of $\\Phi$ and $G$, which are the inner product of respective fields and thus $\\mathcal{O}(P^2T^2)$. Finally, the output is the matrix-vector product of the NTK and $\\Delta$ of dimension P, and then integrated over T time steps following the PDE of Eq. 3. This leads to $\\mathcal{O}(P^3T^3)$.\n\n- **Noise of low-width networks**:\n\n   The training process is stochastic (over initialisation), and quantities like the NTK rely on the self-averaging properties characteristic of the $N\\to\\infty$ limit. Concretely, this means that in the limit, the network's dynamics become deterministic and instead, at finite widths, these observables are a partial/incomplete snapshot of the underlying process, introducing more stochasticity.\n\n- **Typos and notation inconsistencies**\n\n    Thanks for pointing these out. We will correct these in the final version of the paper.\n\n---\n\n *We hope that we have adequately addressed your questions. If you have further questions or comments, we remain at your disposal.*", "claims_and_evidence": "The main claims of the paper are :   \n1- The effect of width scaling on CF depends on the network parametrization. The evidence to support it is : \n- Experimental : Measuring CF as a function of the width for the Split-CIFAR-10 and Permuted MNIST tasks.\n2- The existence of the Edge of Laziness (EoL), a region which separates to regimes. The lazy regime where the features don't change much and CF is low, the rich regime where the features changes significantly more as well as CF. \n- Figure 3, a, b, c : The variation of CFr as a function of gamma_0 \n- Figure 4, a : The variation of the average loss as a function of gamma_0\n3- The optimal gamma_star for the plasticity stability tradeoff is almost identical regardless of the network width : \n- Figure 4, b : The pareto front between CF and Learning error\n4- High feature learning is only beneficial with highly similar tasks\n- Figure 5 C : The average error as a function of the task similarity and gamma_0 on Permuted MNIST", "methods_and_evaluation_criteria": "The proposed evaluation methods are sensible. The authors use the tasks Permuted MNIST, CIFAR 10 and Split-TinyImageNet for their experimental validation.\nThe last observation about the Pre-Training effect in Section 6.2 is very interesting and intriguing. The authors show that on Split-TinyImageNet the law is different compared to the other two tasks. I think this experiment is very important to highlight the validity of the law on training setups and datasets closer to real world data.", "theoretical_claims": "- There seems to be a notation inconsistency between the definition of the NTK across tasks (Eq 2) and its use in Eq 12. The definition only has a single time index, while Eq 12 has two time indexes. Also, I suggest clarifying the time index in the definition of Eq 2, in the weights matrix. Currently the definition doesn't explicitly highlight the time / task index.\n- General question : Out of curiosity, is it tractable to derive the expression of forgetting at t=infinity from the PDE in Eq 12 ?\n\nI skimmed through the proof but haven't checked it in detail.", "experimental_designs_or_analyses": "I checked the soundness of all the experiment designed presented in the main paper, but I didn't check the ones in the Appendix.\n\nThe authors considered the Permuted MNIST, CIFAR 10 and Split-TinyImageNet tasks for their analysis. \nThe first two tasks show clearly that the law is satisfied and the third one shows a different behaviour in the rich regime which the authors explain with forward transfer. \nOne question I have is if there is a specific reason not to consider the CIFAR-100 task, as it lies between the two datasets and would help determine the boundary in terms of data distribution where the pre-training effect would apply or not.", "relation_to_broader_scientific_literature": "This paper relates to the broader literature in the following ways : \n- [1] and [2] provide a theory of CF in the lazy regime, formulating theoretical bounds and a closed form of CF in the NTK regime.\n- [3] and [4] observe that increasing the width of the neural networks reduces CF and study the impact of the model architecture on CF.\n- [5] study theoretically the effect of overparametrization on CF for linear models. \n\nThe paper also relates to the rich feature learning literature, I am not familiar with this research area.\n\n- [1] Doan, Thang Van et al. “A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix.” International Conference on Artificial Intelligence and Statistics (2020).\n- [2] Bennani, Mehdi et al. (2020). Generalisation Guarantees for Continual Learning with Orthogonal Gradient Descent. ArXiv, abs/2006.11942.\n- [3] Mirzadeh, S., Chaudhry, A., Hu, H., Pascanu, R., Gorur, D., & Farajtabar, M. (2021). Wide Neural Networks Forget Less Catastrophically. International Conference on Machine Learning.\n- [4] Mirzadeh, S., Chaudhry, A., Yin, D., Nguyen, T., Pascanu, R., Gorur, D., & Farajtabar, M. (2022). Architecture Matters in Continual Learning. ArXiv, abs/2202.00275.\n- [5] Evron, Itay, Daniel Goldfarb, Nir Weinberger, Daniel Soudry and Paul Hand. “The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting - An Analytical Model.” ArXiv abs/2401.12617 (2024): n. pag.", "essential_references_not_discussed": "I am not aware of any essential references not discussed :)", "other_strengths_and_weaknesses": "The paper was really interesting to read and it challenged several intuitions I had. I found the explanations very clear and I particularly appreciated the experimental evidence provided to support and illustrate the claims in practical settings. \n\nAlso, I think the contribution is significant because the analysis links between several prior findings and further provides new insights about the impact of parametrization on a scale.\n\nI didn't note any major weaknesses, in the other sections I noted some clarification questions.", "other_comments_or_suggestions": "Some typos : \n- In the contributions section : I think NTP and muP are not defined beforehand, they are defined in the next page.\n- Figure 6 - d : The title is clipped\n\n- L163 : Could you briefly explain the cubic complexity without going much in detail ?", "questions_for_authors": "(The questions below are clarification questions and wouldn't impact the final score)\n\n- In  Figure 1 and Figure 4, the law of the smallest width (64) is significantly more nosy than the other larger widths. Could you share an intuition about why it might be the case ?\n- In Figure 6 : Given that ImageNet has a hierarchy of class similarities, how do you control for this similarity when splitting the tasks and classes ?\n- Also about the pre-training effect, wouldn't it be expected for it to occur in CIFAR-10 as well. Is it sensible to measure it with the forward transfer metric and compare for TinyImageNet and CIFAR-10 ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qUe4oVEuq4", "reviewer": "Reviewer_CVSx", "summary": "This paper investigates the effect of model scale and the degree of feature learning in continual learning. It identifies a transition called Edge of Laziness influenced by task similarity, where the model exits an effectively lazy regime with low forgetting to enter a rich regime with significant forgetting. Technically, it extends the DMFT theory to non-stationary learning. Infinite-width simulations and real-world experiments support its claims.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Thank you for taking the time to read and review our paper. We are glad to hear that you found our paper clearly written, and that you consider our experiments well-designed, providing a systematic study of the topic and supporting well both claims and theoretical results.*\n\n---\n\n1. >**Do the claims in this paper hold for the Transformer architecture trained with Adam optimizer?**\n\n    Despite the success of transformers in DL, their adoption in CL is still limited and fairly understudied [1,2,3]. If transformers in CL are already rare, their optimization with adaptive optimizers (e.g., Adam) is even more so, whereas vanilla SGD is the standard optimizer even when training transformers [4,5]. For these reasons, in our work we decided to study the ResNet architecture with SGD. Nevertheless, we agree that exploring our claims even for transformers and for adaptive optimizers is an interesting avenue for future research.\n\n2. > **Can the DMFT be extended to an adaptive optimizer like that in the TP 4b paper?**\n\n    The DMFT could potentially be extended from [6]. However, in this work, we were more interested in understanding the dependence of the dynamics on $\\gamma_0$, which requires lengthy derivations even for a two-layer linear network trained under gradient flow.\n\n3. > **In practice, we may train a neural network with $O(width)$ steps by the scaling law. Can we extend the experiments and the DMFT theory to that case?**\n\n    The DMFT relies on the assumptions of the number of datapoints and timesteps of order O(1), beyond which the theory may break down. In this study, an extension to width-dependent training times is out of scope. However, we point to the experiments presented in Appendix B.1, in which we investigate the effect of training time in the NTP. We find that longer training time systematically yields high feature evolution even in the NTP, and therefore it leads to CF.\n\n4. > **Can we predict the existence of a phase transition theoretically?**\n\n    We are currently unable to predict the phase transition theoretically. Nevertheless, we are actively working towards this result, and we were recently able to characterize the infinite-time solution of CF in the rich regime of the linear case. In this setting, we observe a highly non-trivial relation between the task similarity $\\rho$ and $\\gamma_0$, where the second-order coefficient of CF is a sixth-grade polynomial in $\\rho$. The complexity of this relationship - even in the linear case - points to the non-triviality of the behavior and thus to the difficulty of getting a general characterization of the phase transition theoretically. Nevertheless, we will continue working to extend this result to the non-linear setting for future work.\n\n5. > **In my opinion, the $\\gamma_0$ is just an output multiplier hyperparameter in existing mup papers (e.g., TP 5), so it should transfer across different widths. Am I right?**\n\n    Thanks for the interesting question, which allows us to clarify the role of $\\gamma_0$ as well as the novelty of our findings.\n\n    Firstly, $\\gamma_0$ is not only an output multiplier, instead it also quadratically modulates the LR (cfr Tab 1 in our paper). This implies that one cannot infer transfer properties of $\\gamma_0$ from those of the LR, as the two are non-trivially related.\n\n    Secondly, and crucially, the transfer properties of LR (and other hyperparameters) have only been shown - but not proven analytically - in the stationary setting, and not in the non-stationary and CL scenario. We therefore deem it non-trivial and surprising to observe that, with $\\mu$P, $\\gamma_0$ shows these transfer properties even in the non-stationary scenario.\n\n    In hindsight, given that width scaling does not affect forgetting under mean field scaling, it is intuitive to expect that $\\gamma_0$ should transfer. However, if width scaling were to affect the dynamics (e.g., by lowering forgetting due to increased capacity), then one could in principle expect $\\gamma_0$ to scale differently with width. We hope this clarifies the subtlety of these results and the coherence of our findings.\n\n---\n\n*We hope we have adequately addressed your questions and reservations that hindered your recommendation for acceptance. We remain at your disposal in case you might have further comments or doubts.*\n\n---\n\n### References\n- [1] Ramasesh et al., \"Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics\", ICLR 2021\n- [2] Mirzadeh et al., \"Wide Neural Networks Forget Less Catastrophically\", ICML 2022\n- [3] Lu, et al. \"Revisiting neural networks for continual learning: An architectural perspective.\" arXiv preprint arXiv:2404.14829 (2024).\n- [4] Ramasesh et al., \"Effect of scale on catastrophic forgetting in neural networks\", ICLR 2021\n- [5] Mirzadeh et al., \"Architecture matters in continual learning.\" arXiv preprint arXiv:2202.00275 (2022).\n- [6] Bordelon et al., \"Infinite Limits of Multi-head Transformer Dynamics.\", Neurips 2024", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I did not check the proof carefully, but it seems good, and the experiments support the theoretical results.", "experimental_designs_or_analyses": "Yes. The experiments are conducted on different datasets and support their claims and theoretical results well. For the MNIST dataset, the infinite-width simulations motivated by the DMFT further verify the real-world experiments in finite width.", "relation_to_broader_scientific_literature": "NA.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is written clearly.\n2. The paper designs a systematic study on the impact of model scale and the degree of feature learning in continual learning, and the experiments support their claims well.\n3. As far as I know, this paper takes a first step to extend the DMFT to continual learning.", "other_comments_or_suggestions": "I maintain my score after the rebuttal.", "questions_for_authors": "1. Do the claims in this paper hold for the Transformer architecture trained with Adam optimizer?\n2. Following question 1, can the DMFT be extended to an adaptive optimizer like that in the TP 4b paper?\n3. In practice, we may train a neural network with $O(width)$ steps by the scaling law. Can we extend the experiments and the DMFT theory to that case?\n4. In this paper, DMFT theory is used to conduct simulations to verify the findings in finite-width experiments, which is expensive. Can we predict the existence of phase transition theoretically?\n5. In my opinion, the $\\gamma_0$ is just an output multiplier hyperparameter in existing mup papers (e.g., TP 5), so it should transfer across different widths. Am I right?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NBBjfNSS7a", "reviewer": "Reviewer_6Eo8", "summary": "This paper studies how neural network parameterization (at the extremes, NTP and $\\mu$P) shapes the effect of network width on catastrophic forgetting.\n\n## Update after rebuttal\n\nMy assessment of the paper remains unchanged. I think this is an interesting contribution, but I am still skeptical of the added value provided by the DMFT analysis.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We are grateful for your time reading our paper, and thank you for your thoughtful review.*\n\n---\n\n## The role of DMFT in the paper\n\nWe agree that the results of the DMFT are somewhat abstract, however we believe this is an inherent limitation of this theory: it is hard to simplify the results and reach interpretable results without sacrificing its completeness. Committed to this direction for future work, we are currently working on integrating minimal assumptions that could help us simplify the results without trivializing them.\n\nNevertheless, we would like to address your opinions that \"(1) the DMFT results do not substantially strengthen the claims based on experiments\", and that \"(2) the technical novelty here is limited\".\n\n1. The DMFT equations allow us to actually reach the infinite-width limit, something that would have been impossible otherwise. Although it is unusual to lay the theory at the service of the experimental evidence, we believe that the infinite-width curves crucially strengthen our finite-width insights of the experiments.\n\n2. We agree that the DMFT techniques are a known approach to analyze the network behavior. Nevertheless, we believe that our extension to the Continual Learning scenario is notable and highly valuable for the community, as we\n\n     - introduce the NTK across tasks as a new entity of the DMFT formalism\n     - are the first work to analyze feature learning in CL\n     - propose a perturbative approximation which is different from previous approaches (e.g., Bordelon and Pehlevan), as we also expand the residuals in powers of $\\gamma_0$, and we apply it to the CL scenario. This allows us to study the complex dependence between $\\gamma_0$ and the task similarity $\\rho$, obtaining a closed-form solution for the coefficients of CF, up to the second order, in the linear case. Seeking the \"conceptual meat\", we were recently able to obtain further insights on the role of $\\gamma_0$ in the linear case, where we find that in the infinite-time solution, the second-order coefficient of CF is always positive, certifying that larger $\\gamma_0$ yields higher forgetting. Moreover, we find that this relationship has a non-trivial dependence on $\\rho$ through a sixth-order polynomial, where the maximum CF is obtained at richness-dependent levels of $\\rho$. You can find the preliminary plot [here](https://ibb.co/9mQhGZg1).\n\n## Discussion of related work\n- **Petrini et al., 2022.**, **Vayas et al., 2023.**, **Yang et al., 2023**. Thanks, they should definitely be included in the related works and cited accordingly.\n- **Atanasov et al., 2025.** We would like to point out that this work is concurrent and will be presented at ICLR 2025. Although the paper is related to ours, we feel it is unfair to list as a weakness the missing discussion of such a recent work.\n\n    Nevertheless, their findings are interesting and might be a good starting point for future work. As you suggest, the observed *lazy-rich transition* might be related to our choice of LR scaling (LR scales quadratically with $\\gamma_0$): they observe that this scaling is optimal for the lazy regime, but not anymore for the rich and ultra-rich regimes (Fig. 1b), where instead sub-quadratic LR scaling is optimal. Our transition might therefore be due to a change of the effective LR, shifting from the optimal LR towards a too-large LR and thus approaching divergence. Note that this is just an intuition that requires further study.\n\n## Other comments\n- **Title:** following also the review of DJW3, we have decided to avoid referring to Edge of Laziness, and will therefore opt for a more informative title, as you suggest.\n\n- **Representational Changes:** We have quickly implemented this interesting experiment, and you can find the results for the Permuted-MNIST, restricted to samples 0 and 1, [here](https://ibb.co/Zzwp7QTB). The drop in the alignment after the tasks switch represents an intriguing insight that serves as an interesting starting point for future analysis. We will add this experiment to the Appendix.\n\n- **Implementation of DMFT:** thanks for pointing out the missing details. We use a modified version of the one by Bordelon \\& Pehlevan in the L=1 case, extended to support multiple tasks as per our training setting. We will clarify this in the final version.\n\n- **Network Capacity:** we refer to *network capacity* as the network size (i.e. the width), irrespective of $\\gamma_0$. In l. 352 we state that \"the increased capacity (i.e. size) of the network does not benefit performance\".\n\n- **Figure lables:** we agree that the labels are slightly notation-heavy, but we believe that the CKA measure might be obscure to readers not familiar with the kernel literature; the labels provide a useful guidance to such a reader.\n\n- **Claim in the discussion:** we agree and will adapt this claim.\n\n---\n\n*We hope that we have adequately addressed your questions. If you have further questions or comments, we remain at your disposal.*", "claims_and_evidence": "The claims are well-supported.", "methods_and_evaluation_criteria": "The methods are appropriate.", "theoretical_claims": "The theoretical claims are straightforward extensions of the DMFT results of Bordelon & Pehlevan (2021), and appear sound.", "experimental_designs_or_analyses": "The experiments are generally well-designed, and the authors provide a good selection of additional figures in the Appendices.", "relation_to_broader_scientific_literature": "This paper bridges two bodies of work in machine learning: that on optimal network parameterizations, and that on catastrophic forgetting. It is well-situated within the literature, and should be of broad interest to the ICML audience.", "essential_references_not_discussed": "The authors do a generally good job of reviewing related prior art, but there are few missing references. \n\n- Petrini et al. (\"Learning sparse features can lead to overfitting in neural networks\", NeurIPS 2022) studied how very rich feature learning can lead to overfitting to spurious features. This bears a close conceptual relation to the results here on severe catastrophic forgetting in very rich networks. Moreover, it offers a previous example setting where more feature learning is not better. \n\n- Vayas et al. (\"Feature-Learning Networks Are Consistent Across Widths At Realistic Scales\", NeurIPS 2023) show that beyond some minimal width, $\\mu$P-parameterized networks of different widths behave very similarly, which is related to the authors' findings in Figure 2.", "other_strengths_and_weaknesses": "- I do not think the title does a good job of conveying the main contributions of the paper. I would suggest something that makes the main conclusion obvious. To give a very rough example, perhaps something like \"Rich feature learning can accentuate catastrophic forgetting\". \n\n- One substantial concern I have is with the role DMFT plays in the paper, as I think the space devoted to it in the main text might better be used to present additional experimental results (for instance, the experiments on training time could be promoted to the main text). To be very clear, I am by training a statistical physicist, so I think it's nice that the authors have the DMFT description. However, it is not enough to write down the self-consistent equations; there must be some conceptual meat extracted from them. My concerns are (1) that the DMFT results do not substantially strengthen the claims based on experiment, and (2) that the technical novelty here relative to the cited work of Bordelon and Pehlevan is minimal. What do we gain conceptually from the perturbative approximation? \n\n- The authors do not adequately discuss how some of their findings (particularly the relatively sharp increase in CKA with $\\gamma\\_0$) relate to those of Atanasov et al. 2024 (\"The Optimization Landscape of SGD Across the Feature Learning Strength\"), which documents similar phenomenology. This is also relevant to their choice of scaling of learning rate with $\\gamma\\_0$.\n\n- I think the paper would benefit from some futher analysis of representational changes across tasks. For instance, the authors could plot as a function of training time the kernel-target alignment (as in Atanasov et al. 2024) for each task. This would help clarify precisely what structure is learned and forgotten. Moreover, it would help relate this work to the abovementioned work of Petrini et al. 2022. \n\n- The paper is missing some methodological details on how the authors solved the DMFT equations. In line 771, the authors state that they \"implemented simple computational physics discrete-time dynamics\", but this is too vague to be useful. Presumably they just modified the solver from Bordelon & Pehlevan, but this should be stated.", "other_comments_or_suggestions": "- In figure lables, the authors should just write 1-CKA instead of \"Features evolution, 1-CKA\". \n\n- Mentions of Bordelon et al (2023) for depth scaling in ResNets should cite also the contemporaneous work of Yang et al. \n\n- Please make sure to state the dataset and number of samples used in figure captions (e.g. this is missing in Figure 4). \n\n- I think the claim in the Discussion that \"our findings complete the picture on feature learning in modern NNs\" is too broad, particularly given previous work on settings where feature learning harms generalization.", "questions_for_authors": "I am a bit puzzled by the use of \"capacity\" at a few points in this paper, e.g. in line 352 where it is stated that increasing $\\gamma\\_0$ increases capacity. I would associate capacity more with expressivity than with learning-related features. Can you clarify this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q72CPewfXy", "reviewer": "Reviewer_DJW3", "summary": "This paper explores the relationship between scaling regimes and catastrophic forgetting using the lens of dynamical mean field theory (DMFT). The authors demonstrate theoretically that in feature learning regimes, catastrophic forgetting is more likely. In particular, there is a sharp transition between the lazy and rich regimes with respect to forgetting, which the authors term as edge of laziness (EoL). They show that in continual learning settings, there is an optimal choice of laziness level, which is transferable across model capacities. The authors further note that as forgetting becomes less important (as tasks are more similar), the optimal level of richness increases (optimal laziness decreases).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Thank you for taking the time to read and review our paper. We are glad to hear that you found our paper well-written and that it offers a novel contribution to the space of continual learning.*\n\n---\n\n## Edge of Laziness and Phase Transition\n\nThank you for your thoughtful comment, and we are sorry for the confusion caused by the terminology of Edge of Laziness.\n\nFirstly, we would like to take the opportunity to clarify our position on the observed phase transition. In our results, it is clear that a phase transition happens, however, it does not always appear as a sharp, discontinuous transition between the two regimes, but more like a non-linear continuous function of $\\gamma_0$ in the area where the transition occurs. For this reason, we do not make any claims about the nature of the transition, being it *discrete* or *sharp but still continuous*.\n\nSecondly, our choice of terminology, \"Edge of Laziness,\" was not intended to rigorously characterize the phase transition. Instead, it was intended to be evocative of a boundary between an effectively lazy region of the $\\gamma_0$-space.\n\nThirdly, we are currently unable to demonstrate the nature of the phase transition theoretically, a result that we believe to be highly non-obvious. Nevertheless, we are actively working towards this result, and we were recently able to characterize the infinite-time solution of CF in the rich regime of the linear case. In this setting, we observe a highly non-trivial relation between the task similarity $\\rho$ and $\\gamma_0$, where the second-order coefficient of CF is a sixth-grade polynomial in $\\rho$. The complexity of this relationship - even in the linear case - points to the non-triviality of the behavior and thus to the difficulty of getting a general characterization of the phase transition theoretically. Nevertheless, we will continue working to extend this result to the non-linear setting for future work.\n\nFollowing your comment, and determined to avoid any confusion or misunderstanding with future readers, we have rephrased the parts of the paper that introduce and discuss the concept of the edge of laziness. We decided to refer to it as *lazy-rich transition*, avoiding the word \"edge\". We are also in the process of changing the title to reflect this change of terminology accordingly. We are grateful for the opportunity to improve the clarity of our claims and paper.\n\n## Other Comments\n\n- **DMFT with one hidden layer NN**\n\n    We plan to extend the DMFT to deeper networks in future work. In the meantime, however, we would like to stress the striking overlap between the results in the shallow and deep networks, suggesting that the one-layer derivation is already able to represent well the behaviors of deep networks.\n- **Edge of Laziness, tradeoff, and optimal $\\gamma_0$**\n\n    Thanks for the interesting question, which allows us to touch upon two distinct phenomena that influence the overall optimal level of richness.\n\n    Firstly, we have observed that the task similarity has a direct and powerful impact on the feature evolution itself, meaning that a higher similarity effectively reduces the amount of feature learning for fixed $\\gamma_0$ (Fig. 5a). This suggests that the task similarity (or in general the degree of (non-)stationarity of the data) non-trivially interacts with the regimes of the network. Including this perspective into the DMFT formalism is non-trivial, and we have just started exploring this avenue (see Appendix E3, F). However, we believe this direction would be very interesting and valuable for future work.\n\n    Secondly, the optimal $\\gamma_0^\\star$ reflects the plasticity-stability tradeoff, and therefore, $\\gamma_0^\\star$ can shift from the $\\gamma_0$ at which the *lazy-rich transition* happens. Generally, we find that when the tasks are close to stationary, the optimal $\\gamma_0^\\star$ is always 1, i.e. the maximum of the range tested. In other words, when the data is stationary, higher plasticity helps reach better performance. By contrast, as the amount of non-stationarity is increased, stability is necessary to avoid losing performance on the old data and the optimal $\\gamma_0^\\star$ is consistently lower than 1. From this point of view, the $\\gamma_0$ at which the *lazy-rich transition* happens represents the minimal $\\gamma_0^\\star$ as any value below it will not increase stability (because the network is effectively lazy) and it will not increase plasticity.\n\n    While we do observe the plasticity-stability tradeoff even in the theoretical experiments at infinite width (Fig. 4a), we have not specifically investigated it theoretically.\n\n- **Fig. 6 labels**\n\n    Thanks for pointing out the cut-off labels of Fig. 6, and for suggesting an increased size of the Figure. We will correct these in the final version of the paper.\n\n---\n\n*We hope that we have adequately addressed your questions. If you have further questions or comments, we remain at your disposal.*", "claims_and_evidence": "Overall, the claims made in the paper are generally well supported by theory and experiments.\n\nOne area of potential concern is with the terminology \"edge of laziness.\" The experimental results in Figure 3 could be interpreted either as a discrete transition between a lazy and rich regime, or a sharp, but still continuous transition. If the latter is the case, then I believe \"edge of laziness\" may not be the correct term to describe the results. Can the authors justify why the former is the case? Is there a phase transition at this point that the authors can show theoretically?", "methods_and_evaluation_criteria": "Evaluations are conducted on Permuted-MNIST and Split-CIFAR, which are standard benchmarks.", "theoretical_claims": "The theoretical claims in the main text appear correct. They seem to rely on standard arguments in DMFT.", "experimental_designs_or_analyses": "Experimental setup as described in the captions and throughout the main text appears sounds.", "relation_to_broader_scientific_literature": "As the authors note, previous works have already considered applying scaling limit results to continual learning. The key difference with this work is that they consider large networks in the rich regime under finite data, a previously unexplored setting for continual learning.", "essential_references_not_discussed": "As far as I am aware, the relevant literature is discussed.", "other_strengths_and_weaknesses": "Overall, the paper is well-written and presented and makes a novel contribution to the space of continual learning theory.\n\nA key weakness of the theoretical results is that the DMFT techniques used by the authors applies to one hidden layer neural networks. Experimentally, the authors consider deeper networks though, which somewhat alleviates this concern.", "other_comments_or_suggestions": "It looks like the labels in Figure 6 are cut off. I would also recommend increasing the size of the Figure (perhaps some of the whitespace could be removed).", "questions_for_authors": "As mentioned above, can the authors theoretically demonstrate a phase transition between the lazy and rich regime (for edge of laziness)?\n\nAlso, can the authors theoretically show a connection between the edge of laziness and the optimal stability-plasticity tradeoff? For what kinds of tasks is it true that the optimal $\\gamma_0$ exists at the edge of laziness?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission presents a theoretical and experimental investigation into the effects of neural network parameterization on catastrophic forgetting, extending previous work that focused on the lazy regime (i.e. low plasticity). The authors identify a spectrum of training regimes characterized by a parameter, $\gamma_0$, revealing that variations in parameterization lead to different scaling laws related to forgetting. A notable finding is the *Edge of Laziness*, marking a transition influenced by task similarity in which forgetting shifts from low to high. The paper combines Dynamic Mean Field Theory (DMFT) with empirical validation on standard CL tasks such as Permuted MNIST, CIFAR-10, and TinyImageNet.

Reviewers unanimously appreciate the clarity of writing and the potential significance of the contribution, noting that the analysis bridges prior findings and offers new insights into the influence of parameterization on catastrophic forgetting. Some reviewers comment that the terminology "edge of laziness" lacks clarity in terms of the real contribution, and the extent to which DMFT enhances the experimental claims. There were also several clarifications and corrections requested -- especially the toning-down of the claims made in the discussion and the adjustment of the title to better articulate the content and contributions of the work.

Overall, the reviewers found the work intriguing, well-supported by both theoretical and experimental data, and a valuable contribution to the understanding of the dynamics of CL. The authors are encouraged to incorporate the clarifications presented in rebuttal as there are several that help articulate the subtleties of the theoretical predictions and empirical verification.
Decision: Accept (poster)