The Dual Nature of Plasticity Loss in Deep Continual Learning: Dissection and Mitigation
Authors: Haoyu Wang
Wei P Dai
Jiawei Zhang
Jialun Ma
Mingyi Huang
Yuguo Yu
Venue: NeurIPS 2025 poster ()
Keywords: Deep Continual Learning
Loss of Plasticity
Finite-time Lyapunov Exponent
Primary area: deep_learning

Abstract: Loss of plasticity (LoP) is the primary cause of cognitive decline in normal aging brains next to cell loss.
Recent works show that similar LoP also plagues neural networks during deep continual learning (DCL). 
While it has been shown that random perturbations of learned weights can alleviate LoP, its underlying mechanisms remain insufficiently understood.
Here we offer a unique view of LoP and dissect its mechanisms through the lenses of an innovative framework combining the theory of neural collapse and finite-time Lyapunov exponents (FTLE) analysis.
We show that LoP actually consists of two contrasting types: (i) type-1 LoP is characterized by highly negative FTLEs, where the network is prevented from learning due to the collapse of representations; (ii) while type-2 LoP is characterized by excessively positive FTLEs, where the network can train well but the growingly chaotic behaviors reduce its test accuracy.
Based on these understandings, we introduce Generalized Mixup, designed to relax the representation space for prolonged DCL and demonstrate its superior efficacy vs. existing methods.

The Dual Nature of Plasticity Loss in Deep Continual Learning: Dissection and Mitigation
Haoyu Albert Wang 1.2
Wei P. I4.f. weidai@fude edu en
Jiawei Zhang 1.3
maoywangg8@@fudan edu. cn
Zhangj fudan. edu. cn
Jialun Ma
Mingyi Huang!. 1.2
Yuguo Yu
21307130025mm. fudan. edu. cn
myhuang200fudan. edu. cn
yuyuguoffudan, edu, cn
1. Research Institute of Intelligent Complex Systems, Fudan University.
2. Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University.
3. State Key Laboratory of Brain Function and Disorders and MOE Frontiers Center for Brain Science, Institutes of Brain Science, Fudan University.
4. Shanghai Artificial Intelligence Laboratory.
f These authors contributed equally to this work.
I Corresponding author.
Abstract
Loss of plasticity (LoP) is the primary cause of cogmitive decline in normal aging brains next to cell loss. Recent works show that similar LoP also plagues neural net- works during deep continual learning (DCL). While it has been shown that random perturbations of learned weights can alleviate LoP, its underlying mechanisms re- main insufficiently understood. Here offer unique view of LoP and dissect its mechanisms through the lenses an innovative framework combining the theory of neural collapse and finite-time Lyapunov exponents (FTLE) analysis. We show that LoP actually consists of two contrasting types: (i) type-1 LoP is characterized by highly negative FTLEs, where the network is prevented from learning due the collapse of representations; (ii) while type-2 LoP is characterized by exces- sively positive FTLEs, where the network can train well but the growingly chaotic behaviors reduce its test accuracy. Based on these understandings, we introduce Generalized Mixup, designed relax the representation space for prolonged DCL and demonstrate its superior efficacy vs. existing methods.
1 Introduction
Loss of neural plasticity has been identified the main reason for the progressive cognitive decline in normal aging brains without neurodegenerative diseases (i.e., not due to loss of neurons) [4. Deep learning systems, despite their success plasticity (LoP) with widespread effects in deep reinforcement learning (DRL) [23 [ 20 and deep continual learning (DCL, c.g., Fig. Ih) 7. Unlike catastrophic forgetting, which describes neural network's 's poor performance on previous tasks after learning new tasks, LoP is characterized by the deteriorating performance on new tasks which eventually prevents the system from learning continually. Moreover, given the size of datasets and high energy demand of modern Al training [30]. the cost of time and resource required re-train the existing networks from scratch to adopt new tasks is increasing prohibitive [31]. Therefore, it is crucial that we understand and tame LoP as the need higher than ever for adaptive AT that can efficiently learn new tasks continually.
various domains, also struggle with similar loss of
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Related Work The study of the declining test performance of a continually trained deep neural networks (DNN) began with the efforts to understand different phases of training [2 and incremental learning [5] Later LoP gained its name, i.e. loss of plasticity, and the most attention within the
community of DRI,, as DRL networks are constantly facing shifting objectives and changing input
distribution with the evolving policies [14 16
24 23 20
21 22
12
28
7
6
1 [0]. Recent works
also show that supervised DCL suffers from the same pathology 7 8
17
18
22
Plenty of studies have attempted to explain LoP. One commonly cited cause is the accumulation of dormant or saturated neurons 28 8 7 Activations such as ReLU and tanh are known suffer from such effects where their outputs becomes permanently zero or constant, leading to vanishing gradients which reduce the network's expressivity. Another line of work attributes LoP to a decrease in feature rank [T, Z7220,, which characterize the drop in the diversity of internal representations as training progresses, weakening the network's ability to adapt to new tasks. Other hypotheses include the growth of weight norms [22], which may destabilize training, and the loss of gradient curvature[I8], where training dynamics restrict gradient directions to subspace, thereby constraining the exploration of alternatives. These studies provide valuable insights toward the understanding of LoP. However, some argue that LoP can still occur in absence the aforementioned conditions Therefore, more comprehensive understanding of LoP is in need.
21
Various methods have also been proposed to mitigate LoP. They can be roughly categorized into reset-based, regularization-basse based. and architectural or optimizer- approaches. Based on the observation that DNN exhibit the highest plasticity at initialization, an earlier work suggested retraining newly initialized networks on a replay buffer[14). While effective, this strategy incurs significant computational cost. To reduce overhead, later works only apply resets on specific neurons instead, e.g., on the less "utilized" neurons 66] 77. on neurons with "zero firing rate" (IO) or whose activation are relatively low within each l2yer[28). Regularization baaed methods are also motivated by the network's 's plasticity at initialization but take softer approach, i.e., they encourage the network to retain some properties of its initial state. For instance, by applying an L2 regularization between the current weights and the initial weights (L2-init(17), or with the Wasserstein-2 distance [18] for more proximity to the initialization. Similarly, distillation of initial features (2), as well as a "shrink and perturb" of the weights before each optimization 3] are proposed to maintain feature diversity throughout learning. Architectural and optimizer-badn based approaches are more intricate, including concatenating two ReLU activations to mitigate dormancy [1] (at the cost of increased model size)
and applying layer normalization [21 [22] Additionally, Lyle et al. [21] demonstrated that adjusting the decay rate of momentum resetting the optimizer help restore plasticity.
(a)
Type-1 LoP
Task
VS.
within-class input
Collapse within class
Continual Learning
Accuracy
train
FTLE < O
FTLE + -X
Task
vs.
test
Task N
Tasks
input space
rege. space
rear space
(b)
(d)
Type-2 LoP
Chaos boundary
FTLE
ADI
UFM W
train
near-boundary inpurt
Accuracy
Input
Output
test
Tasks
input
rept.
space
Figure Illustration of LoP, (a) Setup of "Continual ImageNet". adapted from 77 (b) The effective framework to analyze LoP using FTLE and UFM. (c) and (d) Illustrations of the difference between type-1 and type-2 LoP the level of task performance and their contrasting underlying causes, characterized by FTLE.
Contribution As mentioned above, despite various approaches to understand and mitigate LoP, we are still not sure what drives LoP fundamentally. In this work, we comprehensively dissect and address LoP from a new perspective to provide deeper understandings along with better treatment and less computational cost.

---

Specifically, (i) We offer an innovative framework combining finite-timi time Lyapunov exponent (FTLE) analysis [2 with the Unconstrained Feature Model (UFM) used in the theory 3C
(Fig Fig[b) where FTLE quantifies how representation space transforms during training and UFM provides the analytical tractability for optimization representation space despite DNN's ' non-convexity. Using
this framework, (ii) we identified the two types of LoP during DCL. On the level of task performance, they differ only in training accuracy, however, (iii) we further unveil that their underlying causes are exactly opposite: the collapsing of representation space within class (corresponding to increasingly negative FTLEs) is the direct cause of type-1 1 LoP, while chaotic behavior near the boundaries (corresponding increasingly positive FTLEs) is the direct cause of type-2 LoP, (Fig [c. d). (iv) Based on these understandings. we proposed and tested a generalized version of Mixup [35] as the better prescription for LoP.
2 Preliminaries
Here, briefly list the preliminaries of work, i.e., finite-time Lyapunov exponent (FTLE), the theory of Neural Collapse (NC) with unconstrained feature model (UFM), and set up the problem of LoP based existing studies. Readers familiar with the subjects can safely skip this section.
FTLE of DNN A DNN with L, hidden layers apply a composition of multiple nonlinear mappings, ( on input x to its hidden layer representations (()), vl E [} (Fig. [p), and the penultimate layer representation reads:
h (L) = -GLL) ((L-1) o (x),
The propagation of a local change in input, òx, is characterized by the network's 's Jacobian composed of derivatives of the activation D (e) and weight matrices (
8h ( (") ("(
This layer-by-layer mapping from the input space to its deep representation space can be interpreted as the evolution of a dynamical system, with layers discrete time steps [29]. therefore the network's 's sensitivity local changes in input can be analogously described by the maximal Lyapunov exponent used for states y(t in dynamical systems, but with finite time, i.c., FTLE:
(8y()) [0)
1
(
A(y) = lim log
(x)
= I log (x)
By definition, a positive FTLE indicates significant local divergence in the representation, often corresponding classification boundary; and negative FTLE suggests convergence in the representation space for local inputs, usually belonging to the same class. Note that, we use the unnormalized
form form of FTLE, X LX(4) this paper.
Neural Collapse and the Unconstrained Feature Model Neural collapse (NC) deseribes remarkable phenomenon observed during the terminal phase of training (25), where the network's 's last-layer features converge the means within their class, and align with the classifier vectors optimally to form a simplex equiangular tight frame that enhance test performance even when the training error approaches zero.
Theoretical analysis of NC adopts the Unconstrained Feature Model (UFM), where the last-layer representations, hb for E {1 .1.... n), in each class i are treated free optimization variables just as the classifier weights W (Fig to circumvent the non-convexity of DNN, enabling analytical tractability (36] 34 91. Omitting the bias term in the last-layer output, the optimization of UFM is
simply the sum of loss L over K different classes against their labels, y/
target
K M 1=1
min W
C(WH
target
(4)
K
where matrix H = | A1 Prior studies on single classification tasks have demonstrated that
the solution to this optimization inherently satisfies the properties of NC [36] 9 27
3

---

Problem setup of LoP The training paradigm in DCL can be formally defined by considering sequence of T tasks (Ti. T2.... }r where each task T₁ associated with its own training set, urain and test set, D(C) The goal of DCL is to train a DNN that continually adapts to new tasks, updating the parameters of the network O2 after each task Tj such that 0: = arg ming L(D( where L denotes the loss function [T. LoP then formally refers to the phenomena that the test performance of network with continually updating parameters, 6,, becomes increasingly worse than that of a network trained from scratch with initialized parameters, Og- In terms of loss, given a (t)
D(1)
(test) 0c-1),
test: 0j) < L(D(s) test: 0z) for all tasks Tr, Vt' > T", where f denotes the parameters of reinitialized network after training on task Tr. Recently. Dohare et al. [7] has demonstrated the ubiquitous existence of LoP in DCL, particularly in the binary classification tasks where the network continually learns with consistent difficulty. Without loss of generality, this study focuses mostly on binary classification as well for ease of comparison, but we also tested our method on class-incrementee tasks (Sec. 4
large enough number of trained tasks, T", LoP occurs in the network when L(D
3 Main Results
Although the theory NC has successfully described the optimal representational structure trained on classification tasks, whether the theory fits the context of continual learning remains open question. Moreover, the theory utilized UFM for analytical tractability, which ignores the mapping from input space to representation space altogether (Sec. 2 Thus, dissect the underlying mechanisms of LoP, we develop a framework in Sec. that combines the analysis of FTLE (which quantifies the geometric mapping between the input space and the representation space, see Sec. 2 with NC to provide the whole picture of training dynamics (Fig. [T)). Extending this framework to DCL, we identify two distinct types of LoP and their causes, which are verified with toy models in Sec and we check the existence of two types of LoP for networks trained on ImageNet. Finally, Sec 3.5 we introduce generalized version of Mixup address both types of LoP, grounded in our theoretical understanding of their underlying mechanisms.
3.3 In Sec 3.4
3.1
Stretching and compressing Representation Space by Training
The gradient of cross-entropy (CE) loss with respect to the classifier weights and representations consists of "push" term and a "pull" term (34]. This argument also holds for the mean squared error (MSE) loss.
Consider a binary classification task. Let h. E Rd be the penultimate layer d-dimensional rep- resentation of the i-th sample in class h; and let Wa E IRd denote the classifier weight vector of class k. where Is h4 can be written in the corresponding matrix form, Y!arger = [1. O: 0, 1]. Thus, the MSE loss reads LMSE = y 2 Taking its gradient over Wy for l = A, B, we have:
or B. The network output for paired samples can take the matrix form, Y, [w] Whih w W w h w wh Then, the corresponding one-hot labels of class A and B
0LMSE Owk
(1-(1-
Σ M (0-Wāhi) (.
where the first term pulls the weight vector We closer to the same-class representations hy. The second term pushes it away from the representations hy of class k' * k. Similarly, the gradient over by shows two other "pull" and "push" terms acting on the representations:
0LMSE ahk
(1 (1- (1-wThh) Wa + (0 (0-w wiihh WEs
(6)
Altogether, they induce maximal class separability, from which have the following theorem:
Theorem 3.1.1 (Optimal Solution for Binary Classification) Under UFM and the assumption that both classes contain samples, |A| B = n, consider the following optimization prob-
lem:
fWW H) = WH - Y 1 + Aw W.
AH H
min
11 11?
(7)
2
2n

---

where xw and Ry are the coefficients of the two regularization terms that control the magnitude of the classifier weights. W. and deviation of the learned representations H from the initial representations Hi orthogonal to W. The optimal solution obeys:
W = B--WB:
h Ne. v /w//AH We + 1
(8)
where k = A or B, and L. R1.0 is the k-component of Ho- (For proof, see Supplementary
B
Assumption 3.1.1 Distribution of the real-world data satisfies the measure of concentration and the empty space phenomenon, two critical characteristics of figgh-dimensonall data [2).
We empirically verified that the datasets used in our experiments satisty Assumption Supplementary for details). This validation allows us directly employ FTLE analysis on local pairs of samples to quantify the transformation of the representational space, as the within-class regions and areas near classification boundaries are affected by local sample points only.
(sec
Theorem 3.1.2 (FTLE in Representation Learning) The distance between two samples and from classes and k' in representation space, Ahij k.k' h -hý can be quantified by FTLE, do and Mained each denotes its values before and after training, respectively.
(1) For samples from different classes (k + K)), training produces positive FTLE ridge, stretching the space in- in-between, and the magnitude of stretch is captured by stretching factor, oz
k.,k' x
k. X0
log a = mmined
> 0,
(2) For samples of the same class (k- = k),, training carves out a valley of negative FTLE, compressing the space in-between, and the degree of compression is given by compressing factor, 3:
log 3 = Atrained
Xo < 00
(10)
(For proof, see Supplementary D
3.2 Type-1 LoP Induced by the Collapse of Representation Space
According to Theorem 3.1.2] within-class regions are compressed with factor of 3r < 1 during the training of the same class samples. In the DCL setting, after learning T tasks, the FTLE evolves as: k.k 1 = Xb= + log II? St. When the representational space is high-dimensional, the compression factor 3g is close to 1; and 3g decreases with lower effective dimensionality (for details,
see Supplementary D) Over many tasks, the compressing effect accumulates:
trained + -0 as I 3, +
(11)
resulting in the collapse of representation space, where different samples (of the same class) become indistinguishable. This severely impairs the network's 's ability to learn new representations near the collapsed area for new tasks.
Definition 3.2.1 (Type-1 1 LoP) In the settings of DCL, after trained by T tasks, Type-1 LoP is resulted when the representations from different class, ioj and h2 of a new task TT+1 come near the collapsed area, quantified by a highly negative FTLE N Troine, (resulted from previous training). On the level of task performance, both training and test accuracy drop significantly, indicating loss of capacity in learning.
(a)
Task Te
(b)
1.0
(c)
(d)
train test
X
Task
Accuracy
Probability
Ticon
Mean FTLE
X
Tit1
Task T5
-16
X
-20
10
15
20
-15 -10
15 20
A: O
B:x
Group Index (every 50 tasks)
FTLE
Group Index (every 50 tasks)
Figure 2: Type-1 LoP in a low-dimensionnl minimalistic toy model. (a) Illustration of the continual XOR classification task. (b) Training and test accuracy rapidly degrade over tasks. (c) Sampled distribution of FTLE shifts to lower values. (d) Mcan FTLE are negative and decrease over tasks.
5

---

To intuitively understand Type-1 1 LoP, we construct minimalistic DCL task based on XOR classifi- cation (Figure 2a)). The inputs lie in a 2-D plane partitioned into four regions by two orthogonally crossing boundaries (diagonally opposite regions belong to the same class). After each task, the boundaries are rotated randomly by 0, and the classification head is reinitialized. The random rotation simulates the lack of alignment between task boundaries typically observed in real-world scenarios, where consecutive tasks often exhibit a wide range of correlation. The network is a 12-layer MLP with 10 neurons per hidden layer, using tanh activation and MSE loss, As suggested by our theory, low-dimensional representational spaces are more susceptible to Type- LoP, making this architecture an ideal testbed.
The experimental results strongly support our analysis. As the number of tasks increases, both training and test accuracy degrade sharply and eventually to chance level (Fig. (Fig.Zp). The sampled FTLE distribution (Fig- 2 shifts toward highly negative values as the number of tasks increases, indicating a growing proportion of indistinguishable representations of inputs. The trend in mean FTLE (Figure(2d) further reveals the progressive compression of the representational space, driving the network to state where it can no longer accommodate new task representations.
(b)
1.0
(c) 0.14
(d)
Task
4.0
Accuracy
0.9
train
Probability
3.5
0.8
was best
Tack Tion
3.0
Mean
0.7
Task T2co
2.5
0.6
0.00
2.0
0
50
100
150
200
2 4 3
0
50
100
150
200
Task
Task Taaa
Task ID
FTLE
Task ID
Figure 3: Type-2 LoP in a high dimensional minimalistic toy model. (a) Illustration of a 3-D projected 30-D toy model, where the hyperplanes classification boundaries. (b) Training accuracy sustains near perfect while test accuracy degrades over time. (c) Sampled FTLE distribution quickly shifts toward larger values. (d) Mean FTLE are positive and increase over tasks.
3.3 Type-2 LoP Induced by the Over-stretched Boundaries and Chaotic Behaviors
According to Theorem 3.1.2 training a network with samples near the regions that cross the bound- aries stretches the representation space with a factor of a; > 1. In the DCL setting, after learning k.k'' I' tasks, the FTLE evolves as: tranned = Ao k,k' + log IT:., az. This becomes more pronounced in high-dimensional representation spaces. Over many tasks, the accumulated effect yields:
k,k'' T
m 0: 0C.
(12)
From the perspective of dynamical mean field theory in DNNs (26), large FTLE values signal the onset of chaotic dynamics. This manifests extreme sensitivity to small input perturbations, where similar inputs are mapped to highly divergent representations. Despite the chaos, "uncollapsed" representation space support the network with enough expressive power to fit the training sets of the new tasks well. However, if the test samples come near the chaotic regions, the resulting representation becomes disordered and the network fails to generalize.
Definition 3.3.1 (Type-2 LoP) In the setting of DCL, after trained by T tasks, Type-2 LoP resulted when the representation of sample, h4, from new task Tr41 comes near overly stretched area K.KK' (resulted from previous training) quantified by a highly positive FTLE T-iiedd On the level of task performance, training accuracy is guaranteed by the highly expressive chaotic representation space while test accuracy degrades, indicating a loss of capacity in generalization.
X.R
To empirically validate the emergence of Type-2 LoP, we designed a minimalistic toy model with 30-D input space (a projection of which is illustrated in Fig. 3 since Type-2 LoP is rooted in chaotic dynamics, as mentioned above. Data points are sampled uniformly from 1330 Three parallel hyperplanes partition the space into four regions with equal size, the same class assigned to every other region. For each task, the orientation of the hyperplane is chosen randomly and the classification head reinitialized. The network is a 12-layer MLP with 300 neurons per hidden layer, using tanh activations and MSE loss. As predicted by our theory, representation spaces with higher dimensions are more susceptible to stretching, making this architecture an ideal testbed for Type-2 LoP.

---

The results clearly demonstrate Type-2 LoP. The mean of FTLE increases steadily with each task as well as its sampled distribution, indicating progressive expansion throughout the representation space (Fig. Be and d). On the level of task performance, training accuracy sustains, while test accuracy quickly declines (Fig. 3)), revealing growing generalization gap the hallmark of Type-2 LoP.
3.4 Verifying LoP in Real Datasets
In Sec. 3.2 and 3.3 we constructed minimal synthetic toy models to illustrate two distinct types of LoP and validated theoretical predictions. From the proof of theorem 3.1.2 we know that the dimensionality of representation space affects the rate of compression and expansion (see Supplemen- tary DD but it does not necessarily prevent either one. In DCL, the prolonged training can gradually activate both mechanisms, leading to Type-1 LoP, Type-2 LoP, or both. The specific form depends on whether the representation space where the new data points reside, collapses or expands.
To validate our theory in a real-world setting, we evaluate LoP behaviors on the "Continual ImageNet" benchmark (7). sequence of binary classification tasks designed for DCL (see details in Sec. [4). We use the same CNN setup from 7 for the high-dimensional case, and a narrowed version to model the low-dimensional case (see Supplementary for details).
Low Dim Space
High Dim Space
(a)
Type
(b)
Type 2
Accuracy
1.00
--. test
Probability
0.3
Task Teooo
Task Tinod
Task Tioo
Accuracy
test
train
Probability
0.25
Task Tipo
Task Tiooo
Task Teno
Chaos
Collapse
0.60
5000
-15-10-5 0 FTLE
5000
155
10-
Task ID
Task
FTLE
Type 1+ Type?
(d)
Type 1+ Type
1.00
1.00
Accuracy
train test
Probability
0.08
Task Task/Tooco
Tioo Task T1oco
Accuracy
train
Probability
Task Tinco
Task
Task7oooo
Chaos
Collapee
Collapse
0.60
0.65
5000
-15-10-5
5000
-15-10-5 5 10 15 20
Task ID
FTLE
Task ID
FTLE
Figure 4: Representative LoP scenarios in a real dataset. (a), (c): www-dimensional representation
space (a) Type-1 LoP only; (c) Co-occurrence of Type-1 and Type-2 LoP. (b), (d): High-dimensional representation space- (b) Type-2 LoP only; (d) Co-occurrence of both types.
Fig. J four representative cases that capture typical LoP behaviors with representation space of different dimensionality (D): In the low-D D case of Fig. 44. Type- LoP dominates, as indicated by a sharp drop in both training and test accuracy and a shift in FTLE distribution toward strongly negative values. In another low-D case (Fig. 4b), both types of LoP emerge. The FTLE distribution broadens significantly, with negative values reflecting collapse (Type-1) and large positive values reflecting chaotic expansion (Type-2). In the high-D case of Fig. Ap. only Type-2 LoP is observed. The FTLE distribution shifts toward positive values as network generalization degrades despite full training accuracy. In another high-D case (Fig. It), two types of LoP co-occur. Initially, Type-2 LoP dominates due to rapid stretching near boundaries. As more tasks are learned, within-class regions begin collapse, confirmed by the bimodal expansion of the FTLE distribution.
3.5 Mitigating LoP with Generalized Mixup
Since we have shown that there are two types of LoP, an effective mitigation strategy must alleviate intra-class collapse and inter-class boundary expansion at the same time. Here, we draw inspiration from the classical Mixup 35 It generates synthetic data by interpolating pairs of samples and their labels:
(13) where follows Beta distribution. This formulation linearizes the decision boundaries and is effective against the excessive boundary stretching in Type-2 LoP. Specificalll.chaotic behavior in high FTLE conditions, (8h(4)] =e (=))__))) let small input perturbations to be exponentially
x
m + (1 m)a"". = + (1-m)y* m)
7

---

amplified in the representation, often in the direction along the classifier wK. Classical Mixup encourages approximately linear transitions:
Wh(mx* K m))* = mWh(1") (1 m) m))(("))
(14)
thereby suppressing such instabilities and regularizing the representation near the decision boundary.
However, classical Mixup does not address Type-1 LoP, which is caused by representational collapse within a class. To mitigate this, we introduce Generalized Mixup (G-miuup), which modifies the label assignment during intra-class interpolation. For two samples x!! and xy define:
R from the same class,
M M10.5 ml, 2
yk yk M M ml. (15)
2.th ma? (1 m)ay .
in the case of binary classification, where M is confidence amplification factor. This encourages the network expand the intra-class representation, countering collapse and maintaining variability within the class (Figure 5a). For example. if the original labels for dog and cat are [0.8, 0.2] and
[0.2, 0.8). inter-class Mixup uses the standard convex interpolation (Fig b): y mm00., 0.2] (1 - m) (0.2, 0.8). while intra-class G-Mixup (e.g., dog-dog mix) adjusts to (Fig 5 a): y"
0.4|0.5 ml. 0.4[0.5 mm]]
We provide PyTorch-like pseudo-code for implementing G-Mixup with both MSE and NLL losses in Supplementary
4
Experiments
We evaluate the effectiveness of G-Mixup on two continual learning benchmarks 7 Continual ImageNet and Class-Incremental CIFAR-100, Continual ImageNet binary rask-incremental setting with 5000 tasks total, where the network is trained with MSE loss and a CNN backbone. This setup enables long-lifeting plasticity analysis. In contrast, Class-nncrementll CIFAR-100 uses CE loss and ResNet-18 backbone under 20-task class-ineremental learning, offering complementary insights into differrnnt continual learning scenarios.
In both benchmarks, we consider two baselines: (1) Reinit, which trains each task from scratch by leveraging the inherent plasticity of randomly re-initialized parameters, and (2) No Intervention (No Intv.), a continual learning model trained without any LoP mitigation trick. Additionally, compare G-Mixup with three representative LoP prevention methods across different strategies: regularization-base based (I.2 Init (17),, architecture-bat (LayerNorm (1)). and reset-based (Continual Backpropagation, CBP (
Continual ImageNet In this benchmark, the model sequentially learns to distinguish between 5000 binary classification tasks randomly sampled from the ImageNet dataset. Each class contains 600 training and 100 test images. Networks are trained for 250 epochs per task using SGD (batch size 100). Only classification head is reinitialized the beginning of each task simulate task-incremental learning.
We consistent architecture with three convolutional layers followed by three fully connected layers. We compare a wide network (matching (7) representing high- nigh-dimenssonal space, with a nar- rower version for the low-dimensional case. Implementation details are provided in Supplementary G
(a)
Intra- Class-Clas Mixup
(b)
Inter- Class Mixup
[0.2.088
[0.8.0.2]
[0.5.0.5]
2,0
20.
0.8.0.2]
Original
25
00.5
=0.5
[1b]
[0.08.0
0.92]
0.8.0.2
0.8.0.2
[0.38.0.60
0.2.0.8
Man 50..
m 0.75
Mixed
m
0033
m07
Streching space
Streching space
Linearizing boundary space
Figure Schematic of Generalized Mixup mitigating both type LoP. (a) Preventing intra-class representation collapse (Type-1). (b) Avoiding excessive inter-class boundary stretching (Type-2).

---

All models use ELU activation, except for CBP, which employs ReLU as originally proposed and achieves better performance with ReLU under the same setting. All results are averaged over five random seeds. Standard deviations are reported in Supplementary H
Table SmallConv Acc. on Continual ImageNet
Table 2: ConvNet Acc. on Continual ImageNet
Task (x1000)
0-1
1-2
2-3
3-4
4-5
Task (x1000)
0-1
1-2
2-3
3-4
4-5
No Intv.
0.817
0.805
0.562
0.500
0,500
No Iniv.
0.794
0.778
0.604
0.537
0.500
Retrained
0.853
0.845
0.845
0.840
0.840
Retrained
0.857
0.851
0.850
0.849
0.846
init
01804
0.796
0.786
0.785
0.788
1.2
0.814
0.805
0.800
0.807
Layernorm
0.753
0.760
0.759
0.751
0.751
Layernorm
0.782
0.768
0.752
0.749
0.755
CBP
0.834
0.847
0.846
0.847
0.857
CBP
0.848
0.867
0.864
0.863
0.878
G-mixuptours]
0.866
0.881
0.885
0.880
0.879
G-mixupjours] 0.875
0.896
0.899
0.894
0.896
Our findings show that while all methods mitigate LoP to some extent, G-Mixup consistently outperforms other methods across tasks and settings, demonstrating superior capacity to preserve plasticity over long task sequences.
We further analyze the FTLE distributions throughout training (see supplementary D G-Mixup stabilizes FTLE within a moderate range, preventing both overly negative values (Type-1 LoP) and overly positive values (Type-2 LoP), confirming its role in controlling representation dynamics. Ablation studies (Supplementary ( validate that the benefits arise from G-Mixup itself, which effectively mitigates LoP, whereas classical Mixup severely suffers from representation collapse.
Class-Incremental C1F In this setting, the network sequentially learns new classes at each task, totaling 20 tasks to cover all 100 classes. Performance is evaluated on the full set of learned classes. Each task is trained for 200 epochs using SGD (batch size 100, weight decay 0.0005). The classification head remains fixed, with new heads added when new classes arrive, thereby simulating task-agnostic continual learning.
We adopt ResNet- 18 as the backbone. We compare the standard ResNet- 18 (high-dim) with 0.25x scaled version (low-dim), where all channels and fully connected layers are reduced proportionally, We consider the setting of G-Mixup under standard CE loss, which implicitly uses soft labels and may potentially improve model performance. To ensure fairness, all other methods are also trained with the same level of soft-label strength. All methods show noticeable performance gains after incorporating soft labels. Full configuration details are included in Supplementary K
Table 3: 0.25x Resnet- 18 Acc. on CIFAR100
Table 4: Resnet- 18 Acc. on CIFAR100
Task (x4)
0-1
1-2
2-3
3-4
4-5
Task (x4)
0-1
1-2
2-3
3-4
4-5
Intv.
0.927
0,812
0.754
0.708
0.661
No Intv.
0.907
0.847
0,812
0.778
0.743
Retrained
0.926
0.800
0.745
0.702
0.666
Retrained
0,901
0.842
0.815
0.788
0.767
1.2 init
0.925
0.788
0.724
0.682
0.649
1.2 init
0.922
0.829
0.785
0.752
0.723
Layernorm
0.851
0.755
0.723
0.674
0.641
Layemorm
0.847
0.782
0.763
0.728
0.697
CBP
0.923
0.812
0.760
0.713
0.678
CBP
0.907
0.847
0.816
0.788
0.768
G-mixup[ours]
0.932
0.825
0.772
0.732
0.697
G-mixup[ours]
0.928
0.864
0,832
0.800
0.768
As CIFAR- 100 involves more complex multi-class predictions, task difficulty increases gradually and may exceed the capacity of narrower networks. Methods relying on random initialization randomly sampled features require longer adaptation times in such settings. In contrast, G-Mixup facilitates more efficient learning and achieves higher accuracy across all tasks, even under low-capacity network contigurations.
5 Conclusions
In this work, we present unified theoretical framework that integrates FTLE analysis 29 with the UFM from the theory of Neural Collapse (36). offering a new perspective on the dynamics of representation space during continual learning. Through this framework, we reveal that LoP is composed of two contrasting types which arises from two distinct mechanisms: Type- 1 LoP results from intra-class collapse of the representation space, marked by increasingly negative FTLES, whereas Type-2 LoP stems from chaotic behavior near decision boundaries, indicated by increasingly positive FTLEs. Although both types can lead to test accuracy degradation, they differ in their geometric origins. Building this insight, propose Generalized Mixup, a method that explicitly

---

regularizes the geometry of the representation space to counteract both collapse and chaos. By jointly addressing the two types of LoP. Generalized Mixup consistently outperforms existing plasticity- preserving strategies while maintaining the computational efficiency. This work not only advances our understanding of plasticity loss through the lens of dynamical systems theory but also introduces a simple and effective approach for sustaining stable and adaptive continual learning.
Relationship to Catastrophic Forgetting(CF) CF and LoP represent two complementary facets of the stability-plaasiiity dilemma in continual learning. CF reflects excessive plasticity, where new knowledge overwrites existing representations, whereas LoP denotes the opposite failure mode-a gradual loss of the ability to acquire new knowledge, even without explicit stability constraints. While CF has been extensively studied as a stability problem, our findings reveal that LoP emerges intrinsically from cumulative optimization and geometry distortion in the representation space. In contrast, stability-driven regularization methods such as EWC (15) or LwF(19) induce a different, externally imposed reduction in plasticity by design. Recognizing these dual failure modes provides a more complete picture of continual learning: sustaining long-term performance requires balancing memory retention against continual adaptability.
Limitations While our proposed Generalized Mixup effectively mitigates both types of LoP in image classification tasks, it has several limitations. First, the method is specifically designed for supervised classification problems and may not generalize easily to other learning paradigms such as reinforcement learning, where the notion of label interpolation is less meaningful. Second, Generalized Mixup leverages the inherent label tolerance of image classification-nanan relative magnitudes of the output neurons are what determine the prediction, while the absolute values largely irrelevant. This allows flexible interpolation of targets without significantly affecting performance. However, this assumption breaks down in tasks such as regression or structured prediction, where the output must precisely match the ground truth. Interpolating inputs and targets in such settings may introduce noise or bias, leading to degraded performance.
that the
Future Directions Directly regulating FTLE is a principled way to prevent LoP. However, esti- mating FTLE on real datasets is computationally expensive, so our direct-control experiments were verified only in toy settings. In this work, we mitigate LoP indirectly by constraining FTLE magni- tudes via Generalized Mixup, which regularizes intra-class compression and inter-class boundary stretching without explicit Jacobian computation. A promising direction is to develop numerically efficient FTLE estimators and practical surrogates to enable online FTLE-aware training at scale.
Acknowledgment We thank the support from the Science and Technology Innovation 2030 Brain Science and Brain-Inspired Intelligence Project (2021ZD0201301), the National Natural Science Foundation of China (12201 125, 125,9257020), Shanghai Municipal Science and Technology (24JS281 142810000 and HXXD14044000 We thank Shanghai Institute for Mathematics and Interdisci- plinary Sciences (SIMIS) for their financial support SSMIISID-002-NNC The computations in this research were performed using the CFFF platform of Fudan University. Finally, we appreciate the discussions with Prof. Dongrui Wu at HUST.
References
[1] Z. Abbas, R. Zhao, J. Modayil, A. White, and M. C. Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pages 620-636. PMLR, 2023.
[2] A. Achille, M. Rovere, and S. Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018
[3] J. T. Ash and R. P. Adams. On Warm-Starting Neural Network Training, Dec. 2020. URL http://arxiv, org/ htt//bs//1991 08475. arXiv:1910.08.. [es].
[4] S. N. Burke and C. A. Barnes. Neural plasticity in the ageing brain. Nature reviews neuroscience, 711:330-40, 2006.
[5] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr. Riemannian walk for incremental learn- ing: Understanding forgetting and intransigence. In Proceedings of the Earopean conference on computer vision (ECCV), pages 532-547, 2018.
10

---

[6] S. Dohare, R. S. Sutton, and A. R. Mahmood. Continual Backprop: Stochastic Gradient De- scent with Persistent Randomness, May 2022. URLhttpp//arxii wttp:/wwwwrrggba//////00// 06325 trXiv:2108.06322 [es].
[7] S. Dohare, J. F. Hernandez- Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026):688-744 Aug. 2024. ISSN 0028-0836, 1476-4687. doi: 10.
URL. httpt::////ttt http:////wwnnnuurueeeee nature
http/arrtccess//4158-2222222
[8] S. Dohare, J. F. Herranez--Garcia, P. Rahman, A. R. Mahmood, and R. S. Sutton. Maintaining Plasticity in Deep Continual Learning, Apr. 2024. URL httt:://ariii http:///aaaivv.oraaa/////// MiXiv:2306.138/ [cs].
13812
[9] C. Fang, H. He, Q. Long, and W. J. Su. Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training. Proceedings of the National Academy of Sciences, (18(4))e2103091) 2103091118. URL httphhttp:/aatee http://axxivo..gg///// wttp://.rrgbbb///0000// 12699 arXiv:2101. [cs].
Oct. 2021. ISSN 0027-8424, 1091-6490. doi: 10. 10.133//p
[10] V. Farias and A. D. Jozefiak. Self-normalized resets for plasticity in continual learning. arXiv preprint arXiv:2410.20098, 2024.
[11] H. M. Fayek, L. Cavedon, and H. R. Wu. Progressive learning: A deep learning framework for continual learning. Neural Networks, 128:345-357, 2020.
[12] C. Gulcehre, Srinivasan, J. Sygnowski, G. Ostrovski, M. Farajtabar, M. Hoffman, R. Pascanu, and A. Doucet. An Empirical Study of Implicit Regularization in Deep Offline RL, July 2022. URL CRL/http:/// //ariiv//axxv...../ wwwrsii.orrgaeoo httgg/ab//22000 02099 arXiv:2207.02D1 .02099 [es].
[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[14] M. Igl, G. Farquhar. J. Luketina, W. Bochmer, and S. Whiteson. Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning, Sept. 2021. URL http://axxxi org /bs/ 2006. 05826 2.X2:20006005266 [es].
[15] Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, Milan, J. Quan, T. Ramalho, A. Grabska-Barwinske Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3222 Mar. 2017. ISSN 10. 16.1073/paas.11118351111 [es].
1091-6490. doi: wtrggabs//GGi0 w61.aa//......e.aae. 00796 arXiv:1612.00796
161
http:t/tt//axx/ www.
[16] A. Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit under- parameterizationn inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010. 14498, 2020.
[17] S. Kumar, H. Marklund, and B. Van Roy. Maintaining plasticity in continual learning via regenerative regularization. In Conference on Lifelong Learning Agents. PMLR, 2024.
[18] A. Lewandowski, H. Tanaka, D. Schuurmans, and M. C. Machado. Directions of Curvature as an Explanation for Loss of Plasticity, Oct. 2024. Utt.http://arxi, www.ooggggb......... abs.aa////1. 2312.0002222 ar arXiv:2312.00246 [cs].
[19] 7. Li and D. Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935- 294477995-799, 2017.
[20] C. Lyle, M. Rowland, and W. Dabney. Understanding and preventing capacity loss in rein- forcement learning. In International Conference Learning Representations, 2022. URL
httts://openreviier
[21] C. Lyle, 7. Zheng, F. Nikishin, B. A. Pires, R. Pascanu, and W. Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pages 23190-23211 PMLR, 2023.

---

[22] C. Lyle, Z. Zheng. K. Khetarpal, H. Hasselt, R. Pascanu, J. Martens, and W. Dabney. Disentangling the Causes of Plasticity Loss in Neural Networks, Feb. 2024. URL http 7/arxiv. www.orggga..ooooo... hts//bbb/24402 18762 arXiv:2402.18762 [es].
[23] E. Nikishin, M. Schwarzer, P. D'Oro, P.-L. Bacon, and A. Courville. The Primacy Bias in Deep Reinforcement Learning, May 2022. URL (RLhtt/://are wwwxiv.or/aaae org/aos////o//eoooooo 2205 07802 arXiv:2205.0785 07802 [es].
[24] E. Nikishin, J. Oh, G. Ostrovski, C. Lyle, R. Pascanu, W. Dabney, and A. Barreto. Deep Reinforcement Learning with Plasticity Injection, Oct. 2023. URL 2305. 15555 1rXiv2335511555 [es].
org/abs/
[25] V. Papyan, X. Y. Han, and L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):
24652 84652-24663, Oct. 2020. ISSN 0027-8424 1091-6490. doi: 10. 10..11//pna.22055591111 URL
https://paa org/ http///offfff/oooooeeeoeo 1ttpul111110111 1073/pnas 2015509117
[26] B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli. Exponential expressivity in deep neural networks through transient chaos, June 2016. http://///ar org/abs.
arXiv:1606.088
[stat].
[27] M. Seleznova, D. Weitzner, R. Giryes, G. Kutyniok, and H.-H. Chou. Neural (Tangent Kernel) Collapse, Oct. 2023. URL http://arxii orgg a305. 16427 arXiv:2305. 16427 [cs].
[28] G. Sokar, R. Agarwal, P. S. Castro, and U. Evei. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning, pages 32145-32168. PMLR, 2023,
[29] L. Storm, H. Linander, J. Bec, K. Gustavsson, and B. Mehlig. Finite-Time Lyapunov Exponents of Deep Neural Networks. Physical Review Letters, 132(5):057301, Feb. 2024. ISSN 0031-9007, 1079-7114. doi: 10.1 Revl Let.
wtt.:e/eee.......1111 132.05301. https:/itp:// aps org/doi/10
1103/hyyRReLLett. 132. 057301
[30] C. E. Tripp. J. Perr-Sauer, J. Gafur, A. Nag, A. Purkayastha. S. Zisman, and E. A. Bensen. Mcasuring the energy consumption and efficiency of deep neural networks: An empirical analysis and design recommendations. arXiv preprint arXiv:2403.08155, 2024.
[31] E. Verwimp, M. De Lange, and T. Tuytelaars. Rehearsal revealed: The limits and merits of revisiting samples in continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9385-9394, 2021.
[32] M. J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
[33] Z. Wang, E. Yang, L. Shen, and H. Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[34] Y. Yang, S. Chen, X. Li, L.. Xie, Z. Lin, and D. Tao. Inducing Neural Collapse in Imbalanced Learning: Do We Really Need Learnable Classifier the End of Deep Neural Network?, Oct. 2022. http::/tt:////ttt //trvv/.a/vvv////////// org/ abs/2203 09081 arXiv:2203.099: [cs].
[35] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In International conference on learning representations, 2018.
[36] Z. Zhu, T. Ding, Zhou, Li, C. You, J. Sulam, and Q. Qu geometric analysis of neural collapse with unconstrained features. Advances in Neural Information Processing Systems, 34: 29820-2988 29834, 2021.
12

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction sections state the claims made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the work are discussed at the end of the paper.
13

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The assumption and proof are provided in section 3.1 to 3.3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The information for reproducing the experiments are provided the supple- mentary document.
Guidelines:
The answer NA means that the paper does not include experiments.
14

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The data and codes are available on request from the authors.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pblic/guideeee http://oeeuuiiiiiooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
15

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The training and test details are provided in Methods section 3.5 and supple- mentary document.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification:
The paper does not contain experiments that requires statistical significance
test.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resource information is provided in the supplementary document.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////.arpsssscccpppb//tttta
Answer:
[Yes]
Justification:
The research conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This work focuses on the loss-of -aapstiiiit phenomenon in deep neural net- works and is not intended to address any direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
17

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The used open-access data and code explained and cited in the main text and supplementary document accordingly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpssppppssseeeeeeeeeeeee thcode wwt.cooo//aaaae comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The details of the code and model are included in the submission and supple- mentary document.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
18

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM this work does not impact the core methodology, scientific rigorous- ness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy htte::///eerrpsssccc/ooeeeeere/e for what should or should not be described.
19

---

A
List of Symbols Used in This Paper
Symbol
Description
h(0)
Representation at layer & of the network
((0)
Nonlinear transformation layer
W(O
Weight matrix at layer f Penultimate layer representation
h(D)
Small perturbation in input
5h10
Perturbation at layer
(x)
Finite-time Lyapunov exponent (FTLE) of input Unnormalized FTLE, i.e., À = LX
A
h
Representation of sample in class Classifier weight vector for class k
W;
H
Matrix of representations for class k Classifier weight matrix
W
Y
Label matrix for all samples Loss function (typically MSE) Regularization coefficient for W Regularization coefficient for Initial representation orthogonal to W Stretching factor (inter-class expansion) Compression factor (intra-class collapse)
dw
AH
Hø
k.K'
FTLE between class and k' after training
mminne
N
K.
FTLE between class and k' initialization t-th task in continual learning Network parameters after task
T
D(o)
Training set of task /
train
D
Test set of task t Mixup interpolation coefficient Interpolated input and target in mixup Confidence amplification factor in Generalized Mixup
m
m y
M
B Proof of Theorem 3.1.1
We prove Theorem 3.1.1 in main text that we restate as follows.
Theorem B.1 (Optimal Solution for Binary Classification) Under UFM and the assumption that both classes contain samples, |A| B n, consider the following optimization problem:
1 min J(W, H) = WH b1 W.H
Aw
AH 2n
Y
H
Ho
(16)
where xw and Au are the coefficients of the two regularization terms that control the magnitude of the classifier weights, W, and deviation of the learned representations H from the initial representations HJ orthogonal to W. The optimal solution obeys:
=
VDw/>n we + 1
(17)
where & = A or B, and 1 is the k-component of H..
We begin by computing the gradients of the loss function with respect to W and H, and setting them obtain the optimality conditions.
The gradient of C with respect to W given by
VwL = WH + b1
-YHHT + +A W = O.
(18)
20

---

The gradient with respect to H is
VnC= - w ww (WH+ b1 T
- -Y)
Ah (H H 0.
(19)
To connect the two optimality conditions, multiply both sides of Equation 18 on the left by W yielding
W (WH + b1 Y)H + Aw W w W
= 0.
(20)
Substituting Equation
19
into the first term of Equation 20 gives
Aи (H- Aw
(21)
72
Rearranging terms, we obtain the following relationship between the optimal weight matrix W and the learned representation matrix H:
Aи (H HH
Ww W
(22)
Based on prior results from Neural Collapse theory 36 the optimal classification weight matrix converges Simplex Equiangular Tight Frame structure. the binary classification case, this reduces to simple antipodal configuration:
W
(23)
Assuming the classification weights have reached their optimal configuration, the training dynamics of the representation h under the MSE loss yield the following gradient:
LLME dhk
= = - w h WA + (0 (0-wIth
withh Wk'
(24)
where & is the correct class and k' is the incorrect class index.
This gradient lies entirely the subspace spanned by Wa and wx, implying that the updates to h only occur in the plane defined by the classification weight vectors. Consequently, the component of hy orthogonal to this plane remains unchanged during training.
Thus, the representation hy can be decomposed as:
h h! + h
where h!! e span (wk, wey and hp, 1 span ( wa. wer }. The term h denotes the component the initial representation that is orthogonal to the subspace spanned by the classification vectors wy and Wk.
Then Equation 22 becomes:
As
Aw WTw
H H
where H!!! denotes the component of the initial representations that is parallel to the subspace spanned by the classification weight vectors.
This constraint provides closed-fome relationship between the classifier weights and the representa- tion geometry optimality.
For simplicity, will denote || || E and thus H
RAW E.
A
We now continue the proof by deriving a lower bound of the MSE loss. Our goal is to express the lower bound terms of the Frobenius norm || ||.
Recall the MSE loss:
LMSE
Σ 2
Wh, + b1 -
-v:l12
21

---

Since the orthogonal component ho, remains unchanged with Who, 0, we can express the MSE
loss as:
CMSE Σ
Wh! + vi|
To enable a tight application of the reverse triangle inequality, we re-center the target labels by defining:
1 b: (yA +BB),
and rewrite the labels as:
9: = y4 - b E = (yA b)}
Now, the MSE loss becomes:
MSE = Σ
Wh! 5
We now apply the reverse triangle inequality:
|Wi! ie 2 |1: ||
which becomes tight when Wh! !!
0r and squaring both sides yields:
Wh ; | I|b!!
Next, since h!! is aligned with the classification direction Wk. we can write (see a breif proof in Section B.I):
|Wh!l = | ||
Therefore, we obtain:
Wh 36 || || Wil (MIl=-116!!1) IIb!!1))
Substituting into the MSE loss gives:
Σ
[1@%|l - | [n!!1))
AMEE
2 2
From earlier, we have || 11 = E, |1
112
ndw E, where E = | Let n denote the total Au
number of samples. Define the average squared norm:
| 1==
1 ndw E.
n
n
Now apply Jensen's inequality to the convex function f(x) = (b a/2x)2 for a > 0, to get:
1 (1ε7l) | -wal==-la!!) II!!1))² > || W|
Σ Ell!!ip
V
Substitute the average squared norm:
2
n Σ [| [ = |
> ||xal| [
Aw
E
X
Recall || ||- F,, so we obtain:
2
CMSE 2 n
||%a||
Aw Ry
2
22

---

Thus, the original optimization objective becomes:
dw
C = MSE +
WW
2
H
1111
(25)
2
2n
2
n
||
Aw
E
+
Xw
+
Aн
nxw
E
(26)
2
2
2n
Aw
||ūel
F.
AW F..
(27)
This gives a clean lower bound entirely expressed as function of E W Since the loss function is continuous and coercive in F:, the minimum achieved at some finite E".
The conditions for the inequality to hold require representations h! have equal norm (which can also be intuitively obtained from the symmetry conditions )
h const,
Vi.
Also we have the following constrain equation 22
Ww||W
1
|1!!112
Aw AH
W
Therefore, have:
Aw
||
||
for all i in class k.
Ah
Thus the representation becomes:
Aw
W. + h,
λ
where K. E ssan(W))
( 1 is the fixed orthogonal component inherited from the initial representation.
B.1 A brief proof of [Wh = ||
||!
Under the neural collapse assumption 23 that the classification weight matrix structured
W
and we define
h/! = T.W. A. for some т; E R,
to emphasize that the representation lies along the direction of w.s and r; is arbitrary.
We have:
wh
w W WA
T. w 1
1 [-]
Ti
Thus, the squared norm becomes:
Wh 27 1|
On the other hand:
||!!!|| [7s|- l|w.ll-
[W] 1| + wall all
Therefore.
Whh! /2 |T|| |/ V2. 1// |m|] ||w (w]---11!!
This confirms that the equality holds exactly under this structural assumption.
[W || W||= 1
23

---

C Verification of High-Dimensional Data Properties in ImageNet
In the main text, we assume that real-world data distributions exhibit two critical characteristics of high-dimensional spaces: the measure of concentration and the empry space phenomenon. To verify these assumptions, we conducted the following experiments on the ImageNet dataset:
1. Measure of Concentration: We calculated the average pairwise distance between samples in the representation space, along with the variance of these distances. This analysis helps to verify whether the distances concentrate around their mean, as expected in high-dimensional spaces.
2. Empty Space Phenomenon: To assess this property, randomly sampled 1000 points in the representation space and computed the minimum distance between each sample point and these randomly generated points. This test evaluates the relative sparsity of data in high-dimensional spaces.
The results, shown in figure concentration and the empty space phenomenon. These findings validate our assumption about high-dimensional nature of real-world data distributions.
O confirm that the ImageNet dataset adheres to both the measure of
Concentraiion Yeasure Across lases
Emssyy Space Phennmmenn Acrosa Taaks
810
Task
Figure 6: Verification of High-Dimensional Data Properties in ImageNet.
D Proof of Theorem 3.1.2
We prove Theorem 3.1.2 from the main text that we restate as follows.
Theorem D.1 (FTLE in Representation Learning) The distance between two samples i and j from classes ks and k' in representation space, Ahij k. = # - hy: can be quantified by FTLE, Xn and Arriied each denotes its values before and after training, respectively.
(I) For samples from different classes (k W)), training produces positive FTLE ridge, stretching the space in-between, and the magnitude of stretch is captured by a stretching factor, a:
log a
X k.k'
k.k'
> 0,
(28)
(2) For samples of the same class (k K'), training carves out a valley of negative FTLE, compressing the space in-ben between, and the degree of compression is given by compressing factor, 3:
log 3
A
k.
A
J.
0.
(29)
According to Theorem 3.1.1 the optimal representations satisfy:
AAw
k.
W. + 12,0
A
where the trainable component lies entirely in the classification plane spanned by (WA, W and the orthogonal component (.. remains unchanged from initialization.

---

The trained representations become:
Aw
Aw
1
WA 1000 1
B. wB + h330
λ
Hence, the inter-class representational difference after training becomes:
dw
A.B B Atrainedhiig
B.
n
500
+
(WA WB).
In contrast, the inter-class representational difference before training is given by:
A.B Dohi
h B.L.
+
h0,0
w
h5.00
WA
|w
|wall
Wwl
|Wwll
where the initial representations are projected onto the direction of WA-
The change in inter-class representational distance due to training satisfies:
A.B - -1| A
A Aminnad 155
(30)
Aw
1/ -w
K h0:0
W
W
1
WA
W
(31)
Ah
| A
[
[
Aw
(h:30 3.00 WA)
||wall
(32)
|| w alll
))
Aw
12.00
h 5,0
WA
-11
1
(33)
An
W
Similarly, the change in intra-class representational distance due training is given by:
Atrasnedt A.A 1.
A Auhi A.A ||²
(34)
Ingir - 11 1²
- 1|15500 - -n12oll
(35)
2
(Mo0 A.1
Aw
+
(WA wa)
(36)
AnL
n1500
h
WA
h1500
W
WA
(37)
|Wall
||w
|wall
(( 1100
h1301
WA
WA
(38)
[[aall
[ All
A WA)
|wall
1|2
W
(39)
W
W
h/
(40)
|wall
[
(41)
Thus, the key to evaluate the following projection term:
K
2
<110 - K
WA
3,00
|WA]]
25

---

This can be decomposed as:
2
WA
h
WA
Iwall
[wwl
Note that this expression captures the difference the initial projections of the representations h.K and 100 onto the normalized classification direction w *a/l|wal /|ll--
Under this assumption, the original expression reduces to geometric comparison between three dom unit vectors in high-dimensional space: two initial representations 21, 22, and one classification direction w. Let define:
cos 0 = cos (<(=1, w)), cos 02 cos (Z(zz, w))
We are interested in the expected deviation:
cos 01 cos 02)²
According to Lemma D.I the distribution of cos 0 between two independent unit vectors in d- dimensional Euclidean space follows the density:
T
plcos 0) =
(1- -1 cos c1-cos ()d-33/22
(1
Let cos 0 p(x), cos ~ p(y). The distribution of their absolute difference | y1 can be written as:
p=(2) S p( p(x)p(v) 8(2 - | y12) do dy-
By symmetry, this simplifies to:
C² P=(2) -===)==== - - -1- (1d32)(d-3)/2 (1 (xr + (d-3)/2 dr.
where C' is the normalization constant ensuring 8 p()) dr = 1.
Although the exact form of (2(2) is analytically intractable, it can be efficiently evaluated numerically. the following figure, we plot the cumulative distribution function (CDF) of z. Here, dimension d refers to the width of the representation space, i.e., the dimensionality of the representation E Rd As increases, the distribution becomes increasingly concentrated near zero. Conversely, when the dimension d is small, the distribution of z becomes more dispersed and exhibits heavier mass at larger nonzero values.
CDF: PIZ for different dimension of representation space
Cumulative Prebabilty
d=20
d=50
d-100
d=500
2=12
0.0
0.5
1.0
1.5
2.5
3.0
35
4,0
Figure 7: Numerical solution of the CDF of 2.
26

---

Within-Class Representation Collapse. For the within-class region, in Eq shown that training changes the pairwise distance between representations as:
40
we have already
Auainsa K.
1|
Doh
K. K. 1
KK
h4,0
23.00
WA
WAl
As we have analyzed earlier, the scalar projection term
2 h000
h J.03
W
A
follows distribution that becomes increasingly concentrated near zero as the representation dimen- sionality d grows, and conversely, spreads more widely when d is small.
Define the within-class shrinkage factor as:
urineed his
K.K =1= 3. |1
1-110 K.
with 3 <
Then the FTLE after training is given by:
Aeained (x) = 2o (x) log ((x).
This implies that the FTLE will slightly decrease remain nearly unchanged / S 1,when d is large, but may shrink significantly when d small, due to larger projection fluctuations.
Importantly, while this within-task shrinkage effect may appear mild for single task, in the continual learning setting it can accumulate over time, especially when repeated over multiple tasks. This accumulated shrinkage can lead to significant compression of the local representation space, ie., representation space collapse. This analysis is consistent with our empirical observations: in low-dimensional settings, collapse occurs reliably even for small number of tasks, while in higher- dimensional spaces, collapse may or may not occur depending on the specific training setup and number of tasks.
Boundary Space Over-Streiched. For the inter-class region, Eq. 40 shows that training modifies the pairwise distance between representations of different classes as:
Dunincd A.
Aw λœ
(
h1000
B
WA
> )r
j.0.
wwll
To estimate the first term, we recall from the optimization analysis of Equation the total loss satisfies:
27 in Section that
2 12.
dw Hy
((E)
|
E
+ E,
where the minimum occurs approximately at
Au
An
An
E
-|
Aw
n
Xw
This is because n typically large compared to Ais and Aw ~ An in magnitude. Hence,
Aw
|
||
4
Twwwl 1 22.
Aw
In Figure we overlay this estimate (red dashed line) on the empirical CDF of the squared projection term
B
W
2 =
5.0
W
we observe that in both low- and high-dimensional spaces, most values of fall below this estimated threshold. This indicates net expansion of the representation space between classes during training.
27

---

Moreover, the distribution of the projection term becomes more concentrated around zero as the representation dimensionality increases, while it spreads more widely when d small. Therefore, although expansion dominates across dimensions, it is more significant in high-dimensional spaces, where the lower baseline projection leads to faster growth.
We formalize this expansion using a stretch factor:
Arained/ A.B "1
||
A.B 11,
a>l.
The resulting FTLE becomes:
K.K' rained
(1)
2k. (x) log ((z),
indicating positive FTLE ridge across inter-class boundaries, and reflecting a local "stretching effect" that enhances class separability.
While the effect of class boundary expansion is immediately pronounced in high- high-dimensional settings, its manifestation in lowddimensional networks can be subtle within a single task. However, under continual learning, where multiple tasks accumulate over time, the expansion effect may progressively intensify and eventually give rise chaotic behavior. Consequently, we consistently observe clear boundary stretching in high-dimensional models, whereas in cow-dimensional models, such dynamics may emerge only after sufficient task accumulation. This observation aligns with our findings reported in the main text.
Lemma D.1 In high-dimensional spaces, the angle 0 between two randomly sampled vectors 2, 2dd follows the distribution:
r (
Pa(0)
sin d 0.
T
where I(-) is the gamma function 32
E Experimental Setup for Verifying LoP in Real-World Datasets
To validate our discovery of two different types of LoP across realistic datasets, we use the exper- imental setup described Supplementary [H] using identical architectures, training protocols, and hyperparameters. Across all settings, we consistently observe the two types of LoP as well as hybrid cases where both types coexist, differing only in their onset speed and severity.
In the main text Fig. we visualize model performance under four representative single-run configu- rations:
(a) ReLU + CE with the architecture in Table
(b) Tanh + MSE with the architecture in Table
(c) Tanh + MSE with the architecture Table
(d) ReLU CE with the architecture in Table 6
All other experimental conditions remain identical across these runs.
Generalized Mixup Pseudo Code
Generalized Mixup (G-Mixup) is designed simple, plug-and-play module that can be seamlessly integrated into existing training pipelines. It minimally modifies the classical Mixup [35] by adapting the label interpolation rule depending on whether the two samples belong to the same class (intra- class) or different classes (inter-class). It supports both Mean Squared Error (MSE) and Negative Log-Likelihood (NIL.) loss.
Key idea:
For inter-class mixup, standard Mixup applies: convex interpolation of both input and label.
28

---

For intra-class mixup, only the dominant class label is amplified by factor depending on 0.5 to encourage intra-class diversity while conserving class identity.
In this work, we the intra-class amplification factor to MM 0.2. To prevent label probabilities from exceeding 1, we pre-process all labels by assigning the dominant class a value of 1 M, and the remaining M equally distributed across the other classes.
Algorithm 1 Generalized Mixup (G-Mixup) for a Mini-Batch
Require: Mini-batch [11; y( } mixup parameter amplification factor M, number of classes C
1: Preprocess labels: for each vt construct as:
Vi [k] = 1 M, where k = arg max(y!"")
yı [therss] = T-1
M
2: Sample m ~ Beta(o, a)
Shuffle mini-batch to get (1)
4: Compute mixed imputs: z!"" ma, + (1 m)r?
5: if arg max(y)) (B:)
arg max( (intra-class) then
6: yf" [k] = wi[k] + M M 10.5 ml
7: y [others) 1-
8: else
9: y/"h = my. + (1 - m)y// (Use M- M-adjusted labels)
10: end if
11: Compute prediction J = f(x"')
12: if loss is MSE then
13: L N Σ, |iū y/ 1|
14: else if loss is NLL then
15: L N Σ,
CrossEntropyy(o...."
16: end if
17: Backpropagate L
G Experimental Setup for Continual ImageNet
We evaluate G-Mixup on the Continual ImageNet benchmark, following a widely adopted binary task formulation. The full dataset used in this task consists of 1,000 classes, each with 700 images. For each class, 600 samples are used for training and 100 for testing. A continual learning sequence is constructed by randomly pairing classes to create binary classification tasks. Each task comprises 1,200 training samples and 200 test samples, drawn from two classes, Models are trained for 250 epochs using mini-batches of size 100. All images are downsampled to 32 x 32 resolution to reduce computational cost.
For all experiments, we adopt convolutional neural network consisting of three convolutional layers with max-pooling, followed by three fully connected layers. To study the impact of representation dimensionality, we employ two variants: a wide network with larger hidden dimensions, identical to the one used in [7] (see Table representing the high-dimensional regime; and narrow network with reduced layer widths (see Table corresponding to the low-dimensional case. The final output layer has two units representing the current task's classes. At the start of each task, the output heads aechmarkk--lthuute
are reset to zero-a common practice in continual learning this introduces privileged information about task boundaries. All methods are trained using SGD with momentum (set to 0.9), and models are initialized once at the beginning of the entire continual learning sequence. Learning rates are set 0.01; results reported are averaged over 5 independent runs.
we note that
We compare G-Mixup with multiple baselines and LoP mitigation strategies. Two reference points are included: (1) Reinit, which re-initializes the model for every new task, and (2) No Intervention (No Inty.), which applies naive continual training without any LoP countermeasures. For fair comparison,
29

---

We also include representative methods from three major strategies for mitigating LoP: L2 Init [17 (a
regularization-basee based method, with the weight decay coefficient fixed 5 x 10 4). LayerNorm [2] (an rrchitecture-based approach, where LayerNorm is applied before the activation function at every inclu-in both convolutional and fully connected seers-exceptt for the final output layer),
and CBP [
(a reset-based method).
Unless otherwise specified, all methods use the ELU activation function and MSE loss. The only exception is CBP, which uses ReLU as in its original implementation [7]. This choice is motivated by the fact that CBP relies on tracking the utility of individual neurons, and ELU-due due its non-zero minimum- can distort utility estimation by introducing persistent low-level activation. We further confirmed that CBP performs better with ReLU than with ELU under our experimental settings. All other hyperparameters and training schedules are kept consistent across methods to ensure fair comparison.
Table 5: SmallConvNer Architecture Details.
Layer 1:
Convolutional Max-Pooling
Number of Filters
16
Activation
ELU/Tanb/Rel.UU
Convolutional Filter Shape
(5,5)
Convolutional Filter Stride
(1.1)
Max-Pooling Filter Shape
(2.2)
Max-Pooling Filter Stride
(2,2)
Layer 2: Convolutional + Max-Pooling
Number of Filters
32
Activation
EL ELU/Tanb/ReLU
Convolutional Filter Shape
(3,3)
Convolutional Filter Stride
(1.1)
Max-Pooling Filter Shape
(2,2)
Max-Pooling Filter Stride
(2,2)
Layer 3:
Convolutional Max- Sax-Pooling
Number of Filters
64
Activation
ELU/Tanh/ReLU
Convolutional Filter Shape
(3.3)
Convolutional Filter Stride
(1,1)
Max-Pooling Filter Shape
(2,2)
Max-Pooling Filter Stride
(2,2)
Layer 4: Fully Connected
Output Size
16
Activation
ELU/Tanh/ReLU
Layer 5: Fully Connected
Output Size
16
Activation
ELU/Tanh/ReLU
Layer 6: Fully Connected
Output Size
2
Activation
None (Linear)
Table 6: ConvNet Architecture Details.
Layer 1: Convolutional Max-PPollin
Number of Filters 32
Activation
ELU/Tanh/ReLU
Conv. Filter Shape
(5,5)
Conv. Filter Stride
(1,1)
Max-Pool Filter Shape
(2,2)
Max-Pool Filter Stride
(2,2)
Layer 2: Convolutional Max-Pooling
Number of Filters
64
Activation
ELU/Tanh/Rel.U
Conv, Filter Shape
(3,3)
Cony, Filter Stride
(1.1)
Max-Pool Filter Shape
(2.2)
Max-Pool Filter Stride
(2.2)
Layer 3: Convolutional Max- Max-Pooling
Number of Filters
128
Activation
ELU/Tanb/ReLU
Conv, Filter Shape
(3,3)
Conv. Filter Stride
(1,1)
Max-Pool Filter Shape
(2,2)
Max-Pool Filter Stride
(2,2)
Layer 4: Fully Connected
Output Size
128
Activation
ELU/Tnnh/RLL
Layer S: Fully Connected
Output Size
128
Activation
EL.U/TanhRRl.UU
Layer 6: Fully Connected
Output Size
2
Activation
None (Linear)
30

---

H
Continual ImageNet Results
The performance of wide and narrow networks on Continual ImageNet is summarized in Table and
Table g
Table 7: SmallConv Test Accuracy Results on Continual ImageNet
Task (x1000)
0-1
1-2
2-3
3-4
4-5
No Intv.
0.817 (#0.070)
0.805 (+0.085)
0.562 (tt.0
0,500 (=0.000)
0,500 (#0.000)
Retrained
0.853 (#0.065)
0.845 (#0.066)
0.845 (#0.068)
0.840 (#0.067)
0,840 (#0.072)
L2 init
0.804 (+0.055)
0.796 (#0.059)
0.786 (+0.054)
0.785 (#0.052)
0.788 (+0.055)
Layernorm
0.753 (+0.056)
0.760 (+0.052)
0.759 (+0.051)
0.751 (#0.049)
0.751 (+0.056)
CBP
0.834 (#0.052)
0.847 (#0.052)
0.846 (+0.050)
0.847 (=0.050)
0.857 (+0.052)
G-mixup
0.866 (#0.052)
0.881 (#0.048)
0.885 (#0.048)
0.880 (=0.050)
0.879 (#0.054)
Table 8: ConvNet Test Accuracy Results on Continual ImageNet
Task (x1000)
0-1
1-2
2-3
3-4
No Intv.
0.794 (#0.063)
0.778 (+0.073)
0.604 (20.135)
0.537 (#0.074)
0.500 (+0.000)
Retrained
0.857 (#0.060)
0.851 (+0.060)
0.850 (+0.060)
0.849 (=0.060)
0.846 (+0.064)
init
0.814 (#0.047)
0.805 (#0.050)
0.800 (#0.052)
0.803 (#0.052)
0.807 (+0.054)
Layernorm
0.782 (#0.052)
0.768 (#0.056)
0.752 (+0.049)
0.749 (#0.050)
0.755 (+0.055)
CBP
0.848 (#0.053)
0.867 (#0.049)
0.864 (#0.046)
0.863 (#0,048)
0,878 (#0.047)
G-mixup
0.875 (#0.049)
0.896 (#0.043)
0.899 (#0.042)
0.894 (#0.045)
0.896 (#0.047)
I FTLE Analysis of G-Mixup
We analyze the evolution the FTLE distribution under Generalized Mixup at different stages of continual training. Our results reveal that G-Mixup consistently maintains the FTLE within stable and moderate range throughout training. This controlled trajectory of representational dynamics effectively prevents both representation collapse and boundary chaotic behavior- different types of LoP. The robustness of G-Mixup in sculpting the representation space is thus supported by its capacity to regulate FTLE evolution. The corresponding results are presented in Fig.
0.35
0.6
Task T100
Task T100
Task T1000
Task T1000
Task 75000
Task 75000
0.00
0.0
-10
0
10
20
-10
0
10
20
Figure Analysis of FTLE distributions different training stages of Generalized-mmxxu.
J
Ablation Study
As discussed in the main text, Mixup encourages smoother transitions in the representation space by interpolating inter-class samples and labels, which helps mitigate Type-2 LoP (over-stretch), However, it leaves intra-class labels unchanged (fixed one-hot targets), limiting its ability to prevent
31

---

Type- 1 LoP (collapse). In contrast, G-Mixup introduces controlled intra-class label variation thereby regularizing both inter- and intra-class geometry.
To isolate and highlight this intra-class effect, we perform an ablation study in which both Mixup and G-Mixup restricted intra-class sample pairs only, removing inter-class interpolation entirely. This setup ensures that both methods generate the same level of sample diversity while differing only in their treatment of intra-class label variation. To accelerate the onset of collapse and better expose representational instability, we slightly increase the learning rate to 0.02.
As shown in Table 9 continues to support progressive adaptation across tasks.
standard Mixup rapidly leads to collapsed representations, while G-Mixup
Table 9: Test Accuracy Results for Ablation Study
Task (x500)
0-1
1-2
2-3
3-4
G-Mixup (SmallConv)
0.864
0.871
0.879
0.878
Mixup (SmallConv)
0.864
0.581
0.500
0.500
K
Class-Incremental CIFAR-100
We evaluate all methods under the class inaremenrlllltal setting of CIFAR-100, where the model pro- gressively learns 100 classes over 20 tasks, with 5 new classes introduced each task. The model is trained on all accumulated classes at every stage and evaluated on the full set of seen classes, simulating task-agnostic continual learning. Each class contains 600 images, partitioned into 450 for training. 50 for validation, and 100 for testing.
For each increment, the model is trained for 200 epochs using SGD with a momentum of 0.9, weight decay of 0.0005, and a mini-batch size of 100. The learning rate is reset at the beginning of each increment and follows a decaying schedule: 0.1 for the first 60 epochs, 0.02 for the next 60, 0.004 for the following 40, and 0.0008 for the final 40 epochs. For G-Mixup, as the number of classes increases, inter-class interpolations become increasingly dense, making the representation space more entangled and the optimization landscape more complex. To stabilize training under this challenging regime, all learning rates are further reduced by a factor of 0.2 once the number of classes exceeds 10. The same learning-rate schedule is applied to all other baseline methods for fairness. However, under this schedule, their performance deteriorates markedly. Therefore, we present the results obtained under their best pest-performin configurations. During training, the best-performing model on the validation set saved at each increment, and training for the next increment starts from this checkpoint, effectively applying early stopping across increments.
We adopt ResNet-18 as the backbone architecture and evaluate two variants: standard high- dimensional version (ResNet-18) and a compressed www-dimensional variant (0.25x ResNet- 18), where all channel widths and the final fully connected layer are reduced proportionally. Additionally, fully connected layer with width of 16 is appended as the representation layer in the fow-dimensional variant. The output layer grows dynamically by adding 5 new units at each increment, with newly added weights initialized via Kaiming initialization and biases set to zero. Convolutional and linear layers follow Kaiming initialization, while batch normalization weights are set 1.
All input images are normalized to [O, IJ. channel-wise standardized using dataset statistics, and augmented during training with random horizontal flips, 4-pixel padding followed by random crops, and random rotations between [-15%, 15%). These augmentations are applied only to the training set.
We compare G-Mixup against multiple basclines and LoP mitigation strategies under the class- incremental CIFAR- 100 setting. Two primary baselines are considered: (1) No Intervention (No Intv.), which performs standard continual training without any mechanisms to counteract LoP, and (2) Reinit, which re- initializes the model from scratch at each task.
To ensure comprehensive evaluation, we further include representative methods from three major categories of LoP mitigation: 1.2 Init [17], a regularization-basse based method with a fixed weight decay of 5 x 10-1 and no additional 1.2 regularization on the weight; LayerNorm (2I. an architecture-base
approach where BatchNorm replaced with LayerNorm in all layers; and CBP [7) a reset-based approach targeting underutilized neurons. For fair comparison, and to rule out the possibility that
32

---

G-Mixup's s improvement originates from label softening rather than its geometric regularization effect, all baseline methods are trained with the same soft-label smoothing coefficient of 0.1.
All methods are evaluated under identical training protocols as described above. The comparative results presented below.
Table 10: 0.25x Resnet-18 Acc. on CIFARIO0
Task (x4)
0-1
1-2
2-3
3-4
4-5
No Intv.
0.927 (+0.027)
0.812 (#0.018)
0.754 (#0.015)
0.708 (#0.014)
0.661 (+0.008)
Retrained
0,926 (+0.035)
0.800 (+0.020)
0.745 (#0.013)
0.702 (#0.012)
0.666 (+0.007)
L2 init
0,925 (+0.032)
0.788 (+0.020)
0.724 (#0.013)
0.682 (#0.014)
0.649 (#0.007)
Layernorm
0.851 (+0.051)
0,755 (+0.028)
0,723 (=0.016)
0.674 (#0.015)
0.641 (+0.008)
CBP
0.923 (+0.038)
0.812 (+0.020)
0.760 (#0.017)
0.713 (#0.011)
0.678 (#0.006)
G-mixup[ours]
0.932 (+0.030)
0.825 (#0.019)
0.772 (#0.014)
0.732 (#0.014)
0.697 (=0.007)
Table I: Resner-18 Acc. on CIFARI00
Task (x4)
0-1
1-2
2-3
3-4
4-5
No Intv.
0.907 (+0.029)
0.847 (#0.014)
0.812
(#0.013)
0.778 (#0.013)
0.743 (+0.008)
Retrained
0.901 (+0.037)
0.842 (#0.017)
0.815 (#0.012)
0.788 (+0.013)
0.767 (+0.005)
1,2 init
0.922 (+0.027)
0.829 (#0.016)
0.785 (#0.013)
0.752 (#0.012)
0.723 (+0.006)
Layernorm
0.847 (+0.065)
0.782 (+0.035)
0.763 (#0.019)
0.728 (#0.019)
0.697 (+0.019)
CBP
0.907 (+0.027)
0.847 (+0.015)
0.816 (#0.013)
0.788 (#0.011)
0.768 (#0.008)
G-mixup[ours]
0.928 (20.025)
0.864 (20.016)
0,832 (#0.012)
0.800 (#0.009)
0.768 (#0.006)
L Computing Infrastructure
Table 12: Computing infrastructure the primary server
CPU
EMC Core Processor, sockets (384 threads total)
GPU
NVIDIA RTX 4090
Memory
512 GB
Operating system
Ubuntu 20.04.6 LTS
Simulation platform
Python 3.11 with PyTorch 2.1.1
Table 13: Computing infrastructure of the secondary server
CPU
AMD EPYC 9354 NVIDIA RTX 4090
GPU
Memory
512 GB
Operating system
Ubuntu 20.04.6 LTS Python 3.11 II with Py PoTch 2.1.1
Simulation platform
The simulations and analyses in this study are conducted on two high-perfhheeee performance computing servers equipped with AMD EPYC processors and NVIDIA RTX 4090 GPUs. The primary server features dual AMD EPYC 9654 CPUs (each with 96 cores), totaling 384 threads, and supports simultaneous multithreading. The secondary server is equipped with AMD EPYC 9354 CPU and identical GPU and software configuration. Both systems are configured with 512 GB of memory, enabling efficient execution of memory-intensive simulations and deep learning workloads. Experiments were run under Ubuntu 20.04.6 6 LTS using Python 3.11 1 and Py Torch 1.1, providing a robust and up-to-date scientific computing environment.
33


Reviews:
[{"review_id": "UwWCH3d3LA", "reviewer": "Reviewer_BXEg", "summary": "The authors study the Loss of Plasticity (LoP) through the lens of the theory of neural collapse and finite-time Lyapunov exponents (FTLE) analysis. Their claim is that LoP can manifest in two separate regimes:\n(1) Type 1 LoP is characterized by highly negative FTLEs, where the network is prevented from learning due to the collapse of representations\n(2) Type-2 LoP is characterized by excessively positive FTLEs, where the network can train well but which test accuracy on new tasks degrades\nThe authors propose an approach called Generalized Mixup, which allows to mitigate LoP and which is inspired by these 2 regimes.\n\nThe authors conduct toy and real-datasets experiments to showcase their proposed Generalized Mixup strategy as well as to demonstrate two types of the LoP.", "questions": "* What is the reasoning behind architectural design choices for toy experiments for Type 1 & Type 2?\n\n* What is the difference in the experimental regime for Figure 4.a & 4.c and Figure 4.b and 4.d?\n\n* Could the story which is suggested by the authors be simply explained by (a) lack of network capacity (for Type 1), and by (b) overfitting (for Type 2), as opposed to their theoretical framework?", "limitations": "Please see weaknesses.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* A novel perspective on the LoP through the theory of Neural Collapse and finite-time Lyapunov exponents (FTLE) analysis. Their theory identifies two separate regimes of the LoP.\n* A new empirical method for augmenting data, Generalized Mixup, which can tackle the LoP. I think it is particularly interesting because generally LoP approaches do not focus on data augmentations\n* Experimental results showcasing that Generalized Mixup could help with the LoP\n* (Controversial) Toy experiments showcasing two regimes of LoP through the lens of the proposed theory\n\nWeaknesses:\n* In my opinion, theoretical contribution is a bit limited. The proposed approach feels more like a \"diagnostic\" of what is happening, rather than a theory which explains exactly why the LoP happens. My opinion is based on Figure 4, where in both Low dim and High Dim spaces, there are both Type 1 and Type 2 regimes happening. Why is this the case? When I check the toy experiments, it looks like Type 1 happens when the network capacity is low (because authors use 10 hidden units per layer) while Type 2 happens when the network capacity is high (because they use 300 hidden units). Based on this, I would have assumed that In figure 4, in low dim space, only Type 1 would happen, but there is Figure 4,c where both happen. Similarly for Figure 4,b and d. Therefore, it feels that this perspective on Type 1 and Type 2, is something which we can look a-posteriori as a diagnostic of what went wrong, but not as something which can be used to predict LoP.\n\n* The authors don't really explain in the main text why they use 10 hidden units per layer for Type 1 toy experiment and 300 hidden units per layer for Type 2 toy experiment. It feels arbitrary.\n\n* Same criticism, for Figure 4, the authors don't explain the difference between (a) and (c); and (b) and (d).  It's unclear why we have Type 1 and then suddenly Type 1+ Type 2.\n\n* Based on weakness I mentioned above, could the story which is suggested by the authors be simply explained by (a) lack of network capacity (for Type 1), and by (b) overfitting (for Type 2), as opposed to their theoretical framework.\n\n* Based on these, I think the significance of the work is a bit limited.\n\n* I have no concerns regarding the Generalized mixup, this looks like something which helps for addressing LoP.", "was_revised": true, "final_justification": "The authors provided a clarification for my mis-understanding of their contribution and I now acknowledge that their theoretical contribution is novel.", "rebuttal": "We sincerely thank the reviewer for the constructive feedbacks. We would like to start by an overall reply to provide general clarifications about the issues raised here, given that some of the central idea of our work might have not come through. Then, we will follow with responses to the specific concerns.\n\n### Overall reply\n\nThis paper uncovered the fundamental mechanisms underlying LoP in continual learning (CL), i.e., the compressing/stretching of the representational(repr.) space. This primary contribution is a theoretically grounded diagnosis of the emergence of two types of LoP though a-posteriori. It is based on how CL compress and over-stretch the repr. space by repeated exposure to overlapped input distribution from intra-class and inter-class samples, respectively (as proved in theorem 3.1.2). The dynamics of the manipulation of repr. space geometry is characterized by FTLE. Recently, LoP has been associated with various observations, such as gradient vanishing or an increase in dead neurons. However, as pointed out in Lyle et al., (2024), these singular phenomena are insufficient to fully explain LoP on their own. This points to the lack of unified theoretical understanding of LoP since those phenomena are not the direct causes. Despite this, numerous methods have been proposed to mitigate LoP. Their strategies can broadly be classified into two categories, as the following.\n\nOne approach targets specific manifestations of LoP, such as large weight norms. For instance, Dohare et al. (2024) investigated the use of L2 regularization to constrain weight norms and found that it can partially mitigate LoP. Our theoretical framework provides a deeper understanding of how these measures work: the network used in their study is relatively wide and hence susceptible to Type 2 LoP. In such cases, L2 regularization prevents excessively large weights, thereby mitigating Type 2 LoP. However, in networks with initially lower repr. dimensions (narrow networks), which are more prone to Type 1 LoP, L2 regularization can actually impair performance. (We want to emphasize here that the size of repr. dimensionality does not equal to the network capacity, as initially large/small repr. space can still be easily stretched or compressed extremely during CL.)\n\nAnother approach builds on the observation that plasticity is highest at initialization. Techniques such as continual weight re-init (e.g., CBP) or regularization toward initial weights (e.g., L2-init) aim to keep the network in a more plastic regime. While effective in alleviating LoP, these methods suffer from several drawbacks: weight re-init can erase previous learning, while regularization toward initial weights can constrain the network's expressivity. Our work, in contrast, mitigate LoP based on the theoretical understanding that the plasticity can come from maintaining a reasonably sized (for Type-1 to not collapse) and smoothed (for Type-2 to not become chaotic) repr. space without going back to the initialized state, thus avoiding the drawbacks.\n\n(relates to )While we appreciate the reviewer’s insightful attempt to relate Type 1 and Type 2 LoP to limited capacity and overfitting respectively, we respectfully note that this interpretation does not fully capture the mechanisms we aim to describe. This is precisely why we introduced the FTLE framework to characterize the dynamics of repr. space rather than relying on conventional notions of capacity or overfitting. We appreciate the chance to reiterate the essence of our work here.\n\nOur theory demonstrates how repr. geometry crucially drive the network dynamics into two regimes. On the higher level, the dimensionality of the repr. space governs the general direction of the dynamics, i.e., as the dimensionality (which can be set by network width only `initially`) increases, the collapse of intra-class space (related to type-1 LoP) weakens and the chaotic expansion of inter-class space (related to type-2 LoP) dominates. On the lower level, only the local geometry can determines , effectively measured by FTLE. In addition to the theory described in the main text, we support this claim with empirical evidence shown in the following table:\n\nIn the following two tables, we report the FTLE stats for networks of varying widths. In **Table A1**, input samples are uniformly distributed, conformed to our theoretical assumption. As network width increases, the minimum FTLE quickly approaches zero much more quickly than its imbalanced counterpart.\n\nHowever, when the input distribution is highly imbalanced—as shown in **Table A2**, where one class dominates the majority of the space—the intra-class representation vectors become less orthogonal and more prone to collapse, demonstrated by more negative min FTLE vs. the ones for uniform input distribution.\n\n- **Table A1: FTLE statistics under uniform input distribution**\n\n| HiddenDim | Max    | Min      | Avg     |\n| --------- | ------ | -------- | ------- |\n| 10        | 7.6742 | -13.0385 | -3.9194 |\n| 150       | 9.0213 | -3.3198  | 1.4912  |\n| 300       | 9.2577 | -2.4663  | 2.2920  |\n\n- **Table A2: FTLE statistics under class-imbalanced input distribution**\n\n| HiddenDim | Max     | Min      | Avg     |\n| --------- | ------- | -------- | ------- |\n| 10        | 8.8215  | -14.0616 | -4.5067 |\n| 150       | 9.9708  | -5.2987  | -0.0250 |\n| 300       | 10.1752 | -5.1530  | 0.6825  |\n\nBeyond this simplified toy model setting, data distributions are structured, various activation functions and optimizer introduces additional constraints in real training scenarios. These factors modulate the global and local dynamics of the collapse and expansion of the repr. space, but do not alter the general prediction from our theory:\n\n- In low-dim spaces, repr. space collapses (Type 1 LoP), though expansion can still occur.\n- In high-dim spaces, repr. space expands chaotically (Type 2 LoP), though collapse may still be present.\n\nOur empirical observations also align with this: we never observed pure Type 2 LoP in low-dimensional networks, nor pure Type 1 LoP in high-dimensional ones. In Figure 4 of the main text (detailed experiment settings can be found in Appendix E), we chose two widely used activation functions—ReLU (scale-invariant) and tanh (saturating)—to illustrate all four theoretically predicted regimes. One can observe that the narrower ReLU networks in Figure 4C exhibit significantly faster onset of Type 1 LoP than the wider network in Figure 4D, again consistent with our theoretical expectations.\n\nTherefore, Type 1 and Type 2 LoP are the emergent behavior driven by the evolving geometry of the repr. space during CL:\n\n- Type-1 LoP arises from repr. collapse. While narrower networks are more susceptible to such collapse, wider networks tend to collapse more slowly, though they may still exhibit collapse depending on various factors.\n- Network capacity alone cannot explain Type 1 LoP. Increasing capacity (width) may slow down collapse but cannot eliminate it.\n\nRegarding overfitting, it typically refers to excessively tailoring a model to a specific dataset, impairing generalization. While overfitting can account for some aspects of Type 2 LoP, it is not the primary cause in our CL setup. In our setting, the model learns 5000 tasks of comparable difficulty, and performance degradation emerges gradually, unlike the abrupt generalization loss associated with overfitting.\n\nGeneralization in DNN is supported by mapping similar inputs to similar representations. At initialization, FTLE values are approximately zero, indicating distance-preserving mappings conducive to generalization. However, as training proceeds, high-FTLE boundaries proliferate the repr. space, stretching and distorting these local mappings and degrading generalization. This happens regardless even when the learnt boundaries are perfectly generalizable for standalone tasks. Therefore, in contrast to overfitting—which is characterized by overly complex boundaries—Type 2 LoP reflects widespread instability caused by the aggregation of the stretching effect by more and more learnt boundaries (regardless of their generalizability) that finally distort the repr. space in various direction causing chaotic behavior.\n\nWe deeply appreciate the opportunity to engage in this discussion. The reviewer’s thoughtful comments helped sharpen our own understanding, which we will integrate into the final manuscript to improve the clarity and impact of the work.\n\n### Regarding specific concerns\n\n- **W1**: With the overall reply above, we'd like to argue that the provided theory offer much more than a-posteriori diagnosis of LoP, and it reaches the direct cause of the two types of LoP, which also prompt the design of G-mixup that mitigate LoP in advance.\n\n- **W2 & Q1:** Our core motivation is to disentangle the mechanisms behind LoP. We deliberately selected toy models that isolate Type 1 and Type 2 LoP. Therefore, we used a narrower network with 10 hidden units which is prone to Type 1 LoP vs. a wider network of 300 hidden units which is prone to Type 2 LoP. These choices of setups serve as archetypes to illustrate the distinct underlying mechanisms. Networks of all widths follow our theory; what differs is the dominance and the onset speed of each type of LoP.\n\n- **Q2 & W1 & 3:** As discussed, collapse/expansion dynamics depend on activation function, data distribution, and training details. Figure 4 employs ReLU (Figure 4a,d) and tanh (Figure 4b,c) to depict typical real-world cases where both types can co-occur.\n- **Q3 & W4:** While on the surface-level similarities exist between Type 1/2 LoP and capacity/overfitting, the former reflect dynamic changes in representation during CL, instead static model properties that governs single task learning.\n\nWe are willing to further clarify during discussion."}, {"review_id": "OdLJmwVT7B", "reviewer": "Reviewer_FyTC", "summary": "The authors introduce an analytical perspective on why deep networks lose their ability to learn new tasks during continual training by combining Finite-Time Lyapunov Exponent (FTLE) analysis with the Unconstrained Feature Model from Neural Collapse theory. They show that loss of plasticity” (LoP) arises in two distinct modes, Type 1, where representations collapse within each class (highly negative FTLEs), and Type 2, where feature maps near decision boundaries become chaotic (highly positive FTLEs). They verify these phenomena on synthetic XOR/hyper-plane tasks and on Continual ImageNet and class-incremental CIFAR-100. Building on this insight, the authors propose Generalized Mixup, a simple extension of Mixup that amplifies labels during same-class interpolation to prevent collapse while still regularizing boundary regions, thereby stabilizing FTLEs.", "questions": "Q1. Why do you employ soft (i.e. non-one-hot) original labels in the Mix-up? What benefits does this confer?\n\nQ2. Did you apply the same ‘preprocess labels’ stage to all baseline methods? Please clarify.\n\nQ3. How can the activation layer operation be represented as a matrix multiplication? Please provide a clear justification or derivation.\n\nQ4. Please clarify the definition of $\\lambda_{trained,T}^{k,k}$ in section 3.2, especially the validity of $\\lambda_0^{k,k}$.\n\nQ5. Could you show the FTLE curves for the other baseline techniques as well?\n\nQ6. Your theoretical analysis and proofs appear limited to binary-class settings. Can your key theorems and FTLE-based findings be generalized to multi-class classification, and if so, can you provide either theoretical extensions or empirical support?\n\nQ7. Please report backward-transfer or average accuracy by periodically re-evaluating earlier tasks (or using a shared, growing head) to show whether your method also preserves past knowledge, not just forward plasticity\n\nQ8. How sensitive is your method to the confidence-amplification factor M and to the choice of Beta priors for inter- and intra-class mixing?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS1. The authors quantify how the representation space transforms during continual training with theoretical analysis as well as experimental results.\n\nS2. The paper provides an intuitive explanation of Type-1 and Type-2 mechanisms, along with a figure that makes them immediately understandable. Moreover, the concepts of FTLE and neural collapse (NC) are aptly extended to the continual learning setting\n\nWeaknesses:\n\nW1. In standard classification tasks, labels are represented as one-hot vectors. However, in the proposed Mix-up scheme, the original labels are not one-hot, and the manuscript does not explain why this choice is made.\n\nW2. It is unclear whether the ‘preprocess labels’ stage was applied to the baselines in the experiments. Without this clarification, the experimental validity is questionable—if the baselines also perform the same preprocessing, their degraded performance might be an artifact of that step.\n\nW3. Activation layers are typically expressed as element-wise nonlinear functions applied to features, because they cannot be written as matrix multiplications. Yet Equation (4) models the activation as a matrix product; this needs to be corrected or justified. Moreover, the framework does not explicitly account for front-end non-linearities (e.g., BatchNorm, residual scaling, optimizer dynamics), and the approximation errors resulting from these simplifications are not clearly quantified or empirically examined. \n\nW4. It is not specified whether the continual learning scenario involves binary classification with labels that merely flip at each task step, or whether entirely new labels (i.e.\\ new classes) are introduced. If the latter is the case, then in Section 3.2’s definition of $\\lambda_{trained,T}^{k,k}$, the term $\\lambda_0^{k,k}$ cannot exist: the first k refers to the initial label and the second to the label at task T, and they cannot be the same.\n\nW5. The paper presents the FTLE curves for only the proposed method. Without showing the FTLE for the other baselines, the credibility of the claim is weakened.\n\nW6. The analysis focuses on representation geometry, yet other LoP drivers—e.g., weight over-consolidation, dead-unit accumulation, curvature stiffening, learning-rate decay, or sheer capacity shortage—are neither modelled nor monitored. As a result, a system might lose plasticity even when FTLE tails look healthy. \n\nW7. The evaluation is restricted to image classification, so it is unclear whether the reported gains are due to mitigating Type-1/2 LoP specifically or merely reflect the general regularization benefits of Mixup-style augmentations.\n\nW8. By resetting classifier heads after each task, the study does not measure performance on prior tasks, leaving the risk of unobserved catastrophic forgetting.", "was_revised": true, "final_justification": "Although the issue of catastrophic forgetting remains to some extent, the authors have provided satisfactory responses. Therefore, I maintain my positive score.", "rebuttal": "First of all, we'd like to thank the reviewer for the invaluable suggestions, especially the ones regarding soft labels, LoP mitigation vs. mixup-style augmentation and Forgetting Measure, together they help improve the quality and impact of our work. We will make sure to include the results here into the revised manuscript.\n\n**W1, W2 & Q1, Q2**\nIn our original experiments (expt's), we did not preprocess the baseline methods with soft labels. To address this concern, we have now conducted additional expt's on both ImageNet and CIFAR-100 datasets, where all baselines are also trained using soft labels. On ImageNet with MSE loss, the impact of soft labels is negligible. On CIFAR-100, where cross-entropy(CE) loss is applied, soft labels offer a modest improvement during the learning of later tasks. This is because that soft labels help prevent the representational (repr.) collapse or over-expansion that can occur when logits are trained to match one-hot labels. Due to character limitations, we present results using CE loss as a representative case in this section. However, we have conducted all the experiments on the benchmarks presented in the main text, and would be happy to share those results upon request.\nFor all methods, the intra-class soft label smoothing factor was consistently set to 0.1 throughout our experiments. With these additional expt's, we can now show that G-Mixup consistently outperforms all baselines regardless of whether soft labels are used, as shown in the following table:\n\n**Resnet-18 Acc. on CIFAR100 with SoftLabel**\n|Task(×10)|0-1|1-2|\n|--|--|--|\n|NoIntv.|0.850|0.813|\n|Retrained|0.856|0.817|\n|L2init|0.860|0.783|\n|Layernorm|0.848|0.773|\n|CBP|0.836|0.821|\n| **G-mixup[ours]** | **0.884** | **0.828** |\n\nRegarding the use of soft labels in G-Mixup (assuming the reviewer refers to our method): G-Mixup was originally designed for MSE loss, using one-hot interpolation for inter-class mixing and \n(1 + $\\alpha$,0)–style labels for intra-class mixing. To enable the application of G-Mixup under the CE loss, we adopt soft labels to ensure the label remains a valid probability distribution (i.e., sums to one). This allows same-class samples to have different targets based on the mixing ratio m (see Eq. 15). \n\n**W3 & Q3**\n\nIndeed, real-world DNN contain layers with nonlinear activation where matrix multiplications cannot be directly applied. To circumvent this complication, we adopt unconstrained feature model (UFM, Eq. 4), where the post-activation representations are treated as free, trainable variables. This widely adopted simplification have been proven effective and necessary to advance the theoretical study of DL, especially for works on neural collapse and repr. dynamics (Zhu et. al., 2021 NeurIPS (ref. 34)). It allows us to reduce DL to an optimization problem between the last two layers, enabling analytical investigation of repr. dynamics. Finally, we have justified this with numerical experiments of both toy models and real-world datasets.\n\nAs the reviewer points out, nonlinearities—including BatchNorm, residual scaling, and the choice of optimizer—can all affect the learned representation. While our theoretical framework does not explicitly model these, we did conduct empirical validations under these settings, suggesting the simplified approach still works in the real-world setting. Specifically:\n\n- CIFAR-100 benchmark uses both BatchNorm and ResNet, thereby incorporating BN and skip connections.\n- On ImageNet, we test our method under various optimizer, including SGD, Adam, and AdamW.\n\n**W4 & Q5**\n\nIn our setting, we consider a continual learning (CL) scenario where for each new task $T$, the two units in the final-layer are reset to zeros. It could be said that these are new labels (the later case), but actually they are still $[1, 0]$ or $[0, 1]$ in the end (the former case). However, we want to emphasize that the notation $k$ does not concern with the labels, rather, they refer to the input distributions associated with the class. Note that in our CL setups, we allow the samples from a class to reappear and participate in multiple tasks. These scenarios occurs frequently in real-world CL when the definition of a single concept is learnt gradually against other concepts to form the map of knowledge, it is also frequently encountered in RL when policy rules changes under different context while the object class remains the same.\n\nTherefore, what's important is that any given pair of input samples of the same class $k$ share the same label in each task regardless of the vector being $[1, 0]$ or $[0, 1]$. This ensures their representations are consistently driven toward alignment and away from samples of other classes.\n\nIn conclusion, the use of $\\lambda_0^{k,k}$ and $\\lambda_{trained,T}^{k,k}$ is well justified regardless of label encoding. The differences in labels across tasks primarily affect the classifier weights, but not the underlying repr. structure in the feature space.\n\n**W5, W6 & Q6**\nPresenting FTLE curves for other methods would indeed strengthen our claims. However, due to the NeurIPS rebuttal format restrictions and character limitations, we only report the FTLE distribution statistics for baseline methods in the table below. Results for other methods are also available and can be provided upon request.\n\nTable: FTLE Distribution Across Tasks for Different Methods (ConvNet on Continual ImageNet)\n\n| Algo| Task ID | FTLE < -10 | -10~0 | 0~10 | FTLE>10 |\n|-|-|-|-|-|-|\n| No Intv. | 1000 | 0| 0 | 0.10| 0.90|\n| | 4900 | 0.61| 0 | 0| 0.39|\n| L2 init | 2000| 0 | 0.11| 0.89| 0 |\n|| 4500| 0 | 0.09| 0.91| 0 |\n| G-mixup | 2000| 0 | 0 | 1| 0 |\n|| 4500| 0 | 0 | 1| 0 |\n| CBP | 2000| 0 | 0 | 1| 0 |\n|| 4500| 0 | 0 | 1| 0 |\n\nAs shown above, methods that aim to address LoP also maintain FTLE within a healthy range, albeit with different mechanisms. In contrast, baselines exhibit significantly lower(higher) FTLE values, indicating repr. collapse and expansion.\nTheoretically, we've also demonstrated with theorem 3.1.2 that extreme FTLE values characterize the collapse and over-stretch of repr. space, which would lead to LoP.\nAlthough it is hard to _rigorously prove_ that a system will not experience LoP at all when FTLE looks health, we consider these evidences to be enough to claim that such events are highly unlikely.\n\n**W7**\n\nWe note that the smoothing effect of mixup-style augmentations align well with our theoretical framework, which highlights the importance of maintaining a well-conditioned (avoiding extreme FTLE values) repr. space during continual learning, but it could be overshadowed by the effect of sample diversity. Mixup promotes smooth transitions in the repr. space by interpolation of intra-class samples and labels, which can mitigate Type-2 LoP. However, mixup retains unchanged intra-class label, i.e, no smoothing, which limits its ability to prevent Type-1 LoP.\nTherefore, to empirically demonstrate that most of the performance improvement from G-mixup comes from mitigating LoP, rather than from the increased sample diversity, we restrict both mixup and G-mixup to intra-class sample pairs, thereby removing their `inter-class` smoothing effects and equalizing sample diversity. Whereas, G-Mixup still employs label enhancement, which systematically varies intra-class labels (Eq. 15), offering smoothing for the intra-class repr. space.\nAs shown in the table below, mixup rapidly goes into type 1 LoP, whereas G-Mixup consistently mitigate type 1 LoP, supporting continual learning.\n\n| Task(x500) | 0-1    | 1-2    | 2-3    | 3-4    |\n| ---------- | ------ | ------ | ------ | ------ |\n| G-Mixup    | 0.8641 | 0.8715 | 0.8792 | 0.8778 |\n| Mixup      | 0.8635 | 0.5813 | 0.5000 | 0.5000 |\n\n**W8 & Q7**\n\nIndeed, the stability–plasticity dilemma has been a central topic in the CL community. However, our study is positioned within the emerging line of research that focuses on _plasticity loss_—a perspective that emphasizes the opposite concern: maintaining the network's ability to learn new tasks effectively.\n\nFor a typical LoP benchmarks like Continual ImageNet which involves up to 5,000 tasks, backward consistency cannot be guaranteed. E.g, methods like CBP explicitly reset parts of the network at each step, fully embracing the loss of old task performance in favor of forward generalization. A related example is in reinforcement learning, where preserving value functions learned under outdated policies can hinder new learnings.\n\nThat said, we agree with the reviewer that empirical measurement of stability is valuable. To this end, we have evaluated our method using the Forgetting Measure, and include the results in the following table:\n\nConvNet on Continual ImageNet\n| Algo| FM|\n|-|-|\n| L2 init| 0.27 |\n| CBP | 0.27 |\n| G-mixup [Ours] | 0.28 |\n\nSmallConvNet on Continual ImageNet\n| Algo| FM|\n|-|-|\n| L2 init| 0.25 |\n| CBP | 0.23 |\n| G-mixup [Ours] | 0.30 |\n\n\n**Q6**\nAlthough our current theoretical analysis is primarily developed for binary classification tasks to ensure clarity and analytical tractability, we have empirically validated our approach on the Class-Incremental CIFAR-100 benchmark (see L299), where the number of classes increases to 100. We think that there are no fundamental obstacles to extending our framework to the multi-class classification setting.\n\n**Q8**\nWe appreciate the reviewer’s emphasis on hyperparameter robustness, which is indeed a crucial aspect for practical deployment. To examine this, we conducted additional experiments using both SmallConvNet and ConvNet on Continual ImageNet, exploring the amplification factor $M \\in [0.1, 0.3]$ and Beta prior $\\in [0.5, 1.5]$. Results show only mild performance variation: SmallConvNet ranged from 0.875 to 0.886, and ConvNet from 0.895 to 0.904, indicating that G-Mixup is reasonably robust without fine-tuning."}, {"review_id": "3PtrO0daWX", "reviewer": "Reviewer_m8rC", "summary": "The paper utilizes the unconstrained feature model (UFM) and finite-time Lyapunov exponent (FTLE) to investigate loss of plasticity. Two different types have been found: 1) decreasing FTLE, 2) increasing FTLE. Both types can coexist and are affected by the dimensionality of the network. Based on these theories, the paper proposed G-mixup as a way to smooth the between-class differences and increase the within-class differences. Such a method outperforms the existing methods.", "questions": "N/A", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) The paper proves that the training will causes a compression/stretching behavior in the penultimate layer representations.\n2) The paper finds a marker for LoP (a large FTLE causes chaotic behavior, and a small FTLE causes compression).\n3) The paper proposes a new method (G-mixup) to mitigate LoP by smoothing the boundaries and increasing the variability within the class.\n\nWeakness:\n1) FTLE in the framework seems unimportant: the major theorem 3.1.2 (since both type 1 and type 2 lop are defined on it) only introduces FTLE after the whole proof to reduce the notation of stretching/compressing $h$ representations. \n2) The paper gives much proof over the stretching/compressing behavior of the penultimate layer representations, but there is a gap between the behavior of $h$ and LoP. It would strengthen the paper if one could prove that the accuracy will decrease if the sample follows in the stretching/compressing region, and will cause the FTLE to increase or decrease. It would also strengthen the paper if one could empirically show that when the sample follows in the stretching/compressing region, and will cause the FTLE to increase or decrease (i.e., L227 and L228). The current paper shows an overall relationship between FTLE and LoP, but this does not connect to the theory.\n3) Empirically, the authors show G-mixup can mitigate the LoP and control the range of FTLE. To more closely relate to the theorem, it would be better to show results with directly controlling FTLE (e.g., set FTLE as a penalty if it deviates too much from task 1, or dropping some samples if it causes a too wild FTLE).", "was_revised": true, "final_justification": "The authors present a compelling, geometry-inspired analysis of “loss of plasticity” in continual learning—using finite-time Lyapunov exponents (FTLE) to distinguish between representation collapse and chaos—and couple it with a simple yet effective Generalized Mixup strategy. In the original version, the connection between the strategy and the theory is not strongly established. However, in the authors' rebuttal, they provide a direct method for mitigating LoP through controlling FTLE. There remains a theoretical gap between FTLE and accuracy, but the authors offer an intuitive explanation in the rebuttal, and hopefully future work can address it. As pointed out by Reviewer FyTC, catastrophic forgetting is heavily involved in continual learning, while LoP is less common. This reduces the overall significance of the current paper. In sum, I rate it as borderline acceptable.", "rebuttal": "We sincerely thank the reviewer for their thorough reading, thoughtful feedback, and constructive suggestions. We deeply appreciate the time and effort the reviewer devoted to evaluating our work. We have carefully addressed the reviewer’s comments in the following rebuttal, and we hope our clarifications successfully resolve the raised concerns.\n\n**Weakness 1**\nIn our work, FTLE is introduced only to quantify rather than to understand how the geometry of representation evolves during continual learning. Therefore, we'd like to argue that this is not a weakness of our work since our main contributions and emphases have been on the understanding of how the geometry of representational space evolves during continual learning, rather than FTLE itself.\n\nThat said, we still want to justify the application of FTLE in our work. FTLE is a widely adopted metric used in physics for describing the spatio-temporal dynamics of various systems, and was first introduced to deep learning analogously by Storm et. al., 2024 (ref 27) to describe how distance in the representational space relates to the distance in the input space. Here, we further develop a theoretical framework utilizing this metric, specifically for the dynamical characterization of the local geometry of representational space during continual learning. It has successfully described how the distance between representations of same-class samples and different-class samples evolves as learning continues. Therefore, we think it is fair to say that FTLE is very essential to our work as a quantifying tool.\n\n**Weakness 2**\nWe thank the reviewer for this insightful suggestion, which indeed helps strengthen the connection between the theoretical framework and empirical observations in our study.\n\nWe address this concern from both an _intuitive_ and an _empirical_ perspective:\n\n#### 1. Theoretical Intuition\n\nThe penultimate layer representation $h(x)$ is fed into a linear classifier to perform final prediction. For the linear classifier to perform well, representations must be linearly separable. Therefore:\n\n- In **compressing regions** of representational space with lower effective dimensions (prone to Type 1 LoP), if representations from _different classes_ are mapped to nearby or overlapping points (i.e., the compression collapses multiple classes into the same representation space), linear classification becomes impossible, leading to _accuracy degradation_.\n\n- In **stretching regions** of representational space of higher effective dimensions (prone to Type 2 LoP), small input perturbations are mapped to significantly different representations. When _same-class samples_ are send apart in various directions, the local geometry become distorted, making it difficult for the classifier to assign consistent predictions, again leading to lower _accuracy_.\n\nThese effects suggest that regions with extreme FTLE values (on either end) correspond to representational spaces where the local geometry make same-class alignment and inter-class separation progressively infeasible, both are detrimental to classification performance.<!--  Due to time and character limit, we are unable to provide a detailed proof, however we believe a rigorous rewrite of these intuitive understandings into mathematical proofs is possible. -->\n\n#### 2. Empirical Verification\n\nTo further substantiate this link, we visualize classification accuracy over training time for samples grouped by FTLE ranges. Initially, when LoP has not emerged, FTLE values are small and classification accuracy is high. As LoP progresses, FTLE values become either too large or too small (strong stretching/compression), and classification accuracy for these regions drops significantly. These findings empirically support the theoretical link between FTLE, local representational distortion, and classification accuracy.\n\n####  Accuracy and Sample Distribution within Different FTLE Ranges across Tasks\n\n| Task id |FTLE range      | <-10       | -10~0      | 0~10        | >10         |\n|-------|-------------|------------|------------|-------------|-------------|\n| 100   | Accuracy    | 0.000      | 0.000      | 0.800       | 0.000       |\n|       | Sample Count| 0          | 0          | 200         | 0           |\n| 1000  | Accuracy    | 0.000      | 0.000      | 0.813       | 0.804       |\n|       | Sample Count| 0          | 0          | 16          | 184         |\n| 5000  | Accuracy    | 0.518      | 0.000      | 0.000       | 0.713       |\n|       | Sample Count| 112        | 0          | 1           | 87          |\n\nWe thank the reviewer again for pointing out this gap, and we will include these new results in the appendix to clarify this connection between LoP dynamics and performance impact.\n\nAs for the other empirical support suggestion: \"if one could empirically show that when the sample follows in the stretching/compressing region, and will cause the FTLE to increase or decrease\", we believe there's a misunderstanding. As proved by theorem 3.1.2, same-class samples decrease the FTLE values for the region of interest while different-class samples increase them. This happens regardless of whether the samples are following the stretched or compressed region (through the force of back-propagation). Then based on which pair of samples (same-class or different-class) are put into these regions, the samples become less likely to be correctly classified, as supported by the theoretical intuition and empirical support above.\n\n**Weakness 3**\nWe thank the reviewer for this highly insightful suggestion. Indeed, when initially designing our approach, we also considered directly penalizing the FTLE in the loss function to preserve plasticity. However, we ultimately opted for G-Mixup due to practical feasibility.\n\nDirectly computing FTLE involves evaluating the local Jacobian of the feature extractor and estimating its maximum singular value, which is computationally expensive and may introduce gradient stability issues during training. This may make it hard and costly to use FTLE as a regularizer in large-scale settings like ImageNet.\n\nThat said, inspired by the reviewer’s suggestion, we implemented a **proof-of-concept FTLE regularization experiment** in a toy model. Specifically, we approximated the FTLE for each sample using the following procedure:\n\nLet $h(x)$ denote the feature representation of input $x$, and let $J = \\nabla_x h(x)$ be the local Jacobian. We approximate the FTLE using power iteration:\n\n$$\n\\\\begin{aligned}\n&\\\\text{Initialize } v \\\\sim \\\\mathcal{N}(0, I), \\\\quad v \\\\leftarrow \\\\frac{v}{\\\\|v\\\\|} \\\\\n&\\\\text{Iteratively update:} \\\\quad v \\\\leftarrow \\\\frac{J^\\\\top (Jv)}{\\\\|J^\\\\top (Jv)\\\\|} \\\\\n&\\\\text{Then:} \\\\quad \\\\text{FTLE}(x) = \\\\log \\\\left( \\\\| Jv \\\\| + \\\\varepsilon \\\\right)\n\\\\end{aligned}\n$$\n\nWe then add the following regularization term to the loss:\n\n$$\nL_{\\\\text{FTLE}} = \\\\lambda_{\\\\text{FTLE}} \\\\cdot \\\\left( \\\\text{FTLE}(x) \\\\right)^2\n$$\n\nThis term penalizes large deviations of the FTLE from 0, encouraging local smoothness in representation dynamics.  \nIn practice, to reduce computational cost, we used a 20-layer network in the toy experiment, which tends to collapse easily under the setup of Toy Experiment 1 as shown in Figure 2 of the main text. In contrast, when FTLE regularization is applied, the network is able to continually adapt to new tasks without collapsing. To make the FTLE computation more efficient, we performed 5 iterations per sample during the power iteration procedure. Additionally, we employed a linearly increasing FTLE regularization coefficient, set as \\\\( \\\\lambda_{\\\\text{FTLE}} = 0.05 \\\\times \\\\text{epoch} \\\\) (per task), to balance task learning and the structure of the representational space.\n\nAs shown in the table below, the FTLE regularization effectively mitigates LoP in toy settings. We thank the reviewer for the insightful suggestion and will discuss this alternative approach in the revised main text.\n\n#### Classification Accuracy over Task Ranges\n\n| Task Range | 1–10  | 11–20 | 21–30 | 31–40 | 41–50   |\n|---|-------|----|-----|-------|---------|\n| $ L_{classification} $        | 0.829 | 0.761 | 0.603 | 0.584 | 0.565   |\n| $ L_{classification} + L_{FTLE}$| 0.920 | 0.969 | 0.942 | 0.968 | 0.971 |\n\n\n\nWe thank the reviewer for this valuable suggestion. We will include these results in the appendix and highlight the discussion in the final version. Exploring scalable and efficient FTLE-regularization is an exciting direction for future work."}, {"review_id": "TF85BWLoEK", "reviewer": "Reviewer_GUee", "summary": "This paper presents a novel framework for understanding and mitigating the Loss of Plasticity (LoP) in deep continual learning.  The authors posit that LoP is a dual-natured problem, dissecting it into two distinct types. Specifically, Type-1 LoP is characterized by the collapse of the representation space within a class; Type-2 LoP is characterized by an over-stretching of the representation space near decision boundaries.  Based on this new understanding, they propose Generalized Mixup (G-Mixup), an extension of the standard Mixup technique. G-Mixup is designed to specifically counteract both types of LoP by not only regularizing the space between classes but also actively expanding the representation space within a class to prevent collapse. The authors validate their theory and the effectiveness of G-Mixup through experiments on real-world datasets, including ImageNet and CIFAR-100, where G-Mixup is shown to outperform existing LoP mitigation techniques.", "questions": "1. It is well-known that certain regularization-based methods designed to mitigate forgetting (e.g., EWC[3], LwF[4]) can inadvertently harm plasticity. It is not immediately clear how this phenomenon fits into the proposed dual-nature framework. Could you elaborate on whether the plasticity reduction caused by such stability-focused methods can be explained as a form of Type-1 (collapse) or Type-2 (chaos) LoP, or if it represents a different mechanism altogether?\n\n[3] Overcoming catastrophic forgetting in neural networks. PNAS 2017\n\n[4] Learning without forgetting. TPAMI 2017", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper's key contribution is its insightful reframing of LoP . By categorizing LoP into two distinct types based on the dynamics of representation space (collapse vs. chaotic expansion), the work offers a convincing and novel perspective on this challenging problem. The empirical evidence presented, particularly in Figure 4, provides strong support for this dual-nature hypothesis.\n2. The paper is exceptionally well-written and easy to follow. The authors effectively use clear visualizations (e.g., Figures 2 and 3) to build intuition around the core theoretical concepts\n3. The proposed G-Mixup method is directly and logically derived from the theoretical analysis. The experimental results confirm its effectiveness across different settings.\n\n**Weaknesses**\n1. A direct comparison between G-Mixup and Mixup is crucial for validating the central claims of the paper. I suggest the authors add this comparison to the **main text** of this paper.\n2. The paper focuses almost exclusively on plasticity, which is its stated goal. However, in continual learning, plasticity is inherently linked to stability (i.e., mitigating catastrophic forgetting). A more comprehensive evaluation would benefit from a discussion of how G-Mixup affects previously learned tasks. Evaluation metrics could refer to [1] [2].\n\n[1] Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective. ICML 2025\n\n[2] Improving plasticity in online continual learning via collaborative learning. CVPR 2024", "was_revised": true, "final_justification": "Given that this paper provides an insightful analysis of the loss of plasticity and proposes a practical solution, I believe it makes a valuable contribution to the field of continual learning and deserves acceptance.", "rebuttal": "We sincerely thank the reviewer for their time, thoughtful feedback, and positive assessment of our work. We are particularly grateful for the encouraging remarks regarding the novelty and significance of our contributions. We address the questions in detail below and are happy to further discuss any remaining concerns.\n\n**Weaknesses 1**: We are grateful to the reviewer's suggestion on adding the comparison between Mixup and G-Mixup to the main text, which is indeed crucial for demonstrating our method's true effectiveness. As pointed out in the main text mixup promotes smooth transitions in the repr. space by generating interpolated inter-class samples and labels, which can mitigate Type-2 LoP (i.e., over-stretch). However, it retains unchanged intra-class labels (i.e., fixed one-hot targets), which limits its ability to prevent Type-1 LoP. In contrast, G-mixup introduce intra-class label difference through Eq. 15.\n\nTherefore, to clearly demonstrate the difference between the effect of the two mixup-style methods. We conduct an additional experiment that restrict mixup and G-mixup to intra-class sample pairs only, removing the inter-class smoothing effects for both party, and focus on the intra-class repr. space where mixup and G-Mixup provide the same amount of sample diversity but G-Mixup further promotes intra-class representation enhancement.\n\n| Task(x500) | 0-1    | 1-2    | 2-3    | 3-4    |\n| ---------- | ------ | ------ | ------ | ------ |\n| G-Mixup    | 0.8641 | 0.8715 | 0.8792 | 0.8778 |\n| Mixup      | 0.8635 | 0.5813 | 0.5000 | 0.5000 |\n\nAs shown in the table above, mixup rapidly leads to collapsed representations and stagnated learning, whereas G-Mixup continues to support learning subsequent tasks. This result clearly demonstrates the critical role of G-Mixup in addressing LoP, apart from providing sample diversity. We will include these results in the main text of the revised manuscript.\n\n**Weaknesses 2**: We thank the reviewer for raising this insightful point. As correctly noted, the stability–plasticity dilemma is fundamental in continual learning. Our study explicitly focuses on _plasticity loss_, a relatively new but increasingly important perspective that emphasizes the degradation of a network’s capacity to adapt to new tasks, even in the absence of explicit constraints to retain prior knowledge.\n\nBenchmarks such as Continual ImageNet, which involve thousands of sequential tasks (up to 5,000), push the network into regimes where backward consistency is often infeasible or undesirable. In such settings, many methods—including CBP—explicitly reset parts of the model after each task, sacrificing stability in favor of forward plasticity. Similar motivations are found in reinforcement learning, where preserving previously learned value functions from outdated policies may hinder rather than help future performance. Therefore, in our work, the primary objective is to ensure that the network retains sufficient plasticity to continue adapting to novel tasks, rather than preserving accuracy on past tasks.\n\nThat said, we fully agree that empirical assessments of stability are important. In response to the reviewer's suggestion, we report two key metrics of stability—Average Forgetting (AF) and Relative Forgetting (RF)—as defined in the reviewer's referenced work [1], to better illustrate how performance on previous tasks may deteriorate. Furthermore, since Continual ImageNet is formulated as a binary classification problem, we additionally report the average deviation from chance level (i.e., 0.5) after learning all 5000 tasks, reflecting the network’s retained knowledge on previously encountered tasks.\n\nTable A Stability metrics on Continual ImageNet using SmallConvNet\n| Algo            | AF   | RF   | Std |\n| --------------- | ---- | ---- | ---------------- |\n| L2 init         | 0.25 | 0.32 | 0.15      |\n| CBP             | 0.23 | 0.29 | 0.12      |\n| G-mixup [Ours] | 0.30 | 0.36 | 0.13      |\n\nTable B Stability metrics on Continual ImageNet using ConvNet\n| Algo            | AF   | RF   | Std |\n| --------------- | ---- | ---- | ------ |\n| L2 init         | 0.27 | 0.33 | 0.15      |\n| CBP             | 0.27 | 0.33 | 0.07      |\n| G-mixup [Ours] | 0.28 | 0.34 | 0.11      |\n\nAs shown in the tables above, all methods experience some degree of forgetting on previously learned tasks. When evaluating the final model—after training on all 5000 tasks—on earlier tasks, we observe different behaviors in terms of variance from the 0.5 chance-level baseline:\n\n- **CBP** exhibits the lowest variance, suggesting that its representations are largely decorrelated from earlier task classifiers. This is consistent with the fact that CBP resets part of the network during each task, leading to representations that are less aligned with previously learned classification heads.\n- **L2 init** achieves the highest variance. This reflects the fact that L2 init constrains weights to stay close to their initialization, which helps maintain representational stability, but its performance largely depends on how well the representations induced by the initial weights align with the current task.\n- **G-Mixup** shows higher AF and RF values, as it allows the representation space to adapt more freely in order to optimize performance on the current task. While this may introduce some deviation from earlier representations, it supports effective continual learning by maintaining sufficient plasticity.\n\n\n**Question**: Regarding the reviewer’s thought-provoking question about the relationship between regularization-based stability methods (such as EWC and LwF) and our dual-nature framework of plasticity loss:\n\nWhile these methods also lead to impaired plasticity, the underlying mechanisms are fundamentally different. Regularization-based techniques impose constraints on network dynamics—e.g., freezing important weights (EWC) or forcing new predictions to remain close to prior outputs (LwF)—which restrict the network's ability to adapt. In contrast, _loss of plasticity_ as defined in our work refers to a spontaneous decline in adaptability that emerges _even without any external constraints_, purely due to cumulative learning and optimization dynamics.\n\nTherefore, the plasticity reduction caused by stability-focused methods does not neatly fall into the Type-1 (representational collapse) or Type-2 (representational chaos) categories we describe. Instead, they represent a separate, externally induced mechanism that trades plasticity for stability by design.\n\nNevertheless, there is an interesting conceptual overlap. Some recent studies in the plasticity loss literature have borrowed ideas from stability methods to _preserve plasticity_. For example:\n\n- **L2Init**: a plasticity-preserving counterpart to EWC that penalizes deviations from initial weights, under the hypothesis that the initial network state is maximally plastic.\n- **Initial Feature Regularization** (e.g., Lyle et al. 2022): constraining representations to remain close to those from random networks (conceptually similar to LwF), preserving expressivity and adaptability.\n\nWe sincerely thank the reviewer for these important references and suggestions. We will include this discussion in the revised manuscript to clarify the relation between our framework and stability-based methods, and to enhance the theoretical richness of our work."}]

Author rebuttal:

Meta-review: This paper offers a unique perspective on loss of plasticity, with clarity and effective use of visualizations. Notably, G-Mixup is derived using theoretical analysis rather than being an ad-hoc solution. However, reviewers emphasized the need for more comprehensive comparisons and questioned whether the FTLE-based theory truly explains loss of plasticity beyond being a diagnostic tool. The authors are encouraged to address these concerns in the camera-ready version.
Decision: Accept (poster)