Title: The Dual Nature of Plasticity Loss in Deep Continual Learning: Dissection and Mitigation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deep Continual Learning

Loss of Plasticity

Finite-time Lyapunov Exponent
Abstract: Loss of plasticity (LoP) is the primary cause of cognitive decline in normal aging brains next to cell loss.
Recent works show that similar LoP also plagues neural networks during deep continual learning (DCL). 
While it has been shown that random perturbations of learned weights can alleviate LoP, its underlying mechanisms remain insufficiently understood.
Here we offer a unique view of LoP and dissect its mechanisms through the lenses of an innovative framework combining the theory of neural collapse and finite-time Lyapunov exponents (FTLE) analysis.
We show that LoP actually consists of two contrasting types: (i) type-1 LoP is characterized by highly negative FTLEs, where the network is prevented from learning due to the collapse of representations; (ii) while type-2 LoP is characterized by excessively positive FTLEs, where the network can train well but the growingly chaotic behaviors reduce its test accuracy.
Based on these understandings, we introduce Generalized Mixup, designed to relax the representation space for prolonged DCL and demonstrate its superior efficacy vs. existing methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel framework for understanding and mitigating the Loss of Plasticity (LoP) in deep continual learning.  The authors posit that LoP is a dual-natured problem, dissecting it into two distinct types. Specifically, Type-1 LoP is characterized by the collapse of the representation space within a class; Type-2 LoP is characterized by an over-stretching of the representation space near decision boundaries.  Based on this new understanding, they propose Generalized Mixup (G-Mixup), an extension of the standard Mixup technique. G-Mixup is designed to specifically counteract both types of LoP by not only regularizing the space between classes but also actively expanding the representation space within a class to prevent collapse. The authors validate their theory and the effectiveness of G-Mixup through experiments on real-world datasets, including ImageNet and CIFAR-100, where G-Mixup is shown to outperform existing LoP mitigation techniques.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper's key contribution is its insightful reframing of LoP . By categorizing LoP into two distinct types based on the dynamics of representation space (collapse vs. chaotic expansion), the work offers a convincing and novel perspective on this challenging problem. The empirical evidence presented, particularly in Figure 4, provides strong support for this dual-nature hypothesis.\n2. The paper is exceptionally well-written and easy to follow. The authors effectively use clear visualizations (e.g., Figures 2 and 3) to build intuition around the core theoretical concepts\n3. The proposed G-Mixup method is directly and logically derived from the theoretical analysis. The experimental results confirm its effectiveness across different settings.\n\n**Weaknesses**\n1. A direct comparison between G-Mixup and Mixup is crucial for validating the central claims of the paper. I suggest the authors add this comparison to the **main text** of this paper.\n2. The paper focuses almost exclusively on plasticity, which is its stated goal. However, in continual learning, plasticity is inherently linked to stability (i.e., mitigating catastrophic forgetting). A more comprehensive evaluation would benefit from a discussion of how G-Mixup affects previously learned tasks. Evaluation metrics could refer to [1] [2].\n\n[1] Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective. ICML 2025\n\n[2] Improving plasticity in online continual learning via collaborative learning. CVPR 2024\n\nQUESTIONS:\n1. It is well-known that certain regularization-based methods designed to mitigate forgetting (e.g., EWC[3], LwF[4]) can inadvertently harm plasticity. It is not immediately clear how this phenomenon fits into the proposed dual-nature framework. Could you elaborate on whether the plasticity reduction caused by such stability-focused methods can be explained as a form of Type-1 (collapse) or Type-2 (chaos) LoP, or if it represents a different mechanism altogether?\n\n[3] Overcoming catastrophic forgetting in neural networks. PNAS 2017\n\n[4] Learning without forgetting. TPAMI 2017\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper utilizes the unconstrained feature model (UFM) and finite-time Lyapunov exponent (FTLE) to investigate loss of plasticity. Two different types have been found: 1) decreasing FTLE, 2) increasing FTLE. Both types can coexist and are affected by the dimensionality of the network. Based on these theories, the paper proposed G-mixup as a way to smooth the between-class differences and increase the within-class differences. Such a method outperforms the existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The paper proves that the training will causes a compression/stretching behavior in the penultimate layer representations.\n2) The paper finds a marker for LoP (a large FTLE causes chaotic behavior, and a small FTLE causes compression).\n3) The paper proposes a new method (G-mixup) to mitigate LoP by smoothing the boundaries and increasing the variability within the class.\n\nWeakness:\n1) FTLE in the framework seems unimportant: the major theorem 3.1.2 (since both type 1 and type 2 lop are defined on it) only introduces FTLE after the whole proof to reduce the notation of stretching/compressing $h$ representations. \n2) The paper gives much proof over the stretching/compressing behavior of the penultimate layer representations, but there is a gap between the behavior of $h$ and LoP. It would strengthen the paper if one could prove that the accuracy will decrease if the sample follows in the stretching/compressing region, and will cause the FTLE to increase or decrease. It would also strengthen the paper if one could empirically show that when the sample follows in the stretching/compressing region, and will cause the FTLE to increase or decrease (i.e., L227 and L228). The current paper shows an overall relationship between FTLE and LoP, but this does not connect to the theory.\n3) Empirically, the authors show G-mixup can mitigate the LoP and control the range of FTLE. To more closely relate to the theorem, it would be better to show results with directly controlling FTLE (e.g., set FTLE as a penalty if it deviates too much from task 1, or dropping some samples if it causes a too wild FTLE).\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce an analytical perspective on why deep networks lose their ability to learn new tasks during continual training by combining Finite-Time Lyapunov Exponent (FTLE) analysis with the Unconstrained Feature Model from Neural Collapse theory. They show that loss of plasticity” (LoP) arises in two distinct modes, Type 1, where representations collapse within each class (highly negative FTLEs), and Type 2, where feature maps near decision boundaries become chaotic (highly positive FTLEs). They verify these phenomena on synthetic XOR/hyper-plane tasks and on Continual ImageNet and class-incremental CIFAR-100. Building on this insight, the authors propose Generalized Mixup, a simple extension of Mixup that amplifies labels during same-class interpolation to prevent collapse while still regularizing boundary regions, thereby stabilizing FTLEs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1. The authors quantify how the representation space transforms during continual training with theoretical analysis as well as experimental results.\n\nS2. The paper provides an intuitive explanation of Type-1 and Type-2 mechanisms, along with a figure that makes them immediately understandable. Moreover, the concepts of FTLE and neural collapse (NC) are aptly extended to the continual learning setting\n\nWeaknesses:\n\nW1. In standard classification tasks, labels are represented as one-hot vectors. However, in the proposed Mix-up scheme, the original labels are not one-hot, and the manuscript does not explain why this choice is made.\n\nW2. It is unclear whether the ‘preprocess labels’ stage was applied to the baselines in the experiments. Without this clarification, the experimental validity is questionable—if the baselines also perform the same preprocessing, their degraded performance might be an artifact of that step.\n\nW3. Activation layers are typically expressed as element-wise nonlinear functions applied to features, because they cannot be written as matrix multiplications. Yet Equation (4) models the activation as a matrix product; this needs to be corrected or justified. Moreover, the framework does not explicitly account for front-end non-linearities (e.g., BatchNorm, residual scaling, optimizer dynamics), and the approximation errors resulting from these simplifications are not clearly quantified or empirically examined. \n\nW4. It is not specified whether the continual learning scenario involves binary classification with labels that merely flip at each task step, or whether entirely new labels (i.e.\\ new classes) are introduced. If the latter is the case, then in Section 3.2’s definition of $\\lambda_{trained,T}^{k,k}$, the term $\\lambda_0^{k,k}$ cannot exist: the first k refers to the initial label and the second to the label at task T, and they cannot be the same.\n\nW5. The paper presents the FTLE curves for only the proposed method. Without showing the FTLE for the other baselines, the credibility of the claim is weakened.\n\nW6. The analysis focuses on representation geometry, yet other LoP drivers—e.g., weight over-consolidation, dead-unit accumulation, curvature stiffening, learning-rate decay, or sheer capacity shortage—are neither modelled nor monitored. As a result, a system might lose plasticity even when FTLE tails look healthy. \n\nW7. The evaluation is restricted to image classification, so it is unclear whether the reported gains are due to mitigating Type-1/2 LoP specifically or merely reflect the general regularization benefits of Mixup-style augmentations.\n\nW8. By resetting classifier heads after each task, the study does not measure performance on prior tasks, leaving the risk of unobserved catastrophic forgetting.\n\nQUESTIONS:\nQ1. Why do you employ soft (i.e. non-one-hot) original labels in the Mix-up? What benefits does this confer?\n\nQ2. Did you apply the same ‘preprocess labels’ stage to all baseline methods? Please clarify.\n\nQ3. How can the activation layer operation be represented as a matrix multiplication? Please provide a clear justification or derivation.\n\nQ4. Please clarify the definition of $\\lambda_{trained,T}^{k,k}$ in section 3.2, especially the validity of $\\lambda_0^{k,k}$.\n\nQ5. Could you show the FTLE curves for the other baseline techniques as well?\n\nQ6. Your theoretical analysis and proofs appear limited to binary-class settings. Can your key theorems and FTLE-based findings be generalized to multi-class classification, and if so, can you provide either theoretical extensions or empirical support?\n\nQ7. Please report backward-transfer or average accuracy by periodically re-evaluating earlier tasks (or using a shared, growing head) to show whether your method also preserves past knowledge, not just forward plasticity\n\nQ8. How sensitive is your method to the confidence-amplification factor M and to the choice of Beta priors for inter- and intra-class mixing?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the Loss of Plasticity (LoP) through the lens of the theory of neural collapse and finite-time Lyapunov exponents (FTLE) analysis. Their claim is that LoP can manifest in two separate regimes:\n(1) Type 1 LoP is characterized by highly negative FTLEs, where the network is prevented from learning due to the collapse of representations\n(2) Type-2 LoP is characterized by excessively positive FTLEs, where the network can train well but which test accuracy on new tasks degrades\nThe authors propose an approach called Generalized Mixup, which allows to mitigate LoP and which is inspired by these 2 regimes.\n\nThe authors conduct toy and real-datasets experiments to showcase their proposed Generalized Mixup strategy as well as to demonstrate two types of the LoP.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* A novel perspective on the LoP through the theory of Neural Collapse and finite-time Lyapunov exponents (FTLE) analysis. Their theory identifies two separate regimes of the LoP.\n* A new empirical method for augmenting data, Generalized Mixup, which can tackle the LoP. I think it is particularly interesting because generally LoP approaches do not focus on data augmentations\n* Experimental results showcasing that Generalized Mixup could help with the LoP\n* (Controversial) Toy experiments showcasing two regimes of LoP through the lens of the proposed theory\n\nWeaknesses:\n* In my opinion, theoretical contribution is a bit limited. The proposed approach feels more like a \"diagnostic\" of what is happening, rather than a theory which explains exactly why the LoP happens. My opinion is based on Figure 4, where in both Low dim and High Dim spaces, there are both Type 1 and Type 2 regimes happening. Why is this the case? When I check the toy experiments, it looks like Type 1 happens when the network capacity is low (because authors use 10 hidden units per layer) while Type 2 happens when the network capacity is high (because they use 300 hidden units). Based on this, I would have assumed that In figure 4, in low dim space, only Type 1 would happen, but there is Figure 4,c where both happen. Similarly for Figure 4,b and d. Therefore, it feels that this perspective on Type 1 and Type 2, is something which we can look a-posteriori as a diagnostic of what went wrong, but not as something which can be used to predict LoP.\n\n* The authors don't really explain in the main text why they use 10 hidden units per layer for Type 1 toy experiment and 300 hidden units per layer for Type 2 toy experiment. It feels arbitrary.\n\n* Same criticism, for Figure 4, the authors don't explain the difference between (a) and (c); and (b) and (d).  It's unclear why we have Type 1 and then suddenly Type 1+ Type 2.\n\n* Based on weakness I mentioned above, could the story which is suggested by the authors be simply explained by (a) lack of network capacity (for Type 1), and by (b) overfitting (for Type 2), as opposed to their theoretical framework.\n\n* Based on these, I think the significance of the work is a bit limited.\n\n* I have no concerns regarding the Generalized mixup, this looks like something which helps for addressing LoP.\n\nQUESTIONS:\n* What is the reasoning behind architectural design choices for toy experiments for Type 1 & Type 2?\n\n* What is the difference in the experimental regime for Figure 4.a & 4.c and Figure 4.b and 4.d?\n\n* Could the story which is suggested by the authors be simply explained by (a) lack of network capacity (for Type 1), and by (b) overfitting (for Type 2), as opposed to their theoretical framework?\n\nLIMITATIONS:\nPlease see weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}