Title: Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Class incremental learning

lifelong learning
Abstract: Class Incremental Learning (CIL) aims to continuously learn new categories while retaining the knowledge of old ones. Pre-trained models (PTMs) show promising capabilities in CIL. However, existing approaches that apply lightweight fine-tuning to backbones still induce parameter drift, thereby compromising the generalization capability of pre-trained models. Parameter drift can be conceptualized as a form of noise that obscures critical patterns learned for previous tasks. However, recent researches have shown that noise is not always harmful. For example, the large number of visual patterns learned from pre-training can be easily abused by a single task, and introducing appropriate noise can suppress some low-correlation features, thus leaving a margin for future tasks. To this end, we propose learning beneficial noise for CIL guided by information theory and propose Mixture of Noise (MiN), aiming to mitigate the degradation of backbone generalization from adapting new tasks. Specifically, task-specific noise is learned from high-dimension features of new tasks. Then, a set of weights is adjusted dynamically for optimal mixture of different task noise. Finally, MiN embeds the beneficial noise into the intermediate features to mask the response of inefficient patterns. Extensive experiments on six benchmark datasets demonstrate that MiN achieves state-of-the-art performance in most incremental settings, with particularly outstanding results in 50-steps incremental settings. This shows the significant potential for beneficial noise in continual learning. Code is available at https://github.com/ASCIIJK/MiN-NeurIPS2025.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Mixture of Noise (MIN), a new approach to exemplar-free class-incremental learning that reframes catastrophic forgetting as \"parameter drift\" (essentially noise that corrupts previous task knowledge). Guided by this perspective, MIN explicitly learns beneficial noise for each new task and injects it into intermediate features to mask spurious patterns. An auxiliary classifier blends these different noise patterns using adaptive weights, allowing the system to handle multiple tasks seamlessly. Tested across six benchmarks including CIFAR-100, CUB-200, ImageNet variants, and Food-101, MIN achieves state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. Recasting forgetting as noise and injecting learned “positive-incentive” noise is a new approach. The paper outlines two core strategies: Noise Expansion (one noise generator per task) and Noise Mixture (learning weights to combine them). Figures and equations provide a coherent view of the training pipeline (e.g., Fig.1 shows the Pi-Noise layer and three-step training).\n2. The experiments are reasonably comprehensive. MIN is tested on six datasets and multiple incremental settings (10/20/50 steps). Across these, MIN outperforms baselines like L2P, DualPrompt, CODA-Prompt, ACIL, FeCAM, RanPAC, etc.\n3. The authors emphasize that only a small, fixed number of parameters are learned per task. Figures 6(a)-(b) (described in Sec.5.3) show that MIN’s total parameters are comparable to other methods, while new-task parameters are lower than most. \n\n**Weaknesses:**\n\n1. The authors make many unclear statements in the introduction regarding the challenges of using pre-trained models (PTMs) in incremental learning, without providing support or evidence. Some notable ones:\n\n1.1 “Pre-trained models inherently exhibit task-specific redundancy”: It is unclear what is meant by \"task-specific redundancy\". Does this mean that pre-trained models carry features irrelevant to future tasks or that some learned features are irrelevant even for the current task?\n\n1.2 “Models inadvertently assimilate non-essential features into the decision boundaries”: The term \"non-essential features\" is ambiguous and needs further clarification. Also, the mechanism by which non-essential features are \"assimilated\" into decision boundaries is vague and lacks supporting evidence. \n\n1.3 “These features induce catastrophic forgetting through two mechanisms”: The two mechanisms are not clearly distinguished. While catastrophic forgetting often refers to the forgetting the previously learned tasks, it appears that the authors conflate catastrophic forgetting with inter-task confusion, treating both as the same: The performance reduction due to task confusion in future tasks should not be categorized as forgetting. Additionally, forgetting is generally attributed to parameter interference or representational overlap, rather than merely feature redundancy.\n\n1.4 “This issue stems from the parameter sensitivity of PTMs”: The term \"parameter sensitivity\" is not defined. Does it refer to high sensitivity to perturbations, fine-tuning instability, or something else?\n\n1.5 “It is advisable to mitigate individual tasks’ dependence on redundant features”: It’s not clear how one would determine or mitigate “dependence on redundant features”. This claim is too general to be feasible.\n\n1.6 “Catastrophic forgetting stems from parameter drift for old tasks, it can be conceptualized as the intrusion of noise during learning a new task, which undermining the original pattern of the old framework.” The reviewer is not convinced about this statement. Where is evidence?\n\n1.7 “As illustrated in Fig. 2, the visualization shows the impact of the proposed methods for subsequent tasks”: Figure 2 only presents heatmaps for individual images from the baseline and MIN, which does not adequately demonstrate \"the impact of the proposed methods on subsequent tasks.\"\n\n\n2. Methodological complexity and clarity issues. The training procedure involves multiple components (analytic classifier updates, auxiliary classifier, noise generators), which are hard to follow. For example, the role of the auxiliary classifier $W_{aux}$ in Eq.(14) and the sequence of updates (train $W_t$, then $P_t$, then $W_t$ again) are dense and confusing. Clear details of how the noise weights $\\omega_i$ are learned (the prototype similarity in Eq. (11)-(12) is not fully justified) would be necessary. Overall, some parts of Section 4.2 and the training pipeline need to be more clearly presented. \n\n3. Experimental clarity. Some details are either missing or need further clarification. It is not specified whether the results were averaged over random seeds for different task orderings, nor are standard deviations reported. In many cases, the performance improvement is minor, and without statistics such as standard deviation, it is not justified to label it as an improvement. In class incremental scenarios, the order of tasks will affect the performance which should be considered in experiments. Additionally, it is unclear whether all baseline methods were re-implemented under identical conditions (e.g., the same backbone, training schedule, and data preprocessing).\n\n4. Ablation studies: The ablation study is not convincing in demonstrating the effectiveness of the proposed method. From Table 4, the improvement with and without Noise Expansion is minimal (less than 1% in some cases). Without standard deviation and using different task orderings, the results are not valid. No ablation study provided for hyperparameters. \n\n**Rationales Behind Reviewer's Ratings:**\n\n**Quality (3: Good):** The proposed method appears to be technically sound, and the experiments are comprehensive. However, there is limited theoretical analysis, and some methodological aspects (e.g. the prototype-based mixing) are poorly justified.\n\n**Clarity (2: Fair):** The paper is not well-written. Many sections (notably Introduction, Section 4.2 and the algorithmic details) are very unclear and dense. \n\n**Significance (2: Fair):** The performance improvement over prior methods is incremental (few percent gain on benchmarks). While the idea of beneficial noise is interesting, it is not significantly novel. \n\n**Originality (3: Good):** The concept of injecting learned noise has appeared in other contexts (c.f. Pi-Noise literature), and using an auxiliary classifier for incremental learning has precedents. The specific combination (Noise Expansion + Mixture) is a novel twist, but it is not a fundamentally new paradigm.\n\nQUESTIONS:\nThe authors are asked to answer all the questions in Weaknesses, including (but not limited only to): \n\n1. Can the authors provide the formal definition of \"non-essential features\", followed by theoretical justification (or at least empirical evidence) to clarify how the non-essential features are assimilated into decision boundaries?\n\n2. Can the authors provide the formal definition of \"task-specific redundancy\" in the context of pre-trained models, followed by theoretical justification (or at least empirical evidence) to clarify how they distinguish between features that are irrelevant for future (new) tasks versus features that are irrelevant for the current task?\n\n3. The noise-mixing weights depend on the temperature $\\tau$ (Eq. 12) and the choice of prototype dimension. How sensitive are results to $\\tau$ or to the dimensionality $d_2$ used in the noise generator? Have you tuned these, and is performance stable across reasonable values? Providing an ablation (or variance over seeds) would clarify robustness.\n\n4. Did you ensure all baselines use identical backbones, optimization hyperparameters, and exemplar-free settings? For example, in original paper MOS for CUB dataset and 10 tasks the average accuracy reported 93.40 but, in your work, it is 92.08.\n\n5. You use a weighted sum of past noises (Eq. 13). Did you compare this to simpler strategies, such as averaging or using only the current task’s noise? The ablation shows both steps, but it might help to see intermediate variants (e.g. mixing without learning weights vs. with random weights) to fully justify the learned mixture\n\nLIMITATIONS:\nThe authors do not explicitly discuss limitations. But, MIN has so far only been validated on image classification benchmarks with ViT backbones; its effectiveness in other domains or other tasks is unknown. Moreover, any assumptions behind “beneficial noise” (e.g. tasks share common distractors) should be clarified.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-21"}

comments: SUMMARY:
This paper focuses on class incremental learning and proposes a new noise perspective to rethink the parameter drift problems in this task. Based on this perspective, a noise mix method was proposed to integrate the incremental knowledge and previous knowledge. Various experiments show that this perspective and method are effective.

STRENGTHS AND WEAKNESSES:
Pros:
1) This paper provides a new noise perspective regarding the parameter drift or forgetting problem, very interesting.
2) Based on this kind of noise perspective, this paper designs some filters or integrating mechanisms to maintain the good noise and remove others.
3) This perspective and framework are general and can be easily extended to other semi-dense (like detection) or dense prediction tasks.
4) The results show consistent improvements, and the whole experiment with ablations is sufficient.
5) The whole writing is very clear and smooth, organized well, and easy to understand.

Cons:
You should add some visualization or other high-level intuition explanations to make readers accept the noise perspective more easily and naturally.

QUESTIONS:
Can your method be extended to the incremental learning in a generative model, for instance, to learn some new concepts to generate, but without destroying the previous concept generation?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript focused on the class incremental learning (CIL) with pre-trained models (PTM). Based on the analytic learning methods [1], this work considered leveraging the beneficial noise and proposed Mixture of Noise (MIN). Specifically, a task-specific adapter is based on the task-specific noise learned from each task. Then, a set of dynamic weights was used to adjust the optimal mixture of different task noises. Based on this design, the beneficial noise was embedded into the intermediate features to mask the response of inefficient patterns. Experiments on some commonly used benchmarks were conducted to demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. This manuscript adopted a special perspective, the beneficial noise, to design the task-specific adapters, which is novel to the reviewer. Although it was still under the framework of parameter-efficient adapters, it brought a new perspective to designing the adapters.\n2. The experimental part is complete. Most recent baseline methods and commonly used datasets have been included. The ablation study is clear.\n3. Source code was provided to help the readers verify the reproducibility.\n\n\n## Weaknesses\n1. Although this manuscript tried to use \"beneficial noise\" to \"mask the irrelevant activation in the features\", the explanation of this design was still not clear for me. Based on my understanding, the MIN module seems like a \"Bayesian Parameter-Efficient Adapter\" since it introduces some uncertainties through noisy sampling within the parameters of the intermediate layer. However, I didn't see any discussion related to the \"Bayesian\" or \"uncertainty\". Also, the processing of the network output (since it seems that we have an extra sampling process for $\\epsilon$) is not clear.\n2. The motivation of the proposed method is still confusing. The connection between the methodology and some claims was not straightforward. The authors introduced some expressions, like \"learn positive noise to suppress the confusing pattern across tasks\" in Line 141, but I didn't find any connections with the design. See the Questions part for more details.\n3. Some methodological steps were not clear to the reader. For example, the motivation for introducing the auxiliary classifier in Eq. (14) is not clear.\n\nQUESTIONS:\n- 1. I noticed that the reparameterization process was introduced within the adapter $P_t$. Then, I have two questions regarding this process:\n    - 1.1 During the inference process, do we need to sample $\\epsilon$ multiple times for each sample $x$? If not, please explain the reason.\n    - 1.2 If so, how do you determine the network output? Based on my understanding, for each input $x$, we need to sample $\\epsilon$ for multiple times $\\epsilon^{(1)}, \\epsilon^{(2)}, \\cdots, \\epsilon^{(k)}$, we will also get multiple $\\epsilon_{t}$ as  $\\epsilon^{(1)}_{t}, \\epsilon^{(2)}_{t}, \\cdots, \\epsilon^{(k)}_{t}$  for  $k$ samplings. How should we compute the weighting process in Eq.(13) and the final output of the neural network?\n\n- 2. In Line 141, the authors claimed that the positive noise can be learned to \"suppress the confusing pattern across tasks through a large interference to mask irrelevant activation in the features\". I guess the irrelevance and relevance were measured by the cosine similarity between the reparameterized noise center $\\mu$. Based on this consideration, how about directly adopting a single parameter $\\mu$ to make it a deterministic adapter? \n\n- 3. About the motivation behind the auxiliary classifier. What is the objective or consideration for introducing this auxiliary classifier? I didn't get it.\n\n- 4. Some discussions regarding the high-level ideas of the proposed method. When I first read the methodological parts, especially Sections 4.1 and 4.2, I naturally connected them with some existing concepts, including \"Bayesian Neural Network\" and \"uncertainty\" which have been widely investigated in deep learning. This manuscript explained the proposed method under the concept of positive noise and ignored some descriptions about the relevant details, e.g., the multiple sampling regarding $\\epsilon_{1}, \\epsilon_{2}, \\cdots, \\epsilon_{t}$. This made some steps confusing. Could the author discuss the relationship between the proposed method and the studies about \"Bayesian Neural Network\" or \"uncertainty\"?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method called Mixture of Noise (MIN) to address catastrophic forgetting during class-incremental learning. \nThe key idea is to view the problem of parameter drift, where the model forgets old information while learning new things, as a form of destructive noise. \nTo combat this, the proposed MIN method learns a beneficial noise for each new task. \nThe method then uses a Noise Mixture strategy to dynamically combine the beneficial noises from all the tasks the model has learned. \nExperiments on several benchmark datasets show the effectiveness of MIN.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper is easy to follow.\n2. The experiments provided show overall high performance in the benchmarks.\n3. Using noise to enhance CIL performance is something new to me, and I believe it will introduce new insight to the community.\n\nWeaknesses:\n\n1. The paper uses Analytic Learning as its baseline. However, as described in Fig. 1(c), the PTM is updated in each stage through the proposed noise mechanism. The solution in Eq. (3) is derived from Eq. (1) under the assumption that the feature extractor F(·) is frozen during all training phases. In the paper's setting, where F(·) is modified, does the solution presented in Eq. (3) still hold? This needs further clarification.\n\n2. Another major concern is that the proposed noise expansion and noise mixture mechanisms are functionally similar to previous prompt-based or adapter-based PEFT methods. They all introduce additional parameters into the transformer to adapt to new tasks. For the current version of this paper, I am more inclined to see this as an improved adapter mechanism with limited technical contribution. I suggest the authors add more insightful discussion, using theory or experiments, to further elaborate on the advantages of their approach over previous adapter-based mechanisms, specifically for CIL.\n\n3. I believe the set of learnable weights  $w$ shown in Eq. (12) is a crucial component for mitigating forgetting, yet the paper provides insufficient experimental analysis for it. I suggest the authors include more ablation studies, for instance, by replacing the weighted summation in Eq. (13) with a simple average of the noises. Furthermore, it would be insightful to analyze or visualize the final learned weights $w$ after incremental tasks are completed.\n\nQUESTIONS:\nPlease see the weaknesses above.\nI encourage the authors to address my concerns, and I am with a open mind on raising my score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}