Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning
Authors: Kai Jiang
Zhengyan Shi
Dell Zhang
Hongyuan Zhang
Xuelong Li
Venue: NeurIPS 2025 poster ()
Keywords: Class incremental learning
lifelong learning
Primary area: general_machine_learning

Abstract: Class Incremental Learning (CIL) aims to continuously learn new categories while retaining the knowledge of old ones. Pre-trained models (PTMs) show promising capabilities in CIL. However, existing approaches that apply lightweight fine-tuning to backbones still induce parameter drift, thereby compromising the generalization capability of pre-trained models. Parameter drift can be conceptualized as a form of noise that obscures critical patterns learned for previous tasks. However, recent researches have shown that noise is not always harmful. For example, the large number of visual patterns learned from pre-training can be easily abused by a single task, and introducing appropriate noise can suppress some low-correlation features, thus leaving a margin for future tasks. To this end, we propose learning beneficial noise for CIL guided by information theory and propose Mixture of Noise (MiN), aiming to mitigate the degradation of backbone generalization from adapting new tasks. Specifically, task-specific noise is learned from high-dimension features of new tasks. Then, a set of weights is adjusted dynamically for optimal mixture of different task noise. Finally, MiN embeds the beneficial noise into the intermediate features to mask the response of inefficient patterns. Extensive experiments on six benchmark datasets demonstrate that MiN achieves state-of-the-art performance in most incremental settings, with particularly outstanding results in 50-steps incremental settings. This shows the significant potential for beneficial noise in continual learning. Code is available at https://github.com/ASCIIJK/MiN-NeurIPS2025.

Mixture of Noise for Pre-Trained Model-Based Class-Incremental Learning
Kai Jiang 1.2, Zhengyan Shi2.3, Dell Zhang', Hongyuan Zhang2.4 and Xuelong Li
School of Artificial Intelligence, OPtics and ElectroNics, Northwestern Polytechnical University 2 Institute of Artificial Intelligence (TeleAl) of China Telecom
School of Computer Science School of Artificial Intelligence, Shanghai Jiao Tong University *University of Hong Kong jdu.u.uuu.uuu.u....uu... jk@mail www.nnppud....mo edu cn, macho2021@sjtu. edu cn, fell.20ieee.com org.
hyzhang98@gmai1 com, xuelong www.eeeieeee.eeeeeeeeeeeeeeeeeee org
Abstract
Class Incremental Learning (CIL) aims to continuously learn new categories while retaining the knowledge of old ones. Pre-trained models (PTMs) show promising capabilities in CIL. However, existing approaches that apply lightweight fine-tuning backbones still induce parameter drift, thereby compromising the generalization capability of pre-trained models. Parameter drift can be conceptualized as a form of noise that obscures critical patterns learned for previous tasks. However, recent researches have shown that noise is not always harmful. For example, the large number of visual patterns learned from pre-training can be easily abused by a single task, and introducing appropriate noise can suppress some cow-corellttion features, thus leaving margin for future tasks. To this end, we propose learning beneficial noise for CIL guided by information theory and propose Mixture of Noise (MIN,, aiming to mitigate the degradation backbone generalization from adapting new tasks. Specifically, task-specific noise is learned from high-dimension features of new tasks. Then, a set of weights is adjusted dynamically for optimal mixture of different task noise. Finally, MIN embeds the beneficial noise into the intermediate features to mask the response of inefficient patterns. Extensive experiments six benchmark datasets demonstrate that MIN achieves state-of- the-art performance most incremental settings, with particularly outstanding results in 50-steps incremental settings. This shows the significant potential for beneficial noise continual learning. Code is available at mtlhttss:// //g::///tttt com/ ASCIIJK/ MAA.JK//MNNNNM Neur IPS2025
Introduction
Unlike human beings can naturally learn new concepts without forgetting, existing AI systems lack the ability for continual learning (1-4). To address this challenge, Class-Incremental Learning (CIL)
is proposed to mitigate catastrophic forgetting (5) of old knowledge when acquiring new concepts. Most traditional CIL works focus on learning- wfro--scratte 12, i.c., learning model without pre- training. However, the pre-training technique has been widely used in real-world applications [13] [4]. Fine-tuning with pre-trained model (PTM) downstream tasks has become a consensus within the Al community. This has sparked significant research interest in the continual fine-tuning of PTMs for downstream tasks.
Due to pre-training on large-scale datasets, PTMS appear strong generalization. Therefore, preserving the generalization ability of PTMs across all incremental phases becomes a key factor maintain
Conresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

the performance in downstream tasks. A critical challenge arises from persistent cross-task feature interference. Pre-trained models inherently exhibit task-specific redundancy, with only limited discriminative features being task-relevant. During continual adaptation, models inadvertently assimilate non-essential features into the decision boundaries of the current task. These features induce catastrophic forgetting through two mechanisms: 1) when the features are inherited from previous tasks, they compromise established decision boundaries of preceding models, 2) adoption of these features by subsequent tasks disrupts current task boundaries. This issue stems from the parameter sensitivity of PTMs. While such models demonstrate remarkable adaptability to new tasks, this flexibility conversely renders them vulnerable to interference from subsequent tasks. Rather than pursuing enhanced feature utilization efficiency during task adaptation, it is advisable to mitigate individual tasks' dependence on redundant features.
Positive-incention noise (Pi-Noise) has been proven effective in diverse tasks []. It works by masking the confusing part among different categories with noise and highlighting the recognizable part. Catastrophic forgetting stems from parameter drift for old tasks, it can be conceptualized as the intrusion of noise during learning a new task, which undermining the original pattern of the old framework. Joint training based on old tasks also changes the original parameters, but the noise introduced by joint training suppresses confusing inter-task patterns, consequently producing positive impact. Therefore. we aim to directly learn this kind of positive noise from new tasks and embed it into intermediate features. The main contributions of our work are three-fold:
We rethink class incremental learning from the perspective of noise, interpreting parameter drift as introducing noise that is destructive to old tasks. Based on this assumption, we propose to learn beneficial noise for new tasks to suppress the response of the confusing pattern among tasks.
We design the Noise Expansion strategy learn new noise generation module for each task. To avoid multiple inference through the backbone, we mix the beneficial noise from different tasks by training a set of learnable weights using an auxiliary classifier and residual loss, dubbed the Noise Mixture strategy. The noise mixture from multiple tasks is embedded with intermediate features to suppress the confusing patterns among tasks.
We validate MIN on six widely used benchmark datasets. Experimental results show that MIN only needs few learnable parameters to learn beneficial noise adapted to the new task and achieve state-of-the--e CAM indicates that MIN effectively suppresses the response of the irrelevant region and strengthens the representation of key pattern.
performance. Additionally, the visualization with Grad-
2 Related Work
Class-Incremental Learning (CIL): aims to learn a unified model from a data stream without forgetting. Conventional CIL approaches can be roughly divided into several types. Rehearsal- based methods [8] 15H18) construct a fixed exemplar-set
preserve a fraction of samples from old
tasks for future training. Regularization based methods add parameter regularization terms [6] [7] or 20 to transfer knowledge from the old model to the new 24
function regularization terms 10
19
one. Architecture based methods 11
12 25] often add new trainable modules to expand the based into a recursive analytic learning process.
21
feature space for new tasks. Analytic learning based methods
26 29 reformulate the CIL procedure
Pre- Tre-Trained Model-Based CIL: aims fine-tune the PTMs in sequential downstream tasks without forgetting while keeping the generalization ability of PTMs. I.2P (30) and DualPrompt [3] construct prompt pool to select the suitable prompts for samples during model inference. CODA-Prompt 33 develops the prompt selection process with attention-base
weighting. SLCA (33) slows the update of the backbone and designs a classifier alignment strategy using Gaussian modeling for previous
classes. APER 34 shows that the prototypical classifier is strong baseline. FeCAM 335 explores
the prototype network and shows that modeling the feature covariance relations is better than previous attempts at sampling features from normal distributions and training a linear classifier. RanPAC (36]
adopts random projection to enhance the linear separability for prototype-based CIL. EASE [37 adds
task-specifer branches for feature expansion. COFiMA 38 selectively weighs each parameter in the weights ensemble by leveraging the Fisher information of the weights of the model. MOS [39
rectifies the model with training- frre self-refined adapter retrieval mechanism during inference.

---

Task
Down
di=eth
N(O, .
MLP
MLP
dax
Noise Expansion
features
Transformer block
P
4
(b) Noise generator
Noise Layer
Noise Mixture
Tronsformer block
Updort Wi with analyti harning
Wx
features
PTM
tralnable
raalubii meddules (c) Truining pipeline for task
Pi-Noise Luyer
frozwn
Figure I: Illustration of MIN. (a) Pi-Noise Layer: is inserted between transformer blocks of PTM for learning positive noise. Noise Expansion adds a generator for each task and Noise Mixture learns to combine positive noise from different tasks. (b) Noise generator: samples signal from a standard normal distribution and then transfers it into noise using the mean and variance vectors generated by MLP layers. (c) Training pipeline for task I: consists of 3 steps. It firstly updates classifier with analytic learning and then learns noise with trainable modules. Finally, it updates classifier again with analytic learning.
Positive-incentive Noise (Pi-Noise): is the random signal that can simplify the target task which is ubiquitous in diverse fields 42). Following the framework of Pi-Noise, [43] proposes approximate method to learn Pi-noise via variational inference, which validates the efficacy of the idea about generating random noise to enhance the classifier. 44] designs auxiliary Gaussian distribution related to contrastive loss to define task entropy and find that the developed Pi-Noise generator successfully learns augmentations fine-tuning method towards vision-language alignment. In addition, [46] apply the Pi-Noise to graph contrastive learning, thus adding edges adaptively with low time and memory burden.
vision datasets.
[45] adopts Pi-Noise design
3 Preliminary
Class Incremental Learning: learns a unified model from a data stream. Assume that the data steam D = 1 D2.... Dr consists of I tasks, with each task containing several disjoint categories. For task t, D. {xx k denotes the current training set with instances. xf E X, is the input image and yf E y is the corresponding label, where X4 and V/ denote the image set and the label set, respectively. Specially, we denote the label of new task and old tasks as Vnew = 3/ and Vold }/1 U V U -.. Uv-1, respectively. with Vnew n Vol 0. |Yeww = K is denoted as the number new categories, while DVoal M is denoted the number old categories. We follow the exemplar-free free setting [30, 39), where no data from previous tasks is retained in subsequent tasks. Only D, can be accessed in task t. After learning each task, the model will be tested on all seen categories. By leveraging the PTM for initialization, we decompose the model F, = {F,, W; Vinto two distinct components, i.e. the pre-trained backbone F, and classifier W,. Following typical the most parameters of pre-trained backbone are frozen except
PTM-based CII. works R0 39
little part for fine-tuning.
Analytic Learning: develops new branch for class-incremental learning [26, 27]. It iteratively solves the classifier weights through linear regression. For each task, only the current task data, the classifier weights from the previous task, and a fixed-size autocorrelation matrix are required to determine the new task classifier weights. Specifically, the classification problem be defined in the form of linear regression, as shown in Eq.
arg min || - 21 W
(1)
where X' denotes the image set, y the corresponding label set, F the feature extractor and W the classifier weight be solved. The optimal solution Eq. ( can be found in
W (F(x) (2) 27) F(
(2)
3

---

vasi
warplane
schooner
O
OOOO
pug
eagle
Figure 2: Grad-CAM visualization of the baseline method and MIN. The first image of each group is the original image, the second is the visualization of the baseline method, and the third image the visualization result of the MIN. In addition, the first row shows the visualization where both methods achieve correct classification. In the second row, the baseline method is misclassified, while the MIN maintains correct.
According to [26]. Eq. can be rewritten the iterative form shown Eq. for solving the classifier weight,
W
(3)
where - denotes the classifier weight at task & - 1, Xr denotes the image set at task t. 3% the corresponding label set. R₃ the autocorrelation matrix, which is found according to Eq. (
B, (x))
(4)
R2 R-- - F(X,) B,F (X)) Re-1.
This method establishes strong baseline 126]. and the classifier derived from it serves as the foundation our work.
The proposed approach: MIN
Although PTMs exhibit strong generalization capabilities, continuously fine-tuning in downstream tasks leads to rapid performance degradation. The degradation of generalization capability stems from parameter drift, which introduced destructive noise to obscure the recognition pattern of the previous task. Therefore, we aim to learn positive noise to adapt a new task while freezing the original parameters. Continually learning tasks in sequence results in multiple noises from different tasks during inference. To avoid multiple inference through the backbone, we need to integrate these noises from different tasks within each module. In this section, we first introduce Noise Expansion add a trainable noise learner for each task. Then, we adopt Noise Mixture to integrate these noises and embed the mixture of noise to intermediate features. As illustrated in Fig. 2 the visualization shows the impact of the proposed methods for subsequent tasks.
4.1 Noise Expansion
Due to the absence of guidance from the information among tasks, fine-tuning PTM is prone to absorb some irrelevant features to determine decision boundaries. It is equivalent to introducing noise for the current task. Since the label information within the task available, this noise cannot affect the current task. But may constitute the decision boundary previous tasks and thus result in parameter drift to affect the classification pattern of old categories. To this end, we model the parameter drift as kind of noise. Following M. noise is not always harmful. For example, the noise introduced by joint training can play a positive role in classification. Therefore, we aim to learn positive noise to suppress the confusing pattern across tasks through large interference mask irrelevant activation in the features.
Suppose that the pre-trained backbone F= (fi.... Sr.} consists of L blocks. Its forward propaga- tion process can be represented by Eq. 5 and 66.
r1 fi (I), r fi (r:1))
(5)
(6)

---

where 2' denotes the input images and fi the 1 block. As illustrated in Fig. | a). we design Pi-Noise layers between these blocks, thereby modifying the forward inference process of the backbone network as Eq. 7
7 = Pi (r-1))).
(7)
where Pi denotes the t Pi-Noise layer. Due to the absence of inter-task information guidance. employing single Pi-Noise proves inadequate for effectively simplifying all tasks, thereby causing beneficial noise to degenerate into negative noise. To address this limitation, we suggest learning distinct noise for each task, with the parameters in these task-specific generators remaining mutually independent.
To improve efficiency of parameter uti- lization, we first reduce the dimensional- ity of the input features to minimize the scale of each noise generator. Specifi- cally, we employ a down-projection layer Wdown [Rdi to reduce the feature di- mension from d1 to d2. After generating the noise, we use an up-projection layer xdi W E Rdaxdi to remap the dimension of noise back to d1. Notably, both the Nown-projection layer Wdown and the up- projection layer Wap are shared across all tasks, with each of their elements sam- pled from a standard normal distribu- tion. For task f, P. is employed to generate the corresponding beneficial noise, which 10 return the updated model is composed of two MLP layers denoted as of and 6% responsible for producing the mean vector μr, and variance vector o,, respectively. We sample random signal from normal distribution and transform it into beneficial noise as described in Eq. (8) Using this reparameterizztioo technique, we can train Pe via backpropagation.
Algorithm 1: Training pipeline for MIN
Input: Incremental Datasets: (DI,... ;Dr),
Pre-trained backbone:
F= {fi..... ff
Output: Incrementally trained model
for
= 1 to 7 do
Update the classifier W; via Eq.
3
Expand the new noise generator P. Initialize the weight w via Eq. (12) Construct an auxiliary classifier W aux
5
Optimize the w, Pa and W aux via Eq.
Update the W, via Eq. Test the model
end
Et ot + Pr.-
(8)
In summary, the process of deriving beneficial noise from the input feature ri illustrated in Eq.
(
= E of (r: down) + of (row))
For an intermediate feature TL obtain set of noise denoted {E1, ) By combining these noise instances, we generate a noise mixture, which then mapped back to the dj -ddimensiooaa space via the up-projection layer Wup adjust the intermediate feature rl.
A = rI + 4 (fF1, ... ..E)) Wup+
(10)
where sp (-) denotes the noise mixing operation, which is detailed in the next section. For the current task, only the noise generator P: is trained, while all noise generators remain frozen in subsequent tasks. The structure of P; is simple, containing only two MIP layers. Since d2 & d1, the number of parameters trained for each task is approximately equivalent to two dg x d2 square matrices.
4.2
Noise Mixture
Through the Noise Expansion, we obtain a set of noise denoted as {ei...., E. ). If each noise is used separately for inference, it causes the test complexity to grow linearly with tasks like (37] and [39]. The simplest approach is to compute their average. However, substantial variations exist among different tasks. For task t, may benefit from certain similar previous tasks, and thus the beneficial noise corresponding to those tasks should be assigned higher weights. Conversely, some tasks might conflict with the current task, making the noise derived from them detrimental to the current task. Therefore, when learning new task, it is critical to holistically consider the complexity of all tasks, ensuring that the noise mixture achieves an optimal trade-off between old and new tasksduring task adaptation. To this end, we adjust all noise instances using set of learnable weights First,
5

---

Table 1: Average and last performance comparison on CIFAR-100 and CUB-200 datasets with ViT-B/16-IN2IK as the backbone. We report all compared methods with their source code. The best performance is shown in bold, and the second-best result is underlined. All methods are implemented without using exemplars.
CIFAR-100
CUB-200
Methods
10 steps
20 steps
50 stegs
10 steps
steps
50 steps
Ap
Ay
2
85.02aa
79. 10007
81.00
73.03
61.83ga
84.29
81.75 a3.5
20.31
78.5
64.5
DualPrompt
$4.89 49.65
85.57
79.27
73,66
63.97-4441
$4.39
73.45 ...3
53.79 coo
72.48
78.06...4
64,50
CODA-Prampt
21
87.51
50.87
45.66
64.15
74.00
ACIL
56.74.0o 91.21.xa:
91,34
86.78
91.51 cam
86.84...
91.74
87. 22
91.83
87,09 -.
91.73
96,87
SLCA
92.67ax*
93.52
90,76
84.52.xar
56.83
$3.38
74.77
76.00ue
FeCAM
59.05
91.86
$7.04
90.92
85.34.c2
92.73 couc
88,
92.89
68.42
91.16
85.89
RanPAC
93.25
59.55
91.81ax
35.69...1
93.30..ur
59.27
APER
92.22
$7.45
40.57
85.03
88,75
82.37
88.32
85.75 45.75
58.34
65.42
48.42
15.29
92.11aa
57.72
91.51ax
15.00.cua
S4.31ax
24.47
90.12
90.62.00
83.72
91.54am4
96.51
COHMA
89 20.77
92.00ga
58.09
88.177n
79 2664
83.5221
24.77a1
MOS
87.39...1
92.08
92.62.0om
88,5
91.99..12
MIN (Ours)
95.12
32.
33
39.82
24
3x
93.44.0.
initialize the w as Eq.
Pr Jkg
81, l ||
(11)
exp (St,/T))
Wi Σj= exp (8t.j/T)*
(12)
where μr denotes the task prototype saved from the task i, 7 is the temperature coefficient and is set to by default. The noise mixing operation is expressed as Eq.
@ (f ... ,E+)) = Σ E,a..
(13)
Then, the output feature ry, of the backbone is given by Eq. (5) and
Les (Sw sux. y 21 Wt).
(14)
For the training pipeline, we first update the classifier W, according to Eq. (3) and construct an auxiliary classifier Wanx with all elements initialized to zero. Then, we update P, and Wanx using the cross-entropy loss E (-) fit the residual between the output logits 2,, and the ground truth
according to Eq. (14). It worth noting that the classifier W, does not participate in gradient updates during training. After training, update W, once more with the updated P, according Eq. (3)
while W Wanx is discarded. We summarize the training pipeline of MIN in Algorithm
Experiments
In this section, evaluate MIN several benchmark datasets and compare it with other SOTA methods to demonstrate its superiority. In addition, we provide an ablation study and a visualized analysis validate the effectiveness of MIN. More experimental results can be found in the supplementary.
CIARR 5
CFAR staps
CUB d806
CUB 50 sposs
100
4,281
Acturtyy 1541
0,761
0,861
2.12
50
ELGA
FeCAM
RasPAC
EASE
Mis(Surs)
Figure 3: The performance of each learning session under different settings of CIFAR-100 and CUB-200 200. All methods are initialized with ViT-B/16-INIK.

---

Table 2: Average and last performance comparison ImageNet-A A and ImageNet-R datasets with ViI-B/16-IN2IK as the backbone. We report all compared methods with their source code. The best performance is shown in bold, and the second-best result is underlined. All methods are implemented without using exemplars.
ImageNet-AA
ImagoNet-R
Methods
10 steps
20 steps
50 steps
10 steps
20 steps
50 steps
π
π
Ap
π
1.2
$4,90211 1o
54.25g1 as
49 2 us
56 20.600n.
68.75
55. 56.88xx
57.86 19
DualPromph
30.3
30.56gt
3687
43.85pa77
20.95g144
70.29
66.50 90.
66.32gnaa
61.77
55.3 35.3
Prompt
a0
2 u0
38.05-1.3
38.24
26.00.a22
76.76 v0.c
70.45
64.68
35.82..a
45.4
ACII
63,30g
53.26one
62.45ga
$2.26gn
62,48
50.9
75.17
68.40 33
75.4
68.19
67.92 75.37
SLCA
82.26
81
76.63
72.57 a9.35
FeCAM
56.04g
46.41cno
55.41ar 55
45.20 40
45. 16
79.02
71.05
63.69ee11
55.03
24
54.05
62.37
46.61-4aa
82.12.vmm
33.5
77.88.eue
72.22
75.62-ear
09.38
APER
60.26sa32
48.51
....
60.84
48.76..om
61.29
48.58.c2
75:45
67.52
74.93.00
67,30
69.29
01.12.0m
58.18
46.21-22
59.86
47.53
m1.75
26.20
81.18-oa
74.62.mm
76.51eea
68.67 maa
COFIMA
57,70
47.00.0
58,43axm
48.12..m
$2.05 aut
76
51.54.0m
75.15
MOS
07.71ax
05.10ax
21.27
27.
81.32a008
75.07ax09
75
MIN (Ours)
64.32 m0.
72.37
63.66-e.2
66.15
57.08-237
85.18 1.
$3.69.cur
78,08
82.26.c11
-5.72
5.1 Experimental settings
Datasets. We conduct experiments on six benchmark datasets, including CIFARIO0 47
CUB200 (48. ImageNet-A 44). ImageNet-R [50), FOOD101 [51 and Omnibenchmark 52
There are 100 categories in CIFARI00, 101 categories in FOOD101, 200 categories in CUB200, ImageNet-A, ImageNet-R, and 300 categories in Omnibenchmark. Detailed information for the datasets shown in the supplementary.
Protocols. Following recent CIL works, we split a dataset into T tasks, denoted as T steps. Each task learns the same number of categories. To further investigate the performance of all methods across various step sizes, we set the range of T from 5 to 50, ie.. T = 5, 10, 20, and 50, Following [37 34 39], the learning order is determined by the random seed 1993. We run each experiment times and report the average result. Additionally, we also report the results using other random seeds in the supplementary.
Evaluation metric. After learning the task f, we test the model on all known categories. Assuming that A, represents the accuracy of the model after tasks. We use two metrics, the average accuracy across T tasks A + Σt= A, and the last task accuracy Ar, to measure the CIL performance.
Comparison methods. We choose state-of-the-art P'MM-based CIL methods for comparison. To be specific, we choose I.2P (30. DualPrompt [31], CODA-Prome Prommpt (32. SLCA (3), FeCAM [5)
RanPAC (36] APER [34], EASE (37), COFiMA 38 and MOS 39 In addition, we also choose ACIL 26 as the baseline of our work.
Implementation details. Since the wide range of PTMs are publicly accessible, we choose two representative models following [31 37 34 39, denoted as VYT-B/16-IN2IK [53] and ViT-B/16- IN1K. They are both initially pre-traines on ImageNet-21K: while the latter is further finetuned on ImageNet- K. In MIN, we set the batch size to 128 and train for 10 epochs using SGD optimizer with momentum. The learning rate is initially set to 0.001 and decays to following a cosine annealing
Imaeshall-4.5
ImageNiciA
Imsegahee-ft
1,571
2.831
9.21
2.891
120
120
Numbor of classes
Nurbbr of dlasses
Namber classes CDF MA
Number of Dl06600
ELCA
FoCAM
RaiPAC
APER
EASE
Min/Ouns)
Figure 4: The performance of each learning session under different settings of ImageNet-A and ImageNet-R R. All methods are initialized with ViT-B/16-INIK.
7

---

Table 3: Average and last performance comparison on FOOD-101 and Omnibenchmark datasets with ViT-B/16-IN2IK as the backbone. We report all compared methods with their source code. The best performance is shown in bold, and the second-best result is underlined. All methods are implemented without using exemplars.
FOOD-101
Omnibeachmarkk
Methods
10 steps
20 steps
50 steps
10 steps
20 steps
50 steps
A/
A
74.03.s..
70.66 78.74
39.60.axx
s0.26-eax
73.01 voco
63.29
61.60
60.54
59.05
DualPrompt
86.25pa
50 172.0
50.48 ABoran
72.78 22.78m
67.23 67.23
58.80
o7.68
67.45
74.13pa
64 88
71 11.28
50,00 00,08p
PODA-Prompi
76.53
51.57-4ar
79.18 muu
70.01
a4.8
65.78
s0.50..a1
60.50
ACIL
86.03
50 30.66
85.8
90.54..
85.38.cm
$4.26
76.
76
45
s4.28...1
15.31
SLCA
91,10
FoCAM
40.77a..
86.50
90 50.75
50,05 m0.
90.08
85.21
a3.74
77.18
85002
16.19
75.55
RanPAC:
15.002015
M4
APER
85. 33.00
58 58.40gr 1e
67.19g
89.00
83. Miges
80.72p
24. 14
80.26ga 31
73 73.36g
50.27gn
73.0
os
79.30aa0.0
85.0
74.992a
73.Murus
COPMA
85.84 90
58. 12 15
11.82
R1.69g
73 38
80.90 en
MOS
92.54px11
80 27..ce
92.02ae..e
87.08-cxx
86.10
80.12
85.22
78.5
$5.06eaaz
77.49
MIN (Ours)
23
cs
93.20.0.12
p9.3
19.47.x.11
87.36.00
87.79xx 30
79.90x.11
$7.58.uss
decay pattern. The dimension d2 of the latent vector within the Pi-Noise layer is set to 192, with more configurations of this hyperparameter detailed in the supplementary materials.
5.2 Comparison with State-of-the-arts
In this section, we compare the proposed MIN with other state-of-the-al methods on six benchmark datasets and different pre-trained weights. Complete comparison experiments are provided in the supplementary.
CIFAR & CUB. Tab. D prrsennss the performance comparison of different methods using ViT-B/16- IN21K on the CIFAR 100 and CUB200 datasets. We report the results under three different settings for each dataset: 10 steps, 20 steps, and 50 steps. As the number of tasks increases, the incremental learning difficulty progressively intensifies. Due to the lack of inter-task information guidance, all methods exhibit significant performance decline in the 50-steps compared to the 10-steps scenario. The proposed method achieves the best performance across all three settings, with its advantage over the runner-up method becoming notably amplified under the 50-steps setting. Additionally, Fig. presents the incremental performance trends of different methods using ViT-B/16-INIK. For each dataset, we report results under both 5-steps and 50-steps settings to demonstrate the impact of the number of tasks on incremental performance. As shown in Fig. 3 the proposed method exhibits a more gradual decline rate in the 50-steps setting compared to other approaches. When contrasting the 5-step and 50-step results on CIFAR 100, the performance advantage of our method over the runner-up approach increases from 0.76% in the 10-steps setting to 4.28% under 50-steps settings. Similarly, for CUB200, the performance gap expands from 0.86% to 2.12% between these two settings.
ImageNet-A/R. Tab. 2 presents the performance comparison of diffferent methods using ViT-B/16- IN21K on the ImageNet-NFF datasets, with results reported under three different settings. The proposed method demonstrates superior performance across all settings, achieving 6%-9% im- provement in final accuracy over the runner-up approach on ImageNet-A, and a 2%-3% enhancement on ImageNet-R. Furthermore, Fig. 4 illustrates the incremental trends of various methods with
Drnibenshmars
1.12
AccuiDey (5)
1.251
Accumity
1:41
2.38
120
Number
Numbber claass
Namber chensess
Nambbr chooses
SLCA
FeCAM
RasPAC
FASE
COF MA
Min(Cunn)
Figure 5: The performance of each learning session under different settings of FOOD-101 and Omnibenchmark. All methods initialized with VIT-B/16-INIK.
8

---

Table 4: Results of ablation study. NE+ē calculates the average noise of all tasks. NE+E, only uses the μ of each task noise. NE+E, only uses the a of each task noise. NE+E, uses the last task noise and NE+E; uses random task noise.
Methods
CIFAR
CUB
IN-A
IN-R
FOOD
Omni.
π
π
π
π
Baseline
91.18
86.82
91.55
87.08
62.23
53.65
75.14
68. a8
90.26
85.57 09
84.55 u..
76.82
w/ NEEe
94.774a3s
90.22-0.e
92,89...a
90.77
71,43-000
62.59
83886a21
78.65
92.67
.9.34....
86.77-045
a0.12
w/ NE+s,
89.27ax
85.63
48.97
85 45
05.8721.n
16.361
76227au
$8.45
83:16a
82.15.00
23.9
NE+s,
91.32 v4.37
93.15
90
70.45
61.32
84.13
79.05
92,90uc.s
89.25
87.06 wun
50.12
w/ NE+5
92.36an
87.92
92.37Gxxx
63.23
54.15
79.15esss
72.71
90,62
86.12am
85.02
77.47
w/ NE.;
92
10
92.15ga r
88 76
90g
54.88ga2 xt
81.39gr
15gum
91 1100grm as
86.57
7477
MIN
96.12
92.12g0
94,00g1
91.22
72.89
64.32g
85.18ex
79.75
98.
90,0
87.36
80.55
VřT-B/16-INIK, where our approach shows amplified advantages against the suboptimal method as task number increases.
FOODI01 & Omnibenchmark Tab. demonstrates the performance comparison of various methods using ViT-B/I6-IN2IK on the FOODI01 and Omnibenchmark datasets. The proposed method maintains superior performance in all three settings. Fig. illluttraees the incremental trends with ViT-B/I6-INIK where the findings further validate the progressive widening of the performance gaps between our method and the runner-up approach as the number of tasks increases.
5.3 Ablation Study and Visualizations
Ablation study. To verify the effectiveness of each com- ponent in MIN, we use the VIT-B/16-IN2IK to conduct an ablation study on all six datasets with the 10-steps setting. Baseline denotes the approach using analytic classifier with the pre-trained backbone (26) (see Sec. It should noted that in the absence of Noise Mixture, the mixing module op- crates by computing the average of all noises. Additionally, instead of employing auxiliary classifier for training, it di- rectly utilizes We to calculate the cross-entropy loss to update Pt. The results presented in Tab. which validate the effectiveness of each strategy.
85
SLOA
FaCAM
APER
MOS
85
Min
Accuraay
75
80
140
Number
95-
Visualizations. We use Grad-CAM visualization to further demonstrate how MIN applies beneficial noise to improve the performance of baseline methods in Fig. / The comparative results demonstrate that while the baseline method also focuses on the primary subject, its activation region exhibits greater dispersion with pronounced responses in irrelevant regions. For example, in the first set of images labeled vase, the bascline method generates strong activation for the background wall and the hanging painting. But the proposed method pays more attention to the main part of the vase, and suppresses the irrelevant background information. This indicates that the proposed method works on improving the accuracy by suppressing invalid features. We provide more visualizations in the supplementary.
SLOA
FaCAM
80-
APER
MOG
Min.
80
75
70-
80
140
200
Figure 6: Incremental trends with random orders on the ImageNet-R dataset using ViT-B/66-IN2IK (Top) and ViT-B/16-INIK (Bottom).
Random seed experiments. We shuffle the learning order with the random seeds, i.c., { [1993, 1994, 1995, 1996, 1997, 1998). Therefore, we obtain six results of different learning order for each method on ImageNet-R using IN1K. The results are shown in Fig. 6 For each method, the solid line indicates the mean of the five outcomes and the shading indicates the fluctuation range. The results demonstrate that MIN consistently outperforms other methods by a substantial margin.
and ViT-B/16-
Robustness of hyperparameters. There are two hyperparameters in the baseline method, i.e., the coefficient 7 of regularization terms and the buffer size. We investigate the robustness by changing these hyperparameters. Specifically, select y from ( (10, 50, 100, 500, 1000) and the buffer size

---

from [1024, 2048, 4096, 8192, 16384]. We report the average accuracy in Fig. 7a). The results indicate that selecting y as 100 500, along with a buffer size of 8196 16384, yields better performance. In addition, we change the dimension of the hidden layer P. to investigate its impact on the model performance. Specifically, we select from (96, 192, 256, 384, 512, 768) as the dimension that W reduce the input feature into. And we report the incremental trends in Fig. (tb). Then, the incremental trends of the number of trainable parameters are shown Fig. 7 c). As shown in the figure, the model achieves the best performance when the dimension is set to 192 (dim=192)) and further increasing this dimension results in performance degradation. Hence, we adopt dim=192 as the dimension of the hidden layer in Pt. In addition, Tab. Tab.B]shows shows the results of changing T in Eq. (12). It indicates that the proposed method is not sensitive for
1000 182.45 83.78 84.38 84.95 85.
E500
Regulorzationeem temm
Trainable params (M)
500-62..8
60
10
200-62..6
83.68 84.34
84.96 85.17
C3.03
50-62.79 83.8 84.47 84.76 85.02
6368
84.32 84.31 84.33
500
Buffer Size
Number of classes
Hidden dim
Figure 7: Further analysis on parameter robustness.
Additional experiments. We set the hyperparameter d2 to 192, which is the same in the comparative experiments, to analyze the number of model parameters. From Fig. 8 a) and (b), it can be seen that the total number of model parameters of MIN is the same as most of the methods, while the number of learnable parameters for new tasks is less than most of the methods. Fig. ((c) shows the impact of Pi-Noise compared to alternatives including Adapter [54). VPT-shallow (55) and
Table 5: Robustness of in the set- ting of ImageNet-R R 10 steps.
A
Ar
0.50
84.92 u0
79.30
1.00
84.79
20.22
79.38 +0.
1.50
84.91 10.19
79.59
2.00
88.18
70.75
VPT-deep [55). Following ACIL (26). we use these alternative approaches to train the model the first task and then learn subsequent tasks by using analytic classifier. The comparison demonstrates
that Pi-Noise shows superiority over other methods.
11724
LaP
EAGE
Avcroge Parametess D4
Transbe Parameters
25
RariAc:
Bapalin WPT Shalh
Bascl endapter
120
Dapeline Deep
2.0
Dapeiin Houee
1.0
80
60
30
100
120
140
100
150
200
Number of clessess
Number classees
Figure 8: More experiments results. (a) presents the average number of model parameters for different methods across 10 steps. (b) presents the number of trainable parameters for different methods across 10 steps. (c) Impact of Pi-Noise compared to alternatives under the setting of ImagenNet-R 10 steps.
Conclusion
Incremental learning a key step to achieve continual Al system. This paper propose mixture of positive-incentive noise (MIN) for class incremental learning based on pre- pre-trained models. Specifi- cally, the proposed method expands a Pi-Noise generator for each task to adapt new tasks without the degradation of backbone generalization. Considering the cross-task collaboration among different Pi-Noises, we guide the model via an auxiliary classifier learn set of adaptive weights for adjusting their mixing ratios. Extensive experiments validate the effectiveness of MIN.
10

---

References
[1] Xuelong Li. Positive-incentive noise. IEEE Transactions on Neural Networks and Learning Systems. 35(6):8708-877 87114, 2024.
[]] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3366-3385 2022.
[3] Zheda Mai, Ruiwen Li, Jihwan Jeong. David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28-51, 2022.
[4] Mare Masana, Xialei Liu, Bartiomiej Twardowski. Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. caassinncrementl learning: survey and performance evaluation image classification. JEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513 5533, 2022.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1(4):128-135, 1999.
[6] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12))2935 2947, 2017.
[7] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytclaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3366-3375, 2017.
[8] Sylvestre-Alliis Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEF conference on Computer Vision and Puttern Recognition, pages 2001-2010, 2017.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[10] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision- FCCC 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XX 16, pages 86-102. Springer, 2020.
[11] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3023, 2021.
[12] Fu- Yun Wang, Da- Wei Zhou, Liu Liu, Han-Jia Ye, Yatao Bian, De-Chuan Zhan, and Peilin Zhao. Beef: Bi- compatible class-incrementie learning via energy- based expansion and fusion. In The Eleventh International Conference on Learning Representations, 2023.
[13] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07199 2023.
[14] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoeneoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[15] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 374-382, 2019.
[16] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and faimess in class incremental learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13208-13217, 2020.
[17] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Rmm: Reinforced memory management for iass-incrementll learning. Advances in Neural Information Processing Systems, 34:3478-3490, 2021.
[18] Zilin Luo. Yaoyao Liu, Bert Schiele, and Qianru Sun. Class incremetall exemplar compression for class insremmacemtt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11371-11380 2023.

---

[19] Arjun Ashok, KJ Joseph, and Vineeth N Balasubramanian. Class-incremental learning with cross-space clustering and controlled transfer. In European Conference on Computer Vision, pages 105-122. Springer, 2022.
[20] Haitao Wen, Lili Pan, Yu Dai, Heqian Qiu, Lanxiao Wang. Qingbo Wu, and Hongliang Li. Class distillation. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 28443 284-2,88222222 2024.
incremental learning with
[21] Yu- Wang, Da- Rei Zhou, Han-Jia Ye, and De- De-Chuan Zhan. Foster: Feature boosting and compression for class-ineremental learning. In European conference computer vision, pages 0984414. Springer, 2022.
[22] Da-Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. model or 603 exemplars: Towards memory-efficiint class-incremental learning. In The Eleventh International Conference on Learning Representations, 2023.
[23] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthicu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9285-929399 (295, 2022.
[24] Zhiyuan Hu. Yunsheng Li, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos. Dense network expansion for class incremental learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1888-1 1867, 2023.
[25] Kai Jiang, Xueru Bai, and Feng Zhou. Recurrent network expansion for class incremental learning. IEEE Transactions on Neural Networks and Learning Systems, pages 14, 2025.
[26] Huiping Zhuang. Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, and Zhiping Lin. Acil: Analytic class-incremental learning with absolute memorization and privacy protection. Advances in Neural Information Processing Systems, 35:1 1602-11614, 2022.
[27] Huiping Zhuang, Run Hc, Kai Tong, Ziqian Zeng. Cen Chen, and Zhiping Lin. Ds-al: A dual-stream analytic learning for exemplar-free free Artificial Intelligence, volume 38, pages 17237-17244, 2024.
lcarning. In Proceedings of the AAAI Conference on
[28] Huiping Zhuang, Zhenyu Weng, Run He, Zhiping Lin, and Ziqian Zeng. GKEAL: Gaussian kernel embedded analytic leamingg for few-shot class incremental task. In 2023 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7746-7755, June 2023.
[29] Huiping Zhuang, Yizhu Chen, Di Fang, Run He, Kai Tong, Hongxin Wei, Ziqian Zeng, and Cen Chen. GACL: Exemplar-free generalized analytic continual learning. In Advances in Neural Information Process- ing Systems. Curran Associates, Inc., December 2024.
[30] Zifeng Wang. Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual Icarning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139-149, 2022.
[31] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference on computer vision, pages 631-648. Springer, 2022.
[32] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal free continual learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11909-11919, 2023.
[33] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learnr with classifier aligmment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 19148 19158, 2023.
[34] Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, pages 1-21, 2024.
[35] Dipam Goswami, Yuyang Liu, Bartoomiej Twardowski, and Joost Van De Weijer. Fecam: Exploiting the heterogeneity of class distributions in exemplar- free continual learning. Advances in Neural Information Processing Systems, 16:6582-6595,,8 2023.
12

---

[36] Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton Van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 16:1222212 12053, 2023.
[37] Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-nncremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23554-23564, 2024.
[38] Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners. In European Conference on Computer Vision, pages 606-324. Springer, 2024.
[39] Hai-Long Sun, Da-We Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Yc. Mos: Model surgery for pre-trained model-based glass-incrementas learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
[40] Siqi Huang, Yanchen Xu, Hongyuan Zhang, and Xuelong Li. .eam beneficial noise as graph augmentation. In Forty-second International Conference Machine Learning, 2025.
[41] Ziheng Jiao, Hongyuan Zhang, and Xuelong Li. Cnn2gnn: How to bridge enn with gnn. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
[42] Jiquan Shan, Junxiao Wang. Lifeng Zhao, Liang Cai, Hongyuan Zhang, and loannis Liritzis. Anchorformer: Differentiable anchor attention for efficient vision transformer. arXiv preprint arXiv:2505.16400 2025.
[43] Hongyuan Zhang, Sida Huang, and Xuelong Li. Variational positive-incentive noise: How noise benefits models. arXiv preprint vcXiv:2306.07611, 2023.
[44] Hongyuan Zhang, Yanchen Xu, Sida Huang, and Xuelong Li. Data augmentation of contrastive learning is estimating positive-incentive noise. arXiv preprint arXiv:2408. 09929,,09999 2024.
[45] Sida Huang, Hongyuan Zhang, and Xuelong Li. Enhance vision- laiionl alignment with noise. In Proceedings of the AAAI Conference Artificial Intelligence, 2025.
[46] Yanchen Xu, Siqi Huang. Hongyuan Zhang, and Xuelong Li. Why does dropping edges usually outper- form adding edges in graph contrastive learning? In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
[47] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases, 1(4), 2009.
[48] Catherine Wab, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ussd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, 2011.
[49] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
[50] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of oft-o--diitribution generalization. ICCV, 2021.
[51] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 1- 2014, Proceedings, Purt 13, pages 446-461. Springer, 2014.
[52] Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision representation through the lens of visual realms. In European Conference on Computer Vision, pages 594-611. Springer, 2022.
[53] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 /9999, 2020.
16x16 words:
[54] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang. and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:16664-16678 2022.
[55] Menglin Jia, Luming Tang. Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser- Nam Lim. Visual prompt tuning. In European conference on computer vision, pages 709-727. Springer. 2022.
13

---

[56] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances neural information processing systems, 32, 2019.
[57] Da- Wei Zhou, Qi-Wei Wang. Zhi- ThinHong& Qi. Han-Jia Ye, De- Pe-Chaan Zhan, and Ziwei Liu. Class-incremental learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: See Abstraction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See supplementary materials
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
15

---

Justification: see Sec. 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes].
Justification: See pseudo-code and implementation details in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes].
Justification:
We have provided the code link in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwiii/gguie///eeeeeeoeic CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification:
See supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: We have provided the standard deviations for all the repeated experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: See the hardware information in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics btttp:://neurippe http///urrs.cccpuui//i/e httppp/csccppp..t.ttttttttteeeea
Answer: [Yes].
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: See Sec. 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: CC-BY 4.0
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworeeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
19

---

Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA].
20

---

Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas/s//eerrippoccc///o/rreees for what should or should not be described.
21

---

Appendix
A Overview
In this supplementary material, we provide more details about MIN. Firstly, we give the implementa-
tion details in Sec. B Then, provide the details of six datasets in Sec. C In Sec. D we provide hardware information used for all experiments. Then, we provide descriptions of all comparison methods in Sec. MIN. In the end. we provide more comparison experimental results in Sec. G Limitations of MIN is provided in Sec.
Sec. F
we provide more visualizations further illustrate the effectiveness of
B Implementation details
We run all the experiments with Py [56] and reproduce all other comparison methods with Pilot [57]. Following 30 32 36 39,, we conduct experiments with the ViT-B/16-IN21K and
ViT-B/I6-INIK. In MIN, we train the model using the SGD optimizer, with batch size of 128 for 10 epochs. The learning rate decays from 0.001 with cosine annealing to zero. There are subtle differences in these hyper-parameter between differrnnt datasets due to scale of datatsets. We set the dimension of hidden layer as 192 according to Sec. 5.3 The hyperparameters of baseline method, ie., regularization term y and buffer size, set to 100 and 16384 according to Sec. 5.3
C
Datasets
The details of six datasets are illustrated in Tab. 6 FOOD101 and Omnibenchmark.
including CIFARIO0, CUB, ImageNet-A/R,
Table 6: Details of six datasets.
Datasets
Classes
Train
Test
Avg size
CIFAR-100
100
50,000
10,000
32x32
CUB-200
200
9,430
2,358
467x336
ImageNet-A
200
5,960
1,515
443
ImageNet-R
200
24,000
6,000
443x427
Omnibenchmark
300
89,697
5,985
764
FOOD-101
101
75,750
25,250
496x475
D Hardware Information
The hardware information is as follows:
CPU: Intel Xeon(R) Gold 6244 CPU
GPU: 2xNVIDIA GeForce RTX 4090
Mem: DDDDR4 SAMSUNG-32GB
E
Comparison methods
We totally choose the 10 approaches for comparison. brief description of these methods is given below.
L2P. L2P 33 freezes the parameters of pre- pre-trained weights and uses visual prompt tuning to learn the new tasks. It constructs a prompt pool and selects a suitable prompt for each sample with a key-value mapping strategy.
DualPrompt. DualPrompt 31 is the improvement work for I.2P. divides the prompt into two types, i.c. general prompts and expert prompts.
22

---

CODA--rompt. CODA-Promphs [32 replaces prompt reweighting with an attention- based prompt recombination during the prompt selection process. Owning the attention module, it needs more parameters and training time.
updates the backbone slowly and rectifies the classifier with pseudo features which are sampled from the Gaussian distribution modeled by the prototype of old categories.
SL.CA. SL.AA 33
applies the Mahalanobis distance to replace the Euclidean distance for clas- sification without updating the backbone. In addition, it shows that modeling the feature covariance relations is better than previous attempts at sampling features from normal distributions and training a linear classifier.
FeCAM. FeCAM 35
RanPAC. RanPAC [66 introduces a frozen random projection layer between the feature output of the pre-trained model and the classifier, combines with a nonlinear activation function, expands the feature dimension and enhances linear separability, so as to alleviate the forgetting problem without updating the backbone.
APER. APER (34] builds the prototype-based classifier and uses cosine classifier for classification. adapts the new data only in the first task with additional modules. We choose the best version, i.e., APER-ai PPR-aadapeer for comparison.
EASE. EASE (37] is based the idea of network expansion. It adds new branch to each transformer block for each task. Although it trains only one branch in training process, it needs to infer multiple times through the backbone for test.
COFiMA, COFiMA (38 integrates the model parameters of the current task and the previous task by introducing the Fisher information weighting mechanism and balances the plasticity and stability of the model.
MOS. MOS [39] alleviates the parameter drift problem through adapter merging technology, and combines the self-optimization retrieval mechanism without training to dynamically correct the module matching error, which solves the catastrophic forgetting problem in the incremental learning of pre-trained models from two perspectives of parameters and retrieval.
F More Visualizations
We provide more visualizations in Fig. 9 to demonstrate the effectiveness of MIN. For instance, three bird species from the CUB dataset are shown, which require fine-grained recognition for accurate differentiation. The backbone network of the baseline method fails to learn cross-task fine-grained features, only achieving coarse-grained discrimination with limited activation values concentrated on the birds' main body regions. In contrast, MIN learns task-specific Pi-Noise that suppresses common features shared across bird species while generating stronger activation responses to discriminative fine-grained characteristics. This indicates that MIN can effectively inhibit the backbone network's 's activation common features across similar categories in different tasks, thereby facilitating the learning of underlying target features and avoiding confusion of decision boundaries between multiple tasks.
G More Experimental Results with SOTAS
In this section, we show more experimental results of different methods. As demonstrated from Fig. 10 to Fig. 21 we report the incremental performance of different methods with ViT-B/16-IN2IK and VIT-B/I6-INIK K on six datasets. MIN consistently outperforms other methods on different datasets by a substantial margin.
H Limitations
MIN relies on an excellent feature extractor to provide a prior information to guide Pi-Noise generator for adaptation of subsequent downstream tasks. With rapid development of pre-training technol- ogy, the weights of the backbone network trained by self-super learning large-scale unlabeled data are easy to access, which reduces this limitations. In addition, MIN requires an additional downsampling layer in each Pi-Noise layer (although they are training-free) Introduces additional
23

---

rusty blackbird
cubcakes
dark eyed junco
pancakes
prothonotary warbler
baklava
saint bernard
west highland white terrier
Figure 9: More visualizations
network parameters (equivalent to 4% of the number of parameters for ViT-B/16). However, the number of actual training parameters much smaller (equivalent to 0.8M parameters). Therefore, is worth trade-off for the simplicity of implementation and low-cost training.
CIFAR100 5 steps
CIFAR100 10 sleps
CIFAR100 20 sleps
CIFAR100 50 steps
100
103
Accuracy (%)
Accuracy (%)
96
Accuracy (%%)
06-
Accuracy (%)
0.711
34
1,931
1.931
2.431
100
40
100
100
100
FeCAM
RanPAC
Min(Durs)
Figure 10: Incremnetal trends of different approaches using ViT-B/16-IN2IK on the CIFAR100 dataset.
24

---

CIFAR100 steps
CIFAR100 steps
CIFAR100 20 steps
CIFAR100 steps
100
88
Accuracy (%)
Accuracc (%%
Accurasy (2%)
Accuraly (1)
95
04
0.76
94
2.01
3,08
92
281
88
86
88
Number classes
Number classes
Number of classes
Number of classes
SLCA
FeCAM
RanPAC
EASE
COFIMA
MOS
Min(Ours)
Figure 11: Incremnetal trends of different approaches using ViT-B/16-INIK on the CIFARI00 dataset.
CUB 5 steps
CUB 10 steps
CUB 20 steps
CUB 50 steps
100
100
36
Accuracy (%%
82
1.471
Accuracy (%)
92
1.441
Accuracy (%)
96-
1.27
Accuracy (%)
2.361
84
84
84
120
160
200
80
120
180
200
120
160
200
1.20
160
200
Number classes
Number classes
Number of classes
Number classes
RanPAC
Min(Ours)
Figure 12: Incremnetal trends of different approaches using ViT-B/16-IN21K on the CUB dataset.
CUB steps
CUB 10 steps
steps
CUB 50 steps
96
Accuracy (%)
0.861
1.681
Accuraty (3%)
Accuracy (%)
96
Accuralcy
82
02-
2.671
02
2. 2.121
86
200
80
200
160
200
200
Number classes
Number classes
Number of classes
Number of classes
SLCA
FeCAM
RanPAC
APER
EASE
COFIMA
MOS
Min(Ours]
Figure 13: Incremnetal trends of different approaches using ViT-B/16-INIK on the CUB dataset.
ImageNet-A steps
ImageNet-A 10 steps
ImageNat-A 20 steps
ImagaNet-A 50 steps
80
8D
Accuracy (%)
6.06
Accuracy (%6
7.18
Apcuracy (%6)
Accuracy (56
9,411
50
50
120
200
80
120
160
200
120
180
200
40
120
200
Number classes
Number classes
Number of classes
Number classes
SLGA
FeCAM
RanPAC
COFMA
Min[Durs]
Figure 14: Incremnetal trends of different approaches using VIT-B/16-IN2IK on the ImageNet-A dataset.
25

---

ImageNet-A sleps
ImageNet-A 10 steps
ImageNet-A 20 steps
ImageNel-A 50 sleps
80
Acairacy (96
70-2
2.83
Accuracy (N)
8.821
Apouracy (%)
10
Accuracy (96)
80
9.681
70
9111
8
30
50
50
200
80
100
200
120
1601
2001
200
Number of
Number
SLCA
FeCAM
RanPAC
CDFIMA
Min(Ours)
Figure 15: Incremnetal trends of different approaches using ViT-B/16-INIK on the ImageNet-A dataset.
ImageNet-R steps
ImageNer-R 10 steps
ImageNet-R 20 steps
ImageNet-R 50 steps
06-3
Accuracy (96
86
30-
1.55
Accuracy (%6
1.951
Acouracy
50
65
2.46
Accurash
2 85
4.65
80
80
73-
70
Number
Number of
Number Min(Ours)
SLCA
FeCAM
RanPAC
COFIMA
Figure 16: Incremnetal trends of different approaches using ViT-B/16-IN2IK on the ImageNet-R dataset.
ImageNet-R steps
ImageNet-R 10 steps
ImageNet-R 20 steps
ImageNet-R 50 steps
50
90
90-
Accuracy (N)
1.57
Accursccy (%)
Accuracy (%)
Accuress (%)
85
2.27
165
1.471
2.891
80
80
26
70-
70
Number of
Number Min(Ours)
SLCA
FeCAM
RanPAC
COFIMA
Figure 17: Incremnetal trends of different approaches using ViT-B/16-INIK on the ImageNet-R dataset.
FOOD101 steps
FOOD101 10 steps
FOOD101 20 steps
FOOD101 stops
100
Apcuracy (%)
Accuracy (16)
Accuracy (%)
82
1.1
92
0.77
32
1.26
2.39
84
Number classes
Number
Number of classes
Number Min(Ours)
SLCA
FeCAM
RanPAC
APER
EASE
COFIMA
Figure 18: Incremnetal trends of different approaches using ViT-B/16-IN2IK on the FOOD101 dataset.
26

---

FOOD101 sleps
FOOD101 10 steps
FOOD101 20 steps
FOOD101 steps
103
85
Accuracy (%)
1.12
Accuracy (%%
1.51
(96)
02
1.29
Accuracy (%)
1.25
88
Number classes
Number
Number of classes
Number classes
SLCA
FoCAM
RanPAC
APER
EASE
CDFIMA
MOS
Min(Ours)
Figure 19: Incremnetal trends of different approaches using ViT-B/16-IN1K on the FOOD101 dataset.
Omnibenchmark steps
Omnibenchmerk steps
Omnibenchmark steps
Omnibenchmark
36
85
Acsiracy (%)
Aamuracy (%
BD
Sn...
Accuracy (1)
0.53T
90
R5
0.43
86
3.2
1.86
80
80-
78
120
180
240
300
120
160
240
300
70
120
150
240
300
60
120
1:0
240
500
Number of classes
Number of classes
Number of clesses
Number of classes
SLCA
FeCAM
RanPAC
APER
EASE
COFIMA
MOS
Min(Ours]
Figure 20: Incremnetal trends of different approaches using ViT-B/16-IN2IK the Omnibenchmark dataset.
Omnibenchmark sleps
Omniieenchmark 10 steps
Ommibenchmark 20 sleps
Ominibenhhmark 50 steps
36
100
100
85
Accuracy (%)
Accuracy (%)
0o_
Accuracy (%)
1.41
Accuracy
30
2.21
80-
3.1 11
2. 2.38
200
300
00
120
240
300
100
200
300
120
300
RanPAC
Min(Ours]
Figure 21: Incremnetal trends of different approaches using ViT-B/16-INIK on the Omnibenchmark dataset.
27


Reviews:
[{"review_id": "kE1nPQaLaM", "reviewer": "Reviewer_MbJ6", "summary": "This paper introduces a new method called Mixture of Noise (MIN) to address catastrophic forgetting during class-incremental learning. \nThe key idea is to view the problem of parameter drift, where the model forgets old information while learning new things, as a form of destructive noise. \nTo combat this, the proposed MIN method learns a beneficial noise for each new task. \nThe method then uses a Noise Mixture strategy to dynamically combine the beneficial noises from all the tasks the model has learned. \nExperiments on several benchmark datasets show the effectiveness of MIN.", "questions": "Please see the weaknesses above.\nI encourage the authors to address my concerns, and I am with a open mind on raising my score.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is easy to follow.\n2. The experiments provided show overall high performance in the benchmarks.\n3. Using noise to enhance CIL performance is something new to me, and I believe it will introduce new insight to the community.\n\nWeaknesses:\n\n1. The paper uses Analytic Learning as its baseline. However, as described in Fig. 1(c), the PTM is updated in each stage through the proposed noise mechanism. The solution in Eq. (3) is derived from Eq. (1) under the assumption that the feature extractor F(·) is frozen during all training phases. In the paper's setting, where F(·) is modified, does the solution presented in Eq. (3) still hold? This needs further clarification.\n\n2. Another major concern is that the proposed noise expansion and noise mixture mechanisms are functionally similar to previous prompt-based or adapter-based PEFT methods. They all introduce additional parameters into the transformer to adapt to new tasks. For the current version of this paper, I am more inclined to see this as an improved adapter mechanism with limited technical contribution. I suggest the authors add more insightful discussion, using theory or experiments, to further elaborate on the advantages of their approach over previous adapter-based mechanisms, specifically for CIL.\n\n3. I believe the set of learnable weights  $w$ shown in Eq. (12) is a crucial component for mitigating forgetting, yet the paper provides insufficient experimental analysis for it. I suggest the authors include more ablation studies, for instance, by replacing the weighted summation in Eq. (13) with a simple average of the noises. Furthermore, it would be insightful to analyze or visualize the final learned weights $w$ after incremental tasks are completed.", "was_revised": true, "final_justification": "Thanks to the authors' efforts to address my concern. I think most of my major concerns are successfully addressed.\nOverall, I will raise my score to `borderline accept' because some of my concerns are addressed, and I encourage the author to give rigorous proof of Eq. 3.", "rebuttal": "We sincerely thank you for the positive feedback highlighting the **clarity** and ease of following our methodology, the **high performance** demonstrated in our experiments, and the **novel insights** presented in our work. Please find our responses to the comments as follow:\n\n## Reply to W1\n\n> W1: If F(·) is modified, does the solution presented in Eq. (3) still hold?.\n\n**Reply**:  The methods such as finetune and PEFT indeed update the ${\\cal F}$ and learn new representations for a new task, so Eq. (3) does not hold. Therefore, as shown in the row \"Finetune\" of the following table (Tab. 1), even with an analytic classifier, fine-tuning the backbone network on each task still **leads to severe catastrophic forgetting**.\n\n*Tab. 1 Incremental trends of replacing the Pi-Noise with the different PEFT approaches under the setting of ImageNet-R 10 steps.*\n\n| Methods     |   task0   |   task1   |   task2   |   task3   |   task4   |   task5   |   task6   |   task7   |   task8   |   task9   |   Avg.    |\n| :---------- | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n| Finetune    |   93.61   |   78.73   |   75.34   |   68.00   |   61.91   |   56.37   |   48.23   |   47.39   |   43.62   |   40.15   |   61.24   |\n| VPT-Shallow |   93.47   |   84.36   |   80.15   |   71.78   |   68.42   |   65.24   |   63.77   |   58.67   |   60.17   |   58.36   |   70.44   |\n| VPT-Deep    |   94.08   |   80.45   |   75.80   |   70.07   |   65.35   |   60.80   |   57.26   |   54.33   |   48.29   |   42.75   |   64.92   |\n| Adapter     |   94.45   |   78.65   |   70.24   |   65.33   |   57.26   |   45.46   |   40.23   |   35.66   |   35.27   |   30.80   |   55.34   |\n| **MIN**     | **94.12** | **91.42** | **88.05** | **86.00** | **85.34** | **83.81** | **82.15** | **81.55** | **80.25** | **79.65** | **85.23** |\n\nInstead of **learning new representation** like finetune and PEFT methods, the proposed MIN introduces additional noise layer to capture the confusing features **while holding the original representations**. The noise layer can be regarded as **a kind of buffer of these confusing features**, so that the representations of current task maintain more stable. Formally, the noise layer can be viewed as converting $z$ to $z+\\epsilon$.\n\nIt essentially still leverages the representational power of the pretrained model. **This noise layer does not disrupt the representation of the previous task**, so Eq. (3) still holds. \n\n## Reply to W2\n\n>W2: More insightful discussion to further elaborate on the advantages of MIN over previous adapter-based mechanisms, specifically for CIL.\n\n**Reply** :  Current PEFT methods primarily focus on performance in new tasks by learning new representations with minimal parameters. However, the knowledge acquired from new tasks is destructive to previous tasks in CIL. \n\nAs shown in Tab. 1, after fine-tuning with three PEFT schemes separately for each task, all these methods (VPT-Shallow, VPT-Deep and Adapter) **result in catastrophic forgetting**. Furthermore, methods with greater learning capabilities (Adapter) exhibit a more rapid decline in performance. \n\nIn contrast, MIN does not learn new representation, so it can avoid drifting. The main differences and advantages are: \n\n1. **Different Motivation:** MIN aims to adapt to downstream tasks through suppressing irrelevant feature representations by **leveraging the powerful representational ability of PTMs** rather than significantly updating the representations. The core premise is that the visual patterns captured by existing pre-trained models sufficiently encompass most downstream tasks.\n2. **Stability caused by noise layer:** The core of MIN is the noise layer and it can be regarded as **a kind of buffer of these confusing features**, so that the representations of current task maintain more stable. Formally, the noise layer can be viewed as converting $z$ to $z+\\epsilon$.\n\n## Reply to W3.1\n\n>W3.1: More ablation studies such as replacing the weighted summation in Eq. (13) with a simple average of the noises.\n\n**Reply**:  We appreciate your suggestion and have included additional ablation study results in Tab. 2.\n\nIn addition to the existing NE and NM components, we implement other **five** distinct methods as the alternatives for NM (*Noise Mixture*). In Tab. 2, **AvgN** denotes the method with a simple average of noises, which is the same as the row 4 of Tab. 4 in the main text, **OnlyMu** denotes the method with generating the deterministic noise only by vector $\\mu$, **OnlySigmma** denotes the method with generating the zero-mean noise only by vector $\\sigma$, **LastN** denotes the method using only the noise from last noise and **RandomSN** denotes the method with generating noise by selecting a task randomly.\n\nA comparison of experimental results for AvgN, LastN, and RandomSN demonstrates that **integrating noise injection across multiple tasks is superior to single-task noise injection alone**.\n\nIn addition, we also add the ablation study suggested by Reviewer Cck3. Comparison of the experimental results between OnlyMu and OnlySigma demonstrates that **the uncertainty introduced by the vector $\\sigma$ is crucial for operation of MIN**. In contrast, the deterministic adapter constructed solely from the $\\mu$ vector negatively impacts the original baseline.\n\n*Tab. 2 Results of more ablation studies.*\n\n|               |     CIFAR      |                |      CUB       |                |      IN-A      |                |      IN-R      |                |      Food      |                |      Omni      |                |\n| :------------ | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: |\n| Methods       |      Avg.      |      Last      |      Avg.      |      Last      |      Avg.      |      Last      |      Avg.      |      Last      |      Avg.      |      Last      |      Avg.      |      Last      |\n| Baseline      |   91.18±0.03   |   86.82±0.07   |   91.55±0.16   |   87.08±0.17   |   62.23±0.45   |   53.65±0.57   |   75.14±0.05   |   68.45±0.27   |   90.26±0.03   |   85.57±0.10   |   84.55±0.33   |   76.82±0.44   |\n| NE+AvgN       |   94.07±0.04   |   90.22±0.10   |   92.89±0.09   |   90.77±0.15   |   71.43±0.55   |   62.59±0.70   |   83.86±0.24   |   78.65±0.11   |   92.67±0.05   |   89.34±0.09   |   86.77±0.15   |   80.12±0.18   |\n| NE+OnlyMu     |   89.27±0.24   |   85.63±0.33   |   88.97±0.46   |   85.45±0.45   |   65.87±1.27   |   55.36±1.02   |   76.27±0.44   |   67.15±0.34   |   88.45±0.09   |   83.16±0.12   |   82.15±0.68   |   73.95±0.70   |\n| NE+OnlySigmma |   94.37±0.05   |   91.32±0.12   |   93.15±0.17   |   90.47±0.09   |   70.45±0.25   |   61.32±0.33   |   84.15±0.20   |   79.05±0.12   |   92.90±0.05   |   89.25±0.04   |   87.06±0.07   |   80.12±0.05   |\n| NE+LastN      |   92.36±0.18   |   87.92±0.27   |   92.37±0.08   |   87.25±0.12   |   63.23±0.75   |   54.15±0.92   |   79.15±0.16   |   71.71±0.28   |   90.62±0.12   |   86.12±0.05   |   85.02±0.06   |   77.47±0.08   |\n| NE+RandomSN   |   92.65±0.21   |   88.10±0.15   |   92.15±0.23   |   88.76±0.26   |   64.39±1.08   |   54.88±1.35   |   81.39±0.65   |   73.15±0.81   |   91.08±0.30   |   86.57±0.19   |   84.77±0.24   |   77.96±0.16   |\n| **MIN**       | **95.12±0.05** | **92.12±0.16** | **94.00±0.15** | **91.22±0.18** | **72.89±0.45** | **64.32±0.89** | **85.18±0.19** | **79.75±0.07** | **93.36±0.07** | **90.04±0.06** | **87.36±0.04** | **80.55±0.10** |\n\n## Reply to W3.2\n\n>W3.2: It would be insightful to analyze or visualize the final learned weights $w$ after incremental tasks are completed.\n\n**Reply**: Thanks for your valuable advice. Due to the limitation of rebuttal rules, we can not upload any figures. We intend to add the visualizations of the learned weights $w$ across 10 tasks in the format of the heat map to illustrate the dynamics of weight $w$."}, {"review_id": "rZ7v8I0MRO", "reviewer": "Reviewer_Cck3", "summary": "This manuscript focused on the class incremental learning (CIL) with pre-trained models (PTM). Based on the analytic learning methods [1], this work considered leveraging the beneficial noise and proposed Mixture of Noise (MIN). Specifically, a task-specific adapter is based on the task-specific noise learned from each task. Then, a set of dynamic weights was used to adjust the optimal mixture of different task noises. Based on this design, the beneficial noise was embedded into the intermediate features to mask the response of inefficient patterns. Experiments on some commonly used benchmarks were conducted to demonstrate the effectiveness of the proposed method.", "questions": "- 1. I noticed that the reparameterization process was introduced within the adapter $P_t$. Then, I have two questions regarding this process:\n    - 1.1 During the inference process, do we need to sample $\\epsilon$ multiple times for each sample $x$? If not, please explain the reason.\n    - 1.2 If so, how do you determine the network output? Based on my understanding, for each input $x$, we need to sample $\\epsilon$ for multiple times $\\epsilon^{(1)}, \\epsilon^{(2)}, \\cdots, \\epsilon^{(k)}$, we will also get multiple $\\epsilon_{t}$ as  $\\epsilon^{(1)}_{t}, \\epsilon^{(2)}_{t}, \\cdots, \\epsilon^{(k)}_{t}$  for  $k$ samplings. How should we compute the weighting process in Eq.(13) and the final output of the neural network?\n\n- 2. In Line 141, the authors claimed that the positive noise can be learned to \"suppress the confusing pattern across tasks through a large interference to mask irrelevant activation in the features\". I guess the irrelevance and relevance were measured by the cosine similarity between the reparameterized noise center $\\mu$. Based on this consideration, how about directly adopting a single parameter $\\mu$ to make it a deterministic adapter? \n\n- 3. About the motivation behind the auxiliary classifier. What is the objective or consideration for introducing this auxiliary classifier? I didn't get it.\n\n- 4. Some discussions regarding the high-level ideas of the proposed method. When I first read the methodological parts, especially Sections 4.1 and 4.2, I naturally connected them with some existing concepts, including \"Bayesian Neural Network\" and \"uncertainty\" which have been widely investigated in deep learning. This manuscript explained the proposed method under the concept of positive noise and ignored some descriptions about the relevant details, e.g., the multiple sampling regarding $\\epsilon_{1}, \\epsilon_{2}, \\cdots, \\epsilon_{t}$. This made some steps confusing. Could the author discuss the relationship between the proposed method and the studies about \"Bayesian Neural Network\" or \"uncertainty\"?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. This manuscript adopted a special perspective, the beneficial noise, to design the task-specific adapters, which is novel to the reviewer. Although it was still under the framework of parameter-efficient adapters, it brought a new perspective to designing the adapters.\n2. The experimental part is complete. Most recent baseline methods and commonly used datasets have been included. The ablation study is clear.\n3. Source code was provided to help the readers verify the reproducibility.\n\n\n## Weaknesses\n1. Although this manuscript tried to use \"beneficial noise\" to \"mask the irrelevant activation in the features\", the explanation of this design was still not clear for me. Based on my understanding, the MIN module seems like a \"Bayesian Parameter-Efficient Adapter\" since it introduces some uncertainties through noisy sampling within the parameters of the intermediate layer. However, I didn't see any discussion related to the \"Bayesian\" or \"uncertainty\". Also, the processing of the network output (since it seems that we have an extra sampling process for $\\epsilon$) is not clear.\n2. The motivation of the proposed method is still confusing. The connection between the methodology and some claims was not straightforward. The authors introduced some expressions, like \"learn positive noise to suppress the confusing pattern across tasks\" in Line 141, but I didn't find any connections with the design. See the Questions part for more details.\n3. Some methodological steps were not clear to the reader. For example, the motivation for introducing the auxiliary classifier in Eq. (14) is not clear.", "was_revised": true, "final_justification": "During the rebuttal period, my questions have been well answered. I tend to maintain my positive rating.", "rebuttal": "We sincerely thank you for recognizing the **novelty** of our beneficial noise perspective, the **completeness of experiments** with recent baselines and ablation studies, and the provision of source code for **reproducibility**. Please find our responses to the comments as follow:\n\n## Reply to W1 & Q4\n\n>W1 & Q4: Strengthen the distinctions and connection with existing theories, specifically Bayesian networks and uncertainty modeling.\n\n**Reply**: Thank you for your insightful observation regarding connections to Bayesian Neural Networks (BNNs) and uncertainty modeling. We acknowledge that the introduction of noise and its generation might partly resemble some aspects of BNNs. \n\nHowever, it is crucial to clarify that our MIN framework fundamentally differs in its objective and mechanism from traditional BNNs or uncertainty modeling. **MIN adopts noise layer to construct a buffer of confusing feature rather than quantifying prediction uncertainty**. The noise generators learn to mask cross-task interference patterns, analogous to targeting \"confusion uncertainty\" in feature space.\n\nSpecifically, although they are similar with each other (from the aspect of terminology and mechanism), the main differences are:\n\n1. The primary function of this noise in MIN is direct feature manipulation for CIL, not uncertainty estimation.\n2. While we sample noise from a standard normal distribution, this serves primarily as a reparameterization trick to enable backpropagation for training the noise generators. This approach is distinct from Bayesian methods, which aim to model posterior distributions to quantify epistemic and aleatoric uncertainties in parameters or predictions.\n\n## Reply to Q1\n\n>Q1: Does the inference require multiple samplings? Explain the reason.\n\n**Reply**: A Pi-Noise layer is integrated into each layer of the model. During inference, each layer samples $\\epsilon$ **once** and injects it into the features of that layer. Therefore, the model actually **samples multiple $\\epsilon$ during each inference pass, yet only needs to operate on a single $\\epsilon$ per layer**. \n\n### **The forward propagation during training:**\n\nDuring the initial stages of training, models may indeed produce uncertain outcomes due to the randomness of injected noise. This uncertainty is precisely what building a robust classifier requires. \n\nSpecifically, a zero-mean noise sampled from $P_t$ parameterized by $\\sigma$ is injected into input features. After supervised training, representations highly relevant to the current task acquire smaller $\\sigma$ values to mitigate predictive uncertainty, whereas less relevant representations adopt larger $\\sigma$ values.\n\nBuilding upon this, ${\\cal F}$ in Eq. (1) exhibits higher uncertainty toward low-correlation representations. Consequently, when calculating weights via Eq. (3), **it reduces the weighting of representations with low correlations**, thereby decreasing reliance on unrelated features. Therefore, the updated classifier exhibits robustness to low-correlation representations, and **its output remains stable even without multiple sampling**.\n\n### **The forward propagation after training (during inference):**\n\nThe main reasons why we just sample once are:\n\n1. After well training MIN, the output remains stable across multiple independent noise sampling. And additional overload is introduced by multiple sampling but it does not improve the performance.\n2. From the additional results in Tab. 1, the std of performance is rather small, which shows the stability without multiple sampling.\n\n## Reply to W2 & Q2\n\n>W2 & Q2:  How about directly adopting a single parameter $\\mu$ to make it a deterministic adapter? Clarify the connection with the design.\n\n**Reply**: Thanks for your suggestion. We have explored more ablation studies in the following table (Tab. 1).\n\n*Tab. 1 Results of ablation study.*\n\n|               |     CIFAR      |                |      CUB       |                |      IN-A      |                |      IN-R      |                |      Food      |                |      Omni      |                |\n| :------------ | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: | :------------: |\n| Methods       |      Avg.      |      Last      |      Avg.      |      Last      |      Avg.      |      Last      |      Avg.      |      Last      |      Avg.      |      Last      |      Avg.      |      Last      |\n| Baseline      |   91.18±0.03   |   86.82±0.07   |   91.55±0.16   |   87.08±0.17   |   62.23±0.45   |   53.65±0.57   |   75.14±0.05   |   68.45±0.27   |   90.26±0.03   |   85.57±0.10   |   84.55±0.33   |   76.82±0.44   |\n| NE+AvgN       |   94.07±0.04   |   90.22±0.10   |   92.89±0.09   |   90.77±0.15   |   71.43±0.55   |   62.59±0.70   |   83.86±0.24   |   78.65±0.11   |   92.67±0.05   |   89.34±0.09   |   86.77±0.15   |   80.12±0.18   |\n| NE+OnlyMu     |   89.27±0.24   |   85.63±0.33   |   88.97±0.46   |   85.45±0.45   |   65.87±1.27   |   55.36±1.02   |   76.27±0.44   |   67.15±0.34   |   88.45±0.09   |   83.16±0.12   |   82.15±0.68   |   73.95±0.70   |\n| NE+OnlySigmma |   94.37±0.05   |   91.32±0.12   |   93.15±0.17   |   90.47±0.09   |   70.45±0.25   |   61.32±0.33   |   84.15±0.20   |   79.05±0.12   |   92.90±0.05   |   89.25±0.04   |   87.06±0.07   |   80.12±0.05   |\n| NE+LastN      |   92.36±0.18   |   87.92±0.27   |   92.37±0.08   |   87.25±0.12   |   63.23±0.75   |   54.15±0.92   |   79.15±0.16   |   71.71±0.28   |   90.62±0.12   |   86.12±0.05   |   85.02±0.06   |   77.47±0.08   |\n| NE+RandomSN   |   92.65±0.21   |   88.10±0.15   |   92.15±0.23   |   88.76±0.26   |   64.39±1.08   |   54.88±1.35   |   81.39±0.65   |   73.15±0.81   |   91.08±0.30   |   86.57±0.19   |   84.77±0.24   |   77.96±0.16   |\n| **NE+NM**     | **95.12±0.05** | **92.12±0.16** | **94.00±0.15** | **91.22±0.18** | **72.89±0.45** | **64.32±0.89** | **85.18±0.19** | **79.75±0.07** | **93.36±0.07** | **90.04±0.06** | **87.36±0.04** | **80.55±0.10** |\n\n**OnlyMu** denotes the method adopting **a single parameter $\\mu$**. However, the strategy of solely using $\\mu$ demonstrates **adverse effects** in the most of datasets. Furthermore, we explore to **adopt a single parameter $\\sigma$** to generate a series of zero-mean noise. The experimental results (**OnlySigmma**) shows excellent performance. \n\nWe speculate that MIN **mainly works by the uncertainties introduced by the $\\sigma$**. And the noise center $\\mu$ provides a simple bias to increase the difference between tasks.\n\nIn the absence of $\\sigma$, the classifier is influenced by numerous low-correlation representations. Directly adjusting the feature center using $\\mu$ induces alterations in certain low-correlation representations that previous tasks rely on, resulting in performance degradation.\n\nConversely, after simultaneous application of $\\mu$ and $\\sigma$, the classifier becomes insensitive to low-correlation representations. Under these conditions, feature center adjustment with $\\mu$ can enhance inter-task differences. \n\n## Reply to W3 & Q3\n\n>W3 & Q3: Clarify the motivation behind the auxiliary classifier.\n\n**Reply to W3 & Q3**: The analytic classifier employed in our study determines the weights through an iterative linear regression approach, as detailed in Sec. 3 of the main text, which means **this classifier does not employ back-propagation for gradient updates**. \n\nConsequently, **an additional FC layer must be utilized as an auxiliary classifier to train the Pi-Noise layer**, enabling proper gradient back-propagation.\n\nWe intend to **incorporate the explanation for employing auxiliary classifiers within the main text**. Appreciate your valuable suggestion."}, {"review_id": "eayez1UsUL", "reviewer": "Reviewer_xsXt", "summary": "This paper focuses on class incremental learning and proposes a new noise perspective to rethink the parameter drift problems in this task. Based on this perspective, a noise mix method was proposed to integrate the incremental knowledge and previous knowledge. Various experiments show that this perspective and method are effective.", "questions": "Can your method be extended to the incremental learning in a generative model, for instance, to learn some new concepts to generate, but without destroying the previous concept generation?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1) This paper provides a new noise perspective regarding the parameter drift or forgetting problem, very interesting.\n2) Based on this kind of noise perspective, this paper designs some filters or integrating mechanisms to maintain the good noise and remove others.\n3) This perspective and framework are general and can be easily extended to other semi-dense (like detection) or dense prediction tasks.\n4) The results show consistent improvements, and the whole experiment with ablations is sufficient.\n5) The whole writing is very clear and smooth, organized well, and easy to understand.\n\nCons:\nYou should add some visualization or other high-level intuition explanations to make readers accept the noise perspective more easily and naturally.", "was_revised": true, "final_justification": "My main concern about the noise explanation is partly resolved. Due to the policy limitation, the authors can not provide the visualization results in the rebuttal, but promise to supplement the visualization results in a future version. I decided to keep my original score.", "rebuttal": "We sincerely thank you for highlighting the **novelty** of our noise perspective on parameter drift, the **effectiveness** of our designed filters, the framework's **generalizability** to other tasks, the **consistent improvements** and the **sufficient** ablation study, and the **clear writing**. Please find our responses to the comments as follows:\n\n>W1: Add some visualization or other high-level intuition explanations to make readers accept the noise perspective more easily and naturally.\n\n**Reply to W1**: Thanks for your insightful suggestion. Due to the limitation of rebuttal rules, we cannot upload any figures. We intend to add the visualization of intermediate features before and after noise injection to intuitively demonstrate the impact of the beneficial noise on these features. Furthermore, heat-map visualizations of learnable weights $w$ will be added to intuitively illustrate the dynamics of noise learning.\n\n>Q1: Can your method be extended to the incremental learning in a generative model, for instance, to learn some new concepts to generate, but without destroying the previous concept generation？\n\n**Reply to Q1**:  It is interesting and promising to extend the MIN to generative model. We find highly related paper [1]. After reading the paper, we believe that MIN could potentially extend to generative incremental learning as its core mechanism.\n\nBy adding Pi-Noise layers to U-Net/DiT, it avoids destructive parameter drift in learning the new concepts to generate. This aligns with generative models' need to preserve core distributional features.\n\nIts ability to suppress irrelevant activations indicates that it could also help generative models maintain fidelity to prior concepts. By masking inefficient patterns in intermediate features, it may prevent new concepts from overwriting existing latent representations. \n\n[1] Continual learning of diffusion models with generative distillation, 3rd Conference on Lifelong Learning Agents (CoLLAs), 2024."}, {"review_id": "qD84zuaaAx", "reviewer": "Reviewer_wYxB", "summary": "The paper presents Mixture of Noise (MIN), a new approach to exemplar-free class-incremental learning that reframes catastrophic forgetting as \"parameter drift\" (essentially noise that corrupts previous task knowledge). Guided by this perspective, MIN explicitly learns beneficial noise for each new task and injects it into intermediate features to mask spurious patterns. An auxiliary classifier blends these different noise patterns using adaptive weights, allowing the system to handle multiple tasks seamlessly. Tested across six benchmarks including CIFAR-100, CUB-200, ImageNet variants, and Food-101, MIN achieves state-of-the-art performance.", "questions": "The authors are asked to answer all the questions in Weaknesses, including (but not limited only to): \n\n1. Can the authors provide the formal definition of \"non-essential features\", followed by theoretical justification (or at least empirical evidence) to clarify how the non-essential features are assimilated into decision boundaries?\n\n2. Can the authors provide the formal definition of \"task-specific redundancy\" in the context of pre-trained models, followed by theoretical justification (or at least empirical evidence) to clarify how they distinguish between features that are irrelevant for future (new) tasks versus features that are irrelevant for the current task?\n\n3. The noise-mixing weights depend on the temperature $\\tau$ (Eq. 12) and the choice of prototype dimension. How sensitive are results to $\\tau$ or to the dimensionality $d_2$ used in the noise generator? Have you tuned these, and is performance stable across reasonable values? Providing an ablation (or variance over seeds) would clarify robustness.\n\n4. Did you ensure all baselines use identical backbones, optimization hyperparameters, and exemplar-free settings? For example, in original paper MOS for CUB dataset and 10 tasks the average accuracy reported 93.40 but, in your work, it is 92.08.\n\n5. You use a weighted sum of past noises (Eq. 13). Did you compare this to simpler strategies, such as averaging or using only the current task’s noise? The ablation shows both steps, but it might help to see intermediate variants (e.g. mixing without learning weights vs. with random weights) to fully justify the learned mixture", "limitations": "The authors do not explicitly discuss limitations. But, MIN has so far only been validated on image classification benchmarks with ViT backbones; its effectiveness in other domains or other tasks is unknown. Moreover, any assumptions behind “beneficial noise” (e.g. tasks share common distractors) should be clarified.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. Recasting forgetting as noise and injecting learned “positive-incentive” noise is a new approach. The paper outlines two core strategies: Noise Expansion (one noise generator per task) and Noise Mixture (learning weights to combine them). Figures and equations provide a coherent view of the training pipeline (e.g., Fig.1 shows the Pi-Noise layer and three-step training).\n2. The experiments are reasonably comprehensive. MIN is tested on six datasets and multiple incremental settings (10/20/50 steps). Across these, MIN outperforms baselines like L2P, DualPrompt, CODA-Prompt, ACIL, FeCAM, RanPAC, etc.\n3. The authors emphasize that only a small, fixed number of parameters are learned per task. Figures 6(a)-(b) (described in Sec.5.3) show that MIN’s total parameters are comparable to other methods, while new-task parameters are lower than most. \n\n**Weaknesses:**\n\n1. The authors make many unclear statements in the introduction regarding the challenges of using pre-trained models (PTMs) in incremental learning, without providing support or evidence. Some notable ones:\n\n1.1 “Pre-trained models inherently exhibit task-specific redundancy”: It is unclear what is meant by \"task-specific redundancy\". Does this mean that pre-trained models carry features irrelevant to future tasks or that some learned features are irrelevant even for the current task?\n\n1.2 “Models inadvertently assimilate non-essential features into the decision boundaries”: The term \"non-essential features\" is ambiguous and needs further clarification. Also, the mechanism by which non-essential features are \"assimilated\" into decision boundaries is vague and lacks supporting evidence. \n\n1.3 “These features induce catastrophic forgetting through two mechanisms”: The two mechanisms are not clearly distinguished. While catastrophic forgetting often refers to the forgetting the previously learned tasks, it appears that the authors conflate catastrophic forgetting with inter-task confusion, treating both as the same: The performance reduction due to task confusion in future tasks should not be categorized as forgetting. Additionally, forgetting is generally attributed to parameter interference or representational overlap, rather than merely feature redundancy.\n\n1.4 “This issue stems from the parameter sensitivity of PTMs”: The term \"parameter sensitivity\" is not defined. Does it refer to high sensitivity to perturbations, fine-tuning instability, or something else?\n\n1.5 “It is advisable to mitigate individual tasks’ dependence on redundant features”: It’s not clear how one would determine or mitigate “dependence on redundant features”. This claim is too general to be feasible.\n\n1.6 “Catastrophic forgetting stems from parameter drift for old tasks, it can be conceptualized as the intrusion of noise during learning a new task, which undermining the original pattern of the old framework.” The reviewer is not convinced about this statement. Where is evidence?\n\n1.7 “As illustrated in Fig. 2, the visualization shows the impact of the proposed methods for subsequent tasks”: Figure 2 only presents heatmaps for individual images from the baseline and MIN, which does not adequately demonstrate \"the impact of the proposed methods on subsequent tasks.\"\n\n\n2. Methodological complexity and clarity issues. The training procedure involves multiple components (analytic classifier updates, auxiliary classifier, noise generators), which are hard to follow. For example, the role of the auxiliary classifier $W_{aux}$ in Eq.(14) and the sequence of updates (train $W_t$, then $P_t$, then $W_t$ again) are dense and confusing. Clear details of how the noise weights $\\omega_i$ are learned (the prototype similarity in Eq. (11)-(12) is not fully justified) would be necessary. Overall, some parts of Section 4.2 and the training pipeline need to be more clearly presented. \n\n3. Experimental clarity. Some details are either missing or need further clarification. It is not specified whether the results were averaged over random seeds for different task orderings, nor are standard deviations reported. In many cases, the performance improvement is minor, and without statistics such as standard deviation, it is not justified to label it as an improvement. In class incremental scenarios, the order of tasks will affect the performance which should be considered in experiments. Additionally, it is unclear whether all baseline methods were re-implemented under identical conditions (e.g., the same backbone, training schedule, and data preprocessing).\n\n4. Ablation studies: The ablation study is not convincing in demonstrating the effectiveness of the proposed method. From Table 4, the improvement with and without Noise Expansion is minimal (less than 1% in some cases). Without standard deviation and using different task orderings, the results are not valid. No ablation study provided for hyperparameters. \n\n**Rationales Behind Reviewer's Ratings:**\n\n**Quality (3: Good):** The proposed method appears to be technically sound, and the experiments are comprehensive. However, there is limited theoretical analysis, and some methodological aspects (e.g. the prototype-based mixing) are poorly justified.\n\n**Clarity (2: Fair):** The paper is not well-written. Many sections (notably Introduction, Section 4.2 and the algorithmic details) are very unclear and dense. \n\n**Significance (2: Fair):** The performance improvement over prior methods is incremental (few percent gain on benchmarks). While the idea of beneficial noise is interesting, it is not significantly novel. \n\n**Originality (3: Good):** The concept of injecting learned noise has appeared in other contexts (c.f. Pi-Noise literature), and using an auxiliary classifier for incremental learning has precedents. The specific combination (Noise Expansion + Mixture) is a novel twist, but it is not a fundamentally new paradigm.", "was_revised": true, "final_justification": "I appreciate the authors' rebuttal to my initial comments and responses to my follow up questions. I am still not convinced that the novelty of this paper is significant enough. Nevertheless, as majority of my concerns have been addressed, I have revised my score.", "rebuttal": "We sincerely thank you for the positive feedback on our **novel** recasting of forgetting, the **comprehensive experiments**, and the **parameter efficiency**. Please find our responses to the comments as follows:\n\n## Reply to W1 & Q1 & Q2\n\n>**W1 & Q1 & Q2: Unclear statements in the introduction**:\n\n**Reply**: Thanks for pointing out some unclear expressions in Introduction. Due to limitations of rebuttal rules, we have to exclude the detail response of this part. We promise that we will polish the inexact statements in the revision. We can also reply these questions during the discussion phase.\n\n## Reply to W2\n\n>**W2: Methodological complexity and clarity issues.**\n\n**Reply:** Thanks  for your suggestions about improving the clarity of Sec. 4.2. We provide the pseudo code of training pipeline in **Sec. 5 of supplementary**.\n\nAccording to your suggestion, we intend to **revise the Sec. 4.2 to extend more details about the optimization process**, such as visualizations of learned weight $\\omega$ across the incremental phase.\n\n## Reply to W3 & Q4\n\n>W3 & Q4:  Experimental details\n\n**Reply**: \n\n1. (Different task orderings) In Sec. 5.1, line 205-206, we state that the learning order is determined by the **random seed 1993** and all methods are **run 3 times** to report the average results. **This setting is the same as most of previous works (MOS, EASE, APER, and RanPAC).** \n\n2. (STD) We run baselines 3 times and report the average results. The results with stds are shown in Tab. 1, 2 and 3. We did not report the stds in the main paper **limited by the table width**. And most previous works also did not report the stds (MOS, EASE, ...).\n\n*Tab. 1*\n\n||CIFAR||||||CUB||||||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||T=10||T=20||T=50||T=10||T=20||T=50||\n|Methods|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|\n|L2P|±0.78|±0.57|±0.54|±0.33|±0.41|±0.26|±0.32|±0.36|±0.18|±0.25|±0.47|±0.32|\n|DualPrompt|±0.55|±0.79|±0.62|±0.85|±0.35|±0.41|±0.54|±0.75|±0.65|±1.05|±0.45|±0.82|\n|CODAPrompt|±0.32|±0.87|±0.70|±0.25|±0.58|±0.48|±0.87|±1.05|±0.37|±0.55|±0.25|±0.32|\n|ACIL|±0.03|±0.05|±0.03|±0.02|±0.07|±0.15|±0.21|±0.16|±0.18|±0.17|±0.33|±0.19|\n|SLCA|±0.89|±0.44|±0.76|±0.55|±0.30|±0.57|±0.44|±0.57|±0.34|±0.28|±0.47|±0.62|\n|FeCAM|±0.15|±0.17|±0.11|±0.26|±0.17|±0.39|±0.27|±0.33|±0.25|±0.20|±0.34|±0.43|\n|RanPAC|±0.07|±0.11|±0.15|±0.12|±0.10|±0.08|±0.14|±0.15|±0.17|±0.20|-|-|\n|APER|±0.05|±0.14|±0.09|±0.11|±0.09|±0.08|±0.15|±0.10|±0.08|±0.05|±0.13|±0.15|\n|EASE|±0.27|±0.88|±0.33|±0.52|±0.25|±0.68|±0.87|±1.02|±0.65|±0.62|±0.44|±0.32|\n|COFiMA|±0.16|±0.08|±0.27|±0.11|–|–|±0.35|±0.88|±1.22|±1.05|-|-|\n|MOS|±0.12|±0.21|±0.06|±0.25|±0.04|±0.18|±0.30|±0.46|±0.35|±0.18|±0.12|±0.20|\n|**MIN**|**±0.05**|**±0.16**|**±0.04**|**±0.29**|**±0.20**|**±0.36**|**±0.15**|**±0.18**|**±0.12**|**±0.14**|**±0.14**|**±0.22**|\n\n*Tab. 2*\n\n||INA||||||INR||||||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||T=10||T=20||T=50||T=10||T=20||T=50||\n|Methods|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|\n|L2P|±1.22|±1.06|±1.35|±2.08|±0.88|±0.62|±0.36|±0.56|±0.36|±0.45|±0.22|±0.18|\n|DualPrompt|±0.85|±0.65|±1.02|±0.84|±0.77|±1.06|±0.25|±0.65|±0.33|±0.40|±0.28|±0.22|\n|CODAPrompt|±1.35|±1.27|±2.08|±1.55|±0.66|±0.82|±0.59|±0.47|±0.56|±0.51|±0.36|±0.12|\n|ACIL|±0.42|±0.66|±0.57|±0.55|±0.36|±0.21|±0.05|±0.22|±0.08|±0.16|±0.09|±0.16|\n|SLCA|-|-|-|-|-|-|±0.26|±0.34|±0.22|±0.28|±0.31|±0.18|\n|FeCAM|±1.05|±0.65|±1.15|±0.94|±0.85|±0.65|±0.22|±0.18|±0.15|±0.26|±0.17|±0.11|\n|RanPAC|±1.35|±0.45|±1.07|±0.65|-|-|±0.31|±0.16|±0.09|±0.15|±0.27|±0.16|\n|APER|±0.72|±0.66|±1.05|±0.36|±0.82|±0.26|±0.11|±0.04|±0.15|±0.02|±0.10|±0.04|\n|EASE|±1.15|±1.34|±0.98|±1.33|±0.78|±0.92|±0.32|±0.19|±0.20|±0.09|±0.31|±0.24|\n|COFiMA|±0.86|±0.78|±0.75|±1.04|-|-|±0.15|±0.12|±0.16|±0.14|-|-|\n|MOS|±0.62|±0.92|±0.44|±0.32|±0.65|±0.45|±0.34|±0.20|±0.22|±0.12|±0.15|±0.08|\n|**MIN**|**±0.45**|**±0.89**|**±0.60**|**±0.26**|**±0.35**|**±0.27**|**±0.19**|**±0.07**|**±0.17**|**±0.15**|**±0.16**|**±0.19**|\n\n*Tab. 3*\n\n||Food||||||Omni.||||||\n|-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n||T=10||T=20||T=50||T=10||T=20||T=50||\n|Methods|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|\n|L2P|±0.65|±0.85|±0.54|±0.52|±0.38|±0.26|±0.33|±0.26|±0.34|±0.16|±0.47|±0.22|\n|DualPrompt|±0.45|±0.40|±0.37|±0.26|±0.58|±0.67|±0.18|±0.45|±0.35|±0.27|±0.56|±0.38|\n|CODAPrompt|±0.33|±0.15|±0.30|±0.26|±1.77|±0.57|±0.42|±0.65|±0.18|±0.22|±0.14|±0.30|\n|ACIL|±0.03|±0.07|±0.08|±0.07|±0.04|±0.05|±0.35|±0.52|±0.26|±0.47|±0.18|±0.44|\n|SLCA|±0.37|±0.17|±0.27|±0.08|±0.16|±0.12|±0.26|±0.19|±0.44|±0.75|±0.21|±0.15|\n|FeCAM|±0.50|±0.42|±0.32|±0.31|±0.24|±0.11|±0.16|±0.22|±0.18|±0.16|±0.26|±0.31|\n|RanPAC|±0.32|±0.15|±0.20|±0.15|±0.16|±0.09|±0.12|±0.18|±0.08|±0.05|±0.10|±0.05|\n|APER|±0.19|±0.05|±0.18|±0.05|±0.08|±0.04|±0.20|±0.14|±0.21|±0.16|±0.08|±0.10|\n|EASE|±0.25|±0.22|±0.28|±0.35|±0.30|±0.33|±0.45|±0.40|±0.37|±0.25|±0.48|±0.37|\n|COFiMA|±0.06|±0.08|±0.15|±0.16|-|-|±0.40|±0.28|±0.42|±0.25|-|-|\n|MOS|±0.11|±0.08|±0.18|±0.22|±0.10|±0.08|±0.11|±0.15|±0.08|±0.12|±0.12|±0.15|\n|**MIN**|**±0.07**|**±0.06**|**±0.12**|**±0.16**|**±0.07**|**±0.12**|**±0.04**|**±0.10**|**±0.03**|**±0.11**|**±0.15**|**±0.19**|\n\n3. (Different learning orders) We perform the experiments of **different random seeds** {1993, 1994, 1995, 1996, 1997, 1998} in ImageNet-R 10 steps setting, and report the results in **Sec. 2 of the supplementary**. The results are shown in Tab. 4.\n\n*Tab. 4*\n\n|Methods|ViT-B/16-IN21K||ViT-B/16-IN1K||\n|:-|:-:|:-:|:-:|:-:|\n||Avg.|Last|Avg.|Last|\n|SLCA|81.69±1.50|76.09±0.52|81.97±0.98|76.62±1.12|\n|FeCAM|78.23±0.54|72.11±1.06|78.43±1.06|72.49±0.74|\n|RanPAC|81.29±0.73|76.44±0.93|81.75±0.92|76.99±0.67|\n|APER|74.98±0.70|67.56±1.06|73.41±1.00|66.71±0.61|\n|EASE|81.49±0.35|75.79±0.65|81.84±0.59|76.28±0.31|\n|COFiMA|81.16±1.71|75.46±0.86|82.04±0.97|76.54±1.61|\n|MOS|82.16±0.27|77.55±0.36|80.66±0.59|74.21±0.16|\n|**MIN**|**85.00±0.28**|**79.52±0.35**|**85.66±0.59**|**80.27±0.52**|\n\n4. (Conditions) We ensure all baselines are compared under fair conditions. First, all baselines adopt **the same backbones** (ViT-B/16-IN21K from timm library and ViT-B/16-IN1K downloading from Huggingface) and **the exemplar-free setting**. Second, they are trained with **the same learning rate, weight decay, optimizer and scheduler**. Third, their own hyper-parameters are kept default. Therefore, their performance may have **slight differences** from the original paper.\n\n5. (MOS for CUB) MOS provides no results under 10 tasks setting. It just reports the results of CUB B0-Inc10, equivalent to the 20 steps setting in our paper since CUB contains 200 categories. Their reported result (93.49) is closed to the one we reported (92.62). In addition, most of reported results are close with the original papers, and **in some datasets (Omni.) even higher than the original papers**.\n\n## Reply to W4 & Q3 & Q5\n\n>**W4 & Q3 & Q5: More ablation studies and sensitivity**:\n\n**Reply**: \n\n1. (STD) **Limited by the table width**, we did not report the standard deviations in the main text. The results of the ablation study with  the standard deviations are reported in the following table (Tab. 5). Compare row 3, 4 and 9, the results show that the contributions of each component are stable and effective.\n\n*Tab. 5*\n\n||CIFAR||CUB||IN-A||IN-R||Food||Omni||\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Methods|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|\n|Baseline|91.18±0.03|86.82±0.07|91.55±0.16|87.08±0.17|62.23±0.45|53.65±0.57|75.14±0.05|68.45±0.27|90.26±0.03|85.57±0.10|84.55±0.33|76.82±0.44|\n|NE+AvgN|94.07±0.04|90.22±0.10|92.89±0.09|90.77±0.15|71.43±0.55|62.59±0.70|83.86±0.24|78.65±0.11|92.67±0.05|89.34±0.09|86.77±0.15|80.12±0.18|\n|NE+OnlyMu|89.27±0.24|85.63±0.33|88.97±0.46|85.45±0.45|65.87±1.27|55.36±1.02|76.27±0.44|67.15±0.34|88.45±0.09|83.16±0.12|82.15±0.68|73.95±0.70|\n|NE+OnlySigmma|94.37±0.05|91.32±0.12|93.15±0.17|90.47±0.09|70.45±0.25|61.32±0.33|84.15±0.20|79.05±0.12|92.90±0.05|89.25±0.04|87.06±0.07|80.12±0.05|\n|NE+LastN|92.36±0.18|87.92±0.27|92.37±0.08|87.25±0.12|63.23±0.75|54.15±0.92|79.15±0.16|71.71±0.28|90.62±0.12|86.12±0.05|85.02±0.06|77.47±0.08|\n|NE+RandomSN|92.65±0.21|88.10±0.15|92.15±0.23|88.76±0.26|64.39±1.08|54.88±1.35|81.39±0.65|73.15±0.81|91.08±0.30|86.57±0.19|84.77±0.24|77.96±0.16|\n|**MIN**|**95.12±0.05**|**92.12±0.16**|**94.00±0.15**|**91.22±0.18**|**72.89±0.45**|**64.32±0.89**|**85.18±0.19**|**79.75±0.07**|**93.36±0.07**|**90.04±0.06**|**87.36±0.04**|**80.55±0.10**|\n\n2. (Sensitivity of $d_2$) We have performed the experiment in sensitivity of the dimensionality $d_2$, and provide the results in supplementary materials (**Sec. 4 in the supplementary**). The results show that it achieves the best trade off between performance and the number of parameters when setting the ${d_2}=192$.\n3. (Sensitivity of $\\tau$) We keep the temperature ${\\tau}=2$ in all experiments in the main text. According to your suggestion, we perform the sensitivity experiment on the temperature $\\tau$. The results are shown in the following table (Tab. 6). The results indicate that the model performance is not sensitive to the hyper-parameter $\\tau$.\n\n*Tab. 6*\n|$\\tau$=0.50||0.75||1.50||2.00||\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Avg.|Last|Avg.|Last|Avg.|Last|Avg.|Last|\n|84.92±0.21|79.30±0.23|84.79±0.22|79.38±0.10|84.91±0.19|79.59±0.12|85.18±0.19|79.75±0.07|\n\n4. (More detailed ablation studies) We have explored more ablation studies with the *Noise Mixture* in Tab. 5. \n\n   In Tab. 2, **AvgN** denotes the method with a simple average of noises, which is the same as the row 4 of Tab. 4 in the main text, **LastN** denotes the method using only the noise from last noise and **RandomSN** denotes the method with generating noise by selecting a task randomly.\n\n   A comparison of experimental results for **AvgN**, **LastN**, and **RandomSN** demonstrates that **integrating noise injection across multiple tasks is superior to single-task noise injection alone**."}]

Author rebuttal:

Meta-review: This paper proposes a simple yet effective method for class-incremental learning (CIL) with pre-trained models by introducing a mixture of noise strategies during finetuning. Specifically, it injects different types of noise (Gaussian, dropout, etc.) into various components of the model (features, outputs, logits) to improve robustness and mitigate overfitting to new tasks. The authors show strong empirical results across standard benchmarks and model backbones, demonstrating both improved accuracy and reduced forgetting. The method is lightweight, compatible with existing models, and easy to implement.

Reviewers generally appreciated the empirical gains and simplicity of the approach. However, several concerns were raised about novelty (i.e., noise injection is not new) and the lack of deeper theoretical analysis or broader contextualization within the literature. While the authors provided useful clarifications and additional analysis in the rebuttal (e.g., on ablations and comparison to related methods), some concerns about motivation and positioning remain. Overall, due to its empirical effectiveness, broad applicability, and methodological clarity, I recommend acceptance, though not at the level of spotlight.
Decision: Accept (poster)