Hybrid Re-matching for Continual Learning with Parameter-Efficient Tuning
Authors: Weicheng Wang
Guoli Jia
Xialei Liu
Liang Lin
Jufeng Yang
Venue: NeurIPS 2025 poster ()
Keywords: Continual Learning
Parameter-Efficient Tuning
Primary area: general_machine_learning

Abstract: Continual learning seeks to enable a model to assimilate knowledge from non-stationary data streams without catastrophic forgetting. Recently, methods based on Parameter-Efficient Tuning (PET) have achieved superior performance without even storing any historical exemplars, which train much fewer specific parameters for each task upon a frozen pre-trained model, and tailored parameters are retrieved to guide predictions during inference. However, reliance solely on pre-trained features for parameter matching exacerbates the inconsistency between the training and inference phases, thereby constraining the overall performance. To address this issue, we propose HRM-PET, which makes full use of the richer downstream knowledge inherently contained in the trained parameters. Specifically, we introduce a hybrid re-matching mechanism, which benefits from the initial predicted distribution to facilitate the parameter selections. The direct re-matching addresses misclassified samples identified with correct task identity in prediction, despite incorrect initial matching. Moreover, the confidence-based re-matching is specifically designed to handle other more challenging mismatched samples that cannot be calibrated by the former. Besides, to acquire task-invariant knowledge for better matching, we integrate a cross-task instance relationship distillation module into the PET-based method. Extensive experiments conducted on four datasets under five pre-trained settings demonstrate that HRM-PET performs favorably against the state-of-the-art methods. The code is available in the https://github.com/wei-cheng777/HRM-PET.

Hybrid Re-matching for Continual Learning with Parameter-efficien Tuning
Weicheng Wang', Guoli Jia Xialei Liu', Liang Lin', Jufeng Yang" 12,55 1.,g
I VCIP & TMCC & DISSec, College of Computer Science, Nankai University, Tianjin, China.
Pengeheng Laboratory, Shenzhen, China.
3 Department of Electronic Engineering, Tsinghua University. Beijing, China.
School Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China.
Nankai International Advanced Research Institute (SHENZHEN Shenzhen, China.
21202306390mail nankai edu. cn, exped12300gmail com Stalei@nankal edu. en, 1inl ianiggieee..... org. yangjuf wnngguunnageaaai...mm edu.
Abstract
Continual learning seeks enable model assimilate knowledge from stationary data streams without catastrophic forgetting. Recently, methods based Parameter-Efficient Tuning (PET) have achieved superior performance without even storing any historical exemplars, which train much fewer specific parame- ters for each task upon a frozen pre-trained model, and tailored parameters are retrieved to guide predictions during inference. However, reliance solely on pre- trained features for parameter matching exacerbates the inconsistency between the training and inference phases, thereby constraining the overall performance. address this issue, we propose HRM-PET, which makes full use of the richer downstream knowledge inherently contained in the trained parameters. Specif- ically, we introduce a hybrid re-matching mechanism, which benefits from the initial predicted distribution to facilitate the parameter selections. The direct matching addresses misclassified samples identified with correct task identity prediction, despite incorrect initial matching. Moreover, the confidence-baad based re-matching is specifically designed to handle other more challenging mismatched samples that cannot be calibrated by the former. Besides, to acquire knowledge for better matching, we integrate a cross-task instance relationship distillation module into the PET-based method. Extensive experiments conducted on four datasets under five pre-trained settings demonstrate that HRM-PET per- forms favorably against the state-of-the-art methods. The code is available in the cww/weiiiiicccooooooc.oomocee..e cheng7777 777/HRM-PHH
Introduction
Faced with the non-stationary data flow in the real world, continual learning (CL) 83 56 65
aims
overcome catastrophic forgetting [76 97 [[]] while acquiring knowledge from a sequence of
tasks [57 49 (69]. CL is crucial for achieving artificial general intelligence (39 82
84] garnering
significant attention and advancements recent years [38 107
52
92
70
this paper, we focus
on the challenging class incremental learning (CIL.) scenario 90
102
which stores portion of historical examples in a rehearsal buffer to alleviate catastrophic forgetting (5) (4, (50). Consequently, the information of old classes accessible when training the new classes. However, this
Rehearsal babeed method is one of the most representative approach
B2
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

100
Task!
HiiDe-Promps
Direct
distribution
Average final accuracy (%)
-urs
Instruct
100% matching
Parameter
Pretrained
90
Split ImageNet-R
Split CIF CIFAR-100
Pool
Model
Task1
Task10
Upper-bound
Confidence-based
Rematching
Eacorrec maiching
Task 10
matching
11.00
10
20
30
Matching error rate (%)
(a)
Figure 1: (a) shows comparisons of matching and average final classification accuracy of our method with PrDeppProptt on CIFAR-100 and ImageNet-R across ten tasks with Sup-21K. (b) presents that during HRM-PET inference, the predicted distribution after the PET is leveraged for re-matching.
strategy raises privacy concerns 771 72), and the performance is sensitive to storage capacity
88
43
Hence, rehearsal sehearsal-frree continual learning [89
23
3 47] receives increasing attention.
Among maintains the powerful feature extraction ability of pre-trained models, thereby shows impressive performance. PET updates few parameters when adapting to downstream tasks, and frozes the pre- trairri model to alleviate the catastrophic forget- ting 158 20 61.. Most PET-based methods 88, 80) maintain a parameter pool, then train isolated parameters for each task. Particularly, the corresponding group of parameters in the pool distin- guished by the task identity [77). During inference, the task identity of samples is unknown. Current methods address this issue by matching appropriate task identity for testing [88 80). DualPrompt (88) employs the QKV mechanism (78) to match task task-specific prompts. LGCL [36] introduces language guidance during the matching. Prommpt [80] enhances the inference of task identity through the approximated distributions derived from uninstructed representations.
Sehearsal-free methods train models without the help of rehearsal buffer [105]
55
40
21
them, Ceaanetee-Efficiens Tuning (PET) 73 95 35
Although many works make efforts improve the parameter matching process, there still exists substantial potential for further enhancement. As shown in Fig. Ta over 15% of data from Split CIFAR- are mismatched by the sota method HiDe-Prompt. Furthermore, when testing on the Split ImageNet-R, the matching accuracy is only 67.58%. As result, the classification accuracy is 4.54% and 8.85% lower than the setting with 100% matching accuracy. To sum up, previous methods determine the task identity through one-shot matching, which performs sub-optimal in CIL setting. Consequently, we propose a re- re-matthing mechanism to calibrate the matching results.
The re-matching mechanism is shown in Fig. lb Specifically, we propose Hybrid Re-Matching Parameter-Efficient Tuning (HRM-PET) method, which contains two re-matching operations. First, owing to the shared knowledge in the parameter pool, the task identity of the predicted class for some samples may be correct, despite the incorrect initial matching. These samples can be detected when the predicted class does not belong to the matching task identity. The task identity is directly replaced with the one obtained from the prediction. Second, we design confidence-based re-matching for all samples with incorrect initial matching. Specifically, for the parameters and the class of data mismatched, they have never been trained together. Therefore, the confidence of these predictions may be lower than the correct matching. and the experimental results in Fig. Ib demonstrated this assumption. To this end, we detect incorrect matching based on the confidence of the predicted distribution, and obtain the corrected task identity from the class with the sub-highest score. Moreover, we aspire to directly re-match more samples in the first scenarios. Therefore, we integrate cross-tass -aak instance relationship distillation [7] into the PET-based methods, which encourages aligning the shared feature space from different task parameters and improves the task-invariant knowledge.
In summary, our contributions are as follows: 1) We propose a hybrid re-matching strategy based on predicted distribution to improve matching and classification accuracy PET-based CL. 2) further introduce cross-task instance relationship distillation to align the shared feature space, which

---

promotes the learning of task-ing insk-invariant knowledge. 3) We empirically validate the effectiveness of HRM-PET on extensive datasets using five distinct pre-trainet models.
2
Related Work
2.1
Continual Learning
Many methods have been proposed for CL 40
63
99
93
2
such as regularization-
2
1
35
rehearsal based
9
24
11 and architecture- based methods [94] 106 30 Recently, PET-
based methods [37
98 show strong performance in rehearsal-fres setting, which leverage parameter-efficient tuning techniques [33 44), such as prompt tuning, to adapt a sequence of tasks upon frozen pre-trained models. L2P
[9 employs the QKV mechanism [78] to select several prompts corresponding to each sample in shared prompt pool. In addition to task-specific prompts, DualPrompt [88] incorporates global prompts to acquire task-invariant knowledge. LGCL [36] introduces language guidance at the task and instance level to improve prompt matching. HiDe- Prompt [80] 81] decomposes the PET-based method into three components, simultancously enhancing each part, and has achieved ofte-of-the-ar performance. Instead of selecting parameters, CODA- prompt 73 transforms the prompt selection process into a weighted mechanism. DAP [35 and APG [75] propose to generate prompts. Moreover, LAE [19] utilizes ensembling [88 to propose a unified CL framework for three Eerameter-Efficieen Tuning. Although MOS 744 also addresses mismatching through a self-refined retrieval mechanism, the assumptions are different from ours: MOS assumes that the sample is mismatched when the task identity of the final prediction is not the same as the initial matching. In contrast, HRM-PET detects and corrects mismatching based on confidence rather than only task identity consistency.
2.2 Parameter-Efficienn Tuning
Parameter-Efficien Tuning 10 is designed to tune pre-trained models with fewer parameters while freezing most pre-trained parameters. For example, Adapter (28] 64] inserts small network modules between the layers of a pre-trained model. Prompt Tuning [66] adds lightweight trainable parameters (prompts) to the input of the pre-trained model. Compared with Prompt Tuming, Prefix Tuning [44) maintains the length of the output sequence unchanged. LoRA (29] introduces low-rank matrices that capture task task- specific adaptations. VPT B3 applies prompt tuning ViT for the first time, designing learnable input prompts that activate the pre-existing knowledge within pre-traines model. The capability of trrameter-Efficieen Tuning acquire sask-specific knowledge renders highly
conducive to continual learning [16 17 96
3 Methodology
3.1 Continual Learning Formulation
Continual Learning aims to learn sequence of tasks without forgetting. In CIL,, each task is vh,)} from a set of several classes,
consisting of n, samples D, = [(a), vf), (ah, o/)..... (s", where is task identity, 2! E X' represents the i-th input image and y E y corresponding class As shown in Fig. 2 at
label. Therefore, a total of T tasks are defined as D = (Di, D2...., Dr time 4, only D, of task is available for training. For inference, unified linear classifier go, must distinguish all classes encountered so far. Note that the classes between tasks will not overlap and task identity is unknown during inference following common CIL settings [5] 67 38). The deep model fe : X' + } parameterized by weights 0 (0,, 0,, } is often considered as incremental learners, which is split as feature extractor ha, and linear classifier ge,- In PET-based methods, lightweight parameters p combined with frozen pre-trained model hpmm is trained as 0, to extract image features. For input test image x from arbitrary tasks, the model outputs probability distributions (h (z; 0y)) 0g) to predict its label ý. For rehearsal-based CIL, when training the model D,, small number of exemplars from [D,.... D,-1> } are available in the storage buffer. This paper deals with the more challenging rehearsal-free setting, where images in the past never stored.
We follow HiDe- HiDe-Prompt 80] as our baseline. Given the image I Dr of current task i, the frozen pre- trainrd model hpm with parameters 6ptm is directly used to extract its features as query
3

---

Rematching
Inference
9
Initial 1
Param Pool P
Classifier 90,
Testing sample
Pretrained ppm
Class
Direct
MLP
A
iD
Task
Selected
tching
Task C
Doc
Yes
No
Pretrained Bytm
Pt
pt
Classifier 9o,
Direct
Confidence-basee based
Task-r training data
dAT
T2 73 ... T--1
Distribution d
4 top-K
Old feature space
Confidence E
[Tu Tg. Tg ... Ti
LOTIRO
"CIRDD
Yes
Param. Pool
Farggaax E(::p)
New feature space
LE
Figure 2: Pipeline of our method. Task ID means task identity. is and i, denote the outcomes of the initial matching and the re-matching, Pi, and Pi, represent the corresponding parameters of ł and i, in the parameter pool. Re-matching consists of two methods: Direct and Confidence-base scenarios. I' is a set of top-N predicted task identities from initial matching.
((x) = h(;;; pptm). Then, the query feature is input to an auxiliary classifier yu parameterized by O~ for the prediction of task identity, d(x) g(q(x);; 0w) represents the output class-level probability distribution, which is optimized by cross-entropy during the training. A parameter (LoRA) pool P (po. Pi.....Ph . is maintained. During training, we get probability distribution d(z) = ((((;; Pt. ppmm): 0g) by using the pre-trained model followed with current parameter Pt- Then, Pr of the current task t and classifier ge, optimized by cross-entropy. To maintain valid prediction distributions for past tasks, we follow HiDe-Prome PrDmmpr to model the categorical features as a Gaussian distribution. For further details, please refer to the Supplementary Materials.
3.2
Hybrid Re-matching
3.2.1 Direct Re-matching
During the inference phase as shown Fig.
previous PET-based methods first employ feature The corresponding parameter is matched from P and attached to the pre-trained model, then predicts the final classification result ý. When the initial matching is incorrect, the is likely to be confused. However, for some simple samples, the features extracted by shared information in the parameter pool can also be distinguished. Therefore, the task identity of predicted label ý for these samples may be correct. Replacing the initial matched task identity with this can readily enhance matching accuracy. Therefore, We propose simple direct re-matching (DRM) solution by mapping the predicted class-level distribution d(xx) obtained with the initial matched parameters back to the task identity and conducting parameter re- re-matching strategy:
query ((x) to predict the task identity 88 80
& = T(rrgmax ((g(x);(0))
i, T (argmax g( (2y ppmm) 0g)),
where T converts class into task identity, & and i, represent the outcomes of the initial matching and re-matching, respectively. Undoubtedly, we cannot confirm the correctness of of To cover all samples of the above scenarios, we restrict DRM to samples where εy + fg:
argmax g(h(2;; Pi, ppmm 0,0,, ift, + Es argmax g(h(2;; Pi, ppem))))))) )5 0g), otherwise.
(2)

---

For samples with Ey * if is the correct task identity, it can directly improve THL If is incorrect, the worst result is to predict the wrong class like iy. Therefore, this design does not compromise the final performance.
3.2.2 Confidence-basse Re-matching
However, if f is identical to E. directly replacing & with i, for re-matching would be meaningless. Considering that each group of parameters is trained using images associated with its corresponding task identity, the classifier often outputs unconfident predicted distribution for mismatched samples during inference [26]. Hence, we further propose an confidence srefiddence-bseed re- re-matching (CRM) strategy. Initially, we calculate the confidence E(d(z)) by function E(-). Given advanced performance and advantages in detecting semantic drift the post-hoc generalized entropy (GEN) function [51] is applied. GEN calculates generalized entropy [13 22] only relies on d(xz) without any training:
E(d(z)) = =- E Cj
(1-dz)).
where y hyperparameter, d' is the first M values after sorting the probability values in d(xr) from descending order, .M represents the top-M highest probability, d(x) is obtained by & in DRM. Subsequently assess whether mismatching may occurr for by the matching detector De(x)):
0, E(d(x)) ST.
De(xi Opem: 0g) = 1, otherwise,
where 7 is the threshold. When the task identity is inaccurately predicted, De(r)) classifies it as 0. Having detected mismatched samples, i.e., De(x) a more suitable parameter from P. Given that it is impractical to establish a definitive boundary between mismatched and correctly matched samples by т, we refrain from directly replacing fy with another task identity. Rather, compare the confidence of final predicted distributions yielded by the top-N identities in initial matching. Specifically, we first obtain the top-N classes with the highest probabilities from the initial matching prediction distribution d(z)):
the critical challenge lies in identifying
r argmax de, (x)) (c,)
where I the set of the top-N classes. Candidate task identities are obtained by converting all classes to task identity in I with T. Then, we compare the confidence scores E of the final prediction distributions generated by inferencing with the parameters p corresponding to all task identities. The task identity with the highest confidence is selected the corrected task identity 1z:
arg max ((g(h(x:PT()) (pt())(()))))))))))))))))))))))) pptm):
(6)
ier
The final class prediction serM is derived from the prediction distribution corresponding to f,:
ÚCRM = argmax g(h( pptm))
Finally, the classification outcome JRra associated with the task identity of highest confidence employed the re-matching result, thereby diminishing the effects of erroneous detections. Since additional forwards will introduced when comparing confidence, set to 2. The discussion of N is in the Supplementary Material.
3.3 Cross-Task Instance Relationship Distillation
Cross-task instance relationship distillation (CTIRD) aims to facilitate the acquisition of task- invariant knowledge across different task parameters. We hope that more samples can be predicted as correct classes or task identities of classes, despite the incorrect initial matching. For the current task I, new parameters pt and old parameters {PI.....y P .Pt-1 of past tasks are available. Combining arbitrary parameter Ps E P with the pre-trained model hytms we can acquire the corresponding class token as feature embedding en (z) h(x; Optms Pk)) for D.. To represent new knowledge,
5

---

100
Incourett :BOT-2IK
Cornect Cornet--OO--2
CRMEDII
Stp-2KK
BOT-IK
Mcto-1K
80
hroorat-Sup-21K
Coerect-Spp21K
Parameter matching accuracy
GRM
PO0T-21K
BING-KK
70
90
Ratio (96)
80
40
30
Split CIFAR-100
Split ImageNet-R
2
3
7
8
9
10
Datascts
Task
Figure 3: Improvement parameter matching accuracy in HRM-PET. We show the enhance- ments brought by CRM and DRM our method compared to the baseline.
Figure 4: Proportion detected incorrect (blue) and correct matching (green) samples by con- fidence score from taskl to task10 for Split ImageNet-R in Sup-21K and iBOT-21K.
extract features for each sample in the current task by task parameters pt, and then calculate the feature similarity between them obtain similarity matrix W,:
1 (x.)) (x
(8)
During representing old knowledge, images of the current task do not align with any previous tasks, rendering infeasible to ascertain the most appropriate historical parameters for each current sample. To explore useful knowledge from past parameters, we utilize trained task classifier IN to identify the appropriate, albeit suboptimal, old parameters for each current sample. incorporate more previous tasks, we select the top-K task identities:
S argmax (x.), (Tlen))%
S = (5,, 2/
where s; represents subset of top-K task identities selected for J;. For batch of samples, we can obtain K sets {S1, Si.... SS.,, each of which represents an identity set with the 4n (1 E [1. K]) highest confidence level of these samples. For example, si represents the identity of the second sample in batch with the third highest confidence in Then, we can obtain set (Ws, Ws..... Ws of similarity matrices recording instance relationships consistency of shared knowledge across tasks, we compute the Kullback-Leei (KI.) divergence between the normalized instance relationship matrix W, of the current task and others as a loss:
different feature spaces. To ensure the
W = (x)( ez (zz),
(10)
ECTIRD(Pe 0,)
Consequently, the final loss function is:
CwrP(P,, 0g) = Lce(Pe 0g) + XeT CerIID(((p....
(11)
The ACT calibrates the trade-off between the acquisition mask-invariant and task-specific expertise.
4 Experiments
4.1 Datasets
first carry out experiments on two widely used datasets CIL performance evaluation: Split CIFAR-100 and Split ImageNet-R. Split CIFAR-100 splits CIFAR-100[41 dataset into 10 tasks. However, CIFAR-10W is relatively easy to classify and susceptible to information leakage when utilizing some pre- mre-trained models. Split ImageNet-R [88) is recently proposed, more challenging

---

Table 1: Results of Split CIFAR-100, Split ImageNet-R, ImageNet-A, and 5-Datasets on five different pre-trained models (PTM). We report the mean and standard deviation of both average final accuracy Ay and average forgetting FN computed across three random seeds.
PTM
Mothod
Spin CIPAR-1OO
Fs
Ay
Split ImageNet-R
Fo 4
Aw
ImageNet-A
Fvy
Aw 1
5-Dattasets
Fyy
DuaiPrompe
68
86.60:20.19 19
1.15
68, 31
4.4920.14
39.7610.18
5.8521.14
76.09:0091
15.70:1.13
S-Prompt++
30
88.81.0.188
0.8
69.6810.12
3.2910.05
39.1010.30
5/4110.96
83.18.10299
Sup- 2p-21K
CODA-Promp
66
86.94
0.04
70.03:0.447
5.17:0.22
30.96:00114
5.58:00.5
62.54:1.34
18,58:1.67
LAB 19
85.10.10.24
5.01
12.2510.62
3.430065
10.3611.25
9,33,10,45
61.16.0.10
10.2510.22
CPromge
83.18
6.98=1156
69.2
1.42
37.00
8.12
71.72=0.55
18.8741.12
InfL.RA
85.28+0.10
4.28-091
72.24+0.54
3.20
42,12+1.97
6.1111.11
7199--067
12.23+1.05
LoRA
98.4320.38
0.16
71.90:0.22
4.33
42.18:00.10
6.15
93.25=0.03
1.0400.03
HRM-PET (Ours)
99,45+0.22
183-0.13
73.66+0.14
1,00+0115
44,28+0.11
5.1+0.322
9399-0012
0611-0114
DualProngg
[8
76.76:00.3
0:84:034
54,5580.53
5,38:0.70
21.00
6.92:0.84
67 677
2.124
S-Prompt++
36
79.14100.65
9.17
55.1610..83
1.0710.16
23.89
5.8610..88
71.1622.94
19.38.14.28
iBOT-2IK
CODA-Prompt
80.83:0.277
05
61.2210.35
9.6610020
28.1040.28
6.3340.76
36.12=3.18
2..23
LAB hD
78.87+0.58
00:55=00.0 40
60.68+0112
467
29.76+0.16
9.77+1.06
63 1::6-1100
13,75+1.J1 1.12
CPromgx
82.3440.81
6.98=1.16
66.64
7.0610.71
31.66:1.66
10.26:1079
67.38 1,90
0..4810.81
InfL.RRA
+
87,95+0.39
4.88-0
20.15+0.24
4.90
13.87
9.04+1446
75 04.17
8.83+0059
HiDe-LoRA
X
88.44+0.17
4:53-0344 14
75.40+0..22
1.770041
38.55-043
7.940.199
91.53-0.31
+.000.10
HRM-PET (Ours)
89,70+00.13
3.75-0.11
75.23+00221
3.68-0117
40.8-+0.32
4.98-0611
94.38-0111
+.7
DualProunge
76.63100:5
0.4
61.5111.005
5.02
21,0010,39
6.4310.64
15
S-Prompts-
X
77.53+00.56
8.07=097
6088200.68
0.16
21,12+1.12
6,52+0.77
71:1==150
6.6400.14
CODA-Prompt
1
79.11
1.69
66.56110.68
0.22
28.69
6.34
34.34
4.08
iBOT-
LAB h9
75.45=00:43
10.55=0036
67.95+00.55
5.56+0.41
27.18=0.21
9.77
67
0::00:144
13.75
CPronge
76.32+0.79
12.49+1.36
68.25+1.49
1.25+1.19
29.70+0.91
8.23+1111
75.21-0.444
9.86+1112
InfLoRA +
85.5110.100
6.28:00166
11.90110110
2.91
32,01:10.70
7.4
05
9.1210.48
HiDe-1 cRA
X0
86.43+0.23
5:23-0344
72.79++0.28
061
35.63+0.25
6.25++013
08-00-
1.70+0.03
HRM-PET (Ours)
87.11:10.15
0.11
74,6410,28
3.92
37.89
5.19:0..1
94.15:0.155
0
DualPrompg
74.9010.211
10.26.10.22
58.57.0.45
5.80.10.11
21.27.10.52
5.16.10.81
6..21
24.04
S-Prompt++
24.974:0.66
1.78=0.66
17.6440.16
5.08
1..58
6.2240.68
75.19=0080
1,76
CODA-Promp
77.50+0.64
8. 10000001
63.15+0.39
6.86+0.11
24 24.28-40.56
6.66++1.22
51 51.52=22110 2.10
27 27.55+33011
DINO
LAE 19
73.62:10:44
12
63.7010084
5.44:0.24
23.10:10,38
166
65:35:1555
1..25
CPrompe 20
74.64+1.60 69
11.79+1.99
65.67+0.57
9.85+0.91
26.49+0-881
7 7,80+087
73 73.40-0.55
17.67+141
InfLoRA
82.37:00.5
6.40:033
68.51:30.00
5.011:069
31.67:1028
5.12:0049
09:00.2333
11:23:10.05
JRDe-LoRA g0
81.85110.21
5.36.10.22
70.42.10.22
4.60.10.19
31.72.10.33
4,98.10.31
93 95.20.00004
1.3410.01
HRM-PET (Ours)
85.93:0.14
5.14:00.3
72.32:10.20
3.49:0.27
33,37:10.35
4.75:00.5
93.90:0.05
DralPrompe
X
77.77+0.68
6.61=1.06 08
52,57++0.82
2.73+0049
15.07+1.05
4.11+1.665
68.17=0.35
23.56+00.46
S-Prompt++
76.30:20.54
14667:064
53.15:1.10
4.111:844
17.96:20.66
4.50::1 1,96
71.75:0055
13,60:00.33
Mo IK
CODA-Promp
tk
26.83+00334
12.60=00002
55.75++0.26
10:46+0.044
13.75+0021
8.34+0.19
5281=241
33,58+2.99
LAE 19
78.31+0.32
15:24:0.19
54.11:10.16
10.21:0.34
21:42::0.15
12.43:1.11
57.44:0.188
17.11::0.36
CPrompt N1
77.3810.99
12.21111.56
59,1712.55
8.6111885
22.16.10.91
6,9610.69
73.1710.84
17.5711.34
IntLaRA
47
82.70:10.39
8.38:20.66
65.033:0.87
5.69
26.00:1:.81
6.50::0.64
74 74.00=0.388
10.4910.42
IiDe-LoRA
[
85.37.10.21
5.121.0.19
68.01.10.59
4,57.00.66
28.30.10.47
6,7410035
92.72-00.07
1.7110.06
HRM-PET (Ours)
86.00:10.14
0.32:0.11
69.32.10.22
4.10:00.35
29.73:10.52
6.2340.61
93.31:0008
1.2310.15
Table 2: Comparisons with other methods relying on pre-trained models. To ensure a fair comparison, we re-implemented our method under the same weights in SLCA [00].
Split CIFAR-100
Split ImageNet-R
ImagoNet-A
5- 5-Daatsets
Mothod
Ay f
Fyy 4
Aw f
Fsy 4
Aw f
Fyy
Aa f
Fgy L
ESN B7
86.34:2..2
4.76:10.14
65.83-0.58
5.9310.31
45.08:0.22
6.25:10.84
L5.7
2.65:0061
SLCA 100
91.53=-0.28
5.33+00311
77.00=033 7..03:10.14
4.34+0032
$288=0040
7.72++0.65
85,24+0.84
11.32++0.92 94.12.1..12
HRM-PET (curs)
91.63:10.63
3.33-..13
3.11:1..55
58.43-1.112
7.89-10.32
0.58:10.14
dataset. collects 30000 images of different styles and is difficult to differentiate for general classification models from ImageNet [14), which is similarly split into 10 tasks.
To investigate performance on datasets with the larger domain gap from the pre-trained ImageNet.
we evaluate HRM-PET on ImageNet-A [27). consisting of images that are misclassified by ResNet
models. It is also divided into 10 tasks and 20 classes for every task [59]. Moreover, 5-Datasets [5]
consisting of CIFAR- 10 [41), MNIST 44),, Fashion-MNIS, 191], SVHN 62] and notMNIST 4 is examined, which are divided into 5 tasks according to different datasets.
4.2 Evaluation Settings
Comparing methods. We firstly compare multiple PET-based methods: DualPrompt [8], S- Prompt++ (86), CODA-Prompt (73). LAE [19) CPrompt [1], InfLoRA [47]. and HiDe-LoRA [8 81. CODA-Prompt, LAE, and InfLoRA do not maintain task-specific parameters and do not require task identification. For fairness, both our method and HiDe-Prompi use LoRA as the PET, and therefore HiiDe-Promp is referred to as HiDe-LoRA in the table. The LoRA modules are integrated into the initial five layers of the ViT. Moreover, we compare the performance of different pre- pre-trained
7

---

Table 3: Ablation experiment to prob DRM, CRM, and CTIRD in ImageNet-R.
Method
Sup-21K
BBOT-21K
iBOT-IK
DINO-IK
McCo-1K
Baseline
71.4700266
73.49+0.31
72.99=0.56
70.40::0.62
67.89:0.29
BuselineDDRM
72.55:+0.10
74.11+0.22
73.48=0.35
71.05::0.27
68.70:0.42
Baseline+CRM
72.80::0.11
74.35:0.226
74.01
71.32:1::23
08.80:0.20
Bascline+DMM=CRM
73.38=+0021
74.80=00155
74. 14.18=0.18 0818
71.85 11.85500.45
69. 00.10
Mncllne-DRMMCRMMCTTR
73.86.1:0.14
75.23:10.21
74.64.10.28
72.32.10.20
69.32.10.22
Table 4: The additional computational time dur-
Table 5: Comparison of different knowledge dis- tillation methods under Sup-21K and iBOT-21K
ing the inference process. We present the Time
(ms) per image on the test set, and Av (%), after the application of DRM and CRM.
pre-trained model.
Distillaion
CIFAR 100
ImageNet
CIFAR-100
ImageNet-R R
Sup-21.
iBOT:21.
Sup-21.
iBOT-21.
Method
A f
Time.
Aw f
Timel
Logits 45
87.66
88.02
72.88
74.51
Baseline
88.40
2.69
71.60
2.81
Features 105
87.40
83.72
71.15
71.64
Raseline+DRM
88.80
2.81
72.60
2.91
IRD
88.59
88.98
73.21
74.58
Beeline-CRM-DRM
89.45
3.01
73.86
3.24
CTIRD
89.45
89,70
73,86
75.23
models: Sup-21K (68] iBOT-IK [104), iBOT-21K [104, DINO-IK 6 and MoCo-1K (25] We
employ the same supervised or self-supervised pre-trained VřT-B/16 as the backbone. In Sup-21 K, for fair comparison, report the results under uniform checkpoint More details shown in Supplementary Material. Additionally, we conduct comparisons with SLCA (1O0) and ESN [87]. Follow 773,, the performance is measured by average final accuracy AN and average forgetting FN after learning N tasks [53 9 88).
4.3 Comparison with the State-of-the-art Methods
To evaluate the effectiveness, perform extensive comparative experiments with state-of-the-arr PET-based methods. This comparison involves five pre-trained models across four datasets: Split- CIFARI00, Split ImageNet-R. ImageNet-A, and 5-Datasets. Subsequently, we present comparative results with 2 sota methods based on the same pre-trained models but not designed with PET.
The results shown in Table Overall, method achieves state-of-the-art results under 20 set- tings of four datasets with various pre-trate trained backbones. On the Split ImageNet-R and ImageNet-A with large intra-class diversity [103] 88), HRM-PET shows substantial performance improvements. is worth noting that our method improves 1.61% % and 2.10% with the model pre-traine pre-trained on Sup- 21K, 1.83% and 2.33% with the model pre-trained iBOT-21 Moreover, HRM-PET also brings improvement on Split CIFAR-106 On 5-Datasets, characterized by large task gaps, our method achieves improvements of 0.74%, 0.85%, 0.67%, 0.65%, 0.59% respectively under different back- bones. This indicates that HRM-PET consistently achieves better performance, and the improvement particularly significant on challenging datasets.
In addition to applying PET in CIL. there exist many other methods, such as direct fine-tuning. We present the comparison between HRM-PET and two representative algorithms in Table 2 On the one hand, considering AN, HRM-PET surpasses other methods 2.03% and 2.58% on Split ImageNet-R and ImageNet-A. Moreover, method achieves 8.41% improvements on 5 5-Datasets. The results demonstrate the superiority of HRM-PET. On the other hand, although our method only tunes a few parameters, HRM-PET stably exhibits competitive performance on all datasets.
4.4 Ablation Study
Effectiveness the proposed components. As illustrated in Table 3 we conduct a comprehensive analysis of each module, across five pre-trained models the Split ImageNet-R datasets. Ax is utilized as measurement. The baseline is trained with Section 3.1 According to the results, we have the following observations: 1) Compared with baseline model, both CRM and DRM calibrate incorrect parameter matching, hence improving performance. Furthermore, CRM aims to detect all mismatched samples based confidence, resulting in powerful performance. 2) Combining CRM and DRM achieves better performance compared with either CRM or DRM. As shown in Fig. [b] there is partial intersection between correct and incorrect matching in the confidence-based distribution. DRM compensates for the limitations on these samples, thereby combining CRM
VIT-BB 16.nn

---

Correct
Mismatched to other task identities
sang OM
Wo SunD
Figure 5: Visualization of attention regions with (Ours) and without CTIRD (W/o). "Correct" means the parameters with the correct task identity. "Mismatched to other task identities" means parameters corresponding to other incorrect task identities.
76.5
76.5
76.5
76.5
35.5
75.5
73.5
Sup NE
75.5
74.5
74.5
74.5
74.5
73.5
73.5
73.5
73.5
72.5
72.5
72.5
0.0
0.1
0.2
0.3
-10
-10
10a
12
(a) ACT
(b) K
(c)
(d) R
Figure 6: Variation of A with hyperparameters. We show influences of 4 hyperparameters within the context of the ImageNet-R dataset with the Sup-21K and iBOT-21 setting: (a) ACT, (b) K, (c)
T, (d) Ri.
and DRM further improves the matching and classification performances. 3) By combining all components, the model attains the best AN, which demonstrates the complementarity of the proposed re-matching and distillation strategies.
Parameter matching accuracy. To prob the effect of re-matching strategies matching accuracy, we conduct extensive experiments Fig. 3 First, on the Split ImageNet-R with large intra-class diversity (88), our re-matching improves matching accuracy by average of 9 9.28% using various pre- trained backbones. Particularly, we achieve at least 2.91 % various settings, which demonstrates the robustness of the re-matching. Besides, the Split CIFAR- 100 with matching accuracy exceeding 80%, combining DRM and CRM still brings an average matching accuracy improvement of 4.13%. After obtaining higher matching accuracy, HRM-PET achieves higher Ay in Table [
Confidence-based detection. To validate the CRM's ability to detect mismatching, we show the proportion of incorrectly and correctly matched samples filtered by the confidence threshold. In all experiments, the threshold т is consistently set as -10. As depicted in Fig. 4 with the help of the confidence-bassee incremental process. Note that about 10-30% of correctly matched images are inevitably incorporated. To handle these data, we design equation to determine whether to replace the task identity.
threshold, CRM is able to detect the most mismatched samples during the
Inference time. When re- matching is conducted based on the predicted distribution, the model needs an additional forward process (N = 2). which leads to an increase in the inference time. Table 4 shows the increased consumption. Results are fairly compared on RTX3090 at the same batch size. Particularly, DRM and CRM only deal with part of samples that are likely to be mismatched, hence we can observe that the average time cost on the test set is only about O. 0.4ms. Importantly, significant performance improvements be achieved with slight time cost. Therefore, our calibrating strategy based on re-matching mechanism promising for future research.
Training time. The training time of HRM-PET is acceptable. In CTIRD, we first utilize the pre- trained task classifier In to obtain the top-K K old task identities with the highest confidence scores for all samples. The time cost of performing a forward through the classifier is negligible. Subsequently, before training, each sample undergoes inferences to obtain K features for knowledge distillation. Compared to training, the inference-nnly forward introduces minimal additional time overhead. Experimentally, we measured the training time baseline and HRM-PET on ImageNet-R on an RTX 3090. Our method incurs only additional 4.4% (i.c., 0.30 hours) of time overhead.

---

Knowledge distillation. Knowledge distillation [45] is a classic mechanism for preserving model stability. Through experimental exploration, we assess the influence of various knowledge distillation techniques. The compared methods contain traditional logits 45). feature distillation [T05] and similar instance relation distillation (IRD) 7 Ta We implement IRD for each old task (IRD*) without our top-K strategy. According to the results, CTIRD surpasses all alternative methods. The previous methods add overly rigid constraints to the model, which impede the plasticity of new task learning. Differently, CTIRD selects appropriate old parameters for each sample by the top-K mechanism, which further prompts the model to learn shared information across tasks.
Visualization. In CTIRD, we visualized the attention area with parameters of different task identities in Fig. [S] The first column displays attention maps generated using the correct task identity, while columns 2-10 correspond to attention maps produced using incorTect task identities. Two main observations can be drawn: 1) On the one hand, there almost no difference in the attention obtained with the correct parameters when CTIRD is used and not used. This suggests that CTIRD does not impair task-specific knowledge. 2) On the other hand, when CTIRD is applied, even if mismatched parameters are attached, the attention still focuses key feature areas of the object. These key feature areas help them still have the correct task identities or classes in the predicted distribution when the parameters are mismatched. Hence, CTIRD improves shared knowledge.
4.5 Hyperparameter Analysis
In this section, the effect of hyperparameters on the HRM-PET is investigated. All experiments are conducted on Split ImageNet-R, for Sup-21k and iBOT-21k pertaining settings. First, we prob the effect of the CTIRD loss coefficient ACT on AN. which is shown in Fig. 6a The performance of HRM--ET improves as λCT increases and achieves the best performance when ÄCT = 0.2. A smaller ACT compromises the PET's capacity to acquire task-invariant knowledge. Conversely, an excessively large ACT would obstruct the acquisition of task-specific knowledge in new tasks. Next, We also present the influence of the number K for CTIRD Fig. 6bb K controls the amount of knowledge distillation with past task parameters, which influences the trade-off between task-specific and task- invariant knowledge. Note that we utilize the common setting K = to reduce the training computational consumption.
In addition, as shown in Fig. 6c the influence the confidence threshold r re-matching explored. A lower calibrates fewer mismatched samples, whereas higher threshold may classify correctly matched samples as mismatched, we set as -10. validated in Fig. B] combining confidence-baat based re-matching strategy, the accuracy of task identity matching can be significantly improved. Finally, we explore the effect hyperparameter Rank (Rt) leveraged in LoRA, which shown in Fig. (dd In HRM-PET, Rt 8 is suitable to learn task-specific knowledge. For more details, please see the Supplementary Material.
Conclusion
In this paper, propose HRM-PET for rehearsal-frt -free continual learning with paraaetee--fffciient tuning. We design a hybrid re-matching mechanism based on the initial predicted distribution to improve parameter matching accuracy in PET-based CL methods. Specifically, the direct re-matching handles some samples by replacing the incorrect initial task identity with the correct one from prediction. In addition, the confidence-basee re-matching addresses more mismatched samples without correct task identity in the predicted distribution. Furthermore, we introduce cross-task instance relationship distillation to reduce the dependence of the model on matching accuracy. Minimizing hyperparameter dependency and improving inference efficiency represent important avenues for future research. Overall, extensive comparison and ablation experiments indicate that HRM-PET achieves state-of-the ofte--of-hh-arr aat performance.
6
Acknowledgments
This work was supported by the Natural Science Foundation of Tianjin, China (No. (0CZXC0020))0 Shenzhen Science and Technology Program (No. JCCJ202408113 14229039), the National Natural Science Foundation of China (No. 623B2056, 624B2072), the Fundamental Research Funds for the Central Universities, the Supercomputing Center of Nankai University (NKSC).
10

---

References
[1] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang. Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. CVPR, 2021.
[2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. ECCV, 2018.
[3] Nader Asadi, MohammadReza Davari, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Prototype- sample relation distillation: towards replay-free continual learning. In /CML, 2023.
[4] Yaroslav Bulatov. Notmnist dataset. Rep.[Onlinej. 2011.
[5] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In NeurlPS, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- superviedd vision transformers. In ICCV, 2021.
[7] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co21: Contrastive continual learning. In ICCV, 2021.
[8] Sungmin Cha, Sungjun Cho, Dasol Hwang, Sunwon Hong. Moontae Lee, and Taesup Moon. Rebalancing batch normalization for exemplar based class innremnncemettll learning. In CVPR, 2023.
[9] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with a-gem. In ICLR, 2019.
[10] Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. Revisiting parameter-effiche tuning: Are really there yet? In EMNLP, 2022.
[11] Xiuwei Chen and Xiaobin Chang. Dynamic residual classifier for class incremental Icarning. In ICCV, 2023.
[12] Chaoran Cui, Yongrui Zhen, Shuai Gong, Chunyun Zhang, Hui Liu, and Yilong Yin. Dynamic prompt allocation and tuning for continual test-time adaptation. Science China Information Sciences, 6806).160109 2025.
[13] Alexander Philip Dawid and Monica Musio. Theory and applications of proper scoring rules. Metron, 12(2):169-183, 2014.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. hierarchical image database. In CVPR, 2009.
Imagenet: large-scale
[15] Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Adversarial continual learning. In ECCV, 2020.
[16] Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cédric Archambeau. Continual learning with transformers for image classification. In CVPR, 2022.
[17] Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cedric Archambeau. Memory efficient continual learning with transformers. In NeurlPS, 2022.
[18] Qiankun Gao, Chen Zhao, Bernard Chanem, and Jian Zhang. R-dfcil: Relation-guided representation learning for data-free class incremental learning. In ECCV, 2022.
[19] Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang, unified continual learning framework with general parameter-efficient tuning. ICCV, 2023.
[20] Xinyuan Gao, Songlin Dong, Yuhang He, Qiang Wang, and Yihong Gong. Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning. In ECCV, 2024.
[21] Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent prompting for rehearsal frree continual learning. In CVPR, 2024.
[22] Tilmann Gneiting and Adrian Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-333 357-378, 2007.
[23] Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski. and Joost van de Weijer, Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In NewrIPS, 2024.
11

---

[24] Tyler L Hayes, Nathan D Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In ICRA, 2019.
[25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
[26] Dan Hendrycks and Kevin Gimpel. bascline for detceting misclassified and out-of-distribution examples neural networks. In ICLR, 2016.
[27] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
[28] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruma Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter--fficient transfer learning for nlp. In ICML, 2019.
[29] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
[30] Zhiyuan Hu, Yunsheng Li, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos. Dense network expansion for class incremental learning. CVPR, 2023.
[31] Linlan Huang. Xusheng Cao, Haori Lu, and Xialei Liu. Class-incremental leamning with clip: Adaptive representation adjustment and parameter fusion. In BCCV, 2024.
[32] Kishaan Jeeveswaran, Prashant Bhat, Bahram Zonooz, and Elahe Arani. Birt: Bio-inspired replay in vision transformers for continual learning. In ICML, 2023.
[33] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[34] Hyundong Jin and Eunwoo Kim. Helpful or harmful: Inter-task association in continual learning. In ECCV, 2022.
[35] Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. rehearsal-free continual learning. In ICCV, 2023.
Generating instance-level prompts for
[36] Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt-based continual Icarning. In CVPR, 2023.
[37] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahhaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In ICCV, 2023.
[38] Dongwan Kim and Bohyung Han. On the slability-llastiityy dilemma of class-incremental learning. In CVPR, 2023.
[39] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barsska-BBriikka,. et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 -326, 2017.
[40] Yajing Kong, Liu Liu, Zhen Wang, and Dacheng Tao. Balancing stability and plasticity through advanced null space in continual learning, In ECCV, 2022.
[41] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
[42] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied document recognition. PIEEE, (6(11)2227822224 1998.
[43] Qiwei Li, Yuxin Peng, and Jiahuan Zhou. Fes: Feature calibration and separation for non-exemplar class incremental learning. In CVPR, 2024.
[44] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
[45] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935 29447777927277,00 2017.
12

---

[46] Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototypes with prompt-tuning for rehearsal-free continual learning. In WACV, 2024.
[47] Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference- low-rank adaptation for continual learning. In CVPR, 2024.
[48] Tian Yu Liu and Stefano Soatto. Tangent model composition for ensembling and continual fine-tuning. In ICCV, 2023.
[49] Xialei Liu, Yu-Song Hu, Xu-Sheng Cao, Andrew D Bagdanov, Ke Li, and Ming-Ming Cheng. Long-tailed class incremental learning. In ECCV, 2022.
[50] Xialei Liu, Jiang- Tian Zhai, Andrew D Bagdanov, Ke Li, Ming- Ming Cheng, Task-adaptive saliency guidance for exemplar-fie free class incremental learning. In CVPR, 2024.
[51] Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax-bass out-of- distribution detection. In CVPR, 2023.
[52] Yaoyao Liu, Bernt Schiele. Andrea Vedaldi, and Christian Rupprecht. Continual detection transformer for incremental object detection. CVPR, 2023.
[53] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems 30, pages 6470-6477 2017.
[54] Zilin Luo, Yaoyao Liu, Bertt Schiele, and Qianru Sun. Class-incremental exemplar compression for class-incremental learning. In CVPR, 2023.
[55] Zhiheng Ma, Xiaopeng Hong, Beinan Liu, Yabin Wang. Pinyue Guo, and Huiyun Li. Remind of the past: Incremental learning with analogical prompts. arXiv preprint arXiv:2303. 2023.
[56] Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners. In ECCV, 2024.
[57] Mare Masana, Xialei Liu, Bartiomiej Twardowski. Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. inssemenere learning: survey and performance evaluation on image classification. TPAMI, 45(5)55513-5533, 2022.
[58] Michael McCloskey and Neal Cohen. Catastrophic interference connectionist networks: The sequential learning problem. In Psychology of leerriing and motivation, volume 24, pages 109-165. Elsevier, 1989.
[59] Mark D McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. In NeurlPS, 2023.
[60] Martin Menabue, Emanuele Frascaroli, Matteo Boschini, Enver Sangineto. Lorenzo Bonicelli, Angelo Porrello, and Simone Calderara. Semantic residual prompts for continual learning. ECCV, 2024.
[61] Zichong Meng, Jie Zhang, Changdi Yang. Zheng Zhan, Pu Zhao, and Yanzhi Wang. Diffelass: Diffussin- based class incremental learning. In ECCV, 2024.
[62] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NeurlPSW, 2011.
[63] Aristeidis Panos. Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard Turner. First session adaptation: A strong replay-free baseline for class-incremental learning. In JCCV, 2023.
[64] Jonas Pfeiffer, Andreas Rücklé Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. EMNLP, page 46, 2020.
[65] Senthil Purushwalkam, Pedro Morgado, and Abhinav Gupta. The challenges of continuous self-supervised learning. In ECCV, 2022.
[66] Guanghui Qin and Jason Eisner. Learning how to ask: Querying Ims with mixtures soft prompts. In ACL, 2021.
[67] Sylvestre-Alviss Rebufli, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremen- tal classifier and representation learning. In CVPR, 2017.
13

---

[68] Tal Ridnik, Emanuel Ben-Baruch, Asal Noy. and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. In Thirty-fithh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[69] Anurag Roy, Vinay K Verma, Sravan Voonna, Kripahandhu Ghosh, Saptarshi Ghosh, and Abir Das. Exemplar-fiee free continual transformer with convolutions. In ICCV, 2023.
[70] Chao Shang, Hongliang Li, Fanman Meng, Qingbo Wu, Heqian Qiu, and Lanxiao Wang. Incrementer: Transformer for class-incremental semantic segmentation with knowledge distillation focusing on old class. In CVPR, 2023.
[71] Reza Shokri and Vitaly Shmatikov. prvacy-preserving deep learning. In ACM SIGSAC, 2015.
[72] James Smith, Yen- Ven-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class inssemnncrmttt learning. In ICCV, 2021.
[73] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Bonnill, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal- free continual learning. In CVPR, 2023.
[74] Hai- Long Sun, Da-Wa Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Ye. Mos: Model surgery for pre-trained model-based iusseinnrrmettal learning. In AAAI, 2025.
[75] Yu-Ming Tang, Yi-Xing Peng, and Wei-Shi Zheng. When prompt-based incremental learning does not meet strong pretraining. In CVPR, 2023.
[76] Gido M Van De Ven, Zhe Li, and Andreas S Tolias. Class-incremental learning with generative classifiers. In CVPR, 2021.
[77] Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185-119) 2022.
[78] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[79] Fu-Yun Yun Wang. Da- Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and compression for cass-innrrementl leamning. In ECCV, 2022.
Da-Wei
[80] Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. In NeurlPS, 2024.
[81] Liyuan Wang. Jingyi Xie, Xingxing Zhang, Hang Su, and Jun Zhu. Towards general framework for continual learning with pre-training. arXiv preprint 10. 13888, 2023.
[82] Liyuan Wang. Xingxing Zhang, Qian Li, Mingtian Zhang. Hang Su, Jun Zhu, and Yi Zhong, Incorpo- rating peuro-inspired adaptability for continual learning in artificial intelligence. Nature Machine Intelligence, 5(12):1356-1368, 2023.
[83] Liyuan Wang, Xingxing Zhang, Qian Li, Jun Zhu, and Yi Zhong. Cosel: Cooperation of small continual learners is stronger than big one. ECCV, 2022.
[84] Liyun Wang, Xingxing Zhang, Hang Su, and Jun Zhu, A comprehensive survey of continual learning: Theory, method and application. TPAMI, 2024.
[85] Wenjin Wang, Yunqing Hu, Qianglong Chen, and Yin Zhang. Task difficulty aware parameter allocation & regularization for lifelong learning. In CVPR, 2023.
[86] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S S-prompts learning with pre-trained transformers: An occam's razor for domain incremental learning. In NewrlPS, 2022.
[87] Yabin Wang, Zhiheng Ma, Zhiwu Huang, Yaowei Wang, Zhou Su, and Xiaopeng Hong. Isolation and impartial aggregation: A paradigm of incremental learning without interference. In AAAI, 2023.
[88] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, 2022.
[89] Zifeng Wang, Zizhao Zhang, Chen Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning prompt for continual learning. In CVPR, 2022.
14

---

[90] Tz- Ying Wu, Gurumurthy Swaminathan, Zhizhong Li. Avinash Ravichandran, Nuno Vasconcelos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pre-trained models. In CVPR, 2022.
[91] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1008.07747, 2017.
[92] Wen Xiao, Chang-Bin Zhang, Jiekang Feng, Xialei Liu, Joost van de Weijer, and Ming-Ming Cheng. Endpoints weight fusion for class incremental semantic segmentation. In CVPR, 2023.
[93] Zhengyuan Xie, Haiquan Lu, Jia-wen Xiao, Enguang Wang, Le Zhang, and Xialei Liu, Farly preparation pays off: New classifier pre-tuning for class incremental semantic segmentation. In ECCV, 2024.
[94] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In CVPR, 2021.
[95] Longrong Yang, Hanbin Zhao, Yunlong Yu, Xiaodong Zeng, and Xi Li. Res-prompt: Learinng prompt rearrange class space for prompt-based continual learning. In ECCV, 2024.
[96] Shu Yang, Muhammad Asif Ali, Cheng-I.ong Wang. Lijie Hu, and Di Wang. Moral: Moe augmented lora for lims' lifelong learning. arXiv preprint arXiv:2402. / /260, 2024.
[97] Jingwen Ye, Yifang Fu, Jie Song, Xingyi Yang. Songhua Liu, Xin Jin, Mingli Song, and Xinchao Wang. Learning with recoverable forgetting. In ECCV, 2022.
[98] Jiazno Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of of-experts adapters. CVPR, 2024.
[99] Jiang-Tine Tian Zhai, Xialei Liu, Andrew D Bagdanov, Ke Li, and Ming- Ming Cheng. Masked autoencoders efficient class incremental leamerr. In ICCV, 2023.
[100] Gengwei Zhang, Liyuan Wang. Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier aligmment for continual learning on a pre-trained model. In ICCV, 2023.
[101] Zhao Zhang. Yong Zhang. Da Guo, Shuang Zhao, and Xiaolin Zhu. Communication federated continual learning for distributed learning system with non-iid data. Science China Information Sciences, 6612); 2023.
[102] Da-W Wei Zhou, Hai-Long Sun, Han-Jia Yc, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In CVPR, 2024.
[103] DeiWei Zhou, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class-incremental Icarning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint XxXi:2303.073336 2023.
[104] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image hert pre-training with online tokenizer. ICI.R, 2021.
[105] Fei Zhu, Xu- Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In CVPR, 2021.
[106] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Sultaanst representation expansion for non-cxemplar class-incremental learning. In CVPR, 2022.
[107] Orr Zohar, Kuan-Chich Wang, and Serena Yeung. Prob: Probabilistic objectness for open world object detection. In CVPR, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 3 describes the detailed methodology. Section validates the claim.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the Supplementary Materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please see Section and the Supplementary Materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
17

---

Justification: The code is provided in the abstract. The dataset uses public datasets, please see Section 4
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етrоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Section 4 and the Supplementary Materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttlesshttt/////t htttp:://heuripssccc// hwwparppp....pbblllc/aaacasaaaaa
Answer: [Yes]
Justification: Please the Supplementary Materials.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups). privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
19

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please see the Supplementary Materials.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo/caac datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please the Supplementary Materials.
Guidelines:
20

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.LM policy (ttrpe//eaarrrp..cc/Cffffeeeeee occ/Conferrencee22255////// for what should or should not be described.
21


Reviews:
[{"review_id": "eCNhXah9v1", "reviewer": "Reviewer_RzxM", "summary": "This paper studies how to perform continual learning with a pretrained model using the Parameter-Efficient Tuning module. It proposes some techniques (direct rematching + confidence rematching) to improve the task identity prediction when selecting PET task module during inference. It also proposes a distillation loss for training to encourage the learn of shared knowledge among different task PET modules.\n\nThe paper presents an extensive evaluation of the proposed method, compared against several baseline methods, on four datasets and different pretrained models. It also presents ablation studies to examine the effectiveness of each algorithm component as well as key hyperparameters.", "questions": "1. The confidence is a key component of the proposed method and is used in both the rematching step and distillation step (e.g. Eq 3 and 7). To compute confidence, the paper mentioned the use of the generalized entropy function with additional hyperparameters $gamma$ and $M$. Why not use other hyperparameter-free confidence-related measures, like predicted probability or entropy? How are $gamma$ and $M$ selected? What is the interplay between generalized entropy function hyperparameter and the threshold parameter?\n\n2. The paper mentioned that the proposed method can be applied to different PETs. Any results to support this claim?\n\n3. As the proposed method involves many algorithm-specific hyperparameters, how are these algorithm-specific hyperparameters selected? What about baseline methods?", "limitations": "yes, but a bit short.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The idea of direct rematching (section 3.2.1) is simple and effective in improving task identification prediction.\n- The paper includes an extensive comparison with baseline methods on four datasets with different pretrain models.\n- The paper provides ablation studies to investigate the effectiveness of the proposed three components,\n- The paper analyzes the effect of some of the key hyperparameters.\n\nWeaknesses\n- Confidence plays an important role in the proposed method, which is used in both rematching and distillation. The authors mentioned the use of generalized entropy to compute confidence. But in the appendix, it seems that different confidence functions are used for different datasets (maxlogits for CIFAR100). It is unclear how to select the confidence measure for different datasets. It is better to include a systematic study on the effect of the confidence measure.\n\n- The proposed method significantly increases inference time. Although the authors mentioned the absolute increase is small (0.4ms), the relative inference time is about 15% (from 2.8 to 3.2 ms). The inference time cost is already considered a big and important problem when employing foundation models in practice. \n\n- The proposed method introduces many hyperparameters, e.g.$\\gamma$, $M$, $\\tau$, N, K, $\\lambda$. How to select a suitable combination of these hyperparameters for a dataset is non-trivial.\n\n- There is no error bar or confidence interval or standard deviation in the ablation studies, e.g. Table 3, 4 and Figure 6.", "was_revised": true, "final_justification": "The paper is technically sound. I acknowledge the strengths of paper mentione by other reviewers. However, due to increased inference time, the limited theoretical depth, and the introduction of algorithm-specific hyperparameters, I maintain my initial rating of 3 at this stage.", "rebuttal": "Thank you for your insightful comments and questions. \n\n**Q1: A systematic study on the effect of the confidence measure** \n\n\n**A1:**\nTo explore the effect of the confidence measure, we conduct ablation experiments on different post-hoc confidence measures in the table.\n\n1) Overall, our method is not sensitive to different confidence calculation methods.\n\n2) Generalized entropy (GEN) performs best across most settings.  This is attributed to its design for semantic shift scenarios, i.e., detecting inputs with semantic categories that are absent from the training set [48].  In CL, the PET parameters for task $t$ are only trained on the semantic categories of task $t$.  When predicted task identity $\\hat{t}$ does not match the true task $t$ for sample $x$ from task $t$, the semantic class of $x$ is absent from the training set in task $\\hat{t}$.  Therefore, GEN is more advantageous in detecting mismatched samples.\n\n3) On CIFAR-100, performance is near the upper bound with few mismatched samples. MaxLogit with a high threshold effectively filters mismatches, so we adopt it for better performance.\n\n| Confidence | Sup-21K (CIFAR-100) | iBOT-21K (CIFAR-100) | Sup-21K (ImageNet-R) | iBOT-21K (ImageNet-R) | Sup-21K (ImageNet-A) | iBOT-21K (ImageNet-A) |\n|:----------:|:-------------------:|:--------------------:|:--------------------:|:---------------------:|:--------------------:|:---------------------:|\n| MSP        |        89.23        |        89.20         |        73.73         |         74.65         |        44.12         |         40.59         |\n| MaxLogit   |      **89.45**      |      **89.70**       |        73.77         |         74.97         |        44.20         |         40.60         |\n| Energy     |        89.24        |        89.29         |        73.60         |         75.12         |        44.14         |         40.73         |\n| GEN        |        89.35        |        89.55         |      **73.86**       |       **75.23**       |      **44.28**       |       **40.88**       |\n\n---\n\n**Q2: Increasing inference time** \n\n\n**A2:** Although the inference time is relatively increased, we achieve state-of-the-art performance across multiple datasets and pretraining settings. We acknowledge its practical significance and investigate the accuracy-efficiency trade-off in Section 4 of the supplementary material. In the future, further improving efficiency remains a promising direction.\n\n---\n**Q3&Q7: Selection of hyperparameters** \n\n\n**A3&A7:** \nOur method is robust to most hyperparameters, and their selection is generally straightforward:\n1) $\\gamma$ and $M$ are from generalized entropy (GEN).  According to GEN paper, we set $\\gamma$ in (0, 1). $M$ represents the top-$M$ highest probability in the distribution. Given that the minimum number of classes encountered during the incremental process is 20 (in the second task) for all datasets, we set it to 20.\n2) Given the inference cost, $N$ is set to 2 for all datasets.  As shown in Section 4 of the supplementary material, $N$ governs the trade-off between accuracy and efficiency.  $N$ can be adjusted based on available deployment resources.\n3) $K$ and $\\lambda$ in CTIRD are 5 and 0.2 for all datasets. We use common settings for $K$, i.e., top-5. In Eq. 8, $\\mathcal{L}_{\\mathrm{CTIRD}}$ is the sum of $K$ KL divergence losses in knowledge distillation. Drawing from prior work such as IRD [7], where the coefficient for a single distillation step is typically set to 1, we scale the overall loss by $1/K$, resulting in $\\lambda = 1/K = 0.2$.  The experiment in Fig. 6 verifies the effectiveness of our choice.\n4) The $\\tau$ affects the accuracy of mismatch detection. We search for the optimal value on the validation set.  Overall, our method is robust to $\\tau$.  As shown in Fig. 6 (c), treating all samples as mismatches with $\\tau=10^3$ still yields an improvement of 0.77 compared to selecting none ($\\tau=-10^3$).\n5) For the baseline methods, we employ the same selection strategy with identical hyperparameters to ensure a fair comparison.\n\n---\n\n**Q4:  The standard deviation in the ablation studies** \n\n\n**A4:** As shown in the three tables below, we supplement the standard deviations of three random seeds for some experiments in Tab. 3, 4, and Fig. 6 (a). The conclusions drawn remain consistent with those in the initial draft. We will include all standard deviations in the revision.\n\n### Tab. 3\n\n| Method                     | Sup-21K           | iBOT-21K          |\n|:----------------------------:|:-------------------:|:-------------------:|\n| Baseline                   | 71.47 ± 0.26      | 73.49 ± 0.31      |\n| Baseline + DRM             | 72.55 ± 0.10      | 74.11 ± 0.22      |\n| Baseline + DRM + CRM       | 73.38 ± 0.21      | 74.80 ± 0.15      |\n| Baseline + DRM + CRM + CTIRD | **73.86 ± 0.14** | **75.23 ± 0.21**  |\n\n\n### Tab. 4\n\n| Distillation | CIFAR-100 (Sup-21K) | (CIFAR-100 iBOT-21K) | ImageNet-R (Sup-21K) | ImageNet-R (iBOT-21K) |\n|--------------|-------------------|--------------------|--------------------|---------------------|\n| Logits       | 87.70 ± 0.14      | 88.10 ± 0.20       | 72.79 ± 0.28       | 74.32 ± 0.33        |\n| Features     | 87.29 ± 0.22      | 83.78 ± 0.18       | 71.19 ± 0.16       | 71.66 ± 0.13        |\n| IRD*         | 88.61 ± 0.15      | 88.86 ± 0.26       | 73.22 ± 0.18       | 74.68 ± 0.30        |\n| CTIRD        | **89.45 ± 0.23**  | **89.70 ± 0.13**   | **73.86 ± 0.14**   | **75.23 ± 0.21**    |\n\n\n###  Fig. 6 (a)\n\n| PTM      | 0.0              | 0.1              | 0.2               | 0.3              |\n|----------|------------------|------------------|-------------------|------------------|\n| Sup-21K  | 73.38 ± 0.21     | 73.40 ± 0.19     | **73.86 ± 0.14**  | 73.18 ± 0.23     |\n| iBOT-21K | 74.80 ± 0.15     | 75.08 ± 0.20     | **75.23 ± 0.21**  | 74.96 ± 0.17     |\n\n---\n\n**Q5: The hyperparameters of generalized entropy** \n\n\n**A5:** Considering advanced performance and advantages in detecting semantic drift, we choose GEN as the main confidence function. According to GEN paper, we set $\\gamma$ in (0, 1). $M$ represents the top-M highest probability in the distribution. Given that the minimum number of classes encountered during the incremental process is 20 (in the second task), we set it to 20. GEN’s hyperparameters influence the range and distribution of confidence scores, which in turn affect the optimal threshold. By fixing $\\gamma$ and $M$, we ensure consistency in the confidence scale, enabling a more stable threshold.\n\n---\n\n**Q6: Results of different PETs** \n\n\n**A6:** As shown in the table, using prompt as PET, we show the $A_N$ on two different pre-trained weights on the ImageNet-R dataset compared to the state-of-the-art prompt-based method. Our method HRM-PET still achieves state-of-the-art performance.\n| Method         | iBOT-21K         | DINO-1K         |\n|----------------|------------------|-----------------|\n| CPrompt       | 64.64 ± 0.87     | 68.25 ± 1.43    |\n| HiDe-Prompt    | 70.83 ± 0.17     | 68.11 ± 0.18    |\n| HRM-PET (Prompt version) | **72.11 ± 0.19** | **69.55 ± 0.28**|\n\n---"}, {"review_id": "f4cFp4v8Mv", "reviewer": "Reviewer_F34S", "summary": "This manuscript focused on the problem of continual learning with pre-trained models (Cl-PTM). Specifically, the authors focused the performance of task identity matching during the inference process, in which the task identity was predicted for each test sample, and then the task-specific parameters (e.g,  prompts) were selected to make the final classification. The authors investigated that the task-identity match between test samples and task-specific parameters was still challenging in the existing methods, and usually suffered from a high mismatching rate. To address this issue, the authors proposed HRM-PET, a parameter-efficient tuning based on a re-matching strategy, to improve the match accuracy in the existing methods. Specifically, the predicted distribution with the parameters selected through the initial matching was further used to improve the matching results, one by direct re-matching and another by confidence-based re-matching. Besides, the authors further introduced cross-task instance relationship distillation to better acquire the task-invariant knowledge.", "questions": "1. About the content between Lines 148–156: I find Section 3.2 clear until Eq. (5), but I struggle to grasp the intuition and derivation behind it. Could the authors elaborate on this equation in more detail? I have checked the supplementary materials but did not find a thorough explanation.\n2. Missing training details: Some training details are lacking, which makes it difficult to evaluate the computational complexity. For example, in Section 3.3, a subset $ \\mathbf{s}_i $ is selected for $ x_i $. However, the computation of this subset, as well as the subsequent steps in Eq. (8), seems to rely on $ e_k(x) $, i.e., inference using previous task parameters. Does this mean that for each new task $ t $, we need to compute the feature embedding $ e_k(x) $ for each sample under task $ t $ using all previous task-specific parameters $ p_k $? If so, this may introduce additional training overhead. Could the authors provide clarification and, if possible, quantitative evidence? It would also be helpful to include an algorithmic summary in the supplementary material.\n3. Minor expression improvements: In the first line of Eq. (7), I suggest using $\\hat{d}_{c_j}(x_i)$ instead of $\\hat{d}_{c_j}$ to improve clarity.\n\nOverall, I do not have major concerns about this work. I look forward to the authors’ clarifications, which will inform my evaluation during the rebuttal phase and further reviewer discussions.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. This study investigated the core and long-existing problem in CL-PTM, the potentially inaccurate task-parameter matching process during the test phase, which is crucial in this area.\n2. The motivation of the proposed method was clear and evidently supported by empirical observations (e.g., Figure 1-a). \n3. The experimental part is relatively complete. Most of the recent baseline methods and the commonly used benchmarks are included. The ablation studies were clear and supportive. The results succeed to support the motivation of the proposed method.\n\n## Weaknesses\n1. One intrinsic trait is the proposed method is the need for extra inference compared to existing methods, only with initial matching. However, according to the empirical analysis, it seems that the extra inference process did not introduce too much computational overhead.\n2. The expressions of some detailed steps are confusing. More clarification can be made to help the readers better understand the technical details. See the Questions part for more details.", "was_revised": true, "final_justification": "I appreciate the effort from the authors during the rebuttal period. After carefully reading the responses from the authors and the reviews from my colleagues, I decided to raise my rating to \"Accept\".", "rebuttal": "Thank you for your insightful comments and questions. \n\n**Q1: Clarification of inference time** \n\n\n**A1:** The minimal additional inference time overhead is mainly attributed to the following three factors: \n1) Not all samples require additional inference in the re-matching. For DRM, we perform extra inference only on samples where $\\hat{t}\\_{f}$ is not equal to $\\hat{t}_{s}$. For CRM, re-matching is conducted on samples whose confidence scores satisfy $\\tau$. Hence, the average inference time per image increases marginally. In the table below, we provide the **Ratio** of samples undergoing re-matching under each strategy for clearer interpretation.\n| Method            | $A_{N}\\uparrow$ | Time$\\downarrow$ | Ratio |\n|------------------|-----------------|------------------|-------|\n| Baseline          | 71.60           | 2.81             | -     |\n| Baseline+DRM     | 72.60           | 2.91             | 12.2  |\n| Baseline+CRM+DRM |  73.86           | 3.24             | 30.3  |\n2) In CRM, $N$ is set to 2, only one additional forward pass through the ViT is required.\n3) In the baseline, each sample requires two forward passes through the ViT: one for task identity prediction and another for class prediction. Re-matching process involves only a single forward pass through the ViT, so the inference time does not double that of the baseline.\n\n---\n\n**Q2: Explanation of Eq. (5)** \n\n\n**A2:** Eq.(5) describes how to find a more appropriate task identity when the sample $x$ is mismatched. Specifically, we first obtain the top-$N$ classes with the highest probabilities from the initial matching prediction distribution $\\hat{d}(x)$:\n\n\n$\n    \\text{$\\Gamma$} = \\underset{\\{c\\_j\\}\\_{j=1}^N}{\\text{argmax}}\\ \\hat{d}\\_{c_j}(x)\n$\n\nWhere $\\text{$\\Gamma$}$ is the set of the top-$N$ classes. Candidate task identities are obtained by converting all classes to task identity in $\\text{$\\Gamma$}$ with $\\mathcal{T}$. Then, we compare the confidence scores $E$ of the final prediction distributions generated by inference with the parameters $p$ corresponding to all task identities. The task identity with the highest confidence is selected as the corrected task identity $\\hat{t}_{s}$:\n\n$\n    \\hat{t}\\_{s} = \\underset{i \\in \\text{$\\Gamma$}}{\\mathrm{arg\\,max}}\\ E(g(h(x;p\\_{\\mathcal{T}(i)},\\theta\\_{ptm});\\theta\\_{g}))\n$\n\nThe final class prediction ${\\hat{y}}\\_{CRM}$ is derived from the prediction distribution corresponding to $\\hat{t}\\_{s}$:\n\n$\n    {\\hat{y}}\\_{CRM} = \\underset{i}{\\text{argmax}} \\, g(h(x;p\\_{\\hat{t}\\_{s}},\\theta\\_{ptm});\\theta\\_{g})\n$\n\n---\n\n**Q3: Missing training details** \n\n\n**A3:** In fact, for each new task $t$, we compute the feature embedding $ e_k(x) $ for each sample under task $t$ using the top-$K$ ($K=5$) previous task-specific parameters, rather than all old parameters. Specifically, we first utilize the pre-trained task classifier $g_{\\omega}$ to obtain the top-$K$ old task identities with the highest confidence scores for all samples. The time cost of performing a forward through the classifier is negligible. Subsequently, before training, each sample undergoes $K$ inferences to obtain $K$ features for knowledge distillation. Compared to training, the inference-only forward introduces minimal additional time overhead. \n\nExperimentally, we measure the training time of baseline and HRM-PET on ImageNet-R on an RTX 3090 with a batch size of 64.  As shown in the table, our method incurs only an additional 4.4\\% (i.e., 0.30 hours) of time overhead, which is acceptable. We will add the discussion in the revision.\n\n| Method        | Baseline    | HRM-PET (ours) |\n|---------------|-------------|----------------|\n| Training Time | 6.75 hours  | 7.05 hours     |\n\n---\n**Q4: The expressions of some detailed steps are confusing** \n\n\n**A4:** Thanks for the helpful suggestion. We promise to fix the confusing expression in the revision.\n\n---"}, {"review_id": "33AqOk7d2q", "reviewer": "Reviewer_b4of", "summary": "This paper introduces HRM-PET, a novel replay-free continual learning method using Parameter-Efficient Tuning. To enhance inference-time parameter matching, HRM-PET proposes a Hybrid Re-matching mechanism that intelligently utilizes the model’s prediction distribution. It further integrates Cross-Task Instance Relation Distillation to cultivate task-agnostic knowledge and align feature representations. Comprehensive experiments demonstrate HRM-PET’s superior, state-of-the-art performance with minimal additional inference overhead.", "questions": "See weaknesses.", "limitations": "The proposed method may incur substantial computational overhead.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\n\n1. The paper proposes an innovative and effective Hybrid Re-matching strategy (DRM & CRM) that cleverly leverages the prediction distribution to address the critical inference-time parameter matching inconsistency in PET methods.\n2. The paper provides comprehensive experimental validation and extensive ablations.\n3. The code is available\n\n### **Weaknesses**\n\n1. The paper lacks precise explanations for subscript notations in equations (3), (5), and (7), leading to comprehension difficulties. Specifically, $\\tilde{d}(x)$ appears to denote a class-level distribution, while $\\hat{d}(x)$ seems to represent a task-level distribution. However, the subscript representations in equation (5) and (7) vary, requiring explicit clarification for the meaning of each subscript.\n2. It is unclear which prompt/parameter (e.g., $\\hat{t}_f$ or $\\hat{t}_s$) is utilized to obtain $\\tilde{d}(x)$ in equation (3). A clear statement specifying the parameter source for this prediction distribution is essential for reproducibility and understanding.\n3. While inference time is discussed, the paper lacks a comprehensive analysis of training time, which appears to significantly increase due to multiple additional forward passes.\n4. The ablation study does not sufficiently clarify the independent contributions and necessity of each re-matching module. \n5. Given that CRM appears to be an independent module from DRM, the rationale for performing DRM prior to CRM, rather than directly applying CRM for all mismatches, remains unclear and warrants further explanation or empirical validation.\n6. The paper exhibits several technical and typographical errors that detract from its formal presentation and readability. For instance, each equation should conclude with appropriate punctuation (e.g., a period or comma). Additionally, on line 115, \"(LoRA)\" appears to be incorrect. The phrase \"Forfair, LoRA is employed as PET in HiDe-Prompt i.e. HiDe-LoRA and our method.\" on line 193 is unclear and grammatically awkward, requiring rephrasing for clarity.", "was_revised": true, "final_justification": "I thank the authors for their detailed rebuttal, which has effectively addressed my primary concerns. While I believe the paper could be further strengthened with more robust theoretical validation and additional refinement of the writing, the clarifications provided are sufficient. Accordingly, I have raised my score to 4.", "rebuttal": "Thank you for your insightful comments and questions. \n\n**Q1: Explanations for subscript notations and distribution**\n\n**A1:** Clarification of subscript: $\\hat{d}(x)$ and $\\tilde{d}(x)$ are all class-level distribution. $\\mathcal{T}$ is used to convert class in $\\hat{d}(x)$ into task identity in equations 5 and 7.\n\nClarification of distribution: $j$ in equation 3, $c_j$ in equations 5 and 7 all represent the index of the category.  We will unify the expression in revision.\n\n---\n\n**Q2: A clear statement specifying the parameter source for this prediction distribution**\n\n**A2:** In fact, we utilize $\\hat{t}\\_{s}$ in DRM to obtain $\\tilde{d}(x)$ in equation 3. DRM has corrected part of the task identities, using $\\hat{t}_{s}$ can reduce subsequent repeated calculations.\n\n---\n\n**Q3: A comprehensive analysis of training time**\n\n**A3:** During training, for each new task $t$, we compute the feature embedding $ e_k(x) $ for each sample under task $t$ using the top-$K$ ($K=5$) previous task-specific parameters. Specifically, we first utilize the pre-trained task classifier $g_{\\omega}$ to obtain the top-$K$ old task identities with the highest confidence scores for all samples. The time cost of performing a forward through the classifier is negligible. \n\nSubsequently, before training, each sample undergoes $K$ inferences to obtain $K$ features for knowledge distillation. Compared to training, the inference-only forward introduces minimal additional time overhead. \n\nExperimentally, we measured the training time of baseline and HRM-PET on ImageNet-R on an RTX 3090 with a batch size of 64.  As shown in the table, our method incurs only an additional 4.4\\% (i.e., 0.30 hours) of time overhead, which is acceptable. We will add the discussion in the revision.\n\n| Method        | Baseline    | HRM-PET (ours) |\n|---------------|-------------|----------------|\n| Training Time | 6.75 hours  | 7.05 hours     |\n\n---\n\n**Q4: The independent contributions and necessity of each re-matching module in the ablation study.**\n\n**A4:** As shown in the table, we conduct more detailed ablations on ImageNet-R. We have the following observations:\n\n1) Compared with the baseline, each re-matching module brings improvements, which demonstrates independence.\n\n2) Combining CRM and DRM achieves better performance compared with either CRM or DRM, which shows the necessity of each re-matching module.\n\n| Method                     | Sup-21K | iBOT-21K | iBOT-1K | DINO-1K | MoCo-1K |\n|----------------------------|---------|----------|---------|---------|---------|\n| Baseline                   | 71.52   | 73.60    | 72.98   | 70.43   | 67.98   |\n| Baseline+DRM               | 72.60   | 74.09    | 73.58   | 71.12   | 68.69   |\n| Baseline+CRM               | 72.80   | 74.35    | 74.01   | 71.32   | 68.80   |\n| Baseline+CTIRD             | 72.18   | 74.08    | 73.64   | 71.04   | 68.12   |\n| Baseline+DRM+CRM           | 73.40   | 74.75    | 74.18   | 71.85   | 69.10   |\n| Baseline+DRM+CRM+CTIRD     | 73.86   | 75.23    | 74.64   | 72.32   | 69.32   |\n\n---\n\n**Q5: The rationale for performing DRM prior to CRM**\n\n**A5:** As shown in the table, we conduct ablations on the relationship between CRM and DRM. We make the following validation:\n\n1) Combining CRM and DRM achieves better performance than using either DRM or CRM alone, demonstrating their complementarity.  In Figure 1 b, there is a partial intersection between correct and incorrect matching in the confidence-based distribution.  DRM compensates for the limitations on these samples, thereby combining CRM and DRM further improves the matching.\n\n2) When CRM and DRM are used together, the individual performance gains relative to the baseline are partially offset.  \nThis suggests a degree of overlap in the sets of samples selected for re-matching by the two methods. To minimize redundant computation, we apply CRM after DRM in our pipeline.\n\n3) The model's performance is not sensitive to the execution order of CRM and DRM, further probing their complementarity and necessity.\n\n| Method             | Sup-21K | iBOT-21K | iBOT-1K | DINO-1K | MoCo-1K |\n|--------------------|---------|----------|---------|---------|---------|\n| Baseline           | 71.52   | 73.60    | 72.98   | 70.43   | 67.98   |\n| Baseline+DRM       | 72.60   | 74.09    | 73.58   | 71.12   | 68.69   |\n| Baseline+CRM       | 72.80   | 74.35    | 74.01   | 71.32   | 68.80   |\n| Baseline+DRM+CRM   | 73.40   | 74.75    | 74.18   | 71.85   | 69.10   |\n| Baseline+CRM+DRM   | 73.61   | 75.10    | 74.22   | 71.80   | 69.18   |\n\n---\n\n**Q6: The technical and typographical errors**\n\n**A6:** Thanks for the helpful comment. Line 193 states: For fairness, both our method and HiDe-Prompt use LoRA as the PET, and therefore HiDe-Prompt is referred to as HiDe-LoRA in the table. We promise to correct grammar, formulas, unclear expressions, and other technical and typographical errors in the revision.\n\n---"}, {"review_id": "g77bHuBwJ4", "reviewer": "Reviewer_4szz", "summary": "This work proposes a hybrid re-matching method called HRM-PET for rehearsal-free continual learning with parameter-efficient tuning. Two types of re-matching are integrated into a whole framework. The direct re-matching is straightforward by replacing the incorrect initial task identity with the different predicted task identity. The confidence-based re-matching handles the more challenging case that direct re-matching fails to improve. The mismatched samples with wrong task identity are detected with a confidence threshold, and then the task identity of highest confidence is chosen from top-N identities. Moreover, cross-task instance relationship distillation is used to acquire task-invariant knowledge. Experiments on four datasets under five pre-trained settings demonstrate the competitive performance of HRM-PET.", "questions": "1. Fig. 5. I can not understand the underlying message what Fig. 5 wants to convey. Which one is better, and why?\n2. Fig. 4. I don't know what this figure aims to showcase. \n3. Fig. 2 appears to be a little mess. And where is the module of instance relationship distillation? \n4. Although re-matching improves the experiment performance for continual learning, is there any explaination or theory? Why can this method work? \n5. Can the re-matching procedure be casted in more reasonable framework, such as probabilistic reasoning, or VAE (I don't know if it can work, just mention some) something like that? Currently this method appears to be like \"if-then-else\" rules, lacking deep thoughts.", "limitations": "See above review for suggestions for improvement.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The experiment evaluation is extensive and solid. Results of tables and figures demonstrate the performance of the proposed method over SOTA algorithms. \n2. The re-matching idea between the current example (with its class label prediction) and the task ID appears to be novel. Two kinds of re-matching schemes seem to be complementary to each other. Moreover, cross-task instance relationship distillation enhances the overall performance. \n\nWeaknesses:\n1. Clarity. I feel very confused at several places when I read this paper first time. For example, if A and B are matched or re-matched, what are A and B in this hybrid re-matching method? It took me much time to answer such questions. Also, some notations should be clearly defined before using these notations. \n2. Quality. Personally I think the techniques used in this work are straightforward and lack some theoretical depth. For example, if \\hat{t}_f is not equal to \\hat{t}_s, the class label is predicted simply with \\hat{t}_s. In the confidence-based rematching, a simple threshold method is adopted. In the instance relationship distillation, the standard meaure of KL divergence is employed. From these aspects, these techniques appears to be not novel. \n3. Originality. As above stated, the techniques used in this paper seem to be less original, except the overal re-matching framework. \n\nMinor points:\n1. Page 7, line 216. \"between HRM-PET and three representive algorithms in Table 2\". \"three\" should be corrected as \"two\".", "was_revised": true, "final_justification": "I thank the authors for answering my concerns and questions. More experimental results were exposed in Author rebuttal, which make me satisfactory to the quality of this work. It appears that the authors' responses solve most of my concerns of this paper, and I would like to maintain my initial rating.", "rebuttal": "Thank you for your insightful comments and questions. \n\n**Q1: Clarity**\n\n\n**A1:** For PET-based CL methods, the model first predicts task identity for each sample $x$. Then, the corresponding PET parameters $p_\\{\\hat{t}_{f}}$ are retrieved from the parameter pool $P$, and are used for subsequent class prediction. When the predicted task identity equals the true task identity, we refer as **matching**. When they differ, it is referred to as **mismatching**.  In our method, we correct the wrong task identity, which we term **re-matching**. We will add the this clarity in the revision.\n\n---\n\n**Q2&Q3: Quality and Originality**\n\n\n**A2&A3:** Our HRM-PET is simple, yet effective and stable. The main contributions are notable: \n\nMost importantly, we explore the core and long-existing problem in PET-based CL, the potential inaccuracy in the task-parameter matching process during the testing phase. \n\nWe propose a hybrid re-matching framework to improve the accuracy of task identity prediction. CTIRD alleviates the reliance on matching, facilitating shared knowledge without compromising task-specific knowledge.\n\nMoreover, empirical results serve as evidence for the rationality and effectiveness of our method.\n\n---\n\n**Q4: Explanation of Fig. 5**\n\n\n**A4:** Fig. 5 compares the attention regions of the model with (Ours) and without CTIRD (W/o). The first column displays attention maps generated using the correct task identity, while columns 2–10 correspond to attention maps produced using incorrect task identities. Two main observations can be drawn:\n\n1) For the first column, with CTIRD, the model still focuses on the key regions, indicating that CTIRD does not compromise the plasticity.\n\n2) For columns 2–10, even when incorrect task parameters are used, the model with CTIRD still attends to the critical features, e.g., the lizard’s head and limbs, the bird’s beak. Hence. CTIRD enhances knowledge sharing and reduces the model’s reliance on task matching, demonstrating the superiority of CTIRD.\n\n---\n\n**Q5: Explanation of Fig. 4**\n\n\n**A5:** Fig. 4 illustrates the ratio of incorrectly and correctly matched samples that satisfy $\\tau$, relative to the total number of actual mismatched and matched samples, respectively, across each task in the incremental sequence. Specifically, the blue curve shows that over 70\\% of all mismatched samples are detected by $\\tau$, demonstrating that CRM can identify most mismatched samples during the incremental process. Additionally, the green curve indicates that 10–30\\% of correctly matched samples are misidentified as mismatches, highlighting the necessity of using Eq. 5 to determine whether task identity replacement is required.\n\n---\n\n\n**Q6: Instance relationship distillation in Fig. 2**\n\n\n**A6:** The module of instance relationship distillation is in the section with **training** at the bottom of Fig. 2. The loss of instance relationship distillation is between the old feature space and the new feature space. We will add the module name in the revision to improve clarity.\n\n---\n\n**Q7: Theory of re-matching improving the experiment performance for continual learning**\n\n**A7:** The accuracy of task identity prediction has been shown to be critical for improving CL performance, as demonstrated in HiDe-Prompt [77]. Assuming that in the class incremental learning (CIL) scenario, a total of $T$ tasks are defined as $D = \\{D_1, D_2, \\ldots, D_T\\}$. In $D_t$ of task $t$, $\\mathcal{X}\\_{t}$ and $\\mathcal{Y}\\_t$ are the domain and label of task $t$. Let $\\mathcal{X}\\_t=\\bigcup\\_{j}\\mathcal{X}_{t,j}$ and $ \\mathcal{Y}\\_t=\\{\\mathcal{Y}\\_{t,j}\\}$, where $j \\in \\{1,...,|\\mathcal{Y}\\_t|\\}$ indicates the $j$-th class in task $t$. \nGiven a pre-trained model $f\\_\\theta$, CIL aims to learn $P(\\boldsymbol{x} \\in \\mathcal{X}\\_{i,j}|D,\\theta)$ for the sample $\\boldsymbol{x} \\in \\bigcup\\_{k=1}^{t}\\mathcal{X}\\_{k}$, where $j \\in \\{1,...,|\\mathcal{Y}\\_t|\\}$ indicates the $j$-th class in task $t$. Based on the theorem of Bayes, the goal can be decomposed as:\n\n   $P(\\boldsymbol{x} \\in \\mathcal{X}\\_{i,j}|D,\\theta) = P(\\boldsymbol{x} \\in \\mathcal{X}\\_{i,j}|\\boldsymbol{x} \\in \\mathcal{X}\\_{i},D,\\theta) P(\\boldsymbol{x} \\in \\mathcal{X}\\_{i}|D,\\theta) $\n\nwhere $P(\\boldsymbol{x} \\in \\mathcal{X}\\_{i}|D,\\theta)$ is the task identity inference probability, $P(\\boldsymbol{x} \\in \\mathcal{X}\\_{i,j}|\\boldsymbol{x} \\in \\mathcal{X}\\_{i},\\mathcal{D},\\theta)$ is the within-task prediction probability. Hence, when the performance of task identity inference is improved, the performance of CIL will be enhanced. For more proof details, we recommend referring to HiDe-Prompt.\n\n---\n\n**Q8: More reasonable framework**\n\n**A8:** We sincerely thank for the insightful suggestion. The idea of framing re-matching within a probabilistic framework is indeed inspiring. Assume the input is $\\boldsymbol{x}$. The posterior probability that it belongs to class $y = c$ is $P(y=c | \\boldsymbol{x}, D, \\theta)$. We can further decompose it into task identity inference and within-task prediction:\n\n$\n\\begin{aligned}\nP(y = (i,j) \\mid \\boldsymbol{x}, D, \\theta) = P(y = (i,j) \\mid \\boldsymbol{x}, \\text{task}=i, D, \\theta) \\cdot P(\\text{task}=i \\mid \\boldsymbol{x}, D, \\theta)\n\\end{aligned}\n$\n\nwhere $y = (i,j)$ is the $j$-th class in task $t$. After training with PET, we obtain a parameter pool $\\boldsymbol{\\mathcal{P}} = \\{p_1, p_2, \\dots, p_T\\}$, where $p_i$ denotes the learned parameters specific to task $i$. Rematching involves inference with multiple $p$. To aggregate these predictions in a principled manner, we draw upon Bayesian Model Averaging (BMA), which supports combining models under uncertainty about model selection. Applying BMA, the final posterior is expressed as:\n\n$\nP(y=(i,j) | \\boldsymbol{x}, D, \\theta, \\boldsymbol{\\mathcal{P}}) = \\sum_{i=1}^T P(y=j | \\boldsymbol{x}, \\text{task}=i, \\theta, p_i) \\cdot P(\\text{task}=i | \\boldsymbol{x}, D, \\theta)\n$\n\nHowever, not all task parameters can be weighted equally. We hope that the correct task identity has a higher weight. Therefore, we calculate the weights $\\phi_t$ by the conditions in our strategy, such as the confidence of CRM:\n\n$\nP(y=(i,j) | \\boldsymbol{x}, D, \\theta, \\boldsymbol{\\mathcal{P}}) = \\sum_{i=1}^T P(y=j | \\boldsymbol{x}, \\text{task}=i, \\theta, p_i) \\cdot P(\\text{task}=i | \\boldsymbol{x}, D, \\theta) \\cdot \n\\phi_t\n$\n\nIn our paper, $\\phi_i$ acts as a binary gate (i.e., $\\phi_i \\in \\{0,1\\}$). Our rematching is uniformly implemented in the above equation. We will include this insight in the revisions.\n\n---"}]

Author rebuttal:

Meta-review: This paper proposes Hybrid Re-matching with Parameter-Efficient Tuning (HRM-PET) for rehearsal-free continual class-incremental learning. The idea of integrating direct and confidence-based re-matching is technically sound and addresses an important challenge. Reviewers appreciated the clarity of the experiments, the practical utility of the approach, and the detailed rebuttal, which helped resolve most of initial questions.

At the same time, some concerns remain: increased inference time,  additional algorithmic hyperparameters, and the lack of deeper theoretical justification, as well as  the writing that could be further refined. While the rebuttal alleviated several issues, these points were not fully resolved.

Overall, this paper received 1 Accept, 2 Borderline Accept, and 1 Borderline Reject. The reviews were split: some upgraded to an acceptance recommendation after rebuttal, while others maintained reservations. Balancing the strengths and remaining weaknesses, this paper makes a meaningful empirical contribution to continual learning, though with limitations. This meta-review agrees on both the positive aspects and the negative aspects pointed out by the reviewers, and leans towards **Accept** this submission.  The authors are encouraged to carefully address all concerns in their revision.
Decision: Accept (poster)