SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Models
Authors: Linglan Zhao
Xuerui Zhang
Ke Yan
Shouhong Ding
Weiran Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: continual learning
class incremental learning
pre-trained models
parameter-efficient tuning
Primary area: machine_vision

Abstract: Continual learning aims to incrementally acquire new concepts in data streams while resisting forgetting previous knowledge.
With the rise of powerful pre-trained models (PTMs), there is a growing interest in training incremental learning systems using these foundation models, rather than learning from scratch. 
Existing works often view PTMs as a strong initial point and directly apply parameter-efficient tuning (PET) in the first session for adapting to downstream tasks.
In the following sessions, most methods freeze model parameters for tackling forgetting issues. 
However, applying PET directly to downstream data cannot fully explore the inherent knowledge in PTMs.
Additionally, freezing the parameters in incremental sessions hinders models' plasticity to novel concepts not covered in the first session. 
To solve the above issues, we propose a Slow And Fast parameter-Efficient tuning (SAFE) framework.
In particular, to inherit general knowledge from foundation models, we include a transfer loss function by measuring the correlation between the PTM and the PET-applied model.
After calibrating in the first session, the slow efficient tuning parameters can capture more informative features, improving generalization to incoming classes.
Moreover, to further incorporate novel concepts, we strike a balance between stability and plasticity by fixing slow efficient tuning parameters and continuously updating the fast ones.
Specifically, a cross-classification loss with feature alignment is proposed to circumvent catastrophic forgetting.
During inference, we introduce an entropy-based aggregation strategy to dynamically utilize the complementarity in the slow and fast learners.
Extensive experiments on seven benchmark datasets verify the effectiveness of our method by significantly surpassing the state-of-the-art.

SAFE: Slow and Fast Parameter-Efficiet Tuning for Continual Learning with Pre-Trained Models
Linglan Zh
Xuerui Zhang
Ke Yan
Shouhong Ding
Weiran Huang
MIFA Lab, Qing Yuan Research Institute, SEIEE, Shanghai Jiao Tong University
Tencent YouTu Lab $ Zhejiang University
(linglanzhao, kerwinyan, ericahding)@@encent.co com
srzhang0121 edu. en, weiran. w huw.raaanannoottoooocoom
Abstract
Continual learning aims to incrementally acquire concepts in data streams while resisting forgetting previous knowledge. With the rise of powerful pre-trained models (PTMs), there is growing interest training incremental learning systems using these foundation models, rather than learning from scratch. Existing works often view PTMs as strong initial point and directly apply parameter-efficient tuning (PET) in the first session for adapting to downstream tasks. In the following sessions, most methods freeze model parameters for tackling forgetting issues. However, applying PET directly to downstream data cannot fully explore the in- herent knowledge PTMs. Additionally, freezing the parameters incremental sessions hinders models' plasticity to novel concepts not covered in the first session. To solve the above issues, we propose a Slow And Fast parameter- Efficien tuning (SAFE) framework. In particular, to inherit general knowledge from foundation models, we include transfer loss function by measuring the correlation between the PTM and the PFT--apliid model. After calibrating in the first session, the slow efficient tuning parameters can capture more informative features, improv- ing generalization to incoming classes. Moreover, to further incorporate novel concepts, we strike balance between stability and plasticity by fixing slow effi- cient tuning parameters and continuously updating the fast ones. Specifically, cross-classification loss with feature alignment is proposed to circumvent catas- trophic forgetting. During inference. introduce entropy based aggregation strategy to dynamically utilize the complementarity in the slow and fast learners. Extensive experiments on seven benchmark datasets verify the effectiveness of our method by significantly surpassing the state-of-the-arr. Code will be available at https:// // //gi thub. Saam/MIFA-LabbSSS
1 Introduction
Continual Learning (CL.) requires deep learning models incrementally incorporate new concepts from open-world data streams, while retaining previously learned knowledge. This presents a more challenging yet practical setting compared to traditional deep learning, which typically recognizes only closed-set categories. A variety of methods have been proposed for continual learning, in- cluding regularization-basse [17, 20, 47], rehearsal-based [5, 14, 31], and dynamic network-based approaches [1, 43, 44]. These methods often assume that the model is trained from scratch, resulting in substantial performance gap when compared the joint training upper-bound.
"Corresponding authors. "Equal contrihution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prozen/ / Leamahle Parameters
Parameter Efficient Tuning (PET)
Pre: Trained Model (PTM)
Inputs
Shared
First Sesslon (t=1)
PTM +
PTM +
PTM #
A degler
Transfer
MLP
Act
PET
S-PET
Norm
Tune dirocily
Inherit generalizahility
L
Shared
A Attecttoon
Sessont t (t>1
PTM +
PTM +
PTM @
Multi head
PET +
S S-PET
Guide
F-PET
Aention
Norm
Aggregatt
Froeze paramelers
Enhance plasticity
VPT
Inpuls
(a) Prevailing PTM-based CL works (left) v.s. Our framework (right)
(b) Classic PET blocks
Figure 1: Comparisons of (a) prevailing PTM-based CL methods [2, 23, 52] and our Slow And Fast parameter- Efficcien tuning (SAFE). The right part (b) illustrates several parameter-efficien tuning (PET) blocks: Adapter [6], Scale & Shift (SSF) [21], and Visual Prompt Tuning (VPT) [16].
Most recently, with the emergence of powerful pre-trained models, there has been growing interest in utilizing these foundational models as starting points for continual learning [23, 49, 52]. Pre-Trained Models (PTMs) which are often trained on vast datasets, encapsulate wealth of general knowledge, effectively enhancing the performance of deep learning models in continual learning scenarios. As shown the left part Fig. (a)), for adapting PTMs from pre- pre-training datasets continual learning datasets, prevailing works resort to parameter-efficient tuning (PET) techniques [6, 16, 21J in the first session. restrain catastrophic forgetting, in incremental sessions, these works set parameters of the adapted model frozen [2, 15, 23, 52] and only update the classification weights in training-free manner (i.e., without gradient updates) to accommodate novel classes.
However, the above methods have two main limitations. First, direct parameter-efficien tuning in the first session will largely lose the general knowledge inherent in PTMs. This is because PTMs are pre-trained on multitude of datasets while the dataset in the first session only contains relatively limited samples. Without proper transfer mechanisms, the knowledge from PTMs may be overwritten by the adapted model, which impedes the model's generalizability to unseen classes. Second, freezing parameters in the following sessions will hinder the plasticity of the model further absorb new concepts not learned in the first session, resulting in a sub-optimal solution. Although several efforts have been made to mitigate the second limitation, existing works still face certain constraints such as additional storage requirement [35, 49], inferior online branch performance [8] and linearly increased model complexity [54].
Based on the above observations, in this paper, we propose Slow And Fast parameter-Efficine tuning (SAFE) address existing challenges. In particular, SAFE demonstrates a unified framework that effectively inherits the generalizability of PTMs using slow parameter-effictn provides sufficient plasticity to learn tasc-fpec knowledge each incremental session using the fast one (F-PET). Meanwhile, SAFE does not require storing class distributions for data replay and only incurs constant-level additional computation and memory costs.
tuning (S-PET) and
To achieve the above goals, SAFE employs distinct strategies for the first and subsequent sessions. In the first session, we focus on explicitly transferring general knowledge from pre-traines models (PTMs) by introducing a knowledge transfer loss. This involves computing a correlation matrix between feature embeddings from the PTM and the model with parameter-efficient tuning (PET). The diagonal elements of this matrix are maximized to ensure that the features remain consistent across both models, effectively aligning the PET-applied model's performance with that of the PTM. Simultaneously, minimizing the off- off-diagonal elements reduces redundancy in the embeddings, enhancing feature discriminability. After this tuning process, parameters can retain generalizable knowledge from the PTM. To prevent forgetting this knowledge, these trained parameters are subsequently frozen, with only the classification weights being updated, thus designating this model the slow learner.

---

In the incremental sessions, to address the plasticity limitations of the slow learner, we introduce a fast learner capable of continuously integrating new concepts. Given the persistent challenge of catastrophic forgetting in continual learning, the slow learner guides the training of the fast learner. Concretely, we employ a feature alignment loss to minimize the distance between the embeddings of both learners on hypersphere. Additionally, cross-classification loss is proposed to ensure compatibility between the features of the fast learner and the classification weights of the slow learner, and vice versa. This approach allows the fast learner to assimilate new knowledge without storing exemplars or distributions, while also mitigating forgetting, For robust predictions, an entropy-based aggregation strategy is implemented during inference to dynamically leverage the complementary strengths of the slow and fast learners.
To summarize, the contributions of our paper are three-fold:
To inherit the generalizable knowledge in PTMs that has been overlooked in existing continual learning works, we propose to explicitly transfer knowledge from the PTM to a slow learner. Once trained, the slow learner can generalize well classes incremental sessions.
For improving the plasticity CL models, we include a fast learner with guidance from the slow learner to continuously incorporate novel concepts. Moreover, by aggregating both slow and fast learners into a unified framework SAFE, robust predictions can be further made.
The superiority of SAFE is validated on seven continual learning datasets where our method consistently achieves remarkable state-of-the-art performance. For example, our method surpasses the second-best result on ImageNet-A over 4%.
2 Related Work
Continual Learning. Traditional continual learning (CL. aims at continuously updating models with data streams from scratch. Existing strategies involve regularization-basse approaches [17, 20, 47, 51] which prevent forgetting by regularizing network weights or predictions, rehearsal-basee approaches which replay historical data stored in fixed-sized buffer [4, 9, 14, 25, 31], and architecture- approaches [1, 43, 44, 55] which dynamically expand models for novel classes. Among these methods, a recent attempt to preserve knowledge based on slow and fast complementary theory has been proposed [3, 22, 45]. Nevertheless, these approaches typically require adjusting all model parameters, which increases the computational burden of the learning process. Contrarily, our Slow And Fast parameter-Efficient tuning (SAFE) framework only requires much fewer learnable parameters as well as fewer resources, while obtaining more favorable performance.
Continual Learning with Pre-Trained Models. With the emergence of powerful pre-trained models (PTMs), it has become a hot topic to integrate pre-trained models with CL [29, 53] for better performance. Prompt-base based methods [30, 33, 39, 41, 42] utilize prompt tuning to adapt PTMs to new tasks. However, these methods are tailored for Transformers [7, 37] and require an expanding prompt pool with the arrival of new data. First session adaptation methods [2. 23, 52] adapt PTMs solely in the first session and then freeze the model afterward suppress forgetting [26, 32]. Nevertheless, these works lack plasticity for classes in subsequent sessions. Contrarily, another line of works focuses on continual adjustment [8, 35, 49, 54] to accommodate evolving information. However, the above approaches either require storing data distributions [35, 49] for replay, only obtain inferior online branch performance [8], or linearly increase complexity with incremental sessions [54]. Compared to existing works, our method provides a flexible framework that boosts generalizability by inheriting PTM's knowledge in the first session and maintains plasticity for incremental classes with constant complexity in replay-free manner.
3 Method
3.1 Problem Definition
Following previous works [8. 23, 52], in this paper, we mainly consider PTM--based CL under a class-incremental learning setting. Formally, the model is trained sequentially on a series of incre-
mental sessions, where D = { N C (x', (x) represents the I-th training set composed of N, samples, for 1 E 21, T). The sample and label space of D' are denoted by X and y where yd is disjoint between different sessions, i.e., Vi,j and i # j. yd nos
Ø. We follow the
3

---

Model Training
Mode Inference
Session 1
/PTM
Session t>1
PTM
Diagonal
Liug
Transfer
uggengaate
Cor: matii
edn
diggonaannal
D
PTM d
Embeddings
slow
Lls
klow
Jask
S-PET
slow
Classticcation weights
SoftM
Sessiont>1
S slow
(Imprinted)
Extropy
Entropy
PTM *
W slow
12s
S-PET
Guide
Fame protetool
Logils
Zelow
Zast
cos
protonal
Pre-Tniined Model (PTM)
D
PTM #
Wist
L + ass
Slow / Fast Parameter
Efficiient Tuming (PET)
F-PET
fast
(Leamable)
Prozen / Leamablee
Figure 2: An overview of our SAFE framework. In the first session, PTM transfers knowledge the slow learner for better generalization. In sessions t >> 1, the fast learner is guided by the slow learner for enhanced plasticity. During inference, robust predictions are made by dynamic aggregation.
replay-free setting, where only D' is accessible in session t. After training in the t-th session, the ... U U In addition, we also
model is evaluated on all the seen classes so far: y = y² U)22... ...... validate our method domain-incremennal learning setting, where the data distribution between sessions shifts significantly, i.e., Vi,j and i * j. P(x") * ). =.V
3.2 Overall Architecture
For tackling the stability dilemma in CL, draw inspiration from the complementary learn- ing systems theory [19] to develop Slow And Fast parameter Efficient tuning (SAFE) framework, as depicted in Fig. In the first session, the slow learner tuned inherit the general knowledge from PTM and is frozen afterward. In the following sessions. the slow learner only updates its classi- fication head using imprinted weights [28], which acts like the neocortex to slowly incorporate novel knowledge without forgetting. Complementary to this, the fast learner with learnable parameters rapidly encodes novel information as the hippocampus for adapting to new classes.
Formally, features extracted from PTM, slow learner and fast learner are denoted as = (z E Rd where / (PTM, slow, fast) and d is the feature dimension. To leverage the knowledge of PTMs with few learnable parameters and resources, feature extractors for the slow and fast leamers are trained using parameter-efficient tuning (PET) [6, 16, 21] which are referred as S-PET and F-PET, respectively. Consistent with prior works [23, 52, 54], we mainly consider three types of PETs: Adapter [6]. SSF [21], and VPT [16], shown in the right part of Fig.
The classification weights in session É for the slow and fast learners symbolized by Wi E di I (slow, fast), where [) is the number classes seen so far from session to session For the slow learner, Wstow is learned in the first session and expanded using feature centroids of training samples within the same classes [28] afterward to preserve learned general knowledge. Contrarily, Wiast is trainable as CL progresses for the plasticity purpose.
In the following sections, we provide the details of slow and fast learner training in Section 3.3 and Section 3.4. After that, discussions about model inference are presented in Section 3.5.
3.3 Slow Learner
Benefiting from pre-training on large-scale resources, pre-trained models (PTMs) inherently possess strong generalizability for downstream tasks. Previous works [23, 35, 52] typically view the PTM as a preferable starting point for continual learning. To bridge the distribution gap between pre-training datasets and downstream datasets, these methods often directly apply PET to PTMs.

---

However, without proper transfer mechanisms, models directly tuned on downstream data cannot effectively inherit the general knowledge from PTMs. More seriously, the intrinsic knowledge in PTM may be overwritten during adaptation to the recent dataset, since it often contains relatively limited samples. To solve the above issues, we propose to effectively squeeze out information from PTMs and explicitly transfer it to adapted models.
Concretely, in the first session, we calculate the cross-correlation matrix M [Rd between the features of the slow learner and the PTM:
N.
Mi
(PPTM(())] [dstow (ftk) )li+ No
where N₂ is the batch size, d is the feature dimension and denotes clement-wise multiplication Moreover, and] j index the dimensions of the features and matrices. In fact, the correlation matrix characterizes the relationship between feature embeddings of PTM and the slow learner. The i-th row and j-th column of M measures the correlation between the i-th feature dimension (also termed as channel or pattern in the literature) of the PTM and the j-th feature dimension of the slow learner.
To encourage the PET-applied model to mimic the performance of the PTM, we maximize the elements in the diagonal. This maximizing term ensures the slow learner can learn invariant feature components that match the statistics of the PTM:
1 d
Ldiag
E M d
(2)
1=1
Additionally, we reduce the redundancy between patterns in embeddings to enhance discriminability. This can obtained by decreasing the off-diagonal elements M with Cydu'
1 d. (d iml
Lida
M
(3)
Combined with the classification loss Lets using cross-entropy (CE);
No. CB(W
Cels = No
slow O Dslow (Iz)), ys),
(4)
where "o" denotes matrix multiplication, the overall loss function during the first training session is defined as:
Lslow = Lels + Adiag Laing + Ardn Erdu-
(5)
In Eq. (5), Ading and Ardn are the balancing hyper-parameters. Intuitively, the joint optimization of three losses makes the adapted model simultancously acquire distributionnsseciffc knowledge based on Cels and inherit general knowledge of the PTM using Laiag and L,dn As a result, the slow model can better generalize to incoming classes even unseen in the first training session.
3.4 Fast Learner
Although solely using the slow learner with general features already obtains competitive performance, the plasticity of the model is hindered due to its frozen parameters in the following sessions. To strike a balance between stability and plasticity, we adopt the fast learner to continuously learn episodic information for novel classes. However, updating representations without data reply will lead to semantic drift [35, 46, 49], causing catastrophic forgetting of previously learned knowledge. Existing works address this problem either store additional data distributions [35, 49] or require sophisticated drift estimations after each session [35, 46). Compared to previous works, our method imposes no such constraints, and aligns the models before and after updates in single embedding space, essentially addressing semantic drift.
Following [14], in this paper, we use la normalization to map features and classification weights onto hypersphere before element-wist or matrix multiplication. Normalizaiinn omitted to simplify notation.
5

---

First, the fast learner is trained with guidance from the slow learner using feature alignment to preserve prior representations. Specifically, the distance of feature embedding from both models is minimized on hypersphere to alleviate forgetting:
1
Loos
(1 Coss (24), Ptast (zx)))
No
where N₂ is the batch size and cos denotes cosine similarity of two vectors.
Furthermore, we utilize cross-classificttion which contains fast-to-slow loss and a slow-to-fast loss to maintain previous decision boundaries. For fast-to-slow calibration, we feed features from the fast learner to the classification layer of the slow learner. This objective makes features from the fast model compatible with the decision boundaries of the slow one to suppress semantic drift. Moreover, since the classification weight vector of cach class can be viewed as prototype of that class [28, 34], we also use these vectors as inputs for further preserving knowledge from previous sessions:
No
Dia-1l Σ J=1
L12s
1 No
CE(W slow O (x:), V.) + D1=t11
CE(W slow slow O W Weltt.), J),
where w-(5) Rdd denotes the j-th column of W fast which is also the prototype for class j in the fast
fast learner. Similarly, slow-to-fast loss La2r can be derived by swapping the fast and slow terms in Eq. (7). After that, the cross-llasiffication loss can be defined as Leeef = Lis + La2i.
Along with the classification loss Lels in Eq. (4) applied to the fast learner, the optimization objective in the incremental phase is defined as:
Liast Less + Leeet + Aens Lrom;
where Acos is the balancing pppeerpprameter. The loss function Liast smoothly adapts the fast learner to new knowledge while enforcing consistency with previously acquired knowledge, which boosts the plasticity of the model without severe forgetting.
3.5 Model Inference
Since the slow learner inherits general knowledge and the fast learner contains task-adaptive knowl- edge, we can obtain robust predictions by utilizing the complementarity of them. We first introduce the inference using single learner and then provide aggregation strategy based on both learners.
Single-learner-Bart based Inference. Following previous work [23, 26], instead of directly using the classification weights W/ and features (xr). / (slow, fast} for prediction, we take advantage of second-order statistics and prototype information for better performance. Formally, given test sample x, the predicted logits of each learner 2 are calculated as:
2 W. @ (G + 31) @ h(( E Dire
where B is hyper-parameter for regularization, hx(x)) E R-M is projected feature of and classifi- cation weights Wi R Mx(iul is composed of summations of projected features with same class labels. Gram matrix G e IR MM is cumulated based on training data from session to t:
M ΣΣ 1
G
hu(z;) O T, hı(2z) = (W T O (z()))
(10)
In Eq. (10), Wrand RdxM is the projection matrix with each column sampled from (00,, o²1), tb a nonlinear activation function and / denotes the identity matrix. Mathematically, Eq. (9) defines a more general form of regular linear prediction. When Wrand is / and is not applied, it degrades to a ridge regression
[13]. Moreover, if G removed, the classifier further reduces to NCM [24].
Aggregation-basee Inference. As discussed in the above sessions, slow and fast learners excel in handling classes from different sessions. Due to its plasticity, the fast learner can better recognize categories from the latest several sessions but shows limited performance on the old ones caused by potential forgetting. Contrarily, despite limited novel concept adaptation, the slow learner can capture historical knowledge thanks to its stability. Intuitively, when dealing with proficient categories, the

---

model exhibits higher confidence in predictions. Motivated by this, we use entropy to measure the confidence and dynamically aggregate the logits for robust predictions.
Given a test sample, we compute the entropy of predictions using H = Pi log p; for each learner, obtaining Halow and Hiast: where entropy indicates less uncertainty in predictions, the confidence of each learner can be represented by [Olstow, Offast softmax[[>> Halow, - Hfatt)), where 7 is scalar control the peakiness of output distributions. After that, the aggregated logits Zaguregate automatically assign higher weights to predictions with higher confidence, and can be obtained using a convex combination:
softmax is predicted probability. As lower
Zuggregate = Osloww
Zeloow
+
Offast
Zeaast
(11)
Finally, the prediction is obtained using the index the max element in Zaggregate in session while using Zelow instead in session 1 since the fast learner is not available in that session.
4 Experiments
In this section, we first introduce the implementation details of our proposed method SAFE and then compare it to the state-of-the-art on seven popular benchmark datasets. After that, detailed ablative experiments are conducted to validate the effectiveness of each component.
4.1 Experimental Setups
Datasets and Evaluation.
Following previous methods [23, 52, 54], our evaluations are conducted
on seven benchmark datasets: CIFAR 100 [18], ImageNet-R (IN-R) [11]. ImageNet-A (IN-A) [12],
CUB200 [38], Omnibenchmark (OB) [50], VTAB [48] and DomainNet [27]. Previous state-of-
the-art PTM- CL methods are chosen for comparison, including L2P [42], DualPrompt [41],
CODAPrompt [33], ADaM [52], RanPAC [23]. SSIAT [35]. and SLCA [49]. We adopt final accuracy
Accy and average accuracy ACCoeg + Σi Acc; as evaluation metrics.
Implementation Details. Consistent to existing works [23]. we adopt VřT-B/I6-INIK and ViT- B/16-IN2IK the PTM and apply Adapter [6], SSF [21] or VPT [16] for parameter-efficient tuning (Appendix A). In each session, we train the model for 20 epochs using SGD optimizer, weight decay of 0.0005, momentum of 0.9, and cosine annealing schedule where learning rate starts from 0.01 and decays to 0. The batch size is set to 48. In addition, 3 in Eq. (9) selected based on the performance the training data similar to [23]. For other hyper-parameters used in method, we find Ading = 0.1, Xrdu = 100, Acos = 50, 7 = 1 is a reasonable set of default choices. Detailed hyper-parameter sensitivity analyses are provided in Appendix D.
4.2 Comparisons with State-of-The-Arts
In this section, we compare the proposed method SAFE with several state-of-the-art approaches across seven datasets: CIFAR100, 100, ImageNet-R, ImageNet-A, Omnibenchmark, CUB200, VTAB and DomainNet. For fairness, all methods are implemented with the same ViT [7] backbones.
The innseinncremetall learning results from the final session reported in Table 2. As shown in Table 2, our method consistently achieves the best performance among all bench- marks. Notably, we significantly surpass the second-best result on I by 4.4%. When compared to methods storing additional data distributions for replay [35, 49], our method is replay-free and can still outperform these methods by significant margin. In addition, we improve the average accuracy over six datasets by 2.1% compared to the previ- ous best approach [23]. The aforementioned superiority contribute to the generalizability and plasticity of our method within a unified framework.
Table 1: Performance on DomainNet.
Method
Final Acc.
1.2P [42]
40.2
S-iPrompts [40]
50.6
ADaM [52]
50.3
RanPAC [23]
66.6
Slow learner
67.04 04.04
Fast learner
67.
SAFE (ours)
67.82
learning, results on DomainNet with 6 different domains are summarized in Table 1. Our method SAFE can outperform the second-best result by 1.2%, demonstrating that the
For
7

---

proposed framework is applicable to scenarios where the data distribution of the first task diverges significantly from that of subsequent tasks.
Table 2: Performance comparisons on six class- inssemincremttlll learning datasets. The final accuracy (%) of each dataset is reported in the table, and the last column presents the averaged accuracy over all the datasets. Methods with/without data replay are noted using "w/" and "w/o", respectively.
Method
Replay
CIFAR
IN-R
IN-A
CUB
OB
VTAB
Avg
SLCA [49]
91.5
77.0
59.8
84.7
73.1
89.2
79.2
w/
SSIAT [35]
91.4
79.6
62.2
88.8
94.5
L2P [42]
84.6
72.5
42.5
65.2
64.7
77.1
67.8
DualProompt [41]
81.3
71.0
45.4
68.5
65.5
81.2
68.8
CODAPrompt [33]
86.3
75.5
44.5
79.5
68.7
87.4
73.7
w/o
ADaM [52]
87.6
72.3
52.6
87.1
74.3
84.3
76.4
EASE [54]
87.8
76.2
55.0
86.8
74.9
93.6
79.1
RanPAC [23]
92.2
78.1
61.8
90.3
79.9
92.6
82.5
SAFE (ours)
w/o
92.8
81.0
66.6
91.1
80.9
95.0
84.6
4.3
Ablation Study
To investigate the factors contributing to the success of SAFE, we validate the effectiveness of our key components: the slow learner (SL) in Section 3.3, the fast learner (FL) in Section 3,4, and the entropy-based aggregation Section 3.5. Experiments are primarily conducted on IN-A dataset.
Effectiveness of the Slow Learner. We assess the effectiveness the slow learner from three perspectives. Firstly, depicted in Table 3, when the slow learner is added the baseline [23], the final accuracy increases by 3.2% and the average accuracy increases by 2. 1.. This observation verifies that the slow learner can generalize well the incremental classes.
Secondly, we expect the slow learner to inherit generalizability from the PTM. To dive deeper into this aspect, we visualize the embeddings of five unseen classes and five seen classes by T-SNE [36] after the first session adaptation. As shown in Fig. 3, the embedding space of the slow learner exhibits distinct separation be- tween the seen and unseen classes. Note that the feature distributions with SL in the grey el- lipse become more separable compared with the baseline method. This illustrates the successful integration of generalization capabilities from the PTM into the slow learner.
* Canoeids
Clases.
O New Clascs
(a) Baseline
(b) Ours
Figure 3: Comparisons with T-SNE visualization.
Furthermore, explore other alternatives for transferring generalizability. including feature align- ment (FA) by distilling PTM's features, logits aligmment (LA) by distilling PTM's predictions, and second-order statistics alignment (SSA) by distilling PTM's covariance. Table 5 presents the average and final accuracy of the substitutions on IN-A, with the best results highlighted in bold. It is observed that our slow learner can consistently outperform these variations, validating its superiority.
Effectiveness of the Fast Learner. As shown in the third row of Table 3, compared to the slow learner, using only the fast learner can obtain 1.1% improvements in the final accuracy. This indicates
Table 3: Overall ablation study on IN-A
Table 4: Ablation study of aggregation.
Method
Final
Avg
Method
Final
Avg
Baseline
62.21
72.31
Features Concatenate
65.59
73.22
Slow Learner
65.44
74.41
Logits Add
65.90
73.31
Fast Learner
66.49
74.50
Logits Max
66.03
73.46
Slow & Fast Learner (SAFE)
66.56
74.71
Entropy- hased Aggregate
66.56
74.71

---

Table 5: Ablation study of the slow learner.
Table 6: Ablation study of the fast learner.
Method
Final
Avg
Method
buss
Cess
Final
Avg
Baseline
62.21
72.31
Baseline
62.21
72.31
Baseline
62.81
73.35
Finelume directly
8.16
30.73
Baseline LA
64.06
73.70
Finetune Luest
65.31
73.88
Baseline SSA
63.20
73.00
Finetune w/ Loo.
66.07
74.20
Baseline Colow (Slow Learner)
65.44
74.41
Fast Learner
66.49
74.50
87.5
85.0
82.5
Accuracy (%)
80.0
77.5
75.0
Slow Learner
Fast Leurner
72.5
Aggregation
70.0
00-19
20-39
40-59
60-79
80-99
100-119
120-139
140-159
160-179 180-199
All Classes
Class Indices
Figure 4: Validations on the necessity of the aggregation on IN-R. We provide detailed classification accuracy of test samples from different sessions. Results of the slow learner, the fast learner and SAFE are presented for comparison.
that the fast learner is properly guided by the slow learner, and thus can continuously adapt novel classes with suppressed forgetting.
Subsequently, present the necessity of each regularization term in the fast learner. As shown in Table 6, without Leer and Coos. the performance drops to lower than 10% due to catastrophic forgetting. To alleviate forgetting, both Lgest and Less are applied. Specifically, solely using Leeef results in an improvement of 3.1 compared to the baseline, while using only Leus yields a gain of 3.9% over the baseline. Moreover, with all the proposed loss functions, the fast learner can obtain the best performance, validating the effectiveness each regularization term.
Effectiveness of Aggregation. As shown in the last row of Table 3, the combination of the slow and fast learners presents the best result. This observation is consistent with the complementary learning systems theory [19] that memory necessitates the presence of both slow learner and a fast learner for improved performance.
0.70
Aggregation Weights
0.65
Slow learner
Fast learner
0.60
To gain deeper insights into the necessity of both learners, we elaborate on their final accuracy of classes from each
0.45
0.40
session. In Fig. 4, the slow learner, mimicking the neocor- tex, initially stores structured information and performs well on relatively old classes (0-119). Conversely, the fast learner, resembling the hippocampus, swiftly adapts novel concepts and excels in recent classes (120-199), From this perspective, combining these two complemen- tary learners leverages their strengths across the training process, resulting in superior model performance.
0.35
0.30
69-19
20.39
40-59
60-79
80-399
100-
130-1
139
140.11 150
16t-1
180-10
Class Indices
Figure 5: Aggregation weights for the
slow learner and fast learner on IN-R.
In addition, Fig. 5 illustrates how the aggregated model dynamically leverages the strengths of both learners. Concretely, the horizontal axis represents the class indices to which each test sample belongs, while the vertical axis shows the average aggregation weights of each learner assigned to these test samples. observed from Fig. that, for classes 120-199, the fast learner consistently shows higher weights, which is consistent with its superior classification accuracy in these classes as

---

depicted in Fig. 4. For classes 0-119, the slow learner obtains higher weights, generally aligning with its demonstrated stability and better performance on these classes shown in Fig. 4. By adaptively balancing the contributions of both learners, our method achieves harmonious trade-off between stability and adaptability.
Moreover, we undertake detailed comparisons to other merging strategies to validate the effectiveness of our aggregation choice. As depicted in Table 4, we compare our entropy-basee aggregation with three alternatives: feature concatenation, logits addition, and logits max. We report the final and average accuracy, where the results elucidate that the entropy- -aaed aggregation fully leverages both leamerrs and achieves the best performance.
4.4 Memory Usage
In this section, investigate the number of learnable parameters in different methods and report the -errnnntte-erfoomancc comparison. Since no exemplars are stored in our method, the primary storage cost attributed the train- able model parameters introduced by parameter- efficient tuning (PET). Although PET entails additional parameters, it is still small relative to overall size of the pre-trained model (PTM). -rrfrttrr-pefoorancceeee trade-
Final Accuracy
70
Accuracy (%)
50-
40
Moreover, as the off shown Fig. 6, our method SAFE utilizes similar scale of parameters existing PTM- based methods while achieving substantial per- formance improvements.
50
ADAM RasPAC Our SSIAT CODAPrompl EASE
Methods (parameter size from small to large)
Figure 6: Memory usage comparison.
Conclusion
In this paper, we introduced SAFE, a Slow And Fast Efffcietr-Efficient tuning framework for continual learning. Our approach leverages the inherent knowledge in pre-trained models (PTMs) while maintaining model plasticity for novel concepts. By incorporating transfer loss function, we ensure the preservation of general knowledge from PTMs. In the first session, we calibrate slow efficient tuning parameters enhance the model's ability generalize to new classes. To balance stability and plasticity, we fix the slow efficient tuning parameters and continuously update the fast ones, employing a cross-lassiffictionn loss with feature alignment to prevent catastrophic forgetting. During inference, we introduce an entropy-based aggregation strategy for dynamic utilization of the complementarity between the slow learner and the fast learner. Extensive experiments seven benchmark datasets demonstrate that our method significantly surpasses the state-of oftte-of-the-ar,, tht,, validating the effectiveness of our approach.
Limitations: Our approach built upon RanPAC [23], and as such, it shares some of the same limitations. For instance, our method relies on strong feature extractor to effectively inherit generalizability from PTMs, making it less suitable for scenarios where training needs to be per- formed from scratch starting from rather small tasks. Additionally, our method introduces three hyper-parameters to balance the loss functions during training, as previously discussed. While our experiments demonstrate that set of default values works well across the benchmark datasets evaluated in our work, we acknowledge that these choices might not be optimal when applied to datasets with essentially different statistical characteristics. Furthermore, slowly updating the slow learner periodically, rather than keeping it fixed in subsequent sessions, may further enhance the model's 's adaptability and could be promising direction for future research.
Acknowledgement
This project was funded by National Natural Science Foundation of China (62406192). The project and raw idea were initiated MIFA Lab Shanghai Jiao Tong University, while the major part of the work was completed at Tencent YouTu Lab.
10

---

References
[1] Davide Abati, Jakub Tomezak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2020.
[2] Kyra Ahrens, Hans Hergen Lehmann, Jae Hee Lee, and Stefan Wermter. Read between the layers: Leveraging intra-layer representations for rehearsal-free continual learning with pre-trained models. Transactions Machine Learning Research, 2024.
[3] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general con- tinual learning method based on complementary learning system. In International Conference on Learning Representations, 2021.
[4] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920-15930, 2020.
[5] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In European Conference on Computer Vision, pages 233 233-248,20 2018.
[6] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 16664-16678, 2022.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
[8] Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. unified continual learning framework with general parameter-efficient tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11483-11493, 2023.
[9] Jinwei Han, Zhiwen Lin, Zhongyisun Sun, Yingguo Gao. Ke Yan, Shouhong Ding, Yuan Gao, and Gui-Song Xia. sachor-based robust finetuning of vision-language models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 26919-26928, 2024.
[10] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter- fine-tuning for large models: A comprehensive survey, 2024.
[11] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages October 2021. [12] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 15262-15271, 2021.
[13] Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 1970.
[14] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 831 1-39,, 2019.
[15] Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-mode in continual learning. NewrlPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications, 2022.
11

---

[16] Menglin Jia, Luming Tang, Bor- Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, 2022.
[17] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetring in neural networks. Proceedings of the national academy of sciences, 1 114(13):352133526 2017.
et al.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[19] Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 20(7):512-533 2016.
[20] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern analysis and machine intelligence,
2947, 2017.
[21] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling shifting your features:
A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35:109-123 2022.
[22] Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class- incremental learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2544-2553, 2021.
[23] Mark D McDonnell, Dong Gong, Amin Parvanch, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 36, 2023.
[24] Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Generalizing new classes at near-zero cost. IEEE transactions on pattern analysis and machine intelligence, 35(1 (5(1)):264-2637 2013.
[25] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11321-11333 2019.
[26) Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: strong replay-free baseline for of the IEEE/CVF International Conference on Computer Vision, pages 18820-18830, 2023.
learning. In Proceedings
[27] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[28] Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5822-5830, 2018
[29] Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2021.
[30] Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. In The Eleventh International Conference on Learning Representations, 2022.
[31] Allvis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental elassifier and representation learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2001-2010, 2017.
12

---

[32] Yujun Shi, Kuangqi Zhou, Jian Liang, Zihang Jiang, Jiashi Feng, Philip HS Torr. Song Bai, and Vincent YF Tan. Mimicking the oracle: An initial phase decorrelation approach for class incremental learning. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 16722-16731, 2022.
[33] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Bascante-Bonille Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909-11919, 2023.
[34] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, 2017.
[35] Yuwen Tan, Qinhao Zhou, Xiang Xiang, Ke Wang, Yuchuan Wu, and Yongbin Li. Semantically- shifted incremental adapter-tuning is a continual vitransformer. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, 2024.
[36] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[38] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd
dataset. Technical report, 2011.
[39] Liyuan Wang, Jingyi Xie, Xingxing Zhang. Mingyi Huang, Hang Su, and Jun Zhu. Hierarchi- cal decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information Processing Systems, 36, 2023.
[40] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans- formers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022.
[41] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal- continual learning. In European Conference on Computer Vision, pages 631-648. Springer, 2022.
[42] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[43] Ju Xu and Zhanxing Zhu. Reinforced continual learning. Advances in neural information processing systems, 31, 2018.
[44] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference Learning Representations, ICLR 2018. International Conference on Learning Representations, ICLR, 2018.
[45] Sung Whan Yoon, Ye-Yeon Kim, Jun Seo, and Jaekyun Moon. Xtarnet: Learning to extract task- adaptivee representation for incremental few-shot learning. International conference on machine learning. 2020.
[46] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang. Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6982- 6991, 2020.
[47] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987-3995, 2017.
13

---

[48] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark, arXiv preprint arXiv: 19.004667, 2019.
[49] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slca: Slow learner with classifier alignment for continual learning pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19148-19158, October 2023.
[50] Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision represen- tation through the lens of visual realms. In European Conference on Computer Vision, pages 594-611. Springer, 2022.
[51] Linglan Zhao, Jing Lu, Yunlu Xu, Zhanzhan Cheng, Dashan Guo, Yi Niu, and Xiangzhong Fang, Few-shot glass-incremente learning via class-awar bilateral distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1888-118477 2023.
[52] Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class- incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, pages 1-21, 2024.
[53] Da-Wei Wei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey. In Proceedings of the 33rd International Joint Conference on Artificial Intelligence (ICCAI), pages 8363-8371, 2024.
[54] Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[55] Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. Model tailor: Mitigating catastrophic forgetting multi-modal large language models. In International conference on machine learning, 2024.
14

---

Appendices
A Parameter-Effiiien Tuning (PET)
Trained on vast amounts of data, models with billions of parameters exhibit remarkable performance across various tasks. However, the expansive scale and computation pose considerable challenges when customizing them for downstream deployment. From this perspective, parameter-effie -efficee-efficient tuning (PET) provides practical solution by effectively adapting the large pre-trained models with limited additional parameters.
Parameter-effice efficceett tuning selectively adjusts a small proportion of the model parameters while keeping the rest frozen. this way, pre-trained models (PTMs) can partially keep the generalization and deal with domain gaps at low resource cost [10]. Motivated by this, PTM-based continual learning models leverage PET their paradigm to achieve desirable results [23, 52, 54]. Typically, continual learning works leverage ViT-B/16- INIK and VTT-B/16-1 6 as the backbones and fine- tune the model mainly with three PET algorithms: Adapter [6], Scale & Shift (SSF) [21] and Visual Prompt Tuning (VPT) [16], which are introduced in the following:
Adapter:
Adapters are small additional layers inserted into the layers of PTM. Each adapter layer generally consists of three parts: down-projection layer Wdown Rd which reduces the input feature dimension, a non-linear activation function (e.g., ReLU), and an up-projection layer Wap E RRr which projects features back to the original dimension. Specifically, given an input Lxd, the output R is expressed as:
y MLP(x) ReLU( @ Wdown) Wup:
(12)
where L, d and r represent the length of the input feature sequence, original feature dimension and projected feature dimension. In the above equation, "O" denotes matrix multiplication.
Scale & Shift (SSF): SSF involves two main operations: scaling, which multiplies each feature by learnable vector to adjust its spread, and shifting, which adds trainable vector to each feature to change its central position. In the context of fine-tuning PTMs, SSF helps to normalize the feature distributions and adjust to new data. This improves performance and robustness by maintaining consistency in distribution. Specifically,
+ 3,
(13)
where y Rd and 3 IRd are the scaling and shifting vectors, respectively. Moreover, 5." represents element-wise multiplication.
Visual Prompt Tuning (VPT): VPT extends original input features with lightweight learnable tokens and the extended features will fed into subsequent transformer blocks of ViT [7] to obtain the final adapted embedding. Concretely, denote the learnable prompts as P E RKxd xd extended features can be expressed as:
[P,z]] where K is the length of the prompt and y E (k))) is the extended feature.
(14)
B Effects of PET to SAFE
In the main paper, we report the remarkable performance the proposed SAFE framework under the same PET setting [23]. In this section, demonstrate that the proposed approach is general framework that is compatible with diverse PET modules. Specifically, we combine SAFE with Adapter [6]. Scale & (SSF)(SSF) [21] and Visual Prompt Tuning (VPT) [16]. As depicted in Table 7, we report the final accuracy on six datasets compared with the baseline method [23].
As shown in Table 7, the proposed SAFE framework outperforms the baseline across various PET modules by a substantial margin. the baseline on ImageNet-A by over 4% with different PET modules. We also achieve performance improvements by 2.2% with Adapter, 3.1% with SSF, and 3.0% with VPT on ImageNet-R. These results demonstrate the general applicability of our framework across PET algorithms.
worth noting that proposed method consistently exceeds
15

---

Table 7: Performances of SAFE and baseline PanPAC [23] with three different parameter-efficien tuning (PET) modules on six datasets. The rows in shadow show improvements compared to the baseline. The best results are in bold.
Method
PET
CIFAR
IN-R
IN-A
CUB
OB
VTAB
Avg
Baseline
92.2
77.8
59.9
90.3
79.6
92.6
82.1
SAFE (ours)
Adapter
92.8
80.0
64.1
91.1
80.3
94.3
83.8
Improve
+0.6
+2.2
+4.2
+0.8
+0.7
+1.7
+1.7
Baseline
90.3
77.9
62.4
89.9
78.8
92.2
81.9
SAFE (ours)
SSF
91.6
81.0
66.6
91.0
79.8
95.0
84.2
Improve
+1.3
+3.1
+4.2
+1.1
+1.0
+2.8
+2.3
Baseline SAFE (ours)
90.0
76.7
61.2
89.7
79.9
91.6
81.5
VPT
79.7
65.7
90.8
80.9
93.4
83,8
Improve
+2.2
+3.0
+4.5
+1.1
+1.0
+1.8
+2.3
C
Pseudo-code
For the detailed training procedure of the slow learner in Section 3.3 and the fast learner in Section 3.4, summarize the pseudo-code of method SAFE training in Algorithm 1.
Algorithm Model Training in Incremental Session
Input: Model from session 1, training data D' from session l.
Output: Updated model in session 6.
Phase 1: Slow learner in session
2: Freeze pre-trained model parameters @PTM-
3: Randomly initialize classification weights Walow and efficient tuning parameters 6S.PET.
4: while not done do
5: {(x. v)) +- sample batch of data from D
6: Calculate the correlation matrix in Eq. (1) and losses Ldiag, frdn in Eq. (2), Eq. (3).
7: Calculate the overall loss function Lstow in Eq. (5).
8: Update (Wshow, 0s.PET) with gradients Lsloww
9: end while
10: Replace Walow with imprinted weights (i.e., feature centroids of each class in DD).
11: Freeze parameters (Walow; (..PET).
12:
13: Phase 2:
Fast Learner session > 14: Expand Wstow (Rdx - R dxiid with imprinted weights using &slow and Dt
15: Expand Wiast (Rix
dxxd) with imprinted weights using diast and D
16: Initialize the fast learner's efficient tuning parameters 8F-PET from session 1.
17: while not done do
18: ((x,y) + sample a batch of data from D
19: Calculate feature alignment loss Lexe in Eq. (6) and cross-classifictiin loss Leest in Eq. (7).
20: Calculate the overall loss function Liast in Eq. (8).
Update Wiast, (F.PTT with gradients V Lisst.
22:
end while
D Further Ablations
Hyper- Papee-Parrseters Sensitivity, Our framework SAFE includes 4 byyer-parameters Adiag and Ardn for the slow learner, Acse for the fast learner, and y for aggregation. In this section, we supply detailed hyper-pane arper-parameter sensitivity analyses on ImageNet-A. Results for Adiag and Ardn are depicted in Fig. while the results for Aoos are shown in Table 8. Moreover, Table presents experiment on It is observed that hyper-parameters remain relatively stable within certain range. For example, the slow learner can achieve satisfactory results with Ading the range from to 1, and Ardn the range from 100 to 500. The fast learner obtain good performance with A in the interval from
16

---

50 to 100. Moreover, the aggregation module works well by simply setting Y to 1. As result, we set Adiag 0.1, Ardn 100, Acos
50. 7
1 as the default choices of nyper-paramettrs as stated in
Section 4.1 1 of our main paper.
63,65
61:15
09.05
61.71
03:11
60.21
64197
65.11
Asing
02.01
62.15
62.44 48
62.67
62.26
62.01
61.42
17,67
16.26
19.15
56.00
57.93
550
3000
57.41
57.74
$7.47
57.41
57.35
57.14
57.21
0.0
0.1
1.0
10.0
100.0
500.0
1000.0
Aeda
Figure 7: Ablations of hyper-parameter sensitivity on Adiag and Arda for the slow learner.
Table 8: Ablation of Acos on the fast learner.
Acce
0.1
10
50
100
FL
18.56
21.33
40.49
65.20
66.49
66.08
Table Ablation of y on the aggregated model.
0
0.1
10
100
SAFE 65.90
66.36
66.56
66.50
66.24
66.03
Teacher Models for the Fast Learner: As discussed in Section 3.4, the fast learner is guided by the slow learner during adapting novel classes. In this section, provide additional experiments the choice of the teacher model which guides the training of the fast learner. We conduct comparisons on training the fast learner directly (None teacher), using the pre-trained model teacher (PTM) and using the fast learner from the last session as teacher (t 1). As shown in Table 10, utilizing the slow learner teacher model surpasses all the alternatives. This because the slow learner can provide generalizable knowledge to the fast learner and simultaneously alleviate forgetting.
Table 10: Ablation of the teacher model for the fast learner.
Fast learner Final
Teacher
Avg
None PTM
8.16
30.73
55.76
65,46
Fast learner (t 1)
63.66
74.25
Slow learner
66.49
74.50
E
Comparisons with RanPAC
While both the proposed SAFE and PanPAC [23] leverage PTMs for continual learning, they target different components of model. Specifically, RanPAC focuses on deriving decorrelated classifi- cation weights for the classification head with frozen features, whereas our method emphasizes the improvement of trainable feature embeddings within the feature extractor. Furthermore, there is distinct difference in the correlation matrices utilized by the two methods. The correlation coefficients
17

---

CIFAR
CUB
IN-A
OB
IN-R
DomainNct
VTAB
Figure 8: Visualization of seven benchmark datasets.
matrix in RanPAC, shown in Figure 2 of their paper, has dimensions RCxC, where C' denotes the number of classes in the classification head. In contrast, our method employs a cross-correlation matrix of dimensions Raxdd with d representing the feature dimension, detailed in Eq. (1).
In addition, we would like to emphasize that our method is orthogonal to RanPAC. In fact, our approach is built upon RanPAC, and as evidenced in Table 2 of our paper, our method consistently outperforms RanPAC by significant margin.
Visualization of Datasets
In this section, we provide visualization results of the seven evaluated datasets: CIFAR100 [18], ImageNet-R (IN-R) [11], ImageNet-A (IN-A) [12], CUB200 [38], Omnibenchmark (OB) [50], VTAB [48] and DomainNet [27]. As shown in Fig. SAFE can perform well datasets with various characteristics. It is noteworthy that SAFE is capable of scenarios where the data distribution between tasks shifts significantly. For example,
method also shows superior performance VTAB and DomainNet which comprise 5 and 6 distinct tasks, respectively.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the main contributions and scope of the paper, accurately reflecting the theoretical and experimental results presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper includes separate "Limitations" section (in Appendix) that dis- cusses the potential weaknesses of the proposed approach.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
The paper does not include theoretical results, hence this question is not applicable.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides detailed descriptions of the experimental setup, including datasets, hyperparameters, and evaluation metrics, allowing for reproducibility of the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Codes at htttp:://githhi ctt. Lab/SAFE.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.c cc/
cublic/guides /ttee/idde//ooeeeaaiiiieeeee (CodeSubmissiooooolo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //ni //nips. peips.cc/pulle public/guidess /ttpg//dddee/oeeeebmimiiiia sonddeSubmissioooollo.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all necessary details about the training and testing processes, including data splits, hyperparameters, and optimization methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper does not report error bars other statistical significance measures for the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides information about the computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://nette /ttp:r///////i//eeeeeee cc cc/public/ http:i////thiihhhieeeeeeeeeeeeee
Answer: [Yes]
Justification: The research conforms to the NeurIPS Code of Ethics, ensuring ethical considerations met throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper does not discuss the potential societal impacts, either positive or negative, of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
The paper includes no such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly credits the creators of existing assets used and specifies the licenses and terms of use.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo///aooo/o datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
The paper does not introduce any new assets, making this question not applicable.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve any research with human subjects, hence IRB approval is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "JmBXIcSCDU", "reviewer": "Reviewer_bXaj", "summary": "The paper introduces the SAFE (Slow And Fast parameter-Efficient tuning) framework for continual learning using pre-trained models (PTMs). The proposed approach combines slow parameter-efficient tuning (S-PET) to inherit general knowledge from PTMs and fast parameter-efficient tuning (F-PET) to adapt to new tasks in each incremental session. The paper validates the effectiveness of the SAFE framework through experiments on multiple datasets, demonstrating its superiority over state-of-the-art methods.", "questions": "The questions are left with weakness points,", "limitations": "The paper includes a limitation section in the appendix. But it does not cover actual limitations of the work. Instead, the discussed limitation is mainly the characteristics of general pre-trained model-based continual learning (i.e., using the pre-trained model), which is actually not a limitation. The authors may discuss the limitation related to the robustness of the hyper-parameters.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well-organized and easy to follow. \n- The method is well motivated and contains clear novelty comparing previous methods. \n- The paper provides extensive experimental validation on various benchmark datasets, showing improvements over state-of-the-art methods.", "weaknesses": "- The motivation and effects of the proposed techniques are unclear. \n    - For example, what are the specific effects of the cross-correlation matrix? What is the specific motivation? This design is quite similar to the de-correlation operation process in RanPAC, although with a different formulation. More solid analyses are required. \n    - The issue is still with the fast learner part. Although ablation studies are conducted for the components, the insights of the designs are unclear. For ablation studies, not only the terms should be ablated, but the designs, such as the usage of W_slow in the loss should be clearly discussed and analyzed. \n- Although the paper is generally well written, some parts have tedious descriptions and blurry presentations, such as line 58 - 81 in the introduction, which need to be improved. \n- The proposed method contains many hyper-parameters, such as the weights for the loss terms. Ablation studies for them are necessary and required. \n- The importance of aggregation for the performance should be discussed based on the ablation study. \n- The aggregation process requires weights for slow and fast results. Although they can be calculated automatically, it should be analyzed and demonstrated how the distribution of these weights for different data samples and datasets. \n\nThe paper can be a good paper. However many details are absent in the papers, making the delivered message unclear and the techniques not justified well. I will reconsider my score based on the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments. We will include these experiments and provide detailed analysis in the revised version.\n\nQ1: What are the specific effects of the cross-correlation matrix and its specific motivation? This design is quite similar to the de-correlation operation process in RanPAC.\n\nA1: The **motivation** behind the cross-correlation matrix 𝑀 is **twofold**. **First**, it aims to create loss functions that **facilitate the effective transfer of general knowledge from PTM** to the slow learner (SL), which is often overlooked in previous works. This transfer enables the SL to generalize well to novel classes that it has not encountered during training. **Second**, M allows for **theoretical analysis** using the Information Bottleneck (IB) theory [1], which can be represented as IB = I(f, θ) − βI(f,X), where X, f and θ denote images, features and model parameters. We show in Fig. 6 in PDF that loss functions derived from 𝑀 can be directly related to the corresponding terms in the IB theory.\n\nThe **effects of M are twofold** and its **superiority over other alternatives is confirmed in Tab. 4** in manuscript. **1)** By maximizing the diagonal elements, embeddings from PTM and SL become similar, making **SL inherit generalizability** from PTM. **2)** By minimizing the off-diagonal ones, output units of embedding contain **non-redundant information to enhance discriminability**.\n\nOur design is **different from RanPAC**. **1)** SL uses trainable features with M but RanPAC uses frozen features. **2)** SL employs M during feature training while de-correlation is operated on classification weights. **3)** 𝑀 incorporates PTM into the training process. **4)** Both SL and SAFE consistently outperform RanPAC.\n\nQ2: The insights of the designs of fast learner (FL) are unclear. For ablation studies, not only the terms but the designs, such as the usage of W_slow in the loss should be discussed and analyzed.\n\nA2: The **loss function of the FL** **comprises** **two components**: L_cos and L_s2f/L_f2s, which alleviates the forgetting of the feature extractor and classification head, respectively. **First**, **L_cos** regularizes the training of FL by **aligning features** from FL and SL **in a hypersphere to preserve prior representations**. **Second**, **L_s2f/L_f2s** are two symmetric losses that employ cross-classification to **maintain previous decision boundaries**. For example, the **first component of L_s2f** ensures that the features from FL are compatible with the classification weights of SL (W_slow). Since our work does not store data or data distributions for replay, in **the second term of L_s2f**, we make sure classification weights from FL (viewed as prototypes for old classes) are correctly classified by W_slow to mitigate forgetting previous knowledge.\n\nQ3: Some parts have tedious descriptions and blurry presentations, such as line 58-81 in the introduction.\n\nA3: We aimed to provide readers with comprehensive understanding of SAFE. We will enhance the clarity in not only lines 58-81 but the rest of the paper.\n\nQ4: Ablation studies for hyper-parameters in SAFE are required.\n\nA4: SAFE includes 4 key hyper-parameters: λ_diag and λ_rdn for SL, λ_cos for FL and *γ* for aggregation. We provide a reasonable set of default choices in line 246 of our manuscript.\n\nIn PDF, we supply **detailed hyper-parameter sensitivity analyses**. Results for λ_diag and λ_rdn are presented in Fig. 7(a), while the results for λ_cos shown in Tab. 9. Experiments for *γ* are in Tab. 10. It is observed that **hyperparameters remains relatively stable within a certain range**. For example, SL can achieve satisfactory results with λ_diag in 0.1-1 and λ_rdn in 100-500. FL can obtain good performance with λ_cos in 50-100. The aggregation module works well by simply set *γ* to 1. \n\nQ5: The importance of aggregation for the performance should be discussed.\n\nA5: In manuscript, we **validate the effectiveness of aggregation in Tab. 2**, which shows that the **aggregated model achieves both the best** final and average accuracy on IN-A. \n\nTo better illustrate its necessity, **experiments on 6 datasets are provided**. As shown in Tab. 7, the final accuracy of the **aggregated model** obtains **0.9% improvement** on IN-R and obtains **0.4% over 6 datasets**.\n\nQ6: The aggregation process requires weights for SL and FL. Although they can be calculated automatically, it should be analyzed and demonstrated how the distribution of these weights for different data samples and datasets.\n\nA6: **First**, **Fig. 7(b) in PDF shows the (softmaxed) average weights** of SL and SL. For test samples from session 1 to session 10, weights of FL increase monotonously while those for SL decrease. This indicates that **FL is more confident in newly learned samples while SL prefers historical ones**, which is consistent with our hypothesis that FL can capture short-term information and SL focuses on long-term structured knowledge.\n\n**Second**, **in combination with the results in Fig 4**, FL and SL primarily exhibits higher weights in sessions 7-10 and 1-6 respectively, where accuracy is higher. This indicates that the **aggregated model can dynamically leverage the strengths of both learners**, validating the improvements observed in the experiments.\n\nQ7: The authors may discuss the limitation related to the robustness of the hyper-parameters.\n\nA7: Our method introduces 3 hyper-parameters to balance loss functions in training. **Although** we find in experiments that **a set of default choices is suitable for 6 datasets** in manuscript, **it may become suboptimal when tested on datasets with significantly different statistical characteristics**. Promising future directions include designing effective hyper-parameter searching mechanisms or generating them based on the relation between PTM pre-trained datasets and downstream task data.\n\n[1] The information bottleneck method. arXiv physics 2000.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i8iRsBAkpB", "reviewer": "Reviewer_JgdG", "summary": "This paper proposes a novel paradigm for continual learning called SAFE, which utilizes both slow and fast parameter updates. The method focuses on continual learning with pre-trained models, using slow updates to preserve the generalization capability of the pre-trained model while employing fast updates to adapt to new downstream tasks. During inference, the two branches are aggregated to achieve more robust predictions. Experiments on 6 different benchmarks demonstrate that SAFE consistently outperforms existing methods, achieving state-of-the-art performance.", "questions": "1. I am curious about SAFE's performance in scenarios where the data distribution between tasks varies significantly, such as in DomainNet.\n2. What is the setup for the ablation study in Table 2? Does \"only the fast learner\" mean that only the SL is used for inference after training, or does it mean that only SL is involved in the training process?\n3. What is the setup for PET in SAFE? Does it include only one of Adapter, SSF, or VPT, or does it use all three simultaneously?", "limitations": "The method proposed in the paper is only applicable to scenarios where the data distribution of the first task is roughly similar to that of subsequent tasks.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper focuses on continual learning with pre-trained models, a highly valuable topic. By designing a slow learner, it retains the generalization capability of the pre-trained model.\n2. The paper also designs a fast learner to balance stability and plasticity, enabling the model to continually adapt to new downstream tasks.\n3. The paper conducts extensive experiments on multiple benchmarks, providing strong evidence for the effectiveness of the proposed method.", "weaknesses": "1. The slow learner assumes that the data distribution of the first task in the continual learning scenario is roughly similar to subsequent tasks, which is not always true in practical applications. When the data distribution of the first task diverges significantly from subsequent tasks, constraining the fast learner to the slow learner may yield negative results.\n2. The design of the method introduces too many loss function terms, which could make the model training overly complex. Additionally, the stability of these hyperparameters is not analyzed in the experimental section.\n3. The main experimental section lacks comparisons with the latest methods, such as DAP[1].\n4. The ablation studies show that the introduction of many components results in only marginal improvements. According to Occam's Razor, these components should be removed.\n\n[1] Jung D, Han D, Bang J, et al. Generating instance-level prompts for rehearsal-free continual learning[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11847-11857.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions. We will include the mentioned experiments and analysis in the revised version.\n\nQ1: The slow learner (SL) assumes that the data distribution of the first task in the continual learning (IL) scenario is roughly similar to subsequent tasks, which is not always true in practical applications. When the data distribution of the first task diverges significantly from subsequent tasks, constraining the fast learner (FL) to the SL may yield negative results.\n\nA1: **First, SL does not have the assumption.** Instead, SL is adapted solely in the first session with the aim of developing a generalizable model capable of being transferred to diverse subsequent tasks [1][2][3].\n\n**Second, constraining FL to SL does not yield negative results both theoretically and experimentally**. SL inherits generalizability from PTMs, which is **theoretically supported by information bottleneck theory** (Fig. 6 in PDF). **Experimentally**, **SAFE achieves SOTA on datasets where the data distribution of the first task diverges significantly from subsequent tasks**, such as VTAB (Tab. 1 in manuscript) and DomainNet (Tab. 7 in PDF). VTAB includes 19 assessment tasks across a variety of domains, while DomainNet features 6 common objects in different domains.\n\n**Third, we find that not constraining FL to SL yields worse results.** As shown in Tab. 11 in PDF, FL guided by frozen SL achieves SOTA.\n\nQ2: The design of the method introduces too many loss function terms, which could make the model training overly complex. Additionally, the stability of these hyperparameters is not analyzed.\n\nA1: **First**, only 4 additional loss functions accompanied with 4 hyperparameters are used in SAFE, maintaining a **reasonable level of complexity**.\n\n**Second**, we provide the **grid search results** in PDF: the results for λdiag and λrdn in Fig. 7(a), for λcos in Tab. 9, and for *γ* in Tab. 10. The **results remain relatively stable within a certain range, making the default set of hyperparameters** (Line 246 in manuscript) **effective** and simplifying the parameter tuning process.\n\nQ3: The main experimental section lacks comparisons with the latest methods, such as DAP.\n\nA3: **We did not compare SAFE with DAP because** DAP involves direct annotation of task identity, simplifying the difficulty during inference [4], which renders such comparisons unfair. We now present the **comparisons with DAP** in Tab. 12 in PDF and find **SAFE surpasses DAP across datasets**.\n\nQ4: The ablation studies show that the introduction of many components results in only marginal improvements.\n\nA4: **The improvement is not marginal in IL**. **First,** the improvement of 0.x% in **average accuracy** **reflects a** **consistent increase** of 0.x% in the accuracy of each session, which is a noteworthy enhancement. **Second**, other works in PTM-based IL also **highlight that the modest improvements are impactful.** For instance, In EASE[5], a notable PTM-based IL work accepted at CVPR2024, the final acc exceeded the competitor by 0.03%, with the average acc increasing by 0.08% on CUB. Additionally, EASE surpassed the second-best result by 0.06% on OmniBenchmark. Another example is SSIAT[3], which was also accepted by CVPR 2024. In experiments on CIFAR, SSIAT achieved SOTA with an increase of 0.09% in final and 0.06% in average acc.\n\n**These components should not be removed, not only because of the performance improvement but collaboration**: 1) Based on **complementary theory**, SL leverages the intrinsic knowledge in PTMs akin to structural knowledge in **neocortex**, while FL continuously learns episodic information for novel classes similar to the **hippocampus**. 2) As shown in Fig. 7(b) in PDF and Fig.4 in manuscript, **FL compensates for the plasticity issue of SL** and shows superiority in new classes. 3) **SL guides the adaptation of FL**, and Tab. 11 in PDF demonstrates that SL is the best teacher for FL. 4) The designs of Ldiag and Lrdn in SL are **theoretically** supported by information bottleneck theory (IB). As Fig.6 in rebuttal PDF shows, Ldiag and Lrdn can be directly related to the corresponding terms in the IB theory. 5) Lf2s/s2f and Lcos in FL **work together** to boosts the plasticity. Lcos regularizes the training of FL by aligning features from FL and SL in a hypersphere to preserve prior representations. Ls2f/Lf2s are two symmetric losses that employ cross-classification to **maintain previous decision boundaries**.\n\nQ5: SAFE's performance in scenarios where the data distribution between tasks varies significantly e.g.DomainNet.\n\nA5: **SAFE is applied to datasets with significantly varying data distributions between tasks**, such as VTAB and DomainNet (Tab. 8 in PDF), and achieves SOTA. VTAB comprises 5 assessment tasks spanning various domains, including natural, professional, and structured images.\n\nQ6: Does only the fast learner in Table 2 mean only SL is used for inference after training?\n\nA6: It means only the slow learner is used for inference after training.\n\nQ7: Does the setup for PET in SAFE include only one of Adapter, SSF, or VPT, or does it use all three simultaneously?\n\nA7: First, following previous work [5][6], we employ only one PET in SAFE for the experiments, and the specific PET used is consistent with that in [5].\n\nSecond, we validate in Tab. 6 that SAFE is compatible with various PET methods, including Adapter, SSF, and VPT, thereby demonstrating its generalizability.\n\n[1] Forward compatible few-shot class-incremental learning. CVPR2022.\n\n[2] First session adaptation: A strong replay-free baseline for class-incremental learning. CVPR2023.\n\n[3] Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer. CVPR2024.\n\n[4] Continual learning with pre-trained models: A survey. ArXiv2024.\n\n[5] Expandable subspace ensemble for pre-trained model-based class-incremental learning. CVPR2024.\n\n[6] Ranpac: Random projections and pre-trained models for continual learning. NeurIPS2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2FHv3OTVoj", "reviewer": "Reviewer_Psm6", "summary": "The paper proposes a novel method named SAFE (Slow And Fast parameter-Efficient tuning) to tackle challenges in continual learning. SAFE introduces a unified framework that combines slow parameter-efficient tuning (S-PET) for inheriting general knowledge from pre-trained models (PTMs) and fast parameter-efficient tuning (F-PET) for acquiring task-specific knowledge in incremental sessions. SAFE demonstrates state-of-the-art performance on six benchmark datasets, showing significant improvements over existing methods.", "questions": "1. Why did you decide to train the slow adapter only in the initial session? Would there be any benefit to periodically updating it with new data to improve adaptability?\n2. How did you determine the values for hyperparameters such as λ, Ldiag, Lrdn, and Lcos? Could you provide more details on the process and any automated search methods used?\n3. Can you elaborate on the specific contributions of each loss function (knowledge transfer loss and feature alignment loss) in the overall performance of SAFE? How do these losses interact, and are there any potential conflicts between them?\n4. Have you considered comparing SAFE with other PET methods that address the stability-plasticity trade-off, such as O-LoRA [1]?\n[1] O-LoRA: Orthogonal Subspace Learning for Language Model Continual Learning, CVPR", "limitations": "The authors acknowledge that SAFE's training process involves complex loss functions and hyperparameters, which could be a barrier to practical implementation.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. SAFE introduces an innovative approach by integrating both slow and fast parameter-efficient tuning within a unified framework. This dual-tuning mechanism effectively addresses the trade-off between stability and plasticity in continual learning.\n2. The paper presents a thorough and well-structured evaluation on six benchmark datasets, clearly demonstrating the effectiveness and superiority of SAFE over existing methods. \n3. The approach does not rely on storage class distributions for data replay and maintains constant computational and memory overheads.", "weaknesses": "1. Training the slow adapter only in the initial session may restrict its ability to adapt to new tasks, potentially limiting the overall flexibility of the model.\n2. Complex Loss Functions and Hyperparameters: The proposed method involves multiple loss functions and hyperparameters, making the optimization process complex and potentially challenging to tune in practice.\n3. While the method shows promising results, additional experiments or ablations specifically addressing the impact of each loss function and hyperparameter choice would strengthen the validation of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions. We will include the mentioned experiments and detailed explanations in the revised version.\n\n \n\nQ1: Training the slow learner (SL) only in the initial session may restrict its ability to adapt to new tasks, potentially limiting the overall flexibility of the model. Why did you decide to train the SL only in the initial session? Would there be any benefit to periodically updating it with new data to improve adaptability?\n\nA1: **First**, **training SL only in the initial session does not impede the overall flexibility** because: **1)** It is a **common practice in incremental learning** [1][2][3] to **develop a generalizable model capable of being transferred to diverse subsequent tasks**. **2)** The **SL** inherits the generalizability of PTMs, providing the **flexibility to adapt to unseen categories** in the future. This is **verified through: (a)** Experiment results show SL **surpasses the baseline by 3.23%** in Tab. 3 in manuscript and performs will on DomainNet in Tab. 8 in PDF. **(b)** Visualization via t-SNE shows **better clustering results on unseen classes** in Fig. 3 in manuscript **3)** In alignment with **complementary theory**, Fig. 7(b) in PDF demonstrates the **adaptation complementary of SL by FL**, as the latter quickly adapts across sessions.\n\n**Second**, we decided to **train SL only in the initial session rather than periodically updating** because: **1)** Continuously updating may **lead to the forgetting of general knowledge** **inherited** from PTMs. **2)** The performance of the **FL** may **deteriorate without the guidance of frozen SL** as shown in Tab. 11 in PDF. **3)** We **have experimented** with updating SL using EMA or a small learning rate, but **observed no noticeable improvement**. We will address this problem in future work.\n\n \n\nQ2: The proposed method involves multiple loss functions and hyperparameters. How did you determine the values for hyperparameters? Could you provide more details on the process and any automated search methods used?\n\nA2: **First**, only **4 additional loss functions** accompanied with **4 hyperparameters** are used, maintaining reasonable complexity. We have **provided a set of valid default parameters** for implementation (line 246 in manuscript).\n\n**Second**, we provide the **grid search results** in PDF: the results for λdiag and λrdn of SL of Fig. 7(a) in PDF, for λcos of FL in Tab. 9, and for *γ* of aggregation in Tab. 10.\n\n**Third**, the **results remain relatively stable** within a certain range, making a **default set of hyperparameters effective** and simplifying the parameter tuning process. For example, SL can achieve satisfactory results with λdiag in 0.1-1 and λrdn in 100-500. FL can obtain good performance with λcos in 50-100. The aggregation module works well by simply setting γ to 1.\n\n \n\nQ3: Additional experiments or ablations specifically addressing the impact of each loss function and hyperparameter choice would strengthen the validation of the approach. \n\nA3: **First,** we **add the following experimental results** in PDF to verify **components**: **1)** Tab. 7 to demonstrate the significant performance improvements across various datasets, elucidating the necessity of each module. **2)** Fig. 7(b) illustrates how the aggregation mechanism effectively leverages the complementary strengths of SL and FL. **3)** Tab. 8 shows the superiority of SAFE on dataset with domain gap (DomainNet).\n\n**Second**, regarding **hyperparameter** selection: **1)** The grid search results are provided in PDF: Fig. 7(a) displays the results for λdiag and λrdn of SL, Tab. 9 presents the results for λcos of SL, and Tab. 10 shows the results for *γ*. **2)** The results indicate that **a default set of hyperparameters (line 246) is practically effective, as they remain relatively stable within a certain range**.\n\n \n\nQ4: Elaborate on the specific contributions of each loss function in the overall performance of SAFE. How do these losses interact, and are there any potential conflicts between them?\n\nA4: **First,** based on **complementary theory**, **SL and FL are complementary counterparts**, **further substantiated** by Fig. 7(b) in rebuttal PDF and Fig.4 in manuscript.\n\n**Second**, the knowledge transfer **loss functions in SL** work collaboratively to **facilitate the effective transfer of general knowledge from PTM**, which is **proposed for the first time**. **Moreover,** the designs of Ldiag and Lrdn in SL are **theoretically** supported by information bottleneck theory (IB). We show in Fig.6 in rebuttal PDF that Ldiag and Lrdn can be directly related to the corresponding terms in the IB theory.\n\n**Third,** Lf2s/s2f and Lcos in FL **work together** to boost the plasticity **without storing data or data distributions for replay**. **Lcos** regularizes the training of FL by **aligning features** from FL and SL **in a hypersphere to preserve prior representations**. **Ls2f/Lf2s** are two symmetric losses that employ cross-classification to **maintain previous decision boundaries**.\n\n \n\nQ5: Have you considered comparing SAFE with other PET methods that address the stability-plasticity trade-off, such as O-LoRA.\n\nA5: We **did not** **compare** SAFE with O-LoRA due to the following reasons: **1)** They are designed for **different application domains**: O-LoRA is specifically tailored for language models, while SAFE is focused on computer vision. **2)** Comparing **disparate PET methods**, such as LoRA and Adapter, may not be entirely equitable. However, **SAFE has demonstrated robustness and generality** across various PET approaches. We are willing to **explore the extension** of SAFE in future work.\n\n \n\n[1] Forward compatible few-shot class-incremental learning. CVPR2022.\n\n[2] First session adaptation: A strong replay-free baseline for class-incremental learning. CVPR2023.\n\n[3] Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer. CVPR2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J1A5hPPSjO", "reviewer": "Reviewer_RFpE", "summary": "This paper proposes a slow and fast parameter-efficient tuning method for continual learning. Slow learner is learned on the first session and fixed with a transfer loss. Fast learner is continually updating for new tasks. Slow learner and Fast learner are further restricted to avoid forgetting.", "questions": "Can you explain the marginal improvements of some components and how they can support the main contribution of the paper?", "limitations": "Yes, it is addressed in the appendix.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The idea of slow and fast learning has been interesting for continual learning, such as in SLCA. This work leverages pre-trained models to design the slow and fast learner, which is novel. And the experimental results compared to other baselines are significant.", "weaknesses": "Although the main idea is sound, it is not well supported by the experimental results. For instance, In Table 2, it shows that with only Fast learner the performance is already as good as the proposed method SAFE. In Table 3, the gain of the aggregation method is marginal compared to some straightforward baselines.  In Table 5, without using $L_{f2s}$ and $L_{s2f}$, the Fast learner obtains the similar results. All these components are claimed as main contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We will include these experiments and provide detailed explanations of the results in the revised version.\n\n \n\nQ1: The performance with only fast learner (FL) is already as good as the proposed method SAFE, which may not support the main idea.\n\nA1: **First**, to further validate the effectiveness of the slow learner (SL) in SAFE, we present **additional ablation results on six datasets** in Tab. 7 in rebuttal PDF. Notably, **1)** SAFE's final accuracy exceeds that of the FL by an average of 0.4% across datasets. **2)** Specifically, **in ImageNet-R, SAFE surpasses the FL by 0.93%** in final accuracy and **1.55%** in average accuracy.\n\n**Second**, it is crucial to recognize the notable progress in continual learning, particularly given the challenge of balancing stability and plasticity. **For example, EASE [1]** improved final accuracy by **0.03%** over the second-best result and increased average accuracy by **0.08%** on the CUB dataset. On OmniBenchmark, EASE outperformed the second-best result by **0.06%**. Although these improvements might seem minor, EASE has been influential and well-regarded, having been **accepted by CVPR 2024 and cited nine times in three months**. **Similarly, FeCAM** [2], **accepted by NeurIPS 2023**, achieved a **0.2%** final accuracy gain on the ImageNet-Subset, although it **did not surpass recent exemplar-based SOTA** on CIFAR. Compared to these recent SOTAs, **our improvements are significantly more substantial**.\n\n**Third**, we emphasize that the final accuracies of the proposed **SL and FL exceed the second-best** baseline by **3.24% and 4.29%**, respectively, demonstrating **both of them are effective**.\n\n \n\nQ2: The gain of the aggregation method is marginal compared to some straightforward baselines in Table 3. Without using Lf2s and Ls2f, the FL obtains the similar results.\n\nA2: **First**, the **competitiveness** of the proposed aggregation lies in its **ease of implementation and effectiveness**. It achieves **at least a 0.53%** improvement in final accuracy compared with straightforward baselines, **without requiring complex modules or hyperparameters**.\n\n**Second**, the result without Ls2f/f2s represents **a relatively** **enhancement over** Lcos, which is more **challenging**. **Additionally**, when **using only Ls2f/f2s in FL**, the final accuracy increased by **3.1% compared with the baseline**, demonstrating its effectiveness.\n\n**Third**, it is important to note that **this improvement is significant**, especially **in the field of continual learning for PTMs**. For reference, **other markable PTM-based continual learning works** also demonstrate that even **small improvements are noteworthy**.\n\nFor instance, **EASE [1]**, which has been **accepted by CVPR 2024** as a notable PTM-based continual learning work, surpassed the second-best result in final accuracy by only **0.03%** and increased the average accuracy by **0.08%** on the CUB dataset. Additionally, EASE surpassed the second-best result by **0.06%** on OmniBenchmark. \n\n**Another example is SSIAT** [2], also **accepted by CVPR 2024**. In experiments on CIFAR, it achieved SOTA results with an increase of **0.09%** in final accuracy and **0.06%** in average accuracy.\n\n \n\nQ3: How each component can support the main contribution of the paper?\n\nA3: **First**, we emphasize that one of the main contributions is **leveraging PTMs to design both SL and FL**. \n\n**Second**, **SL and FL are complementary counterparts and cannot be separated** for the following reasons: **1)** Based on **complementary theory**, **SL** leverages the intrinsic knowledge in PTMs akin to structural knowledge in the **neocortex**, while **FL** continuously learns episodic information for novel classes similar to the **hippocampus**. **2)** As shown in Fig. 7(b) in the additional PDF and Fig.4 in the submitted manuscript, **FL compensates for the plasticity issue of SL** and demonstrates superiority in new classes. **3)** **SL guides the adaptation of FL**, and Tab. 11 in PDF demonstrates that SL is the best teacher for FL.\n\n**Third**, the **knowledge transfer loss functions in SL cannot be considered separately** because the design of two loss functions is **theoretically** **supported by information bottleneck theory (IB)**. We show in Fig.6 in rebuttal PDF that L_diag and L_rdn can be directly related to the corresponding terms in the IB theory, which is used to inherit generalizability and enhance discriminability respectively.\n\n**Fourth**, **the alignment of features**, i.e., L_f2s/s2f and L_cos, **work together** to boost the plasticity of the model without severe forgetting **because** they alleviate the forgetting of the feature extractor and classification head, respectively. **First**, **L_cos** regularizes the training of FL by **aligning features** from FL and SL **in a hypersphere to preserve prior representations**. **Second**, **L_s2f/L_f2s** are two symmetric losses that employ cross-classification to **maintain previous decision boundaries**. For example, the **first component of L_s2f** ensures that the features from FL are compatible with the classification weights of SL. Since our work does not store data or data distributions for replay, in **the second term of L_s2f**, we make sure classification weights from FL (viewed as prototypes for old classes) are correctly classified by SL to mitigate forgetting previous knowledge.\n\n \n\n[1] Expandable subspace ensemble for pre-trained model-based class-incremental learning. CVPR 2024.\n\n[2] Exploiting the heterogeneity of class distributions in exemplar-free continual learning. NeurIPS 2023.\n\n[3] Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer. CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the valuable comments from all the reviewers. We diligently provide detailed explanations for the questions raised in the respective comments section **point-to-point**.

In addition, **supplementary experiments and theoretical analyses** are incorporated into the one-page PDF attached to the global response. Specifically, the PDF encompasses the following:

1. **Detailed hyper-parameter sensitivity analyses.** This supports our claim in the paper that a set of default choices can be used, and our method does not require exhaustive hyper-parameter tuning.
2. **A Broad study of each proposed component over six datasets.** This validates the necessity and effectiveness of each module in our framework.
3. **Theoretical analyses of our method** using the information bottleneck principle which demonstrates the interpretability of the proposed approach.
4. **Additional validation on DomainNet** which verifies our method is robust to data distribution changes in continual learning.
5. Other comparison and ablation experiments suggested by the reviewers.

We will include these experiments and explanations in the revised version of our paper as well as its supplement, for the readers to have a better understanding.

Meta-review: This paper focuses on continual learning. There is a growing interest in using pre-trained models instead of starting from scratch in these problems. Rather than using PEFT, which can limit the exploration of pre-trained models, it reduces the model’s adaptability to new concepts. To address these issues, the authors propose a new framework inspired by slow-fast learning, which includes a transfer loss function to inherit general knowledge from pre-trained models and uses slow-tuning parameters to capture more informative features. To balance stability and adaptability, the framework fixes the slow tuning parameters while continuously updating the fast ones, incorporating novel concepts and preventing catastrophic forgetting. An entropy-based aggregation strategy is introduced during inference to leverage the complementarity between slow and fast learners. Extensive experiments on six benchmark datasets show that this method outperforms its counterparts.

Although there have been some negative comments on the paper, after carefully reviewing both the paper and the discussions between the authors and reviewers, I support accepting it. I believe the idea presented is valuable to the community. I encourage the authors to address the feedback from the reviews and make the necessary revisions to strengthen the paper.
Decision: Accept (poster)