SAFE: Slow and Fast Parameter-Efﬁcient Tuning for Continual Learning with Pre-Trained Models

Authors: linglan  zhao
xuerui  zhang
ke  yan
shouhong  ding
weiran  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 05:03:57

Abstract: Continual learning aims to incrementally acquire new concepts in data streams while resisting forgetting previous knowledge.
With the rise of powerful pre-trained models (PTMs), there is a growing interest of training incremental learning systems using these foundation models, rather than learning from scratch. 
Existing works often view PTMs as a strong initial point and directly apply parameter-efficient tuning (PET) in the first session for adapting to downstream tasks.
In the following sessions, most methods freeze model parameters for tackling forgetting issues. 
However, applying PET directly to downstream data cannot fully explore the inherent knowledge in PTMs.
Additionally, freezing the parameters in incremental sessions hinders models' plasticity to novel concepts not covered in the first session. 
To solve the above issues, we propose a Slow And Fast parameter-Efficient tuning (SAFE) framework.
In particular, to inherit general knowledge from foundation models, we include a transfer loss function by measuring the correlation between the PTM and the PET-applied model.
After calibrating in the first session, the slow efficient tuning parameters can capture more informative features, improving generalization to incoming classes.
Moreover, to further incorporate novel concepts, we strike a balance between stability and plasticity by fixing slow efficient tuning parameters and continuously updating the fast ones.
Specifically, a cross-classification loss with feature alignment is proposed to circumvent catastrophic forgetting.
During inference, we introduce an entropy-based aggregation strategy to dynamically utilize the complementarity in the slow and fast learners.
Extensive experiments on seven benchmark datasets verify the effectiveness of our method by significantly surpassing the state-of-the-art.

Main paper: 
 introduction: 
 Continual Learning (CL) requires deep learning models to incrementally incorporate new concepts from open-world data streams, while retaining previously learned knowledge. This presents a more challenging yet practical setting compared to traditional deep learning, which typically recognizes only closed-set categories. A variety of methods have been proposed for continual learning, including regularization-based [17, 20, 47] , rehearsal-based [5, 14, 31] , and dynamic network-based approaches [1, 43, 44] . These methods often assume that the model is trained from scratch, resulting in a substantial performance gap when compared to the joint training upper-bound. ... [21] , and Visual Prompt Tuning (VPT) [16] .Most recently, with the emergence of powerful pre-trained models, there has been growing interest in utilizing these foundational models as starting points for continual learning [23, 49, 52] . Pre-Trained Models (PTMs) which are often trained on vast datasets, encapsulate a wealth of general knowledge, effectively enhancing the performance of deep learning models in continual learning scenarios. As shown in the left part of Fig. 1 (a), for adapting PTMs from pre-training datasets to continual learning datasets, prevailing works resort to parameter-efficient tuning (PET) techniques [6, 16, 21] in the first session. To restrain catastrophic forgetting, in incremental sessions, these works set parameters of the adapted model frozen [2, 15, 23, 52] and only update the classification weights in a training-free manner (i.e., without gradient updates) to accommodate novel classes.However, the above methods have two main limitations. First, direct parameter-efficient tuning in the first session will largely lose the general knowledge inherent in PTMs. This is because PTMs are pre-trained on a multitude of datasets while the dataset in the first session only contains relatively limited samples. Without proper transfer mechanisms, the knowledge from PTMs may be overwritten by the adapted model, which impedes the model's generalizability to unseen classes. Second, freezing parameters in the following sessions will hinder the plasticity of the model to further absorb new concepts not learned in the first session, resulting in a sub-optimal solution. Although several efforts have been made to mitigate the second limitation, existing works still face certain constraints such as additional storage requirement [35, 49] , inferior online branch performance [8] and linearly increased model complexity [54] .Based on the above observations, in this paper, we propose Slow And Fast parameter-Efficient tuning (SAFE) to address existing challenges. In particular, SAFE demonstrates a unified framework that effectively inherits the generalizability of PTMs using slow parameter-efficient tuning (S-PET) and provides sufficient plasticity to learn task-specific knowledge in each incremental session using the fast one (F-PET). Meanwhile, SAFE does not require storing class distributions for data replay and only incurs constant-level additional computation and memory costs.To achieve the above goals, SAFE employs distinct strategies for the first and subsequent sessions.In the first session, we focus on explicitly transferring general knowledge from pre-trained models (PTMs) by introducing a knowledge transfer loss. This involves computing a correlation matrix between feature embeddings from the PTM and the model with parameter-efficient tuning (PET).The diagonal elements of this matrix are maximized to ensure that the features remain consistent across both models, effectively aligning the PET-applied model's performance with that of the PTM. Simultaneously, minimizing the off-diagonal elements reduces redundancy in the embeddings, enhancing feature discriminability. After this tuning process, parameters can retain generalizable knowledge from the PTM. To prevent forgetting this knowledge, these trained parameters are subsequently frozen, with only the classification weights being updated, thus designating this model as the slow learner.In the incremental sessions, to address the plasticity limitations of the slow learner, we introduce a fast learner capable of continuously integrating new concepts. Given the persistent challenge of catastrophic forgetting in continual learning, the slow learner guides the training of the fast learner. Concretely, we employ a feature alignment loss to minimize the distance between the embeddings of both learners on a hypersphere. Additionally, a cross-classification loss is proposed to ensure compatibility between the features of the fast learner and the classification weights of the slow learner, and vice versa. This approach allows the fast learner to assimilate new knowledge without storing exemplars or distributions, while also mitigating forgetting. For robust predictions, an entropy-based aggregation strategy is implemented during inference to dynamically leverage the complementary strengths of the slow and fast learners.To summarize, the contributions of our paper are three-fold:• To inherit the generalizable knowledge in PTMs that has been overlooked in existing continual learning works, we propose to explicitly transfer knowledge from the PTM to a slow learner. Once trained, the slow learner can generalize well to classes in incremental sessions. • For improving the plasticity of CL models, we include a fast learner with guidance from the slow learner to continuously incorporate novel concepts. Moreover, by aggregating both slow and fast learners into a unified framework SAFE, robust predictions can be further made. • The superiority of SAFE is validated on seven continual learning datasets where our method consistently achieves remarkable state-of-the-art performance. For example, our method surpasses the second-best result on ImageNet-A over 4%.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The abstract and introduction clearly state the main contributions and scope of the paper, accurately reflecting the theoretical and experimental results presented.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Continual Learning. Traditional continual learning (CL) aims at continuously updating models with data streams from scratch. Existing strategies involve regularization-based approaches [17, 20, 47, 51] which prevent forgetting by regularizing network weights or predictions, rehearsal-based approaches which replay historical data stored in a fixed-sized buffer [4, 9, 14, 25, 31] , and architecture-based approaches [1, 43, 44, 55] which dynamically expand models for novel classes. Among these methods, a recent attempt to preserve knowledge based on slow and fast complementary theory has been proposed [3, 22, 45] . Nevertheless, these approaches typically require adjusting all model parameters, which increases the computational burden of the learning process. Contrarily, our Slow And Fast parameter-Efficient tuning (SAFE) framework only requires much fewer learnable parameters as well as fewer resources, while obtaining more favorable performance.Continual Learning with Pre-Trained Models. With the emergence of powerful pre-trained models (PTMs), it has become a hot topic to integrate pre-trained models with CL [29, 53] for better performance. Prompt-based methods [30, 33, 39, 41, 42] utilize prompt tuning to adapt PTMs to new tasks. However, these methods are tailored for Transformers [7, 37] and require an expanding prompt pool with the arrival of new data. First session adaptation methods [2, 23, 52] adapt PTMs solely in the first session and then freeze the model afterward to suppress forgetting [26, 32] . Nevertheless, these works lack plasticity for classes in subsequent sessions. Contrarily, another line of works focuses on continual adjustment [8, 35, 49, 54] to accommodate evolving information. However, the above approaches either require storing data distributions [35, 49] for replay, only obtain inferior online branch performance [8] , or linearly increase complexity with incremental sessions [54] . Compared to existing works, our method provides a flexible framework that boosts generalizability by inheriting PTM's knowledge in the first session and maintains plasticity for incremental classes with constant complexity in a replay-free manner.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper includes a separate "Limitations" section (in Appendix) that discusses the potential weaknesses of the proposed approach.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem definition: 
 Following previous works [8, 23, 49, 52] , in this paper, we mainly consider PTM-based CL under a class-incremental learning setting. Formally, the model is trained sequentially on a series of incremental sessions, where D t = {(x t i , y t i )} Nt i=1 ⊂ {X t , Y t } represents the t-th training set composed of N t samples, for t ∈ {1, 2, . . . , T }. The sample and label space of D t are denoted by X t and Y t , where Y t is disjoint between different sessions, i.e., ∀i, j and i ̸ = j, Y i ∩ Y j = ∅. We follow the replay-free setting, where only D t is accessible in session t. After training in the t-th session, the model is evaluated on all the seen classes so far:Y 1:t = Y 1 ∪ Y 2 • • • ∪ Y t .In addition, we also validate our method on domain-incremental learning setting, where the data distribution between sessions shifts significantly, i.e., ∀i, j and i ̸ = j, P (X i ) ̸ = P (X j ), Y i = Y j .For tackling the stability-plasticity dilemma in CL, we draw inspiration from the complementary learning systems theory [19] to develop a Slow And Fast parameter-Efficient tuning (SAFE) framework, as depicted in Fig. 2 . In the first session, the slow learner is tuned to inherit the general knowledge from PTM and is frozen afterward. In the following sessions, the slow learner only updated its classification head using imprinted weights [28] , which acts like the neocortex to slowly incorporate novel knowledge without forgetting. Complementary to this, the fast learner with learnable parameters rapidly encodes novel information as the hippocampus for adapting to new classes.Formally, features extracted from PTM, slow learner and fast learner are denoted as f l = ϕ l (x) ∈ R d , where l ∈ {PTM, slow, fast} and d is the feature dimension. To leverage the knowledge of PTMs with few learnable parameters and resources, feature extractors for the slow and fast learners are trained using parameter-efficient tuning (PET) [6, 16, 21] which are referred to as S-PET and F-PET, respectively. Consistent with prior works [23, 52, 54] , we mainly consider three types of PETs: Adapter [6] , SSF [21] , and VPT [16] , shown in the right part of Fig. 1 .The classification weights in session t for the slow and fast learners are symbolized by W l ∈ R d×|Y1:t| , l ∈ {slow, fast}, where |Y 1:t | is the number of classes seen so far from session 1 to session t. For the slow learner, W slow is learned in the first session and expanded using feature centroids of training samples within the same classes [28] afterward to preserve learned general knowledge. Contrarily, W fast is trainable as CL progresses for the plasticity purpose.In the following sections, we provide the details of slow and fast learner training in Section 3.3 and Section 3.4. After that, discussions about model inference are presented in Section 3.5.Benefiting from pre-training on large-scale resources, pre-trained models (PTMs) inherently possess strong generalizability for downstream tasks. Previous works [23, 35, 52] typically view the PTM as a preferable starting point for continual learning. To bridge the distribution gap between pre-training datasets and downstream datasets, these methods often directly apply PET to PTMs.However, without proper transfer mechanisms, models directly tuned on downstream data cannot effectively inherit the general knowledge from PTMs. More seriously, the intrinsic knowledge in PTM may be overwritten during adaptation to the recent dataset, since it often contains relatively limited samples. To solve the above issues, we propose to effectively squeeze out information from PTMs and explicitly transfer it to adapted models.Concretely, in the first session, we calculate the cross-correlation matrix M ∈ R d×d between the features of the slow learner and the PTM:EQUATIONwhere N b is the batch size, d is the feature dimension and "•" denotes element-wise multiplication 1 . Moreover, i and j index the dimensions of the features and matrices. In fact, the correlation matrix characterizes the relationship between feature embeddings of PTM and the slow learner. The i-th row and j-th column of M measures the correlation between the i-th feature dimension (also termed as channel or pattern in the literature) of the PTM and the j-th feature dimension of the slow learner.To encourage the PET-applied model to mimic the performance of the PTM, we maximize the elements in the diagonal. This maximizing term ensures the slow learner can learn invariant feature components that match the statistics of the PTM:EQUATIONAdditionally, we reduce the redundancy between patterns in embeddings to enhance discriminability. This can obtained by decreasing the off-diagonal elements in M with L rdn :EQUATIONCombined with the classification loss L cls using cross-entropy (CE):EQUATIONwhere "⊙" denotes matrix multiplication, the overall loss function during the first training session is defined as:EQUATIONIn Eq. ( 5), λ diag and λ rdn are the balancing hyper-parameters. Intuitively, the joint optimization of three losses makes the adapted model simultaneously acquire distribution-specific knowledge based on L cls and inherit general knowledge of the PTM using L diag and L rdn . As a result, the slow model can better generalize to incoming classes even unseen in the first training session.Although solely using the slow learner with general features already obtains competitive performance, the plasticity of the model is hindered due to its frozen parameters in the following sessions. To strike a balance between stability and plasticity, we adopt the fast learner to continuously learn episodic information for novel classes. However, updating representations without data reply will lead to semantic drift [35, 46, 49] , causing catastrophic forgetting of previously learned knowledge.Existing works to address this problem either store additional data distributions [35, 49] or require sophisticated drift estimations after each session [35, 46] . Compared to previous works, our method imposes no such constraints, and aligns the models before and after updates in a single embedding space, essentially addressing semantic drift.First, the fast learner is trained with guidance from the slow learner using feature alignment to preserve prior representations. Specifically, the distance of feature embedding from both models is minimized on a hypersphere to alleviate forgetting:EQUATIONwhere N b is the batch size and cos denotes cosine similarity of two vectors.Furthermore, we utilize cross-classification which contains a fast-to-slow loss and a slow-to-fast loss to maintain previous decision boundaries. For fast-to-slow calibration, we feed features from the fast learner to the classification layer of the slow learner. This objective makes features from the fast model compatible with the decision boundaries of the slow one to suppress semantic drift. Moreover, since the classification weight vector of each class can be viewed as a prototype of that class [28, 34] , we also use these vectors as inputs for further preserving knowledge from previous sessions:EQUATIONwhereW (j)fast ∈ R d denotes the j-th column of W fast , which is also the prototype for class j in the fast learner. Similarly, slow-to-fast loss L s2f can be derived by swapping the fast and slow terms in Eq. (7) . After that, the cross-classification loss can be defined asL s↔f = L f2s + L s2f .Along with the classification loss L cls in Eq. ( 4) applied to the fast learner, the optimization objective in the incremental phase is defined as:EQUATIONwhere λ cos is the balancing hyper-parameter. The loss function L fast smoothly adapts the fast learner to new knowledge while enforcing consistency with previously acquired knowledge, which boosts the plasticity of the model without severe forgetting.Since the slow learner inherits general knowledge and the fast learner contains task-adaptive knowledge, we can obtain robust predictions by utilizing the complementarity of them. We first introduce the inference using a single learner and then provide aggregation strategy based on both learners.Single-learner-based Inference. Following previous work [23, 26] , instead of directly using the classification weights W l and features ϕ l (x), l ∈ {slow, fast} for prediction, we take advantage of second-order statistics and prototype information for better performance. Formally, given a test sample x, the predicted logits of each learner z l are calculated as:EQUATIONwhere β is a hyper-parameter for regularization, h l (x) ∈ R M is projected feature of x and classification weights Wl ∈ R M ×|Y1:t| is composed of summations of projected features with same class labels. Gram matrix G ∈ R M ×M is cumulated based on training data from session 1 to t:EQUATIONIn Eq. ( 10), W rand ∈ R d×M is the projection matrix with each column sampled from N (0, σ 2 I), ψ is a nonlinear activation function and I denotes the identity matrix. Mathematically, Eq. ( 9) defines a more general form of regular linear prediction. When W rand is I and ψ is not applied, it degrades to a ridge regression [13] . Moreover, if G is removed, the classifier further reduces to NCM [24] .Aggregation-based Inference. As discussed in the above sessions, slow and fast learners excel in handling classes from different sessions. Due to its plasticity, the fast learner can better recognize categories from the latest several sessions but shows limited performance on the old ones caused by potential forgetting. Contrarily, despite limited novel concept adaptation, the slow learner can capture historical knowledge thanks to its stability. Intuitively, when dealing with proficient categories, the model exhibits higher confidence in predictions. Motivated by this, we use entropy to measure the confidence and dynamically aggregate the logits for robust predictions.Given a test sample, we compute the entropy of predictions using H =i p i log p i for each learner, obtaining H slow and H fast , where p = softmax(z) is predicted probability. As lower entropy indicates less uncertainty in predictions, the confidence of each learner can be represented by[α slow , α fast ] = softmax([-γ • H slow , -γ • H fast ]), where γ is a scalar to control the peakiness of output distributions. After that, the aggregated logits z aggregate automatically assign higher weights to predictions with higher confidence, and can be obtained using a convex combination:EQUATIONFinally, the prediction is obtained using the index of the max element in z aggregate in session t > 1, while using z slow instead in session 1 since the fast learner is not available in that session.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we first introduce the implementation details of our proposed method SAFE and then compare it to the state-of-the-art on seven popular benchmark datasets. After that, detailed ablative experiments are conducted to validate the effectiveness of each component.Datasets and Evaluation. Following previous methods [23, 52, 54] , our evaluations are conducted on seven benchmark datasets: CIFAR100 [18] , ImageNet-R (IN-R) [11] , ImageNet-A (IN-A) [12] , CUB200 [38] , Omnibenchmark (OB) [50] , VTAB [48] and DomainNet [27] . Previous state-ofthe-art PTM-based CL methods are chosen for comparison, including L2P [42] , DualPrompt [41] , CODAPrompt [33] , ADaM [52] , RanPAC [23] , SSIAT [35] , and SLCA [49] . We adopt final accuracy Acc T and average accuracy Acc avg = 1 T T t=1 Acc t as evaluation metrics. Implementation Details. Consistent to existing works [23] , we adopt ViT-B/16-IN1K and ViT-B/16-IN21K as the PTM and apply Adapter [6] , SSF [21] or VPT [16] for parameter-efficient tuning (Appendix A). In each session, we train the model for 20 epochs using SGD optimizer, weight decay of 0.0005, momentum of 0.9, and a cosine annealing schedule where learning rate starts from 0.01 and decays to 0. The batch size is set to 48. In addition, β in Eq. ( 9) is selected based on the performance on the training data similar to [23] . For other hyper-parameters used in our method, we find λ diag = 0.1, λ rdn = 100, λ cos = 50, γ = 1 is a reasonable set of default choices. Detailed hyper-parameter sensitivity analyses are provided in Appendix D.In this section, we compare the proposed method SAFE with several state-of-the-art approaches across seven datasets: CIFAR100, ImageNet-R, ImageNet-A, Omnibenchmark, CUB200, VTAB and DomainNet. For fairness, all methods are implemented with the same ViT [7] backbones. The class-incremental learning results from the final session are reported in Table 2 . As shown in Table 2 , our method consistently achieves the best performance among all benchmarks. Notably, we significantly surpass the second-best result on ImageNet-A by 4.4%. When compared to methods storing additional data distributions for replay [35, 49] , our method is replay-free and can still outperform these methods by a significant margin. In addition, we improve the average accuracy over six datasets by 2.1% compared to the previous best approach [23] . The aforementioned superiority can contribute to the generalizability and plasticity of our method within a unified framework.For domain-incremental learning, results on DomainNet with 6 different domains are summarized in Table 1 . Our method SAFE can outperform the second-best result by 1.2%, demonstrating that the proposed framework is applicable to scenarios where the data distribution of the first task diverges significantly from that of subsequent tasks. To investigate the factors contributing to the success of SAFE, we validate the effectiveness of our key components: the slow learner (SL) in Section 3.3, the fast learner (FL) in Section 3.4, and the entropy-based aggregation in Section 3.5. Experiments are primarily conducted on IN-A dataset.Effectiveness of the Slow Learner. We assess the effectiveness of the slow learner from three perspectives. Firstly, as depicted in Table 3 , when the slow learner is added to the baseline [23] , the final accuracy increases by 3.2% and the average accuracy increases by 2.1%. This observation verifies that the slow learner can generalize well to the incremental classes. Secondly, we expect the slow learner to inherit generalizability from the PTM. To dive deeper into this aspect, we visualize the embeddings of five unseen classes and five seen classes by T-SNE [36] after the first session adaptation.As shown in Fig. 3 , the embedding space of the slow learner exhibits distinct separation between the seen and unseen classes. Note that the feature distributions with SL in the grey ellipse become more separable compared with the baseline method. This illustrates the successful integration of generalization capabilities from the PTM into the slow learner.Furthermore, we explore other alternatives for transferring generalizability, including feature alignment (FA) by distilling PTM's features, logits alignment (LA) by distilling PTM's predictions, and second-order statistics alignment (SSA) by distilling PTM's covariance. Table 5 presents the average and final accuracy of the substitutions on IN-A, with the best results highlighted in bold. It is observed that our slow learner can consistently outperform these variations, validating its superiority.Effectiveness of the Fast Learner. As shown in the third row of Table 3 , compared to the slow learner, using only the fast learner can obtain 1.1% improvements in the final accuracy. This indicates that the fast learner is properly guided by the slow learner, and thus can continuously adapt to novel classes with suppressed forgetting.Subsequently, we present the necessity of each regularization term in the fast learner. As shown in Table 6 , without L s↔f and L cos , the performance drops to lower than 10% due to catastrophic forgetting. To alleviate forgetting, both L s↔f and L cos are applied. Specifically, solely using L s↔f results in an improvement of 3.1% compared to the baseline, while using only L cos yields a gain of 3.9% over the baseline. Moreover, with all the proposed loss functions, the fast learner can obtain the best performance, validating the effectiveness of each regularization term. As shown in the last row of Table 3 , the combination of the slow and fast learners presents the best result. This observation is consistent with the complementary learning systems theory [19] that memory necessitates the presence of both a slow learner and a fast learner for improved performance.To gain deeper insights into the necessity of both learners, we elaborate on their final accuracy of classes from each session. In Fig. 4 , the slow learner, mimicking the neocortex, initially stores structured information and performs well on relatively old classes (0-119). Conversely, the fast learner, resembling the hippocampus, swiftly adapts novel concepts and excels in more recent classes (120-199).From this perspective, combining these two complementary learners leverages their strengths across the training process, resulting in superior model performance.In addition, Fig. 5 illustrates how the aggregated model dynamically leverages the strengths of both learners. Concretely, the horizontal axis represents the class indices to which each test sample belongs, while the vertical axis shows the average aggregation weights of each learner assigned to these test samples. It is observed from Fig. 5 that, for classes 120-199, the fast learner consistently shows higher weights, which is consistent with its superior classification accuracy in these classes as depicted in Fig. 4 . For classes 0-119, the slow learner obtains higher weights, generally aligning with its demonstrated stability and better performance on these classes shown in Fig. 4 . By adaptively balancing the contributions of both learners, our method achieves a harmonious trade-off between stability and adaptability.Moreover, we undertake detailed comparisons to other merging strategies to validate the effectiveness of our aggregation choice. As depicted in Table 4 , we compare our entropy-based aggregation with three alternatives: feature concatenation, logits addition, and logits max. We report the final and average accuracy, where the results elucidate that the entropy-based aggregation fully leverages both learners and achieves the best performance.Final Accuracy Parameter SizeMethods (parameter size from small to large) In this section, we investigate the number of learnable parameters in different methods and report the parameter-performance comparison.Since no exemplars are stored in our method, the primary storage cost is attributed to the trainable model parameters introduced by parameterefficient tuning (PET). Although PET entails additional parameters, it is still small relative to the overall size of the pre-trained model (PTM). Moreover, as the parameter-performance tradeoff shown in Fig. 6 , our method SAFE utilizes a similar scale of parameters as existing PTMbased methods while achieving substantial performance improvements. 
conclusion: 
 In this paper, we introduced SAFE, a Slow And Fast parameter-Efficient tuning framework for continual learning. Our approach leverages the inherent knowledge in pre-trained models (PTMs) while maintaining model plasticity for novel concepts. By incorporating a transfer loss function, we ensure the preservation of general knowledge from PTMs. In the first session, we calibrate slow efficient tuning parameters to enhance the model's ability to generalize to new classes. To balance stability and plasticity, we fix the slow efficient tuning parameters and continuously update the fast ones, employing a cross-classification loss with feature alignment to prevent catastrophic forgetting.During inference, we introduce an entropy-based aggregation strategy for dynamic utilization of the complementarity between the slow learner and the fast learner. Extensive experiments on seven benchmark datasets demonstrate that our method significantly surpasses the state-of-the-art, validating the effectiveness of our approach.Limitations: Our approach is built upon RanPAC [23] , and as such, it shares some of the same limitations. For instance, our method relies on a strong feature extractor to effectively inherit generalizability from PTMs, making it less suitable for scenarios where training needs to be performed from scratch or starting from rather small tasks. Additionally, our method introduces three hyper-parameters to balance the loss functions during training, as previously discussed. While our experiments demonstrate that a set of default values works well across the benchmark datasets evaluated in our work, we acknowledge that these choices might not be optimal when applied to datasets with essentially different statistical characteristics. Furthermore, slowly updating the slow learner periodically, rather than keeping it fixed in subsequent sessions, may further enhance the model's adaptability and could be a promising direction for future research. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The paper specifies all necessary details about the training and testing processes, including data splits, hyperparameters, and optimization methods.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Justification: The paper does not report error bars or other statistical significance measures for the experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: The paper includes no such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper properly credits the creators of existing assets used and specifies the licenses and terms of use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not introduce any new assets, making this question not applicable.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve any crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve any research with human subjects, hence IRB approval is not applicable.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A Parameter-Efficient Tuning (PET)Trained on vast amounts of data, models with billions of parameters exhibit remarkable performance across various tasks. However, the expansive scale and computation pose considerable challenges when customizing them for downstream deployment. From this perspective, parameter-efficient tuning (PET) provides a practical solution by effectively adapting the large pre-trained models with limited additional parameters.Parameter-efficient tuning selectively adjusts a small proportion of the model parameters while keeping the rest frozen. In this way, pre-trained models (PTMs) can partially keep the generalization and deal with domain gaps at a low resource cost [10] . Motivated by this, PTM-based continual learning models leverage PET in their paradigm to achieve desirable results [23, 52, 54] . Typically, continual learning works leverage ViT-B/16-IN1K and ViT-B/16-IN21K as the backbones and finetune the model mainly with three PET algorithms: Adapter [6] , Scale & Shift (SSF) [21] and Visual Prompt Tuning (VPT) [16] , which are introduced in the following:Adapter: Adapters are small additional layers inserted into the layers of a PTM. Each adapter layer generally consists of three parts: a down-projection layer W down ∈ R d×r which reduces the input feature dimension, a non-linear activation function (e.g., ReLU), and an up-projection layer W up ∈ R r×d which projects features back to the original dimension. Specifically, given an input x ∈ R L×d , the output y ∈ R L×d is expressed as:EQUATIONwhere L, d and r represent the length of the input feature sequence, original feature dimension and projected feature dimension. In the above equation, "⊙" denotes matrix multiplication.Scale & Shift (SSF): SSF involves two main operations: scaling, which multiplies each feature by a learnable vector to adjust its spread, and shifting, which adds a trainable vector to each feature to change its central position. In the context of fine-tuning PTMs, SSF helps to normalize the feature distributions and adjust to new data. This improves performance and robustness by maintaining consistency in distribution. Specifically,EQUATIONwhere γ ∈ R d and β ∈ R d are the scaling and shifting vectors, respectively. Moreover, "•" represents element-wise multiplication.Visual Prompt Tuning (VPT): VPT extends original input features with lightweight learnable tokens and the extended features will be fed into subsequent transformer blocks of ViT [7] to obtain the final adapted embedding. Concretely, denote the learnable prompts as P ∈ R K×d , extended features can be expressed as:EQUATION)where K is the length of the prompt and y ∈ R (K+L)×d is the extended feature.In the main paper, we report the remarkable performance of the proposed SAFE framework under the same PET setting as [23] . In this section, we demonstrate that the proposed approach is a general framework that is compatible with diverse PET modules. Specifically, we combine SAFE with Adapter [6] , Scale & Shift(SSF) [21] and Visual Prompt Tuning (VPT) [16] . As depicted in Table 7 , we report the final accuracy on six datasets compared with the baseline method [23] .As shown in Table 7 , the proposed SAFE framework outperforms the baseline across various PET modules by a substantial margin. It is worth noting that the proposed method consistently exceeds the baseline on ImageNet-A by over 4% with different PET modules. We also achieve performance improvements by 2.2% with Adapter, 3.1% with SSF, and 3.0% with VPT on ImageNet-R. These results demonstrate the general applicability of our framework across PET algorithms. {(x, y)} ← sample a batch of data from D 1 .Calculate the correlation matrix in Eq. ( 1) and losses L diag , L rdn in Eq. ( 2), Eq. (3).Calculate the overall loss function L slow in Eq. ( 5).Update {W slow , θ S-PET } with gradients ∇L slow . 9: end while 10: Replace W slow with imprinted weights (i.e., feature centroids of each class in D 1 ). 11: Freeze parameters {W slow , θ S-PET }.12:13: Phase 2: Fast Learner in session t > 1. 14: Expand W slow (R d×|Y1:t-1| → R d×|Y1:t| ) with imprinted weights using ϕ slow and D t . 15: Expand W fast (R d×|Y1:t-1| → R d×|Y1:t| ) with imprinted weights using ϕ fast and D t . 16: Initialize the fast learner's efficient tuning parameters θ F-PET from session t -1. 17: while not done do Calculate feature alignment loss L cos in Eq. ( 6) and cross-classification loss L s↔f in Eq. ( 7).Calculate the overall loss function L fast in Eq. ( 8).Update {W fast , θ F-PET } with gradients ∇L fast . 22: end whileHyper-Parameters Sensitivity. Our framework SAFE includes 4 hyper-parameters: λ diag and λ rdn for the slow learner, λ cos for the fast learner, and γ for aggregation. In this section, we supply detailed hyper-parameter sensitivity analyses on ImageNet-A. Results for λ diag and λ rdn are depicted in Fig. 7 , while the results for λ cos are shown in Table 8 . Moreover, Table 9 presents the experiment on γ. It is observed that hyper-parameters remain relatively stable within a certain range. For example, the slow learner can achieve satisfactory results with λ diag in the range from 0.1 to 1, and λ rdn in the range from 100 to 500. The fast learner can obtain good performance with λ cos in the interval from 50 to 100. Moreover, the aggregation module works well by simply setting γ to 1. As a result, we set λ diag = 0.1, λ rdn = 100, λ cos = 50, γ = 1 as the default choices of hyper-parameters as stated in Section 4.1 of our main paper. Teacher Models for the Fast Learner. As discussed in Section 3.4, the fast learner is guided by the slow learner during adapting to novel classes. In this section, we provide additional experiments on the choice of the teacher model which guides the training of the fast learner. We conduct comparisons on training the fast learner directly (None teacher), using the pre-trained model as a teacher (PTM) and using the fast learner from the last session as a teacher (t -1). As shown in Table 10 , utilizing the slow learner as a teacher model surpasses all the alternatives. This is because the slow learner can provide generalizable knowledge to the fast learner and simultaneously alleviate forgetting. While both the proposed SAFE and PanPAC [23] leverage PTMs for continual learning, they target different components of the model. Specifically, RanPAC focuses on deriving decorrelated classification weights for the classification head with frozen features, whereas our method emphasizes the improvement of trainable feature embeddings within the feature extractor. Furthermore, there is a distinct difference in the correlation matrices utilized by the two methods. The correlation coefficients matrix in RanPAC, as shown in Figure 2 of their paper, has dimensions R C×C , where C denotes the number of classes in the classification head. In contrast, our method employs a cross-correlation matrix of dimensions R d×d , with d representing the feature dimension, as detailed in Eq. ( 1).In addition, we would like to emphasize that our method is orthogonal to RanPAC. In fact, our approach is built upon RanPAC, and as evidenced in Table 2 of our paper, our method consistently outperforms RanPAC by a significant margin.In this section, we provide visualization results of the seven evaluated datasets: CIFAR100 [18] , ImageNet-R (IN-R) [11] , ImageNet-A (IN-A) [12] , CUB200 [38] , Omnibenchmark (OB) [50] , VTAB [48] and DomainNet [27] . As shown in Fig. 8 , SAFE can perform well on datasets with various characteristics. It is noteworthy that SAFE is capable of scenarios where the data distribution between tasks shifts significantly. For example, our method also shows superior performance on VTAB and DomainNet which comprise 5 and 6 distinct tasks, respectively.NeurIPS Paper ChecklistFollowing[14], in this paper, we use l2 normalization to map features and classification weights onto a hypersphere before element-wise or matrix multiplication. Normalization is omitted to simplify notation. 


Decision: This paper focuses on continual learning. There is a growing interest in using pre-trained models instead of starting from scratch in these problems. Rather than using PEFT, which can limit the exploration of pre-trained models, it reduces the model’s adaptability to new concepts. To address these issues, the authors propose a new framework inspired by slow-fast learning, which includes a transfer loss function to inherit general knowledge from pre-trained models and uses slow-tuning parameters to capture more informative features. To balance stability and adaptability, the framework fixes the slow tuning parameters while continuously updating the fast ones, incorporating novel concepts and preventing catastrophic forgetting. An entropy-based aggregation strategy is introduced during inference to leverage the complementarity between slow and fast learners. Extensive experiments on six benchmark datasets show that this method outperforms its counterparts.

Although there have been some negative comments on the paper, after carefully reviewing both the paper and the discussions between the authors and reviewers, I support accepting it. I believe the idea presented is valuable to the community. I encourage the authors to address the feedback from the reviews and make the necessary revisions to strengthen the paper.

Reviews: review: paper_summary: paper_summary: The paper introduces the SAFE (Slow And Fast parameter-Efficient tuning) framework for continual learning using pre-trained models (PTMs). The proposed approach combines slow parameter-efficient tuning (S-PET) to inherit general knowledge from PTMs and fast parameter-efficient tuning (F-PET) to adapt to new tasks in each incremental session. The paper validates the effectiveness of the SAFE framework through experiments on multiple datasets, demonstrating its superiority over state-of-the-art methods.
strength_weakness: strength_weakness: - The motivation and effects of the proposed techniques are unclear. 
    - For example, what are the specific effects of the cross-correlation matrix? What is the specific motivation? This design is quite similar to the de-correlation operation process in RanPAC, although with a different formulation. More solid analyses are required. 
    - The issue is still with the fast learner part. Although ablation studies are conducted for the components, the insights of the designs are unclear. For ablation studies, not only the terms should be ablated, but the designs, such as the usage of W_slow in the loss should be clearly discussed and analyzed. 
- Although the paper is generally well written, some parts have tedious descriptions and blurry presentations, such as line 58 - 81 in the introduction, which need to be improved. 
- The proposed method contains many hyper-parameters, such as the weights for the loss terms. Ablation studies for them are necessary and required. 
- The importance of aggregation for the performance should be discussed based on the ablation study. 
- The aggregation process requires weights for slow and fast results. Although they can be calculated automatically, it should be analyzed and demonstrated how the distribution of these weights for different data samples and datasets. 

The paper can be a good paper. However many details are absent in the papers, making the delivered message unclear and the techniques not justified well. I will reconsider my score based on the rebuttal.
questions: questions: The questions are left with weakness points,
limitations: limitations: The paper includes a limitation section in the appendix. But it does not cover actual limitations of the work. Instead, the discussed limitation is mainly the characteristics of general pre-trained model-based continual learning (i.e., using the pre-trained model), which is actually not a limitation. The authors may discuss the limitation related to the robustness of the hyper-parameters.
paper_summary: paper_summary: This paper proposes a novel paradigm for continual learning called SAFE, which utilizes both slow and fast parameter updates. The method focuses on continual learning with pre-trained models, using slow updates to preserve the generalization capability of the pre-trained model while employing fast updates to adapt to new downstream tasks. During inference, the two branches are aggregated to achieve more robust predictions. Experiments on 6 different benchmarks demonstrate that SAFE consistently outperforms existing methods, achieving state-of-the-art performance.
strength_weakness: strength_weakness: 1. The slow learner assumes that the data distribution of the first task in the continual learning scenario is roughly similar to subsequent tasks, which is not always true in practical applications. When the data distribution of the first task diverges significantly from subsequent tasks, constraining the fast learner to the slow learner may yield negative results.
2. The design of the method introduces too many loss function terms, which could make the model training overly complex. Additionally, the stability of these hyperparameters is not analyzed in the experimental section.
3. The main experimental section lacks comparisons with the latest methods, such as DAP[1].
4. The ablation studies show that the introduction of many components results in only marginal improvements. According to Occam's Razor, these components should be removed.

[1] Jung D, Han D, Bang J, et al. Generating instance-level prompts for rehearsal-free continual learning[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 11847-11857.
questions: questions: 1. I am curious about SAFE's performance in scenarios where the data distribution between tasks varies significantly, such as in DomainNet.
2. What is the setup for the ablation study in Table 2? Does "only the fast learner" mean that only the SL is used for inference after training, or does it mean that only SL is involved in the training process?
3. What is the setup for PET in SAFE? Does it include only one of Adapter, SSF, or VPT, or does it use all three simultaneously?
limitations: limitations: The method proposed in the paper is only applicable to scenarios where the data distribution of the first task is roughly similar to that of subsequent tasks.
paper_summary: paper_summary: The paper proposes a novel method named SAFE (Slow And Fast parameter-Efficient tuning) to tackle challenges in continual learning. SAFE introduces a unified framework that combines slow parameter-efficient tuning (S-PET) for inheriting general knowledge from pre-trained models (PTMs) and fast parameter-efficient tuning (F-PET) for acquiring task-specific knowledge in incremental sessions. SAFE demonstrates state-of-the-art performance on six benchmark datasets, showing significant improvements over existing methods.
strength_weakness: strength_weakness: 1. Training the slow adapter only in the initial session may restrict its ability to adapt to new tasks, potentially limiting the overall flexibility of the model.
2. Complex Loss Functions and Hyperparameters: The proposed method involves multiple loss functions and hyperparameters, making the optimization process complex and potentially challenging to tune in practice.
3. While the method shows promising results, additional experiments or ablations specifically addressing the impact of each loss function and hyperparameter choice would strengthen the validation of the approach.
questions: questions: 1. Why did you decide to train the slow adapter only in the initial session? Would there be any benefit to periodically updating it with new data to improve adaptability?
2. How did you determine the values for hyperparameters such as λ, Ldiag, Lrdn, and Lcos? Could you provide more details on the process and any automated search methods used?
3. Can you elaborate on the specific contributions of each loss function (knowledge transfer loss and feature alignment loss) in the overall performance of SAFE? How do these losses interact, and are there any potential conflicts between them?
4. Have you considered comparing SAFE with other PET methods that address the stability-plasticity trade-off, such as O-LoRA [1]?
[1] O-LoRA: Orthogonal Subspace Learning for Language Model Continual Learning, CVPR
limitations: limitations: The authors acknowledge that SAFE's training process involves complex loss functions and hyperparameters, which could be a barrier to practical implementation.
paper_summary: paper_summary: This paper proposes a slow and fast parameter-efficient tuning method for continual learning. Slow learner is learned on the first session and fixed with a transfer loss. Fast learner is continually updating for new tasks. Slow learner and Fast learner are further restricted to avoid forgetting.
strength_weakness: strength_weakness: Although the main idea is sound, it is not well supported by the experimental results. For instance, In Table 2, it shows that with only Fast learner the performance is already as good as the proposed method SAFE. In Table 3, the gain of the aggregation method is marginal compared to some straightforward baselines.  In Table 5, without using $L_{f2s}$ and $L_{s2f}$, the Fast learner obtains the similar results. All these components are claimed as main contributions.
questions: questions: Can you explain the marginal improvements of some components and how they can support the main contribution of the paper?
limitations: limitations: Yes, it is addressed in the appendix.

Comments: title: Further Clarifications (2/2)
Further Clarifications (1/2)
Further Clarifications
Further Clarifications
Official Comment by Reviewer Psm6
Further Clarifications (3/3)
Further Clarifications (2/3)
Further Clarifications (1/3)
comment: Q4: The authors provide a new analysis on the aggregation weights in rebuttal - “in combination with the results in Fig 4, FL and SL primarily exhibits higher weights in sessions 7-10 and 1-6 respectively”. How to explain this observation?

A4: Thank you for your question.
We would like to provide **further clarification regarding Figure 7** in the rebuttal PDF. Figure 7 depicts the average aggregation weights of both learners after the whole model is trained in the final incremental session. The horizontal axis represents the session number to which each test sample belongs. For instance, "1" in the horizontal axis indicates test samples belong to classes in session 1 (e.g., classes 0-19 in ImageNet-R). The vertical axis shows the average aggregation weights of each learner assigned to these test samples.
 
Thus, the results presented in Figure 7, in conjunction with those in Figure 4, are intended to illustrate how **the aggregated model dynamically leverages the strengths of both learners**. For example, in sessions 7-10 (classes 120-199), the fast learner consistently shows higher weights, which is consistent with its superior classification accuracy in these classes as depicted in Figure 4. This highlights the fast learner's adaptability. Conversely, in sessions 1-6 (classes 0-119) of Figure 7, the slow learner obtains higher weights, generally aligning with its demonstrated stability and better performance in these classes shown in Figure 4. By adaptively balancing the contributions of both learners, our method achieves a harmonious trade-off between stability and adaptability. This dynamic aggregation is key to the state-of-the-art performance we report across six datasets.

For the readers to have a better understanding, we will replace the session numbers in Figure 7 in our revised paper with the corresponding class numbers which directly correspond to the horizontal axis in Figure 4.

Q5: About the limitation - my comments are only about discussing more essential limitations in the paper.

A5: We appreciate your concern regarding the limitations of our method. Our approach is indeed built upon RanPAC [1], and as such, it shares some of the same limitations. For instance, our method **relies on a strong feature extractor** to effectively inherit generalizability from PTMs, making it less suitable for scenarios where training needs to be performed from scratch or starting from rather small tasks. However, existing continual learning methods that utilize self-supervised learning or otherwise create good feature extractor backbones could potentially draw inspiration from our method for downstream continual learning tasks.

Additionally, our method introduces three hyper-parameters to balance the loss functions during training, as previously discussed. While our experiments demonstrate that a set of default values works well across the six datasets included in the manuscript, we acknowledge that **these choices might not be optimal** when applied to datasets with essentially different statistical characteristics. Furthermore, the search for these hyperparameters currently lacks a strong theoretical foundation, which is an area that could benefit from further research and refinement.

To date, these are the primary limitations we have identified. We believe that acknowledging and addressing these limitations will help in understanding the scope and applicability of our method.

We sincerely hope that this response addresses your concerns, and we remain fully open to any further discussion to ensure the highest quality and clarity of our work.

Thank you again for your time and consideration.

[1] RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS2023.
Thank you for your valuable feedback. We are pleased to note that some of your concerns have been addressed in our previous response and are grateful for your acknowledgment of our contributions. Regarding the remaining issues and suggestions you mentioned, we would like to provide a more thorough discussion in this response. All the analysis, explanations and modifications provided in this response will be included in the revised version of our manuscript.

Q1: The authors may further highlight and clarify the relationship with RanPAC for the correlation matrix part. 

A1: Thank you for your thoughtful suggestion. We appreciate the opportunity to clarify the relationship between our method and RanPAC [1]. While both approaches leverage pre-trained models (PTMs) for continual learning, they target different components of the model. Specifically, RanPAC [1] focuses on deriving decorrelated classification weights for the **classification head** with **frozen features**, whereas our method emphasizes the improvement of **trainable feature embeddings** within the **feature extractor**.

Furthermore, there is a distinct difference in the correlation matrices utilized by the two methods. The correlation coefficients matrix in RanPAC [1], as shown in Figure 2 of their paper, has **dimensions $\mathbb{R}^{C \times C}$**, where **$C$ denotes the number of classes** in the classification head. In contrast, our method employs a cross-correlation matrix $\boldsymbol{M}$ of **dimensions $\mathbb{R}^{d \times d}$**, with **$d$ representing the feature dimension**, as detailed in line 156 of our manuscript.

Moreover, we would like to emphasize that **our method is orthogonal to RanPAC** [1]. In fact, our approach builds upon RanPAC, and as evidenced in Table 1 of our paper, our method consistently outperforms RanPAC [1] by a significant margin.

Q2: Furthermore, if the matrix is learnable, I am not confident about (or at least it is not clarified or analyzed) what kind of correlation can be learned and how to make sure a valid correlation can be learned, especially considering that the matrix and other parameters are learned simultaneously.

A2: Thank you for raising this important point. We would like to clarify that the correlation matrix $\boldsymbol{M}$ in our method is **not a directly learnable entity**. Instead, it is a temporary variable computed based on the learnable features (Eq. 1 of our paper) and is subsequently used to calculate the loss functions $\mathcal{L}\_{\mathrm{diag}}$ and $\mathcal{L}\_{\mathrm{rdn}}$. As such, the correlation matrix itself is not directly learned, but rather it is determined once all model parameters and inputs are set. 

In fact, the correlation matrix characterizes the **relationship between feature embeddings of PTM and the slow learner**. Concretely, the $i$-th row and $j$-th column of $\boldsymbol{M}$ measures the correlation between the $i$-th feature dimension (also termed as channel or pattern in the literature) of the PTM and the $j$-th feature dimension of the slow learner.

Q3: For the ablation studies on aggregation, I want to see how important aggregation inference is and how’s the sensitivity of it, for an already-trained model. It seems Table 2 is the ablation studies of whether slow or fast is included in the whole model, mainly the training process.

A3: Thank you for your insightful question. 

We argue that aggregation is important as it facilitates interaction between the slow and fast learner, aligning with the complementary theory. **The importance of our aggregation** can be demonstrated through the following experiments: 1) As shown in Table 7 of the rebuttal PDF, the proposed aggregation method results in an average improvement of 2.1% in final accuracy across six datasets compared to the baseline. 2) Table 3 of the manuscript further highlights that our aggregation strategy leads to an improvement of at least 0.53% in final accuracy when compared to other competitive aggregation alternatives.

Regarding **the sensitivity of aggregation inference**, it's worth noting that the only hyperparameter involved is $\gamma$. One can observe in Table 10 of the rebuttal PDF that $\gamma$ remains relatively stable within a certain range (i.e., $\gamma$ varies from 0.1 to 10). In our method, $\gamma$ is simply set to 1, as stated in line 246 of the manuscript.
I appreciate the authors’ response. The authors addressed my concerns partially; and I can see the contributions in the paper. I would like to be on the side of acceptance and hope that more detailed analyses and more insightful discussions can be added to improve the paper's quality. Based on the current rebuttal, I would like to maintain my original score. 


- The authors may further highlight and clarify the relationship with RanPAC for the correlation matrix part. Furthermore, if the matrix is learnable, I am not confident about (or at least it is not clarified or analyzed) what kind of correlation can be learned and how to make sure a valid correlation can be learned, especially considering that the matrix and other parameters are learned simultaneously. I suggest some strong analysis and justification for this part. 

- For the ablation studies on aggregation, I want to see how important aggregation inference is and how’s the sensitivity of it, for an already-trained model. It seems Table 2 is the ablation studies of whether slow or fast is included in the whole model, mainly the training process. 

- The authors provide a new analysis on the aggregation weights in rebuttal - “in combination with the results in Fig 4, FL and SL primarily exhibits higher weights in sessions 7-10 and 1-6 respectively”. How to explain this observation? 

- About the limitation - my comments are only about discussing more essential limitations in the paper.
Thank you for your thoughtful feedback. We greatly appreciate the time and effort you have put into reviewing our submission. We have carefully considered your concerns and have endeavored to address them in our rebuttal. Below, we would like to summarize the key points regarding your raised concerns.

1. We provide further explanations to address your misunderstanding that the slow learner operates under certain assumptions. We also experimentally verified that constraining the fast learner to the slow learner yields better results. 
2. We apply the proposed method to datasets with significantly varying data distributions between tasks, such as VTAB and DomainNet, and achieve state-of-the-art results. This demonstrates that our method is effective in scenarios where the data distribution of the first task diverges significantly from that of subsequent tasks.
3. For the concerns related to loss functions and hyper-parameters, we offer additional ablations and details to highlight the stability of the default choices, which are straightforward to implement in practice.
4. Thank you for bringing DAP to our attention. **We will include a discussion and comparison with it in the Related Work** of the revised version, further demonstrating the superiority of our approach.
5. Regarding concerns about the improvements of each component, we provide additional experimental results to demonstrate the significant improvement of each component in the overall effectiveness of our approach and their collaborative impact.

In addition, we would like to draw the reviewer's attention to the **unique contributions and features** of our work, particularly the **theoretical foundations** that distinguish it.
1.  To the best of our knowledge, we are the first to apply **slow and fast learning to parameter-efficient tuning** in addressing challenges in PTM-based continual learning.  
2.  Our slow learner inherits generalizable knowledge from PTMs, underpinned by **theoretical analysis** using information bottleneck theory—a crucial aspect often **overlooked in previous works**. This transfer allows the slow learner to generalize effectively to novel classes it has not encountered during training.
3.  We tackle the current limitations in adaptation by guiding the fast learner with the slow learner, thereby **eliminating the need for access to data or data distributions**. Throughout this process, the proposed fast learner continuously adapts to new knowledge while effectively mitigating the forgetting of previous knowledge.
4. We provide extensive experiments on multiple benchmarks, providing strong evidence for the effectiveness of the proposed method.

We hope these additional details and clarifications address your concerns. **We remain open to further discussion** to ensure the clarity and quality of our work. Given the extensive elaborations and the unique contributions of our research, **we respectfully request a reconsideration of the score**. We believe our work offers valuable insights that would be of great interest to the NeurIPS community.

Thank you again for your time and consideration.
Thank you for your feedback. We greatly appreciate the time and effort you have put into reviewing our submission. We have carefully considered your concerns and have endeavored to address them in our rebuttal. Below, we would like to summarize the key points regarding your raised concerns.

1) Regarding the concerns about training the slow adapter only in the initial session, we have provided both qualitative and quantitative explanations to demonstrate its effectiveness and necessity compared to periodic updates. 
2) For concerns related to loss functions and hyper-parameters, we offer detailed ablations and analyses to highlight their specific contributions and to show the stability of the default hyper-parameter choices, which facilitate practical implementation.
3) Additionally, we provided a specific explanation in the manuscript for why the proposed method was not compared with O-LoRA, an inspiring work in continual learning. Thank you for bringing this to our attention; **we will include O-LoRA** in the Related Work section of the **revised version** for discussion and comparison.

Moreover, we would like to draw the reviewer's attention to the **unique contributions and features** of our work, particularly the **theoretical foundations** that set our work apart.
1. To the best of our knowledge, we are the first to apply **slow and fast learning to parameter-efficient tuning** in addressing challenges in PTM-based continual learning. 
2. Our slow learner inherits generalizable knowledge from PTMs, underpinned by **theoretical analysis** using information bottleneck theory—a crucial aspect often **overlooked in previous works**. This transfer allows the slow learner to generalize effectively to novel classes it has not encountered during training.
3. We tackle the current limitations in adaptation by guiding the fast learner with the slow learner, thereby **eliminating the need for access to data or data distributions**. Throughout this process, the proposed fast learner continuously adapts to new knowledge while effectively mitigating the forgetting of previous knowledge.
4. The clear demonstration of the effectiveness and superiority of our proposed method.

We hope these additional details and clarifications address your concerns. **We welcome any further discussion** to enhance the clarity and quality of our work. Considering the extensive elaborations and the unique contributions of our research, **we would like to humbly request a reconsideration of the scoring**.

We believe our research offers valuable insights and contributions that would be of significant interest to the NeurIPS community.

Thank you again for your time and thoughtful consideration.
Thanks for your detailed responses. However, regarding the concerns I raised, I still remain unconvinced. After considering the opinions of other reviewers and the authors' rebuttal, I have decided to maintain my initial score.
Many thanks to the authors for the detailed answers. 
After considering both the other reviews and the rebuttals, I maintain the original rating.
**C. Unified Framework Synergy**

We would like to emphasize that our approach serves as **a unified framework** for PTM-based continual learning. Specifically, we would like to clarify that:

1) The effectiveness of our framework lies not only in the contribution of each individual component but also in the overall improvement it delivers, which can make the **impact of each component less apparent** when viewed in isolation. A similar situation is **observed in RanPAC** [1], accepted by **NeurIPS 2023**, where Phase 1 (a single component) results in improvements of only **0.3%** on CUB and OmniBenchmark, and **0.2%** on VTAB. In contrast, our two learners achieve a minimum improvement of **0.36%** on CUB, **0.75%** on OmniBenchmark, and **1.11%** on VTAB. **Furthermore**, our method consistently demonstrates **substantial improvements across all six popular datasets**.

2) The **components** of our method are **interdependent** and should not be separated. For example, the **fast learner relies on the slow learner for guidance** during adaptation. As demonstrated in Table 11 of the rebuttal PDF, the performance of the fast learner degrades by **10.73%** and **2.83%** when guided by alternative teachers.



**D. Contributions Beyond Experimental Results**

We would like to draw the reviewer's attention to the **unique contributions and features** of our work (particularly the **theoretical** foundations that set our work apart): 

1) To the best of our knowledge, we are the first to apply **slow and fast learning to parameter-efficient tuning** for addressing challenges in PTM-based continual learning. 
2) Our slow learner inherits generalizable knowledge from PTMs, supported by **theoretical analysis** using information bottleneck theory, a crucial aspect often **overlooked in previous works**. This transfer enables the slow learner to generalize well to novel classes that it has not encountered during training. 
3) We address the **current limitations in adaptation** by guiding the fast learner with the slow learner, **eliminating the need for access to data or data distributions** [2][4]. During this process, the proposed fast learner continuously adapts to new knowledge while effectively mitigating the forgetting of previous knowledge.

---

In conclusion, we believe that the thorough analysis and evidence provided in our response demonstrate the robustness and effectiveness of our proposed method. By addressing each of the reviewer's concerns and presenting detailed comparisons with SOTA approaches, we have shown that our work not only advances the field but also offers unique contributions through its theoretical foundations and innovative components. **We sincerely hope that this additional clarification helps to illustrate the significance of our research and leads to a favorable evaluation.** Thank you once again for your thoughtful review and consideration.
**B. Notable Component-Wise Improvement**

Our method is composed of three key components: a slow learner, a fast learner, and an aggregation module. We would like to emphasize that **each component plays a significant role in the overall effectiveness of our approach**.

1) **Slow Learner**: Our method demonstrates **significant improvements over the baseline and other alternatives**: **1)** As shown in Table 2 of the manuscript, the proposed slow learner surpasses the baseline method RanPAC [1] by **3.24%** on ImageNet-A. **2)** As indicated in Table 7 of the rebuttal PDF, the proposed slow learner achieves an **average improvement of 1.19%** in final accuracy **across six datasets** compared to the baseline. **3)** As shown in Table 4 of the manuscript, the slow learner **outperforms competing methods** that also aim to transfer statistical knowledge in PTMs. The final accuracy of the slow learner improves by **2.63%** compared to first-order information transfer and by **2.24%** compared to second-order information transfer. **4)** The visualization via t-SNE shows **superior generalization** of the slow learner **on unseen classes** in Figure 3 in the manuscript. The key results are summarized in the following table.

|    Method    | Performance (%) |            Competitor             | Data Source |
|:------------:|:---------------:|:---------------------------------:|:-----------:|
| Slow learner |      +3.24      |             Baseline              |   Table 2   |
| Slow learner |      +2.63      | First-order information transfer  |   Table 4   |
| Slow learner |      +2.24      | Second-order information transfer |   Table 4   |

2) **Fast Learner**: Our method shows **notable improvements compared to the baseline and other approaches**. Specifically: **1)** As shown in Table 2 of the manuscript, the proposed fast learner **surpasses the baseline by 4.29%** on ImageNet-A. **2)** As indicated in Table 7 of the submitted PDF, the proposed fast learner achieves an **average improvement of 1.72%** in final accuracy **across six datasets** compared to the baseline. **3)** The following ablation table, which extends Table 5 of the manuscript with a new row, demonstrates that **both $\mathcal{L}\_{\mathrm{cos}}$, $\mathcal{L}\_{\mathrm{f2s/s2f}}$ contribute significant improvements.** Specifically, solely using **$\mathcal{L}\_{\mathrm{f2s/s2f}}$** results in an improvement of **3.10%** compared to the baseline, while **$\mathcal{L}\_{\mathrm{cos}}$** yields a gain of **3.86%** over the baseline. **4)** As shown in Table 11 of the rebuttal PDF, the fast learner guided by the slow learner achieves an improvement in final accuracy of **10.73%** and **2.83%** compared to **other teacher choices**. **5)** As shown in Figure 7(b) in the additional PDF and Figure 4 in the submitted manuscript, **fast learner compensates for the plasticity** issue of slow learner and demonstrates superiority in new classes. 

| Method                                                                            | Final     | Avg       |
| --------------------------------------------------------------------------------- | --------- | --------- |
| Baseline                                                                          | 62.21     | 72.31     |
| Fast Learner w/o $\mathcal{L}\_{\mathrm{f2s/s2f}}$, $\mathcal{L}\_{\mathrm{cos}}$ | 8.16      | 30.73     |
| Fast Learner w/o $\mathcal{L}\_{\mathrm{cos}}$                                     | 65.31     | 73.88     |
| Fast Learner w/o $\mathcal{L}\_{\mathrm{f2s/s2f}}$                                 | 66.07     | 74.20     |
| **Fast Learner**                                                                  | **66.49** | **74.50** |


3) **Aggregation Module**: 
- **First**, the proposed aggregation method demonstrates **significant improvements over the baseline and other alternatives**. Specifically: **1)** As indicated in Table 7 of the rebuttal PDF, the proposed aggregation approach achieves an **average improvement of 2.1%** in final accuracy **across six datasets** compared to the baseline. **2)** As shown in Table 3 of the submitted manuscript, the aggregation method results in **at least a 0.53% improvement** in final accuracy compared to straightforward baselines.
- **Second**, the proposed aggregation method **does not require complex modules or trainable parameters**, making it straightforward to implement.
- **Third**, it is important to recognize that **aggregate** methods are generally more **challenging to improve**. For example, the **LAE**[6] method from **ICCV 2023** achieved only a **0.4%** increase in average accuracy, with a **0.4% decrease** in final accuracy on ImageNet-R. In contrast, our method demonstrates a **2.85% final** improvement on ImageNet-R and an **average** improvement of **2.26%**.
Thank you for taking the time to review our previous response and provide your feedback. We appreciate your concerns and would like to further **clarify the effectiveness of our work from the following four perspectives** for your consideration.

**A. Significant Overall Performance of Our Method**

First of all, as validated in Table 1 of our paper, compared to recent SOTAs, **our method shows substantial improvements across all 6 popular datasets**. For example, we surpass the second-best result on ImageNet-A over **4%** which indicates notable superiority. 

Second, we would like to further highlight the **significant relative improvements** against previous methods. In particular, we compare our method with two representative SOTA methods, RanPAC [1] and SSIAT [2], from NeurIPS2023 and CVPR2024 respectively, and results are summarized in the table below. We remark that 1) SSIAT [2] did not report (denoted by symbol "-") the result on Omnibenchmark (OB) and hence we compute both the average of all available datasets except OB and the average of all 6 datasets; 2) Metrics in the brackets denotes the corresponding increase (↑) / decrease (↓) compared to its previous SOTA.

| Method               | CIFAR | IN-R        | IN-A        | CUB         |  VTAB      |   OB | Avg of first 5 datasets (Avg5)       | Avg of all 6 datasets (Avg6)       |
|----------------------|-------|-------------|-------------|-------------|-------------|------|-------------|-------------|
| RanPAC [1] (NeurIPS 2023) | 92.2  | 78.1        | 61.8        | 90.3        | 92.6        | 79.9 | 83.0        | 82.5        |
| SSIAT [2] (CVPR 2024)     | 91.4 (↓0.8) | 79.6 (↑1.5) | 62.2 (↑0.4) | 88.8 (↓1.5) | 94.5 (↑1.9) | -    | 83.3 (↑0.3) | -           |
| **Ours**            | **92.8 (↑1.4)** | **81.0 (↑1.4)** | **66.6 (↑4.4)** | **91.1 (↑2.3)**  | **95.0 (↑0.5)** | **80.9 (↑1.0)** | **85.3 (↑2.0)** | **84.6 (↑2.1)** |

One can observe from the table that:
- **The relative improvements of our method compare favorably against those of the most recent SOTA SSIAT [2]**. Instead of only 0.3% increase of SSIAT [2] in Avg5 against its previous SOTA RanPAC [1], our method boosts the metric by a significant margin 2.0% compared to SSIAT [2] and outperforms RanPAC [1] by 2.1% in Avg6 metric.
- Our method shows **consistent improvement over all datasets**. Compared to SSIAT [2] even performs inferior to its previous SOTA RanPAC [1] on CIFAR and CUB, our method outperforms previous methods on all the datasets with various characteristics. This validates the robustness of the proposed method.

Based on the above observations, we believe that our method is sufficiently effective when compared with baselines.

In addition, we would like to remind the reviewer that due to the **inherent difficulty of the tasks**, the trade-off between stability and plasticity, and the fact that our method builds upon pre-trained models (PTMs) rather than starting from scratch, these improvements are even more challenging to achieve. **Other methods show less significant improvements** [3][5]. For instance, **EASE** [5], accepted by **CVPR 2024**, achieved a **0.03%** improvement in final accuracy and a **0.08%** increase in average accuracy on CUB compared to the second-best result. On OmniBenchmark, EASE outperformed the second-best result by **0.06%**. **FeCAM** [3], accepted by **NeurIPS 2023** did **not** surpass recent exemplar-based state-of-the-art results on CIFAR.

[1] RanPAC: Random Projections and Pre-trained Models for Continual Learning. NeurIPS2023.

[2] Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer. CVPR2024.

[3] Exploiting the heterogeneity of class distributions in exemplar-free continual learning. NeurIPS2023.

[4] SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model

[5] Expandable subspace ensemble for pre-trained model-based class-incremental learning. CVPR2024.

[6] A Unified Continual Learning Framework with General Parameter-Efficient Tuning. ICCV2023.
Thank you for the responses. However, I remain unconvinced that the improvements are significant, so I will maintain my initial score.