Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning
Authors: Fangwen Wu
Lechao Cheng
Shengeng Tang
Xiaofeng Zhu
Chaowei Fang
Dingwen Zhang
Meng Wang
Venue: ICML 2025 oral (2025)
Keywords: Class-incremental learning
continual learning
Primary area: deep_learning

Abstract: Class-incremental learning (CIL) seeks to enable a model to sequentially learn new classes while retaining knowledge of previously learned ones. Balancing flexibility and stability remains a significant challenge, particularly when the task ID is unknown. To address this, our study reveals that the gap in feature distribution between novel and existing tasks is primarily driven by differences in mean and covariance moments. Building on this insight, we propose a novel semantic drift calibration method that incorporates mean shift compensation and covariance calibration. Specifically, we calculate each class's mean by averaging its sample embeddings and estimate task shifts using weighted embedding changes based on their proximity to the previous mean, effectively capturing mean shifts for all learned classes with each new task. We also apply Mahalanobis distance constraint for covariance calibration, aligning class-specific embedding covariances between old and current networks to mitigate the covariance shift. Additionally, we integrate a feature-level self-distillation approach to enhance generalization. Comprehensive experiments on commonly used datasets demonstrate the effectiveness of our approach. The source code is available at https://github.com/fwu11/MACIL.git.

Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning
Fangwen Wu Lechao Cheng x Shengeng Tang² Xiaofeng Zhu' Chaowei Fang?
Dingwen Zhang 94
Meng Wang?
Abstract
Class cect- in Task t-1
Meao w Compensation
Class-incremental learning (CIL) seeks to enable a model to sequentially learn new classes while retaining knowledge of previously learned ones. Balancing flexibility and stability remains sig- nificant challenge, particularly when the task ID is unknown. To address this, our study reveals that the gap in feature distribution between novel and existing tasks is primarily driven by differ- ences in mean and covariance moments. Building on this insight, we propose a novel semantic drift calibration method that incorporates mean shift compensation and covariance calibration. Specifi- cally, we calculate each class's mean by averaging its sample embeddings and estimate task shifts us- ing weighted embedding changes based on their proximity to the previous mean, effectively cap- turing mean shifts for learned classes with each new task. We also apply Mahalanobis dis- tance constraint for covariance calibration, align- ing class- class-specific embedding covariances between old and current networks to mitigate the covari- ance shift. Additionally, we integrate feature- level self-distillation approach to enhance gener- alization. Comprehensive experiments on com- monly used datasets demonstrate the effectiveness of our approach. The source code is available https://ittubb.comffvll Tttp:/wwl/MMAAIII I/MACIL.gir.
Drift
Class cEct- 1
Task
Covariance Calibration
(a) Semantic Drift
(b) Calibration
Figure I. As new tasks Iearned, the categories from previously tasks in the latest updated model contimuously experience shifts in their means and variances. referred this paper, we calibrate such semantic drift by applying explicit mean shift compensation and implicit variance constraints (b).
(a) Semantic Drift. In
et al., 2017). The primary goal of continual learning (Cheng et al., 2024a;;; Zhang al., 2024; Huang al., 2024a;b) is to facilitate knowledge accumulation and transfer, allowing models to adapt quickly to new, unseen tasks while maintain- ing robust performance on previously learned tasks (Parisi et al., 2019). This capability has broad applications fields such as computer vision, robotics, and natural language processing.
In recent years, the issue of model plasticity has become less prominent in deep learning-basee based approaches, primarily due to two factors: (1) the increasing capacity of deep models allows them to effectively over-fit new data, and (2) large- scale pre-training on extensive datasets equips models with powerful feature extraction capabilities (He et al., 2019). Parameter-efficient fine-tuning based on pretrained models has further enhanced model plasticity, as highlighted in numerous recent studies (Houlsby et al., 2019; Lester et al., 2021; Hu et al., 2022).
1. Introduction
Continual Learning, also referred lifelong learning. aims to enable machine learning models to sequentially learn multiple tasks over their life-cycle without requiring retraining or access to data from previous tasks (Rebuffi
Despite these advancements, existing methods as regularization (Kirkpatrick et al., 2017), memory replay (Lopez-Paz & Ranzato, 2017), and knowledge distillation (Li & Hoiem, 20ile-while improving stability to some extent, introduce additional costs. For example, (1) memory replay methods require storing and retraining old task data, increasing storage and computational demands. (2) knowledge distillation involves additional computational overhead during the distillation process, complicating and slowing down training. These additional costs hinder the
'Zhejiang Lab 'Hefei University of Technology 'Xidian Uni- versity "Northwestern Polytechnical University. Correspondence to: Lechao Cheng <chengle hfu.tt.edu.n>.. Dingwen Zhang <zhangdingwen2006yy @ comi..com>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Navigating Semantic Drift in Task-Agnostic Clase-nneremental Learning
practical deployment of continual learning methods. There- fore, key challenge in the field is improve model stability while minimizing resource consumption and computational overhead (Wang et al., 2024b).
2. Related Works
2.1. Class-Incremental Learning
Class-Incremental Learning (CIL) aims to enable a model to sequentially learn new classes without forgetting previ- ously learned ones. This presents a significant challenge. especially since task-IDDs are not available during inference. To address this issue, several strategies have been proposed, which can be broadly categorized as follows:
In this paper, we build upon successful practices by lever- aging parameter-efficient fine-tuning based on pretrained models to further analyze catastrophic forgetting. Through extensive experimental observations, we discovered that al- though low-rank adaptation (e.g.. LoRA (Hu et al., 2022)) based on pretrained models can effectively maintain model plasticity, the incremental integration of tasks and model updates induces feature mean and covariance thift-watt we also term Semantie Drift. As illustrated in Figure 1, the feature distribution of the original task data undergoes significant mean shifts and changes in variance shape as more tasks are introduced.
Regularization-bann based methods (Li & Hoiem, 2017; Rebuffi et al., 2017; Kirkpatrick et al., 2017; Zenke et al., 2017) fo- cus on constraining changes to important model parameters during training on new classes. Replay-based methods ad- dress forgetting by maintaining memory buffer that stores examples from prior tasks. When learning a new task, the model is trained not only on the current task but also on these stored examples, helping it retain previous knowledge. These methods include direct replay (Lopez- PPzz& & Ranzato, 2017; Riemer et al., 2019; Chaudhry et al., 2019; Liu et al., 2021) as well as generative replay (Shin et al., 2017: Zhu et al., 2021). Optimization--ased methods focus explic- itly designing and modifying the optimization process to reduce catastrophic forgetting, such as through gradient pro- jection (Farajtabar et al., 2020; Saha et al., 2021; Lu et al., 2024) or loss function adjustments (Wang et al., 2021; Wen et al., 2023). Representation-ban based methods aim to maintain stable and generalizable feature space as new classes are added. These include self-supervised learning (Cha et al., 2021; Pham et al., 2021) and the use of pre-trained models (Wang et al., 2022a; Gao et al., 2024a; McDonnell et al., 2023). A key challenge for exemplar-free methods is the shift in backbone features. Recent studies have proposed estimating this shift through changes in class prototypes (Yu et al., 2020; Gomez- Villa al., 2024; Goswami al., 2024). This work investigates the semantic drift phenomenon in both the mean and covariance, calibrating them to mitigate catastrophic forgetting.
Based on these observations, we propose to address seman- tic drift with both mean shift compensation and covariance calibration, which constrain the first-order and second-order moments of the features, respectively. Specifically, we com- pute mean class representations after learning a novel task as the average embedding of all samples in class. The shift be- tween old and novel tasks is approximated by the weighted average of embedding shifts, where the weights are deter- mined by the proximity of each embedding to the previous class mean. This approach effectively estimates the mean shift for all previously learned classes during each new task. We also introduce an implicit covariance calibration tech- nique using Mahalanobis distance (Mahalanobis, 1936) loss to address semantic drift. This method aligns the covari- ance matrices of embeddings from old and current networks for each class, ensuring consistent intra-class distributions. By leveraging the old network as "past knowledge", we compute class-specific covariance matrices and minimize the absolute differences in Mahalanobis distances between embedding pairs from both networks. This approach effec- tively mitigates covariance shift, maintaining model stability while allowing continual learning. As shown in Figure 1(b), these constraints effectively maintain model stability while preserving plasticity. Additionally, we implement feature self-distillation for patch tokens, further enhancing feature stability. In summary, our main contributions are:
2.2. Pre-trained Model based Class-Incrementts Learning
Pre-trained models have become key component in CIL due to their ability to transfer knowledge efficiently. It is prevailing to use parameter-Fffficiet Fine- Fine-Tuning (PEFT) methods to adapt the model computation efficiently, PEFT methods introduce a small portion of the learnable param- eters while keeping the pre-trained model frozen. LoRA (Hu et al., 2022) optimizes the weight space using low-rank matrix factorization, avoiding full parameter fine-tuning; VPT (Jia et al., 2022; Wang et al., 2024d) injects learn- able prompts into the input or intermediate layers to ex- tract task-specific features while freezing the backbone net- work; AdaptFormer (Chen et al., 2022), based on adaptive Transformer components, integrates task-specific informa-
We delve into the exploration of the semantic drift problem in class-incremental learning and further pro- pose efficient and straightforward solutions-mean shift compensation and covariance calibration, which significantly alleviate this challenge.
We orchestrate an efficient task-agnostic continual learning framework that outperforms existing meth- ods across multiple public datasets, demonstrating the superiority of our approach.
2

---

Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning
tion with general knowledge.
task - 1, and it is frozen in task i.
Prompt-based class-incremental continual learning meth- ods dynamically adjust lightweight prompt parameters to adapt to task evolution. Key mechanisms include: dynamic prompt pool retrieval (Wang et al., 2022c), general and expert prompt design for knowledge sharing (Wang et al., 2022b), discrete prompt optimization (Jiao et al., 2024). con- sistency alignment between classifiers and prompts (Gao et al., 2024b), decomposed attention (Smith et al., 2023), one forward stage (Kim et al., 2024), and evolving prompt adapting to task changes (Kurniawan& et al., 2024). Adapter- based methods such as EASE (Zhou et al., 2024b) dynami- cally expand spec-fie subspaces and integrate multiple adapter predictions with semantie-guided prototype synthe- sis to mitigate feature degradation of old classes; SSIAT (Tan et al., 2024) continuously tunes shared adapters and estimates mean shifts, updating prototypes to align new and old task features; and MOS (Sun et al., 2025) merges adapter parameters and employs a self-optimization retrieval mechanism to optimize module compatibility and inference efficiency. LoRA-basee methods, such as InfLoRA (Liang & Li, 2024), introduce orthogonal constraints to isolate low- rank subspaces, effectively reducing parameter interference between tasks. Together, these methods offer efficient and scalable solutions for adapting pre-trained models to class- incremental learning tasks. In this work, we leverage the power of the LoRA in the context of CIL and build our semantic drift calibration modules on top of it.
3.2. Overview
Figure 2 illustrates the overall architecture for class- incremental learning. We employ a frozen pre-trained ViT (Dosovitskiy et al., 2021) model as the backbone with leam- able task-specific LoRA modules. The output class tokens are forwarded through a task-specific classifier, which gen- erates the class scores, while the angular penalty loss (Peng et al., 2022; Tan et al., 2024) is used to compute the classifi- cation loss Lets:
exp(s cos(),))
fels
log
(1)
n2.
cos())))
w,Jo, where cos(O,) = They Fel and n' is the number of training samples in task t. The mean and covariance of each class are stored for each learning session. Before the training process, covariance of each class is precomputed from the class tokens generated by the network trained previous tasks. These covariance matrices are then used to align the distribution of the rep- resentations generated by the current network with that of the old network, based on the Mahalanobis distance. This is referred to as the covariance calibration loss Loov. Fur- thermore, patch tokens are leveraged to preserve knowledge from earlier tasks at the feature level through a distillation loss Laistil-
8 represents the scaling factor,
3. Method
After the training process, the class means are updated through the mean shift compensation process, and the clas- sifier heads are retrained using the calibrated class statistics. The training pipeline is illustrated in Algorithm 1. In sum- mary, the overall training objective is:
3.1. Preliminaries
3.
LEARNING
Consider dataset consisting of T tasks (D*) For each task, the dataset D' = {aa} vt) ) contains n' inputs xf E Rd and their corresponding labels v C We use X and Y to denote the collection of input data and label of task f, respectively and C* = [c) is the label set contains |C'| classes. In the caas-incremeetal setting, for any task i * j. the input data from different tasks follow different distributions, i.e., p(X*)= A p(X)) and labels sat- isfy C' n C* G. The learning objective is to find the model f² IR + g where C² = U represents the total number of classes learned. This model is trained on all training datasets to perform well on all test dataset seen up to task t. In our scenario, the model f is based on pre-trained model, consisting of fz( = W (x)), where oh RD + 2d is a feature extractor composed of frozen pre-trained model and learnable parameters 0 in the LoRA modules, and a classification head W = (w" y for each task, where we have w² E dd For a given task t, the old network f/ (xx) refers to the network trained on
f Lets + Leor + A
3.3. Low-Rank Adaptation
In dass-inncremenal learning (CIL), task IDs are not pro- vided during the inference stage. For methods based on pre-trained models, the use of task-specific PEFT modules often involves task ID prediction step during testing (Wang et al., 2022b; 2024a; Sun et al., 2025). Other approaches avoid task ID prediction, as low prediction accuracy can negatively impact performance. For instance, some methods use weighted sums of prompts to determine the prompts ap- plied during inference (Kurniawan et al., 2024; Smith et al., 2023), or aggregate all previous PEFT modules (Zhou et al., 2024b; Liang & Li, 2024). Alternatively, some methods rely on shared PEFT module across tasks (Huang et al., 2024c; Tan et al., 2024).
Compared to other PEFT modules applied in CIL, such as prompts and adapters, LoRA (Hu et al., 2022) performs
3

---

Navigating Semantic Drift in Task-Agnostic Mlass-nncremental Learning
input of
Multi-layer Network for
During Training Task
patch takens
class
Task
specifpecific Classifier
Classification
patch takens
patch takens
Loss L..
LoRA
LoRA
LoRA
After Training
Self-distillation
Mean Shift Compensation
c [1,c]
Festure Sampling
Task-
Classification
Loss Liaia
specific
Re-training
Classifier
Loss frea
Wir
ApDx
class taken
class token
Calibration
Covariance Calibration Loss fe
Task
patch takens
cless token
Frozen Parameters
Covariance Calibration
Trainable Parameters
Task 8-1
LoRA
LoRA
LoRA
Loss Functions
Loss Lex.
input of 1-1
Multi-layer Network for Task t-1
Figure Illustration of our method at task t. The feature extractor task uses a frozen pre-trained VIT backbone with learnable LoRA modules. The output class tokens (yellow) are passed through classifier to compute the classification loss Lebs, and the mean and covariance each class are stored for each session. During training. class tokens (yellow and blue) are used to align class distributions via covariance calibration loss Lcov. Patch tokens from network (yellow) distill knowledge from network t 1 (bluc) through a distillation loss Ldisin. After training, the class means are updated using a mean shift compensation module, and the classifier heads are retrained with the calibrated statistics.
inference by adding the low-rank adaptation matrices to the original weight matrices. This enables LoRA to ef- ficiently combine the pre-trained model weights with the task-specific adaptation matrices, as well as the information across the different task-specific matrices.
classifier's performance. Thus, it is necessary to impose constraints on the semantic drift and calibrate the means and covariances of class distributions.
3.4.1. MEAN SHIFT COMPENSATION
In this paper, we utilize task-specific LoRA modules, where each task is assigned unique LoRA module. Considering the pre-trained weight matrix Wo E Id the update of the weight matrix is decomposed into the product of the low rank matrices B E Rdxr and A E R2r* where r a value much smaller than the input dimension and the output dimension k. The aggregation of all previous LoRA modules at task 1 can be expressed as W Wo Ei= A, Additionally, structures: the task-shared LoRA module, which uses common LoRA module for all tasks (W = Wy + BA), and a hybrid structure that combines both task-specific and task- shared designs, inspired by HydraLoRAA (Tian et al., 2024). In this hybrid structure, the shared LoRA module's matrix A used across tasks, while the independent LoRA module's matrix B, is specific each task (W- = Wo Ei=1 B,A). During training, only the low-rank weight matrices A, and B, are learnable, and the pre-trained weight Wo is frozen.
We define the mean class representation of class e in the embedding space after learning task as:
> M1 cl0)(2;) 16 No
(3)
where No is the number of samples for class c. The shift between the class mean obtained with the current network and the class mean obtained with the previous network can be defined as:
explore two other LoRA
Apt 1 = p. 1e
(4)
Previous work (Yu et al., 2020) suggests that the shift be- tween the true class mean and the estimated class mean can be approximated by the shift in the current class embeddings between the old and current models. Specifically, the shift of the class embeddings can be defined as:
A where belongs to class c. We can compute &6 (x:)) using the model trained on task t - 1 before the current task training. Then, we compute the drift (x:) and use it to approximate the class mean shift A 1-1
(b(2)) -
(5)
3.4. Semantic Drift
As tasks increase, we longer have access to the data from previous tasks, and therefore cannot compute the true distribution of earlier classes under the incrementally trained network. Both the mean and variance of feature distributions of the old classes change. This phenomenon referred to as semantic drift. When semantic drift occurs, it affects the
Apt-i- E, W. A 6 1- (x))
(6)
Σ,

---

Navigating Semantic Drift in Task-Agnostic Class- Incremeanlllltlll Learning
with
old and new networks:
110g (2r,) - 142 1
w exp(
Loow >>> das ( 1
20
OEC.
(10)
where is the standard deviation of the Gaussian kernel, and the weight wy indicates that embeddings closer to the class mean contribute more to the mean shift estimation of that particular class. The proposed method can be used to compensate the mean shift of all previously learned classes at each new task with m x ii: = A + p
- - (x)) 60 (0 -2-1)
3.5. Classifier Alignment
The model exhibits a tendency to prioritize the categories associated with the current task, resulting in a degradation of classification performance for categories from previous tasks. Upon completing the training for each task, the clas- sifier undergoes post hoc retraining using the statisties of previously learned classes, thereby enhancing its overall performance. It is assumed that the feature representations learned by the pre-trained model for each class follow a Gaussian distribution (Zhang et al., 2023). In this frame- work, the mean Pe and covariance Σ. of the feature repre- sentation for each class c, as described in previous sections, are calculated and stored. A number of 8, samples he then drawn from the Gaussian distribution . MW Σe) for each class as input. The classification head is subsequently retrained using cross-entrope loss function:
3.4.2. COVARIANCE CALIBRATION
In this section, we address semantic shift from the per- spective of the covariance matrix by introducing a novel covariance calibration technique, which is powered by Mahalanobis distance-based loss. The objective is to sure that, for each class in the new dataset, the embeddings generated by both the old and current networks follow the same covariance structure. Specifically, the covariance ma- trices of the embeddings from the current network should be aligned with those from the old network. To achieve this, utilize the old network, trained the previous task, as form of "past knowledge" and use it to calculate the covariance for each class in the current task. Since the Mahalanobis distance directly depends on the covariance matrix, optimizing the difference between embedding pairs from the old and current networks in terms Mahalanobis distance implicitly constrains the shape of the intra-class distribution, thus alleviating covariance shift.
s.|C] Σ log
Lhead
(11)
se
Evec (")(()))
where C denotes all classes learned until current task, is the classifier. With the alignment of semantic drift, the true class mean and covariance are calibrated, which helps mitigate the classifier's 's bias, typically induced by overconfi- dence new tasks, and alleviates the issue of catastrophic forgetting.
Mathematically, the Mahalanobis distance (Mahalanobis, 1936) is defined as the degree of difference between two ran- dom variables r and y. which follow the same distribution and share the covariance matrix Σ:
dm( ( - Tx
(8)
3.6. Feature-level Self-Distillation
To enhance the model's resistance to catastrophic forget- ting, we propose self-distillation approach that focuses on improving the utilization of patch tokens in classification tasks, suggested in (Li et al., 2024; Wang al., 2024c). In Vision Transformers (ViT), the information from patch tokens is often underutilized, which can limit the model's ability to generalize across tasks (Zhai et al., 2024). To address this, we introduce a self-distillation loss based on patch tokens.
In our setting, we calculate the Mahalanobis distance d (0)(%) oz Et-l) using the embedding pairs cal- culated with the data from the current task t and as computed by covariance matrix Σt- of the class from the current task with the old network pt-1
Before training, for each class the covariance matrix is computed as:
In this method, the class token output from the current net- work is treated as the essential feature information that needs to be learned for the current task. The feature-level self-distillation loss encourages the alignment of patch to- kens from the current network output with the class token. Specifically, we compute the angular similarity, sim,, be- tween the current task's patch tokens, denoted as pf. and the class token et, with the features normalized using 1.2
Σ (r (x= 1
(9)
N.
where 1 1 is the class mean of the class e calculated with the old network fr- The loss function for minimizing the absolute difference in Mahalanobis distances of the sample input pairs (Rt, 2.y) between embeddings obtained from the
5

---

Navigating Semantic Drift in Task-Agnostic Clase-nnerelental Learning
normalization.
The loss is then formulated as:
(Krizhevsky, 2009) is widely used dataset in CIL., con- taining 60,000 images across 100 categories. We also split CIFAR-100 into 5, 10, and 20 tasks with each task contain- ing 20, 10, 5 classes, respectively. CUB-200 (Wah et al., 201 is a fine-grained dataset containing approximately 11,788 images of 200 bird species with detailed class labels. ImageNet-A (Hendrycks et al., 2021 a real-world dataset consisting of 200 categories, notable for significant class imbalance, with some categories having very few training samples. We split CUB-200 and ImageNet-A into 10 tasks with 20 classes each.
sim (1), -1p3 11p 11 (12)
Ldistill T.
where p is the j-th patch token for the current task, of is the class token for the current task, p is the patch token from the previous task, and L is the total number of patch tokens for the current task. We disable the gradient updates during angular similarity computation.
The low angular similarity between patch tokens and class tokens suggests that the patch tokens contribute less to the semantic representation of the current task. To encourage better feature reuse, patch tokens with low similarity are encouraged to resemble the patch tokens from the previous network, thereby improving the retention of important task- related features. This approach ensures that patch tokens are more effectively utilized, contributing to the model's 's robust- ness and its ability to mitigate forgetting when transitioning between tasks.
We follow the commonly used evaluation metrics in CIL. We denote ai as the classification accuracy evaluated on the test set of the j-th task (where j S i) after learning tasks in incremental learning. The final accuracy is calculated as A 1 و Di,5 and the average accuracy of all incremental tasks is Aurg + - A,. In line with other studies, our evaluation results are based on three trials with three different seeds. We report both the mean and standard deviation of the trials.
Algorithm Semantic Drift Calibration
4.1.2. IMPLEMENTATION DETAILS.
Require: Incrementally learned model {o))7=1 with learn- ing parameters 0, classifiers [w"]T= dataset (D')T=i
In our experiment, we adopt ViT-B/16 (Dosovitskiy et al., 2021) pre-trained on ImageNet21K (Russakovsky al., 2015) as the backbone. We use the SGD optimizer with the initial learning rate set as 0.01 and we use the Cosine Annealing scheduler. We train the first session for 20 epochs and 10 epochs for later sessions. The batch size is set to 48 for all the experiments. LoRA module is inserted to the key and value of all the attention layers in the transformer. The distillation loss weight À is set to 0.4, the LoRA rank is set to 32, and the scale in the angular penalty loss is set to 20, These values are determined through sensitivity analysis.
1: for task 1 = 1 to T do
2: for C C* do
3: Extract features (x;) using the frozen model
learned from task t- 1;
4: Compute mean = and covariance E
5: end for
6: for Batch { {(x{, v{) sampled from D' do
7: Train ob and wt using L = Lets + Leov + A
8: end for
9: for CE CEU do
10: Estimate and compensate the class mean shift n": ~ Ao 1 + App-
4.2. Comparison with State-of-the-arts
We conduct comparative evaluation of our proposed method against state-of-ther the-art (SOTA) class-ineremental learning (CIL) approaches based on pre-trained models, with a particular focus on techniques utilizing parameter- efficient fine-tuning (PEFT). To ensure a fair comparison, we evaluate all methods using the same ViT-B/16- pre-trained models, identical random seeds, and consistent class orders. Specifically, proaches, including L2P (Wang et al., 2022c), DualPrompt (Wang et al., 2022b), CODAPrompt (Smith et al., 2023), VQ-Prompt (Jiao et al., 2024), OS-Prompt (Kim et al., 2024), and CPrompt (Gao et al., 2024b); adapter-based methods such as SSIAT (Tan et al., 2024), EASE (Zhou et al., 2024b), and MOS (Sun et al., 2025); the LoRA-based method InfLoRA (Liang & Li, 2024) integrated with Classi- fier Alignment, referred to as Infl +RRA+CA; the first-session adaptation method RanPAC (McDonnell et al., 2023), which
11: Sample from N(re Σx) with the calibrated statis-
tics and retrain the classifiers {w He with Lread;
12: Store mean Me and covariance Σg;
13: end for
14: end for
4. Experiments
compare prompt-based ap-
4.1. Setup
1. DATASETS AND METRICS.
We train and validate our method using four popular CIL datasets. ImageNet-R (Hendryeks et al., 2021a) is gener- ated by applying artistic processing to 200 classes from ImageNet. The dataset consists of 200 categories, and we split ImageNet-R into 5, 10, and 20 tasks, with each task containing 40, 20, and 10 classes, respectively. CIFAR-100
6

---

Navigating Semantic Drift in Task-Agnostic Class- Ineremennllllttll Learning
Table Last and average performance results on four benchmark datasets (10 tasks) are reported. The mean and standard deviation of three trials are provided. We compare all methods using the same TTT-B/16-IN21I1 backbone, seeds, and class orders. For L2P and DualPrompt, we use the implementations provided by (Zhou al.. 2024a). Missing implementations on the datasets are denoted as '
ImageNet-R
Image Net-A
CUB-200
CIFAR-100
Method
A
Akyz
ALso
Ane
A1aot
Aaug
Aua
AAu
L2P(Wang et al., 2022c)
70.56:10.51
75.60::.34
84.74=0.4
1..65=1.002
DualPrompt(Wang et al., 2022b)
66.8910.70
71.60:10.31
85.17=0.55
89.18:11011
CODD--Propt(Smihh et al., 2023)
72.82+0.50
18.13++0.52
87.00+0.31
90.68+11
SLCA(Zhang et ul., 2023)
78.95:-0.36
83,20:1:.23
86. 13:1075
91.75=0.442
90.30=0.43
93.32:10.99
RanPACIMeDenmell et al., 2023)
77.944:0.1
82.98:1..31
62.25=0.33
69.92:11.99
89.94.10.52
93.68::.488
92.09-0.133
94.75:10.64
OS- SS-Prropt(Kim al., 2024)
74.76+00.23
80.29:#0.71
86.50=0.11
1.36
VQ-Prumpp(Jiao et ul., 2024)
75.68:10.23
80,02:10.18
86.47:10.40
91.37.1..4
90.27=00066
93,10:10.84
CPrompt(Gao et al., 2024b)
76.38:1046
81.52:1.38
87.63=0.17
91.50:11.10
EASE/Zhou et al., 2024b)
75.91
81.38:10.29
14.93=1.14
63.92-10.66
85.04
90.9311.03
88.22:0044
92.02:10.76
Infl Loo.AAAAAA--aann 14, 2024)
78.78+0.31
83.37+0.54
91.39+0.27
94.06+00888
SSIAT(Tan et al., 2024)
79.55:10.27
83,70:1.388
62.65=1.22
1..14:124
89,6810,48
93.67:0.46
91.41=0.14
94,27:10.75
MOS(Sun et al., 2025)
77.6841.44
82.0611..53
54.75=1.09
63.32:12.38
89.97-1.32
93.4330.600
91.53=00.35
94,2111.91
Ours
81.88++0.07
85.95+0.27
64.14=0.58
71.45+1.35
90.52++0.13
93.93+0.47
91.94+0.17
94.43+0.77
Table Last and average results on ImageNet-R and CIFAR-100 are reported. The mean and standard deviation of three trials are backbone, seeds, and class orders.
provided for 5 and 20 tasks settings. We compare all methods using the same
ImageNet-R
CIFAR-100
Method
5 tasks
20 tasks
tasks
20 tasks
Ausa
Aaua
ALux
Asv
ALax
Awe
Auax
Awe
CODA (Smmpit et al., 2023)
74.91110.33
09.55
68.62.10.52
74.61 +0.31
89. 89.16:10.08
92.46-10.79
01.18
86.62:10.93
SLCA(Zhang et al., 2023)
81.0110.11
84.1810.288
77.23110:40
82.2110.49
91.3010.54
93.97=0.56
88.54
92.6620.78
RanPAC(McDommell et al., 2023)
79.53+0.12
83.69+0.13
75.47+00.22
81.20+0.15
92.68+0.16
94.85+0.554
90.77+0.17
94.00+0.77
Promppp(KKi et al.. 2024)
75.78:10.06
80.49:0.49
72.50:10.78
78.43=1107
88.35+0.23
92.0410.76
81.4610.56
17.15:11.15
CPrompi(Gao al., 2024b)
77.9910.31
82.34:1.322
73.77:10.14
79.51
89.04:10.41
92.26=0.57
84.48::046
19.35::1.09
VQ VQ-Prompt(Jiao et al., 2024)
76.00++0.28
79.84=0.56
74.76++:0.29
79.30=0.34
90.97+0.17
93.50=0.67
09.25=0.43
92.53=+0.68
EASEEZhou al., 2024b)
76.75410.41
81.14:0.31
73.90:10.61
80.26:10.2
89.53:10.15
92. 02.6733
06.30-0..6
90.80:10.99
SSIAT((Tan et ul., 2024)
80.52:10.07
84.25:10.11
78.35:10.34
82.39:1.42
92.0110.115
94.37-10.68
90.07:00.44
93.52:10.65
IntLoRA+CA(Liagg & Li, 2024)
80.92:10.28
84.22:0.30
76.50:10.30
81.57:0.34
92.28:10.06
94.4622..3
90.39=0.01
93.32:10.22
MOS(Sun et 2025)
78.76+0.17
82.37+0.26
75.16+00.59
8..53+0.70
92.31+00.20
94.44++0.74
89.43++037
92.95+00.79
Ours
83.37-40.26
87.00+0.35
79.43++0.3
84.34+0.32
92.40+0008
94.71+0.64
90.51+0.14
93.48-+0.67
only trains PEFT modules using data from the first task; and the fine-tuning method SLCA (Zhang et al., 2023), which also incorporates the Classifier Alignment step as in our approach.
best method, SSIAT, by a significant margin of 2.33%. The Aay also surpasses the second-best SSIAT by 2.25%. On the ImageNet-A dataset, our method achieves a final accu- racy of 64. 14%, surpassing the second-best SSIAT by 1.49%.
These results highlight the effectiveness of our PEFT-based approach in significantly improving performance on datasets with large domain shifts, outperforming both first-session adaptation methods and full fine-tuning methods, as well as other PEFT-based approaches.
Table I summarizes the performance of these SOTA meth- ods across four widely used benchmark datasets. We report both the accuracy on the last task (Atasa) and the average accuracy across all tasks (Awv)), presenting the mean and standard deviation over three independent runs with differ- ent random seeds. The use of random seeds introduces vari- ability in class order across runs, making the evaluation of model performance more challenging. Notably, our method achieves superior performance in both Am and Awg. Our method demonstrates impressive results, particularly on the more challenging datasets with larger domain gaps, such as ImageNet-R and ImageNet-A. On ImageNet-R, our method achieves a final accuracy of 81.88% surpassing the second-
In contrast, on the CIFAR-100 and CUB-200 datasets, our method performs well, though with marginal benefits com- pared to other methods. Notably, on the CUB-200 dataset, our method achieves superior performance. It is also impor- tant to note that the first-session adaptation-based method, RanPAC, performs well on both CIFAR- 100 and CUB-200, likely due to the significant relevance between the pretrain- ing dataset (ImageNet) and these two datasets.
7

---

Navigating Semantic Drift in Task-Agnostic Class- Inssemnncrlllttll Learning
ImagNet-R 5-Task
ImagNet-R 10-Task
ImagNet-R 20-Task
CIFAR-100 10-Task
93
98
90
Accuracy (%6
90
Accuracy (%)
96
87
86
86
Accuracy
94
84
84
82
82
81
92
80
78
90
78-
76
78
75
88
1
2
7
8 9
10
1
3
1113151719
3 4
6 Task
5
7
8
9
10
Task
Task
Task
SLCA
RanPAC
MOS
CPrompt
InfLoRA
EASE
SSIAT
OURS
Figure 3. The performance of each learning session under different settings of ImageNet-R and CIFARIOO, All methods are initialized with ViT-B/16-IN2 These curves plotted by calculating the average performance across three different seeds.
Additionally, we evaluate the performance of our method on both longer task sequences (20 tasks) and shorter task sequences (5 tasks) for CIFAR-100 -10 and ImageNet-R, as reported in Tables 2. Across these varied experimental settings, method consistently outperforms competing approaches, demonstrating its stability and robustness in handling diverse CIL scenarios.
Table 3. The ablation studies for each component contribution evaluated on 10-session ImageNet-R. MSC means Mean Shift Compensation. CC is Covariance Calibration. PD indicates Patch Distillation. The exp-I is the baseline (cLoRA+Las)
Components
MSC
cc
PD
Aus
Axu
Ablations
exp-I
x
x
x
79.36
0.57
84.58 +: 0.67
exp-11
v
x
x
80.81
0.16
85.47 0.36
Figure 3 illustrates the incremental accuracy of each session for three ImageNet-R settings and one CIFAR- setting. The results show that on ImageNet-R, our method consis- tently achieves the best performance, with clear distinction from other methods. On CIFAR-100, tively stable, and the final results are comparable to the first-session adaptation- based method, RanPAC, which is closely related to pretraining data characteristics.
exp-Ill
x
v
x
80,70
= 0.32
85,52 0.42
exp-IV
v
v
x
81,60
0.09
85.88 + 0.35
exp-V
81.88
0.07
85.95 + 0.27
method is rela-
4.3.2. LoRA STRUCTURES DESIGN
While there has been significant exploration of aask-specific and task-shared PEFT modules, particularly concerning prompts and adapters, research on LoRA-based modules is relatively limited. In this paper, we investigate the use of task-specific and task-shared LoRA modules, as well hybrid architecture that combines both, inspired by Hydra-LoRA (Tian al., 2024). In Table 4, we evaluate these designs across four datasets and report the final ac- curacy averages from three trials. The results indicate that the performance differences among the LoRA designs are minimal, with the task-specific design slightly outperform- ing the other two, except for the ImageNet-A dataset, where the task-shared LoRA module achieves marginally higher performance.
4.3. Ablation Study
4. THE IMPACT OF EACH COMPONENT
As shown in Table 3, we systematically assess the con- tributions of different components to the baseline method on the ImageNet-R dataset. The baseline, consisting of a task-specific LoRA structure with angular penalty loss for classification, achieves competitive performance of 79.36% in Alau- In exp-II, we add the MSC module, which, in conjunction with classifier alignment, provides im- provement of over 1.4%. The incorporation of Covariance Calibration with classifier alignment in exp-III also leads to an improvement of approximately 1.35%. These results underscore the importance of both mean shift compensation and covariance calibration aligning feature distributions across tasks, thereby reducing catastrophic forgetting and enhancing stability across task sequences. Combining the MSC and CC modules gives significant boost, improv- ing performance by approximately 2.3% above the bascline method. Finally, the inclusion of patch distillation offers a further marginal improvement, resulting in a state-of-the the art performance of 81. 81.88% for ALa and 85. 85.95% for A confirming the effectiveness of our method.
Table 4. Experimental results of different LoRA structures. We report the final accuracy ALas. Average of three trials
Structure
CIEAR-100
ImageNet-R
ImageNet-A
CUB-200
G-LoRA
91.53:0.119
81.1520.19
64.27-1.088
90.16:20.56
B-LORA
91.94-1.17
81.88=0.07
64.14-t0.58
90.52-+0.13
Hydra LoRA
91.18:10.13
81.00.0.09
63.64.10.59
89.82.10.50
5. Limitations and Future Works
In this study, we have focused on addressing semantic drift by aligning first-order (mean) and second-order (covariance)
8

---

Navigating Semantic Drift in Task-Agnostic Clesemnnallllntal Learning
statisties. While this approach has shown promising results, it is inherently limited in its ability to capture more complex aspects of feature distribution shifts. Specifically, higher- order moments, such as skewness (third-order statistic) and kurtosis (fourth-order statistic), are not considered in this framework. These higher-order statisties could provide addi- tional insights into the shape and tails of the data distribution, which may help in mitigating semantic drift more effectively, especially in tasks with significant feature distribution shifts. Future work will explore this approach by incorporating higher-order statistical moments like skewness and kurtosis into the alignment process.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with a-gem. In ICLR, 2019.
Chen, S., Ge. C., Tong, Z., Wang. J., Song, Y., Wang, J., and Luo, P. Adaptformer: Adapting vision transform- ers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 15:16664-16677 2022.
Cheng, D., Ji, Y., Gong, D., Li, Y., Wang. N., Han, J., and Zhang, D. Continual all-in-one adverse weather removal with knowledge replay on a unified network structure. IEEE Transactions on Multimedia, 2024a.
Cheng. D., Zhao, Y., Wang, N., Li, G., Zhang, D., and Gao, X. Efficient statistical sampling adaptation for exemplar- free class incremental learning. IEEE Transactions on Circuits and Systems for Video Technology, 2024b.
6. Conclusion
We analyze catastrophic forgetting in machine learning models using parameter-efficieet fine-tuning based on pre- trained models. Our experiments reveal that low-rank adap- tations like LoRA induce feature mean and covariance shifts, termed Semantic Drift. To address this, we propose mean shift compensation and covariance calibration to constrain feature moments, maintaining both model stability and plas- ticity. Additionally, we implement feature self-distillation for patch tokens to enhance feature stability. Our task- agnostic continual learning framework outperforms existing methods across multiple public datasets.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference Learning Representations, 2021. URL https:// openreview. net htt///ooumm//// i d
Farajtabar, M., Azizan, N., Mott, A., and Li, A. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773, PMLR, 2020.
Acknowledgements
This work has been supported by the New Comnestoone Sci- ence Foundation through the XPLORER PRIZE, National Natural Science Foundation of China Grant No. (72188101, 62472139, U22A6001), National Key Research and De- velopment Program of China No.2023YFE010866 oneer" and "Leading Goose" R&D Program of Zhejiang No. No.20245SYS0002, the Anhui Provincial Natural Science Foundation, China (Grant No. 2408085QF191), the Funda- mental Research Funds for the Central Universities (Grants No. JZ2024HGTA0178. JZ2023HGQA0097)) and the Open Project Program of the State Key Laboratory of CAD&CG (Grant No. A2403), Zhejiang University.
Gao, X., Dong, S., He, Y., Wang, Q. and Gong, Y. Be- yond prompt learning: Continual adapter for efficient rehearsal-free continual learning. In European Confer- ence on Computer Vision, 2024a.
"Pi-
Gao, Z., Cen, J., and Chang, X. Consistent prompting for rehearsal- frree continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28463-28473, 2024b.
Gomez-Villa, A., Goswami, D., Wang, K., Bagdanov, A. D., Twardowski, B., and van de Weijer, Exemplar free continual representation learning via learnable drift com- pensation. In European Conference on Computer Vision, pp. 473-490. Springer, 2024.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Goswami, D.,
A., Liu, Y., Kamath, S., Twardowski, B., van de Weijer, J., et al. Resurrecting old classes with new data for exemplar-free continual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 28525- 28534, 2024.
References
Cha, H., Lee, J., and Shin, J. Co21: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference on computer vision, pp. 5516-9525, 2021.
He, K., Girshick, R., and Dollár, P. Rethinking imagenet pre- training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4918 4918-4427, 2019.

---

Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021a.
Krizhevsky, A. Learning multiple layers of fea- tures from tiny images. In Technical report, 2009. URL https: seapp.ssmmaticcchllee.. hts/CorppsID:1125550
Kurniawan, M. R., Song, X., Ma. Z., He, Y. Gong, Y., Qi, Y., and Wei, X. Evolving parameterized prompt memory for continual learning. Proceedings of the AAAI Conference Artificial Intelligence, 38 (12):13301-13009 Mar. 2024. doi: 10.1609/aaii.v3388 29231. URL https: /tt.////..ooao///o wwwposssaaaaaaaaooo.aoooao.aaa.a aaai. org/ index. php/ AAAI
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15262-15271, 2021b.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
Lester, B., AI-Rfou, R., and Constant, N. The power of scale for parameter-efficiee prompt tuning. In Moens, M.-F., Huang, X., Specia, L.,, and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computa- tional Linguistics. doi: 10.18653/1/2221.emmmpp 243. URL https:ttttccaaaaeeee http:///aaaattttooggo org/2021. emnlp main.243/.
Hu, E. J., Shen, Y., Wallis, P., Allen- Zhu, Z., Li, Y., Wang, S., Wang. I.., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https:// openreview
Huang. L.. An, Z., Zeng, Y., Xu, Y. et al. Kfc: Knowl- edge reconstruction and feedback consolidation enable efficient and effective continual generative learning. In The Second Tiny Papers Truck at ICLR 2024, 2024a.
Li, J., Wang, S., Qian, B., He, Y., Wei, X., and Gong, Y. Dynamic integration of task-specific adapters for class incremental learning. arXiv preprint arXiv:2409. 14983, 2024.
Huang, L., Zeng, Y., Yang, C., An, Z., Diao, B., and Xu, Y. etag: cluss-incremental learning via embedding distilla- tion and task-oriented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12591-12599, 2024b.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40( (122)29335-2477 2017.
Liang, Y-S. and Li, W.-. Inflora: Interference- low- rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23638-23647) 2024.
Huang, W.-C., Chen, C.-F., and Hsu, H. OVOR: Oneprompt with virtual outlier regularization for rehearsal-free class- incremental learning. In The Twelfth International Confer- ence on Learning Representations, 2024c. URL https: //openrevien net/ forum? wwmmmmddddFFddPeeeede
Liu, Y., Schiele, B., and Sun, Q. Rmm: Reinforced memory management for class-incremente learning. Advances in Neural Information Processing Systems, 34:3478-3499 -4990, 2021.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and I Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Jiao, L., Lai, Q., Li, Y., and Xu, Q Vector quantization prompting for continual learning. NeurlPS, 2024.
Lu, Y., Zhang, S., Cheng, D., Xing, Y., Wang, N., Wang, P., and Zhang, Y. Visual prompt tuning in null space for continual learning. NeurlPS, 2024.
Kim, Y., Li, Y., and Panda, P. One-stage prompt-based continual learning. In European Conference on Computer Vision, pp. 163-179. Springer, 2024.
Mahalanobis, P. C. On the generalized distance in statistics. Proceedings of the National Institute of Sciences of India, 2:49-55 55, 1936.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- -arnska-Brwinskaa,. A., et al. Overcoming catastrophic forgetting in neural networks, Proceedings of the national academy of sciences, 114(13)3351133222 2017.
McDonnell, M., Gong, D., Parvanch, A., Abbasnejad, E., and van den Hengel, A. RanPAC: Random projections
10

---

Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning
and pre-trained models for continual learning. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https: / wwwp//ween..w..com n forum?id=aec5BUER
Conference on Computer Vision and Pattern Recognition, pp. 13252-23262. 2024.
Tian, C.. Shi, Z., Guo, Z., Li, L., and Xu, C. Hydralora: An asymmetric lora architecture for efficient fine-tuning. In Advances in Neural Information Processing Systems (NeurIPS) 2024.
Parisi, G. I., Kemker, R., Part, L., Kanan, C., and Wermter, S. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71, 2019.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-uesd birds-200-2011 dataset. Technical Report CNS-TR-20 201001,,,,,, California Institute of Tech- nology, 2011.
Peng, C., Zhao, K. Wang, T., Li, M., and Lovell, B. C. Few-shot claseennerelental learning from an open-set perspective. In European Conference on Computer Vision, pp. 382-397. Springer, 2022.
Wang. L., Xie, J., Zhang, X., Huang, M., Su, H., and Zhu, J. Hierarchical decomposition prompt-based continual learning: Rethinking obscured sub-optimality. Advances in Neural Information P Processing Systems, 36, 2024a.
Pham, Q., Liu, C., and Hoi, Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34, 2021.
Rebuffi S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Wang, L.., Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: theory, method and application. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024b.
Riemer, M., Cases, I., Ajemian, R., Liu, M., Rish, I., Tu, Y., and Tesauro, G. Learning to learn without forgetting by maximizing transfer and minimizing interference. In In International Conference on Learning Representations (ICLR), 2019.
Wang, S., L., Sun, J., and Xu, 7. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 184-193, 2021.
Wang, Y., Huang, Z., and Hong, X. S-prompts learning with pre-trained transformers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-566 5695, 2022a.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211 211-252, 2015.
Wang. Y., Cheng, L.. Duan, M., Wang, Y., Feng, Z., and Kong, S. Improving knowledge distillation via regulariz- ing feature direction and norm. In European Conference on Computer Vision, pp. 20-37. Springer Nature Switzer- land Cham, 2024c.
Saha, G., Garg, I., and Roy, K. Gradient projection mem- ory for continual learning. In International Confer- ence on Learning Representations, 2021. URL https: //openrevies net forum? 330 j.MDojORCCN JORCNC2.
Wang, Y., Cheng, L., Fang, C., Zhang, D., Duan, M., and Wang, M. Revisiting the power of prompt for visual tuning. arXiv preprint arXiv:2402.02.383 2024d.
Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. Advances neural informa- tion processing systems, 30, 2017.
Wang, Z., Zhang, Z., Ebrahimi, S., Sun, R., Zhang, H., Lee, C.-Y., Ren, X., Su, G., Perot, V., Dy, J., et al. Dualprompt: Complementary prompting for rehearsal learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022b.
Smith, J. S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P.,
Kim, D., Arbelle, A., Panda, R., Feris, R., and Kira, Z. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Pro- ceedings of the IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 11909-11919, 2023.
continual
Wang, Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022c.
Sun, H.-I.,, Zhou, D.-W., Zhao, H., Gan, I.,, Zhan, D.-C., and Ye, H.-J. Mos: Model surgery for pre-trained model- based class-incremental learning. In AAAI, 2025.
Tan, Y., Zhou, Q., Xiang, X., Wang, K., Wu, Y., and Li, Y. Semantically-shifte incremental adapter- tuning is a continual vitransformer. In Proceedings of the IEFF/CVF
Wen, H., Cheng, H., Qiu, H., Wang, L.,, Pan, L.,, and Li, H. Optimizing mode connectivity for class incremental leam- ing. Proceedings of the 40th International Conference

---

Navigating Semantic Drift in Task-Agnostic Class-Incrementn Learning
Machine Learning, volume 202, pp. 36940-36957.
PMIR, 2023.
Yu, L., Twardowski, B., Liu, X., Herranz, L.. Wang. K., Cheng, Y., Jui, S., and Weijer, J. v. Semantic drift compensation for class-incremental learning. In Proceed- ings of the IEFE/CVF conference on computer vision and pattern recognition, pp. 6982-6991 2020.
Zenke, F., Poole, B., and Ganguli, Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995, PMLR, 2017.
Zhai, J.-T., Liu, X., Yu, L., and Cheng, M.-M. Fine- tine-grained knowledge selection and restoration for non-exemplar class incremental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6971-6978, 2024.
Zhang. D., Li, Y., Cheng, D., Wang, N., and Han, J. Center- sensitive kernel optimization for efficient on-device in- cremental learning. arXiv preprint arXiv:2406.0888 2024.
Zhang, G., Wang, L., Kang, G., Chen, L., and Wei, Y. Slea: Slow learner with classifier alignment for continual learning on pre-trained model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19148-19158, 2023.
Zhou, D.-W., Sun, H.-L., Ning, J., Ye, H.-J., and Zhan, D.-C. Continual learning with pre-trained models: survey. In IJCAI, pp. 8363-8371, 2024a.
Zhou, D.-.., Sun, H.-L,, Ye, H.J.,, and Zhan, D.-C. Expandable subspace ensemble for pre-traing pre-trained model- based class-incremental learning. Proceedings of the IEFF/CVF Conference Computer Vision and Pallern Recognition, pp. 23554-23564, 2024b.
7hu, F., Zhang, X.-Y., Wang, C., Yin, F., and Liu, C.-I. Pro- totype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 5871-5880 June 2021.
12


Reviews:
[{"review_id": "N2T2NajROV", "reviewer": "Reviewer_31F2", "summary": "This paper tackles the challenge of class-incremental learning in continual Learning, which enables models to sequentially learn multiple tasks without retraining or accessing data from previous tasks. \nWhile recent advancements in deep learning, such as larger model capacities and large-scale pretraining, have improved plasticity, traditional methods (e.g., regularization, memory replay, and knowledge distillation) still come with significant computational and storage overheads, hindering practical deployment. \nThe authors focus on addressing catastrophic forgetting and the issue of semantic drift, which occurs when feature means and covariances shift as new tasks are added. \n\nThey propose two solutions: \n(1) Mean Shift Compensation, which estimates and corrects the drift in feature means by computing weighted averages of embedding shifts, and (2) Covariance Calibration, which uses Mahalanobis distance to align the covariance matrices of embeddings from old and new tasks. These methods are incorporated into a task-agnostic continual learning framework that outperforms existing techniques across several public datasets, enhancing both model stability and adaptability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 31F2 for the valuable comments. Reviewer 31F2 gives a **positive rating (3-Weak Accept)**, finds \"the proposed method demonstrates superior performance\", and our experimental designs are \"rigorous, diverse and comprehensive\", etc. We address the main concerns below:\n\n> **Reviewer 31F2 suggests that a computational cost analysis would be useful to demonstrate that LoRA-based tuning is more efficient in terms of computation than full fine-tuning.**\n\nThanks. We evaluate the computational cost in terms of the number of trainable parameters and Multiply-Accumulates (MACs). We analyze the computational cost of multiple SOTA methods on ImageNet-R. The results are shown in the table below.\n\n|Method| Trainable Params (M) | MACs(G) |ImageNet-R ($\\mathcal A_{Last}$)|\n|---|---|---|---|\n|L2P|0.19|37.48|70.56|\n|DualPrompt|0.41|35.17|66.89|\n|RanPAC|2.00|17.58|77.94|\n|SLCA|85.8|17.58|78.95|\n|CPrompt|0.26|25.01|76.38|\n|EASE|1.19|17.81|75.91|\n|MOS|3.2|17.64|77.68|\n|InfLoRA|0.32|20.51|78.78|\n|SSIAT|1.19|17.81|79.55|\n|Ours|1.20|20.82|81.88|\n\nThe results highlight that our Params and MACs are minimal. Even under low computational cost conditions across all methods, our approach still delivers outstanding results.\n\n> **Reviewer 31F2 asks how our method scales across different model sizes with varying complexities.**\n\nThanks. We evaluate our method with four different scales of models pretrained on ImageNet-21K at a resolution of $224 \\times 224$.\n|Model|depth|embed_dim|heads|Params(M)| $$\\mathcal A_{Last}$$ | $$\\mathcal A_{Avg}$$ |\n|---|---|---|---|---|---------- |---------- |\n| ViT-Ti/16 |12|192|3|5.7|$60.85$|$70.90$|\n| ViT-S/16 |12|384|6|22|$75.82$|$81.87$|\n| ViT-B/16 |12|768|12|86| $81.83$|$86.27$ |\n| ViT-L/16 |24|1024|16|307| $85.28$ | $89.10$ |\n\nThe results show that our method demonstrates strong scalability with pre-trained model size. The  ViT-Large based model can get better performance.\n\n> **Reviewer 31F2 asks how our method handles imbalanced class distributions.**\n\nThanks. We identify a notable class imbalance in our training datasets, such as ImageNet-R and ImageNet-A. For ImageNet-R, the training set comprises 200 classes with a total of 24,000 samples. The most frequent class contains 349 samples, whereas the least frequent class contains only 38 samples, resulting in a maximum-to-minimum sample ratio of approximately 9.18:1. In comparison, the ImageNet-A training set also consists of 200 classes but includes only 5,981 samples. Here, the most common class has 86 samples, while the least common class has just 2 samples, leading to a ratio of 43:1. Our proposed method is capable of addressing this class imbalance issue during training. \n\nSpecifically, in the classifier alignment phase, we employ the following strategy to alleviate the class imbalance:\nFor each class $ c $, we generate $s_c$ synthetic feature samples from a normal distribution $ \\mathcal N(\\mu_c, \\Sigma_c) $, where $s_c$ is the same for all classes, and thus independent of class frequency. This design ensures that minority classes generate more synthetic samples relative to their original size, effectively mitigating the classifier’s bias toward high-frequency classes.\n\nClass imbalance itself is an important research direction, and we consider extending our method to better address this challenge as a promising avenue for future work.", "claims_and_evidence": "The claims made in the submission are generally well-supported by clear and convincing evidence. The proposed method demonstrates superior performance, especially on challenging datasets like ImageNet-R and ImageNet-A, where it surpasses the second-best method, SSIAT, by significant margins in both A_{last} and A_{avg}. Additionally, the ablation study confirms the effectiveness of key components like Mean Shift Compensation (MSC) and Covariance Calibration (CC), which together improve performance by over 2%. The method’s robustness across task sequences (both long and short) further solidifies its reliability in diverse class-incremental learning (CIL) scenarios.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem of class-incremental learning (CIL) and align well with the challenges inherent in this application. The approach leverages a frozen ViT backbone with task-specific LoRA modules, which is an effective way to retain previous knowledge while allowing task adaptation without excessive memory overhead. LoRA’s ability to integrate low-rank weight updates ensures that task-specific adaptation is computationally efficient, making it well-suited for CIL. Semantic drift (feature mean and covariance shift) is a known challenge in CIL, and the method addresses it explicitly through: Mean Shift Compensation (MSC): Correcting shifts in class mean embeddings. Covariance Calibration (CC): Aligning covariance matrices using Mahalanobis distance, ensuring feature distributions remain consistent across tasks. These techniques align with prior findings on distributional shifts in continual learning and provide a principled way to improve knowledge retention. Results highlight the method’s strengths in handling domain shifts (ImageNet-R, ImageNet-A) while performing competitively on natural datasets (CIFAR-100, CUB-200).", "theoretical_claims": "Overall, the theoretical claims appear reasonable and grounded.", "experimental_designs_or_analyses": "The experimental design is well-structured and rigorous, employing diverse and challenging benchmark datasets, consistent task sequences, and fair comparisons with state-of-the-art methods, ensuring a comprehensive evaluation of model performance. The study’s use of multiple independent runs with identical random seeds enhances result reliability, while the ablation studies effectively isolate the contributions of key components, particularly Mean Shift Compensation (MSC) and Covariance Calibration (CC), demonstrating their effectiveness in mitigating semantic drift. The inclusion of both short (5 tasks) and long (20 tasks) sequences further validates the method’s robustness across different continual learning settings. Additionally, the framework’s reliance on pretrained models with LoRA-based fine-tuning presents a computationally efficient alternative to full fine-tuning approaches. While statistical significance testing and computational cost analysis could further reinforce the claims, the experimental design provides strong empirical evidence supporting the proposed method’s superiority in handling catastrophic forgetting and domain shifts, making it a promising solution for class-incremental learning.", "relation_to_broader_scientific_literature": "The paper advances the field by integrating LoRA-based fine-tuning with semantic drift compensation, covariance calibration, classifier alignment, and feature self-distillation, all of which address major challenges in CIL. It builds on prior work in PEFT-based CIL, feature shift correction, classifier bias mitigation, and ViT self-distillation, but improves upon them by removing reliance on memory buffers, reducing computational overhead, and enhancing feature stability. These contributions align with and extend existing literature, making them highly relevant to the broader research community in continual learning and parameter-efficient adaptation of large models.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "1. How does the proposed LoRA-based fine-tuning scale across different model sizes with varying complexities? In particular, how does it handle imbalanced class distributions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N0vQt9ONHU", "reviewer": "Reviewer_XH2d", "summary": "Balancing flexibility and stability remain a key challenge in class-incremental learning (CIL). To address this, this paper introduces mean shift compensation and covariance calibration to regulate feature moments, preserving both model stability and adaptability. Additionally, a feature self-distillation mechanism for patch tokens is implemented to further enhance feature consistency. The proposed efficient, task-agnostic continual learning framework surpasses existing methods across multiple public datasets, demonstrating its effectiveness and superiority.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer XH2d for the valuable comments. Reviewer XH2d gives a **positive rating (4-Accept)**, finds that \"the paper is well-written and well-organized\" with \"a clear, straightforward motivation\" coupled with \"intuitive illustrations\" and \"demonstrate superior performance\" with \"careful experimental designs\", etc. We address the main concerns below:\n\n> **Reviewer XH2d asks for further clarification on the necessity of integrating the patch distillation module with the main approach that alleviates semantic drift.**\n\nThanks. Patch distillation is not an independent add-on separate from semantic drift calibration; rather, it enhances feature space stability to provide a more reliable basis for mean/covariance calibration. Its mechanism is as follows: semantic drift calibration (MSC/CC) targets high-level semantic features (class tokens), while patch tokens capture fine-grained visual details. Stabilizing local feature representations can reduce the distributional shift at the class token level. In ViT, shallow features encompass general patterns (such as edges and textures) shared across tasks. Enforcing consistency among patch tokens helps prevent the degradation of these foundational features. In summary, Patch Distillation is not an isolated or unrelated component; rather, it complements MSC/CC and plays a crucial role in improving model robustness.\n\n> **Reviewer XH2d asks for a detailed discussion on the gains of our method on CIFAR-100.**\n\nThanks. We highlight that our approach is designed to be general. Notably, it demonstrates significant advantages on more challenging tasks, such as ImageNet-R and ImageNet-A. In our experiments, we applied a unified hyper-parameter setting without dataset-specific tuning, which underscores the versatility of our approach. We are confident that by fine-tuning the hyperparameters of our method for CIFAR-100, further improvements can be achieved.  Within a limited timeframe, we find an optimized hyperparameter configuration using LoRA with a rank of  $r = 20$ and a patch distillation loss weight of  $\\lambda = 0.1$. This setting achieve a CIFAR-100 performance of $\\mathcal A_{\\text{Last}} = 92.20$ and $\\mathcal A_{\\text{Avg}} = 95.15$.", "claims_and_evidence": "The claims made in the submission are well-supported by clear and convincing evidence. The proposed mean shift compensation, covariance calibration, and feature self-distillation methods are validated through extensive experiments across multiple public datasets. The results consistently demonstrate superior performance over existing class-incremental learning (CIL) methods.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the class-incremental learning (CIL) problem. address semantic drift through mean shift compensation and covariance calibration, ensuring both stability and adaptability. The mean shift compensation tracks distribution shifts, while covariance calibration constrains feature variance, collectively enhancing classifier alignment across incremental tasks. Additionally, a feature self-distillation module further stabilizes feature representations over time.", "theoretical_claims": "The formulations are mathematically technically correct.", "experimental_designs_or_analyses": "The experimental design is well-structured and comprehensive, covering component-wise ablations, model design choices, and competitive benchmarking.", "relation_to_broader_scientific_literature": "This work mitigates the Semantic Drift issues in CIL by integrating both mean shift compensation and covariance calibration. It builds upon prior research in parameter-efficient fine-tuning (PEFT) for CIL~\\cite{hu2021lora, valipour2022dylora, hao2024flora}, which focuses on reducing computational overhead while preserving model adaptability. Existing works only study the semantic drift phenomenon by exploring class prototypes shifting, however this work extends the semantic drift phenomenon in both the mean and covariance, calibrating them to mitigate catastrophic forgetting.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strength:\n1. Good writing with well-organized structure.\n2. The illustration looks good and intuitive.\n3. The performance looks good compared with SOTA approaches.\n4. The motivation is clear and straightforward and well demonstrated with carefully designed experiments .\n\nWeakness:\nThe patch distillation module appears to be somewhat incremental and lacks a strong integration with the overall motivation and methodology. While it aims to enhance feature stability, its connection to the core semantic drift calibration and covariance alignment framework is not entirely clear. Specifically, its role in mitigating catastrophic forgetting or improving class-incremental learning (CIL) performance should be more explicitly justified. If it primarily contributes to feature refinement rather than directly addressing semantic drift or classifier calibration, further clarification on its necessity and integration with the main approach is needed.", "other_comments_or_suggestions": "None.", "questions_for_authors": "The gains on CIFAR-100 seem marginal. Could you please provide a detailed discussion on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5pfbdNg1nY", "reviewer": "Reviewer_JCMA", "summary": "This paper identifies that the feature distribution gap between novel and existing tasks is primarily influenced by differences in mean and covariance moments. To address this, a novel semantic drift calibration method is proposed, integrating mean shift compensation and covariance alignment. Specifically, a Mahalanobis distance constraint is applied to align class-specific embedding covariances between previous and current networks, effectively mitigating covariance shift. Additionally, a feature-level self-distillation mechanism is introduced to enhance generalization and improve model adaptation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer JCMA for the valuable comments. Reviewer JCMA gives a **positive rating (4-Accept)**, acknowledges \"a novel semantic drift calibration method\" which is \"well-designed and effectively assessed\" and notes that its \"key components improve performance remarkably\" offering \"improved efficiency and stability\", etc. We address the main concerns below:\n\n> **Reviewer JCMA asks if we could provide a detailed explanation for the performance discrepancy in CIFAR-100 and Imagenet-R.**\n\nThanks. Please refer to our response to the second concerns raised by Reviewer XH2d.\n\n> **Reviewer JCMA asks if we could discuss the limitations and future studies.**\n\nThanks. We discuss the Limitations and Future Work below. In this study, we have focused on addressing semantic drift by aligning first-order (mean) and second-order (covariance) statistics. While this approach has shown promising results, it is inherently limited in its ability to capture more complex aspects of feature distribution shifts. Specifically, higher-order moments, such as skewness (third-order statistic) and kurtosis (fourth-order statistic), are not considered in this framework. These higher-order statistics could provide additional insights into the shape and tails of the data distribution, which may help in mitigating semantic drift more effectively, especially in tasks with significant feature distribution shifts. Future work will explore this approach by incorporating higher-order statistical moments like skewness and kurtosis into the alignment process.", "claims_and_evidence": "The claims in the submission are well-supported by compelling empirical evidence. The proposed method exhibits superior performance, particularly on challenging benchmarks like ImageNet-R and ImageNet-A, where it outperforms the second-best method, SSIAT, by substantial margins in both 𝐴_{last} and 𝐴_{avg}. Ablation studies validate the contributions of key components, Mean Shift Compensation (MSC) and Covariance Calibration (CC), which collectively improve performance remarkably. Furthermore, the method demonstrates robustness across both long and short task sequences, reinforcing its reliability in diverse class-incremental learning (CIL) scenarios.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria appear well-aligned with the problem of class-incremental learning (CIL). The use of challenging benchmarks such as ImageNet-R and ImageNet-A is appropriate, as these datasets introduce real-world complexities, including distribution shifts and adversarial robustness, which test the model’s adaptability. The evaluation metrics, 𝐴_{last} and 𝐴_{avg}, effectively capture both final and average performance across task sequences, providing a comprehensive assessment of catastrophic forgetting and knowledge retention. Additionally, the inclusion of ablation studies on Mean Shift Compensation (MSC) and Covariance Calibration (CC) further validates the method’s contribution. The experiments on both long and short task sequences enhance the evaluation’s credibility by ensuring the method’s robustness across varying levels of incremental complexity. Overall, the methods and evaluation criteria are well-designed for the intended application, effectively assessing both adaptability and stability in CIL scenarios.", "theoretical_claims": "The submission does not include formal theoretical proofs but presents technically grounded mathematical formulations. The formulations for Mean Shift Compensation (MSC) and Covariance Calibration (CC) are well-justified, particularly in their use of Mahalanobis distance constraints for covariance alignment. Additionally, the loss functions and optimization strategy align with established techniques for mitigating catastrophic forgetting. While no theoretical analysis is provided, the effectiveness of these formulations is validated through empirical experiments and ablation studies, demonstrating their practical impact. Overall, the mathematical foundations of the proposed method are sound and effectively supported by experimental results.", "experimental_designs_or_analyses": "The experimental design is generally well-structured and appropriate for evaluating class-incremental learning. The use of challenging datasets, ablation studies, and comparisons with a strong baseline strengthen the validity of the findings.", "relation_to_broader_scientific_literature": "This paper contributes to class-incremental learning (CIL) by integrating LoRA-based fine-tuning with semantic drift compensation, covariance calibration, classifier alignment, and feature self-distillation, addressing key challenges in parameter-efficient adaptation. It builds upon prior research in parameter-efficient fine-tuning (PEFT) for CIL\\cite{hu2021lora, valipour2022dylora, hao2024flora}, which focuses on reducing computational overhead while preserving model adaptability. Unlike standard PEFT methods, this work explicitly tackles feature drift, aligning with studies on feature shift correction\\cite{liu2021swin, zhu2024vision} by introducing covariance alignment and Mahalanobis distance constraints to stabilize class representations over incremental tasks. Additionally, the paper extends classifier bias mitigation\\cite{wu2019large, kang2019decoupling} by ensuring that learned features remain well-calibrated across tasks without reliance on memory buffers. By integrating feature-level self-distillation, it also aligns with research in self-supervised learning and knowledge retention\\cite{zhang2022self, touvron2021training}. These innovations improve both efficiency and stability in incremental learning, making the proposed approach highly relevant to the broader literature on continual learning, parameter-efficient adaptation, and large-scale vision model optimization.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "1. The results in Table 2 show that the performance on ImageNet-R is good but seems not good enough on CIFAR-100. Could you provide a detailed explanation for this discrepancy?\n\n2. Better to discuss the limitations and future studies.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1GgBpy2U55", "reviewer": "Reviewer_o87J", "summary": "This paper proposes a method to improve model stability in class-incremental learning by alleviating the semantic drift phenomenon. The authors leverage the transferability of pretrained models and train parameter-efficient fine-tuning LoRA modules with a frozen ViT backbone. They define semantic drift in two dimensions: the mean and covariance of the features. To address this, they compensate for shifts in the mean and introduce a novel approach that uses a Mahalanobis distance loss function to constrain the covariance. The updated first- and second-order statistics are then used to align the classifier in the post-training stage. Additionally, the authors employ a patch-token-based distillation loss to further improve the model’s performance. The method is extensively validated on four major CIL datasets, with each module demonstrating its effectiveness and achieving SOTA performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer o87J for the valuable comments. Reviewer o87J gives a **positive rating (3-Weak Accept)**, finds our method is \"reasonably designed\" with \"a novel, creative covariance calibration approach\" and mathematical formulations are \"well-defined and correct\", etc. We address the main concerns below:\n\n> **Reviewer o87J asks if the sensitivity analysis of the hyperparameters could be presented.**\n\nThanks.  Here we include the sensitivity study for some important hyperparameters. The results are shown in the table below.\n\n| s    | 10         | 20         | 30         | 40         |\n| ---- | ----------| ---------- | ---------- | ---------- |\n|  $\\mathcal A_{Last}$    | $79.18 \\pm 0.17$ | $81.88 \\pm 0.07$ | $81.27 \\pm 0.07$ | $80.53 \\pm 0.30$ |\n\n| r    | 8          | 16         | 32         | 64         |\n| ---- | ---------- | ---------- | ---------- | ---------- |\n|   $\\mathcal A_{Last}$  | $80.80 \\pm 0.27$ | $81.31 \\pm 0.04$ | $81.88 \\pm 0.07$ | $81.88 \\pm 0.25$ |\n\n|$\\lambda$| 0.2        | 0.4 | 0.6 | 0.8        |\n| ---- | ---------- | ---------- | ---------- | ---------- |\n|  $\\mathcal A_{Last}$   | $81.83 \\pm 0.28$ | $81.88 \\pm 0.07$ | $81.72 \\pm 0.18$ | $81.44 \\pm 0.19$ |\n\n>**Reviewer o87J questions the effectiveness of the patch distillation module.**\n\nThanks. Kindly refer to our response to the first concerns raised by Reviewer XH2d.", "claims_and_evidence": "The authors claim that stability in class-incremental learning (CIL), which is a primary goal of CIL, can be undermined by the phenomenon of semantic drift. This phenomenon induces shifts in feature mean and covariance. The claim is supported by visualization and empirical validation across standard benchmarks.", "methods_and_evaluation_criteria": "The authors propose using low-rank adaptation modules that continually adapt to incremental tasks, and introduce a mean shift compensation and covariance calibration method to alleviate the semantic drift phenomenon, thereby improving stability. The mean shift compensation tracks the mean shift in the feature distribution, while the covariance calibration module adds constraints to the shape of the distribution. The compensated mean and calibrated covariance matrix are then used to update the classifier head at the end of each incremental session. They also incorporate a distillation module to further improve the stability aspect of the CIL task. Overall, the method seems reasonable for addressing the CIL challenge. The proposed method is evaluated on common datasets: ImageNet-R, ImageNet-A, CUB-200, and CIFAR-100, reporting both last and average performances.", "theoretical_claims": "The mathematical symbols, variables, and equations are generally well defined and mathematically correct.", "experimental_designs_or_analyses": "1. The experimental design is valid, including ablations of the effectiveness of each component, different LoRA design choices, and a comparison with a large number of very recent SOTA methods across four mainstream benchmark datasets.\n\n2. Although the authors mention that hyperparameter choices are based on sensitivity analysis, experimental results still need to be presented.", "relation_to_broader_scientific_literature": "This work extends the definition of semantic drift, as described in the literature (e.g., Semantic Drift Compensation for Class-Incremental Learning), to both class mean and covariance perspectives, and proposes a novel covariance calibration method.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength:\nCreatively imposing the Mahalanobis distance as a constraint in the loss function, which implicitly constrains the covariance matrix.\n\nWeakness:\nThe gain from patch distillation is relatively small.", "other_comments_or_suggestions": "No", "questions_for_authors": "See experimental designs or analyses, and other strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Other expertise"]}}]

Author rebuttal:

Meta-review: This paper introduces a novel solution to semantic drift in class-incremental learning (CIL) by addressing shifts in feature mean and covariance between new and old tasks. The authors propose Mean Shift Compensation (MSC) and Covariance Calibration (CC), using Mahalanobis distance constraints to align embedding covariances and reduce drift. Additionally, a patch-token-based self-distillation module is incorporated to enhance model stability, and the method is validated on several benchmark datasets, showing improved performance.

The strengths of the paper lie in its innovative solution to semantic drift through the combination of MSC and CC, demonstrating both theoretical soundness and strong empirical results. The approach performs particularly well on challenging datasets like ImageNet-R and ImageNet-A and shows the effectiveness of its key components through ablation studies.
Decision: Accept (oral)