LCA: Local Classifier Alignment for Continual Learning
Authors: Tung Tran
Danilo Vasconcellos Vargas
Khoat Than
Venue: ICLR 2026 Poster (2026)
Keywords: continual Learning
local robustness
catastrophic forgetting
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: A novel local robustness loss to align classifiers after integrating backbones in continual learning.

Abstract: A fundamental requirement for intelligent systems is the ability to learn continuously under changing environments. However, models trained in this regime often suffer from catastrophic forgetting. Leveraging pre-trained models has recently emerged as a promising solution, since their generalized feature extractors enable faster and more robust adaptation. While some earlier works mitigate forgetting by fine-tuning only on the first task, this approach quickly deteriorates as the number of tasks grows and the data distributions diverge. More recent research instead seeks to consolidate task knowledge into a unified backbone, or adapting the backbone as new tasks arrive. However, such approaches may create a (potential) \textit{mismatch} between task-specific classifiers and the adapted backbone. To address this issue, we propose a novel \textit{Local Classifier Alignment} (LCA) loss to better align the classifier with backbone. Theoretically, we show that this LCA loss can enable the classifier to not only generalize well for all observed tasks, but also improve robustness. Furthermore, we develop a complete solution for continual learning, following the model merging approach and using LCA. Extensive experiments on several standard benchmarks demonstrate that our method often achieves leading performance, sometimes surpasses the state-of-the-art methods with a large margin.

Published as a conference paper at ICLR 2026
LCA: LOCAL CLASSIFIER ALIGNMENT FOR CONTIN- UAL LEARNING
Tung Tran', Danilo Vasconcellos Vargas', Khoat Than
"Kyushu University, Fukuoka, Japan "Hanoi University of Science and Technology. Hanoi, Vietnam tran. tung. son. 9490s kyushu-u.ac. jp
vargaslinff www..i.w...uuuuu..cccc jp khoattg@soict.. hust. vn
ABSTRACT
A fundamental requirement for intelligent systems is the ability to learn contin- uously under changing environments. However, models trained this regime often suffer from catastrophic forgetting. Leveraging pre-trained models has re- cently emerged as a promising solution, since their generalized feature extractors enable faster and more robust adaptation. While some earlier works mitigate for- getting by fine-tuning only on the first task, this approach quickly deteriorates as the number of tasks grows and the data distributions diverge. More recent research instead seeks to consolidate task knowledge into unified backbone, or adapting the backbone as new tasks arrive. However, such approaches may create (po- tential) mismatch between task-specific classifiers and the adapted backbone. To address this issue, we propose a novel Local Classifier Alignment (LCA) loss better align the classifier with backbone. Theoretically, we show that this LCA loss can enable the classifier to not only generalize well for all observed tasks, but also improve robustness. Furthermore, we develop complete solution for con- tinual learning, following the model merging approach and using LCA. Extensive experiments several standard benchmarks demonstrate that method often achieves leading performance, sometimes surpasses the state-of-the-ae methods with large margin.
INTRODUCTION
In real-world scenarios. data arrives continuously, requiring deployed models adapt to evolving distributions while preserving previously learned knowledge. This challenge, known as the stabil- ity-plasticity dilemma in continual learning, captures the difficulty of balancing adaptation with retention. A widely adopted benchmark approximates this setting by partitioning dataset into dis- joint tasks and training a model sequentially without access to earlier data (Wang et al. 2024). At test time, the model must handle all tasks without being given the task identity. This setup referred to as Mhass-nncremental Learning (CIL) (Van de Ven et al. 2022).
Pre-trained models (PTMs) have recently emerged a strong foundation for this setting, in contrast to earlier methods that trained networks from scratch (Li & Hoiem 2017 Kirkpatrick et al. 2017). PTMs can be obtained through supervised self-supervised training, and include large multimodal models such as CLIP (Radford et al. (202)) as well as vision backbones like the Vision Transformer (Dosovitskiy et al. (2220). Their broad generalization ability makes them effective feature extractors, requiring only lightweight adaptation and thereby reducing forgetting.
Nevertheless, naive sequential adaptation still leads to degradation on past tasks. Restricting updates to the first task avoids forgetting but blocks useful transfer to later ones. Since each task introduces unique information, the final model must capture both shared and task-specific components. A natural approach is to incrementally adapt each task and then merge the resulting models into a unified backbone. Insights from Linear Mode Connectivity and the Lottery Ticket Hypothesis
Conresponding author.
Code is available this https URL

---

Published as a conference paper at ICLR 2026
(Frankle al. 2020) suggest that task-specific solutions be connected through low-loss paths and rely on sparse. critical parameters, making them amenable to combination. Advances in model merging (llharco et al., 2022 Yadav et al. 2023) further support this strategy, with both theoretical guarantees (Li et al. 2025) and empirical evidence on adaptation tasks (Akiba et al. 2025) showing that merged models can preserve and even enhance knowledge. Thus, it is reasonable treat each task-specific model as a standalone expert containing complementary knowledge, whose consoli- dation yields a stronger overall backbone. However, merging backbones across tasks introduces new challenge: classifiers trained independently may no longer align with the integrated backbone. Because these classifiers cannot retrained without access to past data, even small parameter shifts can lead to severe drops in performance earlier tasks. Addressing this misalignment is the central focus of our work.
Our contributions in this work are as follows:
We introduce a novel loss, Local Classifier Alignment (LCA), for aligning CIL classifiers This loss not only well aligns the backbone with the classifier, but also ensures robust- ness of the classifier. It also can contribute to reducing overlap between classes and hence improving classifier's performance.
We provide theoretical analysis that decomposes the test error of a CIL model into three fundamental parts, including feature distribution shift, class-wise loss, and robustness. Those parts must be well controlled to assure a high performance for all observed tasks. Such a theory is erucial support reliability of LCA and trustworthy CIL.
We propose a complete CIL solution in which model merging (for PEFT parameters only) is used to slightly adapt the backbone new tasks and LCA serves as the main alignment loss to reduce mismatches between classifiers and the backbone. In our method, each class is represented as a Gaussian in the feature space, and LCA jointly optimizes all classifiers.
We conduct extensive experiments on seven benchmark datasets. Our results show that I.CA consistently improves performance over baselines, enhances robustness under diverse scenarios, and can be integrated with other CII. methods to further boost their effectiveness. Figure shows superiority of LCA on seven benchmark datasets.
RELATED WORKS
Representation-Basel Methods. This line of research leverages the representational power of large pre-trained models for continual learn- ing (Wang et al. 2024). Trained on mas- sive and diverse datasets, these models pro- vide strong transferability and inherent robust- ness against forgetting. One prominent di-
100-
92.8
8
91
[M+1
110.0
90-
85.8 8
BM
08.5
[81.1
143.4
Aonurry
P6.
75.0
70.1
rection adapts prompting techniques from nat- ural language processing, where prompts are modeled learnable parameters attached to the inputs (Wang et al.] (022c]b) by Tran et al. 2025). These prompts act additional instruc- tions that guide model predictions during con- tinual learning. Another direction adapts the pre-trained backbone only once during the first fine-tuning (PEFT) modules and then relies on class pro- totypes for inference (Panos et al. 2023] /Zhou
65
CIBARIO
my
Dec
CUB
VIAB
CARG
Datasets
Figure 1: A comparison between IM and IM+LCA. IM is the result after only done the Incremental Merging step, while IM+LCA has Local Classifier Alignment as the last step.
task by using parameter-efficieee
et al., 2025 McDonnell et al. 2023 Perez al. 2018). Such prototype-based approaches clas- sify via cosine similarity can avoid forgetting because accumulated prototypes across tasks are tasks order-invariant. However, empirical evidence shows that adaptation only in the first task is insuf- ficient, since data distributions in real-world scenarios vary substantially across tasks, which limits this method's applicability in long training horizons.
Incremental Backbone Evolution. A complementary line of research updates the backbone throughout the task sequence. For example, SI.CA (Zhang al. 2023) reduces forgetting by apply-
2

---

Published as a conference paper at ICLR 2026
ing smaller learning rate to the backbone than the classifier, while methods such MagMax (Marczak et al. 2024). EASE (Zhou et al. 2024), and MOS (Sun et al. 2025b) focus on integrating task-specific components into unified backbone. In all cases, past classifiers are typically frozen to avoid bias toward the current task's data, which inevitably creates a mismatch between the evoly- ing backbone and the fixed classifiers. To mitigate this issue, EASE reweights old classifiers using semantic similarity between new and old prototypes, while MOS dynamically selects suitable back- bone adapters inference time. Another line of research seeks to enrich past prototypes through augmentation, as in Fe'TrLL (Petit et al. 2023), PASS (Zhu et al. 2021b), IL2A (Zhu et al. 2021a), and CCFA (Kim al., 2024). or enhance the capacity of current classifiers through boosting, as in FOSTER (Wang et al. (022a). Our approach retrains all classifiers after the unification step using samples drawn from simple Gaussian models, together with a novel loss that emphasizes lo- cal robustness. This procedure reduces the mismatch between the backbone and the classifiers and improves the stability of the overall model.
Model Merging. Another related direction is model merging, which has recently gained attention for constructing unified model from independently trained task-specific ones. Early methods such FisherMerging (Matena & Raffel 2022) use the Fisher Information Matrix to weight parameter importance, while Task Arithmetic (llharco et al.) (2022) represents task updates as vectors, enabling explicit addition subtraction of knowledge. More advanced approaches like TIES-Merge (Yadav ett al.] 2023) address task interference by resolving sign conflicts across task vectors. Although these methods show strong performance in out-of-domain generalization (Rame al. 2022), applying them directly to continual learning often introduces mismatch between the merged backbone and fixed classifiers, and incurs storage overhead since parameters from all past tasks must be retained. In contrast, our approach builds on the spirit of model merging by incrementally consolidating PEFT modules, which reduces memory requirements, and coupling this with a continuous training scheme that solidifies accumulated knowledge across tasks.
3 METHODOLOGY
This section presents our methodology for continual learning, which consists of two complemen- tary components. The first focuses on incrementally merging task-specific backbones into a unified model, while maintaining proximity across tasks by initializing new training from the latest merged backbone. The second addresses the mismatch that arises when frozen task-specific classifiers inter- act with the consolidated backbone, introducing alignment mechanism based on class-wise local regions. Together, these components enable the model to effectively retain past knowledge while adapting new tasks.
3.1 PROBLEM FORMULATION
In class-incremental learning (CIL), the objective is to train a single model sequentially on series of (potentially infinite number of) tasks. Each task is associated with dataset
D. = {(,,W) w) x E Xr, y }}}.
where æ denotes an input sample and y its corresponding label. Here, X' represents the input space and }/ the label space for task i. A defining characteristic of the CIL setting is that the label spaces of different tasks are strictly disjoint:
Vi J. 31 n 0
This assumption implies that each new task introduces set of novel classes that the model has not encountered before. Consequently, the model must continuously expand its knowledge while pre- serving performance on previously learned classes, making the prevention catastrophic forgetting central challenge in CIL.
There are different strategies for constructing classifiers in continual learning, such as Nearest Class Mean (NCM) classifiers, which assign labels by comparing test features to stored class prototypes. In this work, however, we focus on the more general and widely used setup of progressively ex- panding Multi-Layer Perceptrons (MLPs) top of the feature extractor. Instead of training single unified classifier over all classes, a new MLP head is added for each task, with its output dimension matching the number of classes introduced by that task. This approach not only reduces storage but
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 Incremental Merging (IM)
Input: Datasets {Di...., Dr), pretrained model pretrained base PEFT 8pen
Output: Merged PEFT module mmerged
1:
Task vector accumulator b. Sequentially process tasks
2: for i= to do
3: ppott uinetune(ppft.... D.)
D Task-i adaptation
4: Tourr Opont Optt,
Task vector
5:
for = to do
d number of parameters
6: if | (8) then
7:
(Ak) < Tours
(k)
Di Keep larger magnitude, preserve sign
8: end if
9:
end for
merggd poft -T
Update merged PEFT
11: end for
also mitigates forgetting, since earlier classifiers remain frozen and are not modified during subse- quent training. After tasks, the classifier consists of task task-specific heads {Øc .e...... .ch)),))))) and inference is performed by evaluating each head separately and concatenating their outputs:
h (z) = concat ch(cat(((z;; 09%h), h(r;0%)), .... h(x; 0(h)).
3.2 INCREMENTAL KNOWLEDGE CONSOLIDATION
Although pre-trained models possess strong generalization, they still lack the domain- specific knowledge needed to serve as effective feature extractors. As a result, fine-tuning stage is required. To prevent forgetting during continuous fine-tuning, early methods often assume that task distribu- tions remain close and restrict adaptation to the first task. However, as the number of tasks grows and their distributions diverge, performance carlier tasks inevitably deteriorates. Inspired by search on model merging, we propose an incremental integration scheme that unifies task-specific backbones into single consolidated backbone.
We finetune the model-consisting of the pretrained backbone, the PEFT module, and the task classifier parameterized by pretrained: peh, Get. on task dataset D.. To limit drift across tasks, each task is initialized from the most recent PEFT parameters Oper,_, rather than the original base, keeping successive solutions close in parameter space-as property shown to be important for stable merging (Li et al. 2025). The finetuning step finetune uses SGD with cross-entropy loss.
After finetuning task i, we flatten the PEFT parameters into a vector Tpent. Assuming that final parameters should remain close to the pretrained initialization, subtract the base vector Tpef, and select elements based on the largest absolute deviations. To avoid parameter growth, we retain only the previously merged vector and the current task vector during each merging step. The resulting update vector is added back to the base parameters to obtain the merged PEFT module for task as summarized in Algorithm I
We further include an ablation in Appendix comparing Min, Max, and MaxAbs selection rules. All operators yield stable and competitive performance when applied to PEFT modules, reinforcing that merging only PEFT parameters is robust. To our knowledge, no prior work performs parameter- value-based merging without any trimming phase while still achieving effective results.
3.3 LOCAL CLASSIFIER ALIGNMENT
In contrast with a generalized feature extractor, task-specific classifiers should remain well separated to achieve strong classification performance. Because datasets from previous tasks are unavailable, updating earlier classifiers during new training would move them away from their previously opti- mized values. The standard pipeline therefore trains and merges the backbone while freezing the old classifiers, which introduces a mismatch between the unified backbone and those fixed heads. We address this discrepancy with alignment step that bridges the components.
4

---

Published as a conference paper at ICLR 2026
Consider the CIL approach where embeddings of the imput samples, a Gaussian distribution (N or prototype) to represent a class, and classifier. This approach has been investigated heavily and often high-performing CIL methods. At each time step I, one can use a learning method to train a classifier h, based on the classifier h which already fitted for prior tasks and dataset D, for the current task. The overall performance of he depends heavily on the employed learning algorithm. A good algorithm can well align the prototypes and classifier. However, this might not be always the case, and hence can degrade the overall CIL performance.
pretrained model such as VIT to produce high-quality
We propose simple finetuning step, called Local Classifier Alignment (LCA), to better align the classifier and prototypes. Specifically, LCA minimizes the following loss
C Σ
L(). = 1
L
(1)
C
L = RZ-D, [e(h,, z]] + 2, [[((hr,)) =) - E(h,,, ='|l
(2)
where D, contains ii.d. samples from the Gaussian distribution N that represents class in the feature space induced by the backbone, D = D...,.,,),, and C. is the total number of observed classes upto task t.
Basically, LCA tries to simultaneously minimize the loss for each class and keep the loss less sensi- tive to small change in the input samples around the class prototypes. The class loss can be seen from the first term Eg-D, [e(h,, z)]. while the sensitivity comes from the second term in each I4- Such term can be seen as regularizer penalize the classifier for unstable predictions. A larger value for X suggests a stronger penalty for sensitivity of the loss.
It is worth noting that the novelty of LCA comes from the regularization term. It not only helps improve robustness of the classifier, but also can reduce overlapping between classes. Indeed, when using the first term as the main objective for training the classifier, some samples randomly generated by Nr of one class can lie far from the i-th class prototype and hence can be closer to some other class prototypes. This can harm the training for the classifier. The second term can help us reduce the negative effect from those potentially harmful samples, under a suitable choice of X.
3.4 THEORETICAL ANALYSIS
We next analyze the generalization ability of the classifier h4, Until time step I, the classifier h already has learned from C, classes. We want estimate its test error for those learned tasks. To this end, the classical tradition is to estimate L(P, he) = J. E Ez-N; [E(,,, 2)}, which represents the overall expected error for the learned tasks.
Let Ui=1 Z, be the decomposition of the data space into non-overlapping local areas, with 2, the local area with centroid μ4; where μ4 is the mean of the Gaussian distribution N₁, for each index We have the following bound for the expected error of the overall classifier up to time step whose proof appears in Appendix A
Theorem 3.1. Consider model hy learned from a dataset D (DI,... Dr), where D; con- tains n. i.i.d. samples from distribution No for each i S Co and a bounded loss &. De- note P J. Eee N₁ as the overall distribution,
-E,, n4, Emax sup, l(he z), and E;(h,) a...N..a..ll 2) ((h,,8)) : 2,8 2i) for each index i. For any 8 > 0, the following holds with probability at least 1 - ō:
C, 1n 2l101
L(P,h.) S L(D, + Σ (le) + lmax
72
This result shows that the test error of the classifier l4 can be controlled by both the training error and the robustness term E = D ziz, (hx), which tells how robust is the loss w.r.t. to small change in the input of the samples around the class prototypes. A stronger robustness (i.e., smaller 2) can lead to tighter bound on the test error, suggeting better model. When both L(D, hy) and are small, the model must have small test error and hence generalize well on unseen data. On the other hand, bad model will exhibit large training error T.(D, h4) or large Therefore, Theorem 3.1 plays as
5

---

Published as a conference paper at ICLR 2026
the theoretical foundation for our LCA loss ① Training by this loss would arguably improve both performance and robustness of the classifier, which is crucial for real-world CIL tasks.
Corollary 1. Given the notations and assumptions as in Theorem 3. if nz m for all i, then the following holds with probability at least
In4 2 In lm(10)
L(P, S L(D, (1)
Σ (ba) + lmax
(4)
mC,
It suggests that, to assure high generalization, the number m samples for each class should not too small when doing alignment. If one use a small m. the uncertainty part in (4) will be large, meaning some randomness (by noise) can harm the alignment step.
This is direct consequence of Theorem 3.1
Remark 1. Although the LCA loss is introduced to the CIL context, the loss is general enough to be employed in many other contexts. Indeed, one can use LCA loss to train CIL classifier. Also, one easily plug LCA to do finetuming step for the existing CIL methods. In those cases, the theoretical benefits of LCA shown in Theorem 3.1 may remain valid. is worth noting that the result in Theorem 3.1 holds when the feature distributions (or prototypes) are fixed. This means Theorem 3.1 may not directly apply for the cases of prototype changes.
To fully understand the generalization dynamic of the overall CIL classifier, the previous results are insufficient. We need to take the backbone change into consideration. A change to the main backbone will lead to change in the class prototypes that represent the Gaussian N₁. As result, the overall distribution P in Theorem 3.1 will change after learning a new task.
After training task t 1, Ns accurately represent each new class i of task t and Pa-1 cL_ ΣG, Nr represents the overall distribution. After training a new task t, each distribution M may change to N₁ due to backbone change. Therefore, the overall distribution until task can be rewritten as Pt- = 1 EeT No+ 1 C.
C. Ef-cr_, Ng. However using this distribution to analyze generalization ability for all trained tasks may not be accurate. What we should analyze is for the distribution P. d - Ng. We have the following results.
C.
Theorem 3.2. Given the notations in Theorem
3.1 we have
L(Pi, h4) S m TV(P,, Pa) L(),,
(5)
where TV(:;)) denotes the total variation distance. Let D = Uisc,, D.U, Dj, with D, ~ N₁ and Dj ~ N2. contains i.idd samples. For any > O, the following holds with probability at least 1 8:
(he) + Lmax /
Cr 21((1/))
L(P,, 28 TVP..P)) - LLD,,)) -(D.b)) + Σ n
(6)
n
Basically, this theorem says that the test error of model he will be small when can ensure that the LCA loss (L(D,h.) h.) + DC x (h:)) is small and the induced distribution P. is close to P. (the accurate feature distribution). When distributions P: and Pt are far from each other, the backbone really causes catastrophic forgetting and thus can significantly increase the errors for past tasks. As result, bound (6) suggests that good CIL model should not significantly change the feature distributions of past tasks while being both robust and accurate for each class.
Return to our CIL method, both components have their own important roles. IM can train the back- bone to adapt the feature distribution to the new task, but also reduces catastrophic forgetting for past tasks and hence keeps small TV(P,, Pt). The classifier alignment by LCA can simultaneously train the new part, adapt the old part while encouraging robustness of the overall classifier. Those roles very crucial to ensure high performance for CIL. Therefore those analyses reveal the foundational support for L.C..

---

Published as a conference paper at ICLR 2026
Table 1: Average performance comparison on seven datasets with VIT-B/16-INIK the pretrained backbone. The best result is highlighted in bold, while the second best is highlighted in italic.
Method
CIFARI00
IN-R
IN-A
CUB
OB
VTAB
CARS
Overall
APER+ Addpter
90.8 = 0.5
78.8 * 0.6
58,9 1.3
89.7 1.3
80.3 + 0.4
90.7 + 0.6
50.6 1.1
77.1
APER-Finetune
8.9
0.8
58.7
89.5
77.8
91.8
53.2 1.4
75.0
APER+SSF
1.0
78
61.6
89.6
0.6
91.8
51.3
77.5
APER-VPT-Deep
0.9
0.7
5..0
1.2
0.84
91.9 1.4
50.6
76.6
OPER-YPT-Shalloen
8.9
6..3
56,9
8.4
0.9
01.5
50.9
74.8
CODA Prompt
91.0
0.4
4..
7 1.2
7..0
2.66
26.
65.1
DualPrompt
0.6
0.5
1.5
78.9
7..4
84.0
49.4
71.9
EASE
9..
8.5
67.8
89.5
80.8
9.1
48.1
79.1
L2P
84
0.6
1.76
75.8
73.8
8.8
53.
71.9
MOS
0.3
83,3 0.6
6..6
0.6
86.1
9.5
714 19.6
83.9
SLCA
93.7
74.6 2.2
IM
92.8 0.1
66.5
84.6
70.1
IM-LCA
94,8 0.3
85,8
7.5
90.8 0.3
0.5
95.2 1.1
7
85.6
4
EXPERIMENTS
This section presents the details of our experiments seven benchmark datasets, along with an ablation study to examine some design aspects our method.
4.1 EXPERIMENT SETUP
Dataset. We follow the setup in (McDonnel et al. 2023) to select datasets and define the number of classes in each incremental task. Specifically. we evaluate on CIFARI00 (Krizhevsky Hinton 2009), ImageNet-R (IN-R) (Hendrycks et al., 202 ImageNet-A (IN-A) (Hendrycks et al. 202 CUB-200 (CUB) WWelinder et al. (010), OmniBenchmark (OB) (Zhang et al.) (022). VTAB (Zhai al. 2019), and StanfordCars (CARS) (Krause et al. 2013). All datasets split into 10 tasks,
except VTAB which divided into 5.
Baselines. the following methods used for comparison:
Ours: We consider two variants: IM (Incremental Merging) and IM+LCA, where the latter incorporates LCA in the alignment phase.
Pre-trained based CIL. methods: CODA-Prompt (Smith et al. 2023). DualPrompt Wang
al. 2022b), L2P (Wang al. (022c), EASE (Zhou et 2024). MOS (Sun et al. 2025b).
SLCA (Zhang et al. 2023), APER (Zhou et al. 2025).
adopt the implementation from (Sun al. 2025a) for all baseline methods and use the same pre-trained backbone, ViT-B/16 trained on ImageNet- -K (VIT-B/I6IINIK) (Dosovitskiy al. 2020).
For fair comparison and reproducibility,
Training. We apply LoRA (Hu et al. 2022) and ensure a consistent computational budget by fixing the low-rank dimension to 64. All experiments are carried out on single NVIDIA RTX 4090 GPU running Ubuntu 22.04. Detail on hyperparameters is mentioned in Appendix B
Evaluation Metrics. Following standard practice, we report the performance along incremental stages, defined as AA = ΣT= Ac, where A; denotes the average accuracy after done training on task t.
4.2
EXPERIMENT RESULTS
4.2.1 OVERALL BENCHMARK
Table reports the mean and standard deviation of accuracy across three random seeds (1993, 1994, 1995). With LCA, the final model achieves the highest performance on five out of seven benchmark datasets, yielding an overall improvement of nearly 2%. Unlike methods such as EASE (Zhou al.) 2024) and MOS (Sun et al. 2025b), which either expand the backbone to integrate new tasks or rely complex inference procedures, approach requires additional mechanism. The backbone is

---

Published as a conference paper at ICLR 2026
(a) CIBARI
(b) IN-R
(c) TN-A
100.0
100
97.5
965
95.0
Arcurary (16
92.5
Accuracy (50
85
Accuracy (363
70-
80
90.0
87.5
APER Aaggee
AFGR. Aiguor
50
AFER AARgAA
1.5
500
ATER 1S2
EA5e
APER VPSShalion
DealProuge
3405
65
APER 1PPEStatio
DualProape
-405
DesiProupt
AFER -VPSSallo
134-6A
M455
60-
0000 Prange
30
CORA
80
150
150
Number of Classes
Numbor
Number Classes
(d) CUB
OB
(8 VTAB
100
100
100
96-
90
90
Accurecy (%)
90-
Annuracy (98)
85-
80-
80
Adtaker
ATER: Atagtar
70-
APER VP
52.04
APER SPT
AFER -VPDDag
SE.CA
APSK WiSthall
5425
APER vir-stallo
1408
APER WYSStaloo
-6- UM-
70
DualProrge
70
CoDA-rongg
COOA Drange
CODA-rungt
50
100
150
100
150
200
20
30
50
Number Classes
Number
Number Classes
(g) CARS
90
70-
Accuracy
80-
50
APEA VYCDDees
525A
AFER + SP-Shalio
5405
30
DealProrgt
IN-LCA
36
126
156
Number of Classes
Figure 2: Performance curves of different methods across all tasks and datasets. All methods use VřT-B/16-IN1K as the pre-trained backbone without any additional exemplars.
incrementally merged without storing past parameters samples, and the only extra storage is the mean and covariance of each encountered class, which scales as (n)) with the number classes. Figure 2 further shows that LCA consistently outperforms other methods across most datasets, with a notable improvement of 8% on ImageNet-A compared to the runner-up.
4.2.2 ROBUSTNESS MEASUREMENT
Following the standard (Hendrycks Dietterich 2019 Taori et al. 2020). we measure the per- formance of IM and IM+LCA on two robustness benchmarks, CIFARTOO-C and CIFAR 100-P. CIFAR 100-C constructed by applying 19 types of common corruptions (e.g., noise, blur, weather, and digital distortions) 5 severity levels to the original CIFAR- test set, thereby evaluating model robustness under distribution shift. The final corruption robustness accuracy is computed as
19
5
1
Accc 19 X
Ac.ar
where A is the accuracy under corruption type e with severity s.
100-.. in contrast, focuses on prediction stability under perturbations. Each image is per- turbed by transformations such translations, rotations, noise, and the model's consistency is
8

---

Published as a conference paper at ICLR 2026
90
100
85
98
Accumny (N]
96
Accunsy
94-
75
2-00
2-111
92
2-61
70
50
100
150
200
10
20
30
40
50
Number of Classes
Number of Claases
Figure 3: Effect of X on the accuracy datasets.
100
822
Acourac 186)
00
701
22
CIPABL00
105
15-4
0170
C5
VTA0
CARS
Seventy
Meas Accroay
Bolasines BooT
(a) Performance for variants of MOS and SLCA. The postfix LCA means that method uses alignment loss.
(b) Robustness comparison between IM and IM+LCA on CIFARI00 and CIFARI00-P. Metries
include accuracy, accuracy severity level 5, and overall robustness score.
Figure 4: (a) Complementary evaluation of LCA when using LCA for MOS and SLCA. (b) Robust- ness performance of IM and IM+LCA on corruption and perturbation benchmarks.
measured across perturbed versions. The perturbation robustness accuracy is defined
N
1 P
Accp
p,is
where P denotes the set of perturbation types, Np is the number of perturbed samples for type p, and Ap.i is the accuracy on the i-th perturbed sample. Finally, to summarize robustness across both benchmarks, report overall robustness score:
Robustness = 2 (Acce + Acce).
Together, these benchmarks evaluate both accuracy under distributional corruption and resilience of predictions under small perturbations.
Figures 4b and summarize the robustness results. Figure 4b shows that when being trained with LCA, the model obtains clear improvement on robustness, with more than +2% gain in mean accuracy CIFAR100-C and +2.5% gain CIFAR100-1 The radar plots in Figure 5 further reveal that LCA consistently improves accuracy across all corruption and perturbation types. These results confirm that LCA strengthens robustness both average and across diverse perturbations.

---

Published as a conference paper at ICLR 2026
Motion
Blur
Glass Blar
IM IM+LGA
Zoom Blur
Molion Blur
IM-ICA
Blur
Smow
+33 1.91%
Noise
Nolse
Irost
+2.88%
+3.3%
3.45%)
+3300%
Shot
68%
10g
2.99
+3.2%
Gansstan Notse
Brighton
Gaussian
Brighth
33,0%5
Specide
+3.1%
+3.2%
+2.7%
L.8%
Trunsh!
Scale
Spalter
+2.8%
Transfor
Saturate
Pizelata
JPEG Com
Rolate
Till
presaton
Blar
(a) Performance on CIFARI0O-C.
(b) Performance on CIFARIO0-P.
Figure 5: Accuracy performance of IM and IM+LCA under different corruption and perturbation types. The relative difference between IM and IM+LCA is highlighted.
4.2.3 ABLATION STUDY
LCA as a complementary component. We further evaluate the applicability of LCA on SLCA (Zhang et al. (223)) and MOS (Sun et al. 2025b), two methods that also update the backbone pro- gressively. asses the impact of our proposed method, we construct two baselines by omitting the final step of these methods, and retaining only the update backbone part. From these, we build their counterparts, SLCA-LCA and MOS-LCA, which include the aligment step as the final step. We measure the average performance and report in Figure 4a We do not find the optimal hyper- parameters but fixing the value of À at 0.1, yet the variants with LCA show improvements on all scenerios, notably in IN-A, CUB, VTAB, and CARS, in some sults of the original methods (SLCA-LCA achieves 89.0% in CUB compare to 90.0% of the original, MOS-LCA achieves 93.1% in CIFARI00 compare to 94.3%).
matching the reported
Hyperparameter sensitivity. In our method, À controls the strength the robustness penalty in the loss function. While an appropriate choice as Figure 3 shows that overly strong regularization can degrade results. In practice, we find that 0.1 provides stable and reliable performance across all datasets.
important for achieving strong performance
CONCLUSION
This paper investigates the mismatch that may arise between continuously updated backbone and the task-specific classifiers in continual learning. We propose a theoretically grounded loss function, with provable bound on classification error, which enables the training of all classifiers using fea- tures generated from a Gaussian distribution. Extensive experiments on seven benchmark datasets confirm the effectiveness of our method, and additional robustness evaluations under various noisy conditions demonstrate consistent improvements. While this work primarily focuses on addressing the alignment phase, future research will explore integrating the proposed loss into the end-to-end training pipeline. Such approach has the potential to further enhance the robustness of the back- bone itself and the performance the overall CIL method.
Despite having significant contributions, our work still remains some limitations. For example, have not investigated the proposed LCA loss in other contexts. Furthermore, although providing theoretical foundation and novel insights, the developed theory does not take the training of the whole backbone into consideration. Addressing those limitations can open interesting directions for future research.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility of our results. The hyperparameters re- quired for both training and inference explicitly reported in Appendix B The complete source code provided as supplementary material, where users can download and unzip the package, fol- low the installation instructions, and the main script reproduce used in this work are publicly available, implementation details and evaluation protocols are de- scribed in Section |. Together, these resources intended make it straightforward for others to replicate and build upon our work.
experiments. All datasets
ACKNOWLEDGEMENTS
This work was supported by JSPS Grant-in-Aid Challenging Exploratory Research Grant Number JP25K22833 and JSPS Research on Academic Transformation Areas (A) Grant Num- ber JP22H05 194. The authors also gratefully acknowledge the financial support provided by Hattori Hokokai Foundation and Telccommunications Advancement Foundation.
REFERENCES
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 7(2):195-204, 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pp. 3259-3269, PMLR, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 2019,
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distributiin generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. $340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint wrXiv:1907.07177 2021b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799, PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference Learning Representations, 2022. URL /tttp:://ppenrvviiie wwt.:/w.eettrrrreomeoeem
id=nZeVKeeFyYE
Gabriel Ilhareo, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint
arXiv:2212.04088 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European conference on computer vision, pp. 709-727. Springer, 2022.
Taehoon Kim, Jaeyoo Park, and Bohyung Han. Cross-class feature augmentation for class incre- mental learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13168-13176, 2024.
11

---

Published as a conference paper at ICLR 2026
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. Fine-grained categorization and dataset bootstrapping using deep learning with humans the loop. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 580-587, 2013. doi: 10.1109/CVPR.2013..0 URL Ittpp:: www.sstd..ddddooooooooo.. att.
dataset html
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Hongkang Li, Yihua Zhang. Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Ye Yu Chen. When is task vector provably effective for model editing? generalization analysis of nonlinear transformers. arXiv preprint arXiv:2504. 10957, 2025.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2941) 2017.
Dongze Lian, Daquan Zhou, Jiashi Feng. and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning. Advances in Neural Information Processing Systems, 35: 109-123, 2022.
Daniel Marezak, Bartlomiej Twardowski, Tomasz Trzciáski, and Sebastian Cygert. Magmax: Lever- aging model merging for seamless continual learning. In European Conference on Computer Vision, pp. 379-395. Springer, 2024.
Michael S Matena and Colin A Raffel. Merging models with fisghr- averaging. Advances in Neural Information Processing Systems, 15:17703- 2022.
Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton Van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems, 16:12222-1205,, 2023.
Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: strong replay-free baseline for daass-incrmmenta learning. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 18820-18830, 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. Fetril:
Feature translation for exemplar-fre free class duss-incremmntal learning. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 39 1911-1 1-3920, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamoniy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of--istribution generalization. Advances in Neural Information Processing Systems, 35: 10821 2022.
Sylvestre-Alvis Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. ooda-prompt: Continual de- composed attention based prompting for rehearsal frree continual learning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 11909-11991 2023.
12

---

Published as a conference paper at ICLR 2026
Hai-Long Sun, Da-Wa Re-Wei Zhou, De-Chuan Zhan, and Han-Jia Ye. Pilot: A pre-trained model-based continual learning toolbox. SCIENCE CHINA Information Sciences, 68(4):147101, 2025a. doi:
https://doi..or/10.. http://bo..rrgg1011000////11111
142
Hai-Long Sun, Da-Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Ye. Mos: Model surgery for tre-trained model- based inssemnnceme learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 39, pp. 20699-20707, 2025b.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18599 2020.
Quyen Tran, Tung Lam Tran, Khanh Doan, Toan Tran, Dinh Phung, Khoat Than, and Trung Le.. Boosting multiple views for pretrained-based continual learning. In The Thirteenth International Conference on Learning Representations, 2025.
Gido M Van de Ven, Tinne Tuytelaars, and Andreas Tolias. Three types of incremental learning.
Nature Machine Intelligence, 1192:118811977, 2022.
Fu- Yun Wang, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature boosting and com- pression for class-incremental learning. In European conference on computer vision, pp. 398-414. Springer, 2022a.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, 46(8):5362-5383, 2024.
Zifeng Wang. Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European conference on computer vision, pp. 631-648. Springer, 2022b.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 139-149, 2022c.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. Technical report, California Institute of Technology, 2010.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neurul Information Processing Systems, 36:7093-71155 2023.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04800 2019.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19148-19158, 2023.
Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision representation through the lens of visual realms. In European Conference on Computer Vision, pp. 594-611. Springer, 2022.
Wei Zhou, Hai-Long Sun, Han-Jin Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class- incremenaat learning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23554-23564, 2024.
Da-Wei Zhou, Zi- Zen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class- incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, 131(3):1012-103, 2025.
13

---

Published as a conference paper at ICLR 2026
Fei Zhu, Zhen Cheng, Yao Zhang, and Cheng-lin -in Liu. Class-incremental learning via dual augmentation. Advances neural information processing systems, 34: 14306 -40066111,8,,,,--- 2021a.
Fei Zhu, Ya- Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3871-5880, 2021b.
14

---

Published as a conference paper at ICLR 2026
A PROOF FOR MAIN THEOREMS
Proof of Theorem 3.1
Let Pt P(2;) be the probability measure of area 2, and note that
Ei=1 Ps 1.
We first observe that
C.
I(P,h.) T(P,h.)
ni I,N; h1) +
mi L(V, hn) L(D, + h.) (7)
==1
Since L(P,h() = Disi P,L(N,, E. h2) and L(NG, h4) S mmaxs we observe that
C.
C Σ
Ct n
L(P,h:)
L(N,,hs) h2) = P.L((N,,)))) - Σ L(N.
(8)
1=1
T4i
L(Ni, ho)
Pi
1
C₁
74
max
p:
(10)
Note that (N., niC, a multinomial random variable with parameters n and (Pi,..,PC)) PC.). For any > Bretagnolle- Hase--Cbrrrrrraee inequality shows Pr
[
pt
ti 2 2c) I 2 exp(-
In other words, for any >> 0, taking least 1-6 6:
/
C In 4
the following holds true with probability at 2n
C. n ((Ns,te) s ov
C In C1In
2lnd
L(P.h))
(11)
Next we observe the second term:
C. ni I(Ns,h)) - L(D.h.)
C
C. nh L(D,,h) h4)
Tii
L(Ni,ha) - 1=1
(12)
n
1=1
14
[I(N, hx) L(D h,)
(13)
n
14 ((h E 2,))
(14)
n
Ri
(E (Er-N, E(he, s) 2, E Z,)) (15)
1=1
(B (|e(t, E(he, 8)] 2,8 Z.)((6)
n
C
B4
(hx)
(17)
n
Combining and (11) and (17) completes the proof.
15

---

Published as a conference paper at ICLR 2026
Proof of Theorem 3.2 Denote (z( and (z( the density of distributions Pa and P,, respec-
tively. Observe that
I(Pt,
- (P)
(18)
BzWPi [e(he, 2)] 2n [((be; 2)] + L [(Pr,h)) ht)
(19)
J (( 2)) (pr + L ((P,))) hi)
(20)
/ [((be,z) (p((z)dz (c(z) L(Pr,hs)
(21)
S mmxx / (1(()) pc(z)] + L I(P.. h4)
(22)
2 max TV(P,, P.) L(P,,,,))
(23)
which shows the first statement. The second result thus follows using the result Theorem 3.1
B TRAINING DETAILS
Table 2: Training and merging hyperparameters used in all experiments of LCA.
Hyperparameter
Value
Training epochs
10
Batch size
64
Base learning rate
x 10
Weight decay
x 10-4
Optimizer
SGD (momentum = 0.9)
Learning rate scheduler
CosineAnnealing (eta-min 10-6)
Merging coefficient
1.0
Alignment classifier epochs
10
Number of samples per class
512
Alignment batch size
128
LoRA rank
64
LoRA alpha
128
LoRA dropout
0.0
LoRA initialization
Gaussian
C DETAILS OF EXAMPLE GENERATION
In this section, we provide details on how Gaussian distributions used align the classifiers during the incremental training process. Pre-trained models typically produce well-structured rep- resentations, which allows to approximate each class distribution using its empirical mean and covariance.
Storing class statistics. parameters 9, The empirical mean Me Idd and covariance Eg e Rddd are computed as
For each class at training stage t, we extract features using the backbone
D.l
1 = c) d(a:;)));
(24)
K
[D+]
Σ,
(b(2;; 0.) - mee) (bb(a;;0)) Me)
(25)
K
where (1; denotes the feature extractor with backbone parameters 0g, and K -/Da 1(y: c)-
16

---

Published as a conference paper at ICLR 2026
VTAB
100
90
Accuracy (%)
80
Adupee IM
I22-E-5FF-M--CA
IN21 KAdapter- MMMlppprr-MM--CC IN 1K-Adggte--I IM IN21 KAAdaiirr-MM
INIKSSP-M
70
122KKS5F-MM
INLKYPTIM-LIA
INRK IM-LCA INZUKLORADDY
INLKVPTIN
INTK DV RNZARIAA
INZIRVPEIM
60
10
20
30 Number of Classes
40
50
Figure 6: Ablation study on different PEFT strategies with two backbones, ViT-B/16-IK and VIT- B/16-2IK.
Feature replay via Gaussian sampling. Before each alignment step, we regenerate features for every class c Va by sampling from a multivariate Gaussian distribution:
2. (pee Ec).
(26)
We generate approximately five times the batch size of synthetic features per class, which provides sufficient diversity for classifier alignment.
D FURTHER ANALYSIS
Results with different fine-tuning strategies. We further examine the adaptability of incremental merging and LCA with different parameter-efficient fine-tuning (PEFT) strategies the VTAB dataset. The strategies include SSF (Lian et al. [2022), Adapters (Rebuffi et al.) (017). VPT (Jia al. 2022). and LoRA (Hu al. 2022), evaluated with ViT-B/16 pretrained on both ImageNet-IK and 21..
Sigwr that LCA consistently improves performance across all PEFT methods. In particular, VPT without LCA suffers a sharp performance drop. suggesting that incremental merging alone is insufficient to prevent forgetting in some cases. Adding LCA, however, proves beneficial across all methods, demonstrating its adaptability. Notably, despite having relatively few trainable parameters, Adapters emerge as strong candidate within our pipeline, achieving the highest accuracy when combining with LCA.
E FURTHER RESULTS ON ACCURACY
In this section, report additional results of our proposed method when finetuning with the Adapter PEFT module (Houlsby et al. 2019), shown Figure 7 We refer to this variant as IM+LCA Adapter, while the results in the main text correspond to IM+LCA LoRA.
From the figure, we observe that several baseline methods suffer from strong instability under cer- tain settings (e.g., SLCA on IN-A and MOS on CARS), whereas our method remains consistently stable across all three random seeds (1993, 1994, 1995). Furthermore, with Adapter, our approach datasets such CUB and OB. This confirms that the lower
achieves state-of-the-er performance performance reported the main text for these datasets is due to our reproducibillty-oriene straints on PEFT method choices, rather than limitations of the proposed method itself.
con-
17

---

Published as a conference paper at ICLR 2026
Table Average performance on different merge operators. CA means normal classifier alignment, while LCA is our proposed method.
Method
IN-A
VTAB
CARS
IM Max
66.26
84.76
69.90 + 1.47
IM Max CA
70.86 = 1.10
71.88 = 0.49
IM Max LCA
7
94.22
74.85 # 1.50
IM Min
66.49
8..48
69.79 1.36
IM Min CA
7.11
2.33
71.82 + 0.50
IM Min LCA
1.200
9.12
74.28 = 1.29
IM MaxAbs
6.10
8.96
70.1 = 1.50
IM MaxAbs CA
0.82
91.71
72.15 = 0.52
IM MaxAbs LCA
75.0 0.50
95.2 1.10
76.2 + 1.40
(4) CIFAR100
(b) IN-R
(c) TN-A
90-
95.0
30-
92.5-
85
Accurscy
Armurry
80-
20.0
ATOR Mapar
L2P
APE3
APES Aigur
COP
AVER 9770m2
APER VEStee
SLCA
82.5
20
10
50
150
13s
150
175
19s
200
Number
1.0
100
1.0
05-
Accorer EN6
00
Accurery (85)
90-
0:-
85
75
INSE
75
4PE3 SEP
3AST
66-
ATE3 5SF
ATER VISGRLE
3005
70-
3G5
APE VESDEENN
11222
19-1C Main
75
100
125 150
200
25
75
100
135
150
175
200
10
50
Number Classess
Number of Clasons
Number of Chasses
(g) CARS
00
62
50-
40-
ARRE
A 9770m2
SECA
30-
DasFrenge
DeNLCA LARA
Figure 7: Additional results Performance Curve.
F DIFFERENT MERGING OPERATORS
Table 3 shows the benefits of our parameter-sclection design. We evaluate the generalization of method using three merge operators: Min, which selects the parameter with the smallest value;
18

---

Published as a conference paper at ICLR 2026
1.00
MaxAbs
1.0000
0.9987
0.9966
0.9943
0.9921
0.9900
0.9885
0.9861
0.9845
0.9829
-0.98
Merge Method
Cosine Similarity
-0.96
MAX
1,0000
0.9973
0.9932
0.9864
0.9781
0.9670
0.9569
0.9428
0.9309
0.9184 84
-0.94
MIN
1.0000
0.9973
0.9932
0.9865
0.9781
0.9676
0.9578
0.9441
0.9326
0.9207
0.90
Task
Task 2
Task
Task
Task
Task
Task
Task
(a) IN-A
1.00
MaxAbs
1.0000
0.9997
0.9989
0.9982
0.9970
Cosine Similarity
Merge Method
0.96
MAX
1,0000
0.9993
0,9980
0.9961
0,9928
0.94
MIN
1.0000
0.9993
0.9970
0.9950
0.9915
-0.92
0.90
Task
Task
Task 2
Task 3
Task 4
(b) VTAB
1.00
MaxAAbs
1,0000
0.9912
0.9883
0.9772
0.9742
0.9674
-.98
Cosine Similarity
Merge Method
0..6
MAX
1.0000
0.9965
0.9869
0.9783
0.9657
0.9529
0.9377
0.9233
0.9069
0.8877
0.94
MIN
1.0000
0.9964
0.9873
0.9788
0.9667
0.9541
0.9388
0.9246
0.9084
0.8895
0.92
0.90
Task
Task
Task
Task
Task
Task
Task
Task
Task
(c) CARS
Figure 8: Similarity measurements between backbone representations across tasks, computed rela- tive to the backbone obtained from the first task.
Max, which selects the largest; and MaxAbs, which selects the parameter with the largest absolute magnitude. When want reduce the cost of storage by merging only two models, the MaxAbs operator can be viewed as a simplified form of TIES MEES-Mggging without the trimming phase in Yadav al. (023), and is equivalent the merge rule proposed in Marczak et al. (202)).
Importantly, unlike prior works such as Marczak et al. (204)) which apply merging to all model parameters, show that merging only PEFT module parameters is substantially easier and more stable. Under this restricted setting, even very simple operators like Min or Max achieve compet- itive performance across benchmarks, demonstrating that PEFT-focused merging does not require
19

---

Published as conference paper at ICLR 2026
43.49
93. 93.77
93. 78
93.85
92.53
93.50
MOS
93
64.30
85.04
us. 585.48
99.
88 .0.00
88. 688
85.24
86. .0.50
87 79.49
88
MOS-LCA
SLCA
$33.00
83.23
SLCA-LCA
75. 98.88
73.87
74.37
71.41
68. 68.19
70.40
67.58
68.03
Accuracy
CIFAR100
IN-R
IN-A
CUB Datasets
OB
VTAB
CARS
Figure 9: Accuracy of the original methods and their LCA variants across standard datasets.
complex selection strategies. Moreover, observe that the merge coefficient does not need careful tuning as a fixed value 1.0 works reliably across all benchmarks.
G ABLATION ON ROBUSTNESS TERM
We measure the similarity between the backbone obtained at each task and the backbone learned after the first task. The backbone the first task serves an anchor, as it is assumed bridge the gap between the pre-trained dataset and the target domain. Figure @equre88inn Billustraaes how incremental fine- tuning with merging enables the backbone evolve gradually over time, supporting the theoretical intuition discussed in Section 3.4
Table further shows the impact of including the robustness term during training. Across all merge operators, adding the robustness term yields consistent and clear improvements, demonstrating its effectiveness independent of the chosen merge strategy.
H EFFECT OF LCA ON DIFFERENT METHODS
In addition the results reported Figure 4a where we replace the original alignment loss with our LCA loss, we conduct more extensive ablation to evaluate the adaptability of LCA when attached to other alignment-
methods, specifically SLCA (Zhang et al., (023) and MOS (Sun et al.) 2025b). Both methods continually evolve the backbone representations over time, making them natural candidates for incorporating our proposed loss. Figure shows that, across all datasets, the LCA-aaugmented variants consistently outperform their corresponding original methods, indicating a clear accuracy improvement.
We further evaluate robustness following the same protocol used for IM. Specifically, test CIFAR1O0 100-C, the corrupted version of CIFARI00, and CIFAR100-P, the perturbed version. Details of these robustness benchmarks are provided in Section 4.2.2 Figure 10 summarizes the overall
20

---

Published as a conference paper at ICLR 2026
MOS
86.30
86 90
24.98
85.18
SLCA
85
SLCA-LCC
81 19.9
01:33
N00 31-18
77
97 .8
Accuracy (%6
76.51
74.57
75.
75
64.89
65. 01
63
CIFAR1O04 OD-C
CIFAR100-C
CIFAR100-P Meen Accuracy
Overall
Mean Accuracy
Severily
Robustness Score
Robustness Metrics
Figure 10: Accuracy of the original methods and their LCA variants under various corruption and perturbation types.
Zoom Blur
Motion Blur
Glass
MOS MOS-LCA
Zoom Blur
Motion Blur
Glass
Defocus Blur
SUGM-UA
Blur
Snswww
Snow
Impulse
Impulse Nošse
2.24 2200
Noise
7%) 2.4%
Frost
Frost
+1..0% 11.0%
4-2.0%
+4066%
Noioe
Shot
1.33%
10.6%
+1.1%
Shot Noise
Fog
11.8%
+3.6%%
Fog
+1.3%
+2.0%
Gaussian
Gussiaan
Brightse 50
Brightne 5B
22.1%
+2.15
Specide Noise
+2.3%
specilee Nolse
+1..%%
+2.5%
1 L
Contrust
21.0%
Contrast
+2.3%
+2.1
Spatter
$2.3%
Spatter
Transfor
Transfor
Saturate
Saturate
Pixelate JPEG
Pixelate Com
pression
Blur
presslon
Blar
Figure 11: Accuracy of the original methods and their LCA variants CIFAR100-C corruptions.
robustness gains, while Figures II and 12 break down the improvements for each corruption and perturbation type, respectively.
I EFFECT LCA WHEN SELECTIVELY UPDATING CLASSIFIERS
Updating only half of the task-specific classifier heads (five in IN-A and three in VTAB) during alignment still yields competitive performance as demonstrated in figure [3] indicating that full classifier finetuning is not always necessary.
21

---

Published as a conference paper at ICLR 2026
Zooma Blur
Motion Blur
MDS MOS LCA
Zoom Blur
Motion
Blur
SLCV
1.9%
3..3%
9%
Shot Noise
Shot
Snow
Snow
+-0.8%
+2.80
+1.8%%
+2.19
60%s
60%
Brightre 2.0%
1.5%
Causelan Noise
Brighton 1.8%
Gausslan 2.11%
Noise
+1.2%
+-.4%%
+2.2%%
Transiat
Scale
Transiat
Scale
1..99
Figure 12: Accuracy of the original methods and their LCA variants CIFARI00-P perturbations.
(a) IN-A
(b) VTAB
95
105
100
Accuracy (%)
Accuracy (%)
95
80
75
90
85
65
Full classifters classifier
classifters classifiers
80
Pull clascifiers classifier
classifiers
60
3 classifiers
50
100
150
20
40
50
Number of Classes
Number of Classes
Figure 13: Accuracy selectively finetuning a subset classifiers.
22


Reviews:
[{"review_id": "z3PFV1fzEE", "reviewer": "Reviewer_MUvf", "summary": "This paper tackles the problem of class incremental leanring with pretrained model. While pretrained model can act as a good general representation model, they still lack domain specific knowledge and also suffer from the catastrophic forgetting when trained on sequential tasks. To alleviate these problems, the authors propose icremental merging and local classifier alignment method with some theoretical/empirical analysis.", "questions": "See weaknesses", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- Paper is well written and easy to follow. Especially, the authors tried to provide multiple visualisations to help the understanding of the readers.\n- Instead of just giving empirical evidence, the authors also provide theoretical analysis as well", "weaknesses": "- My major concern about this paper is about novelty. To me, it looks like none of the proposed component is novel not just in entire deep learning literature but also in continual learning literature. For example, classifier alignment was studied even from few years ago as in [R1, R2]. Adopting model merging to continual learning is not a novel idea as well as in [R3]. I can not see the distinct novelty of the proposed method compared to the above cited papers.\n\n- Since the pretrained model is evolving over time, there is no guarantee that the feature space structure is maintained. If so, LCA method and its theoretical analysis can not be justified. \n\n- Also, improvements in Table 1 is bery marginal, especially when IM is applied alone. What happens if LCA is applied to the existing methods? \n\n- In Figure2, why the starting point is different? The proposed methods has higher starting point but similar tendency. Due to this, I wonder the improvements comes just from good starting point, not by the proposed methods.\n\n[R1]Zhu, Fei, et al. \"Class-incremental learning via dual augmentation.\" Advances in neural information processing systems 34 (2021)\n[R2]Kim, Taehoon, Jaeyoo Park, and Bohyung Han. \"Cross-class feature augmentation for class incremental learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 12. 2024.\n[R3]Marczak, Daniel, et al. \"Magmax: Leveraging model merging for seamless continual learning.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a8B9fDt0S6", "reviewer": "Reviewer_uBFe", "summary": "The paper proposes a continual learning method based on model merging. Specifically, it alleviates the mismatch between task-specific classifiers and the adapted backbone by a novel Local Classifier Alignment (LCA) loss. LCA keeps the in-task loss less sensitive to a small change in the input samples around the class prototypes. The effectiveness of LCA loss is supported both theoretically and empirically. The paper conducts experiments on multiple CIL datasets. Results show the improved performance compared with various pre-trained based CIL methods. A robustness measurement and ablation study are included.", "questions": "1. Model merging based on magnitude selection is proposed in MagMax (Marczak et al., 2024) as well. What is the main difference/benefits of IM compared to MagMax?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Applying model merging methods to CL is an emerging topic and has potential to reduce forgetting for large scale models. \n\n2. The effectiveness of the LCA loss is justified both theoretically and experimentally. \n\n3. The paper conducts thorough experiments with comparison to recent CL baselines. The visualization of results is good.", "weaknesses": "1. It is unclear how LCA loss solves the mismatch between classifiers and the merged feature extractor. \n- The LCA loss is computed only on in-task samples, reducing $\\epsilon_i$ in Eq. 4. However, the mismatch between classifier and feature extractor seems aiming to reduce $L(\\mathbf D, h_t)$. Although LCA loss improves in-task robustness and reduces the loss upperbound, it’s unclear how it addresses the mismatch across tasks. \n- It could be helpful to show more evidence of the claim ‘LCA can reduce overlapping between classes’ in L220 and how this can reduce negative effect from potentially harmful samples (L224).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6nvzwzo27C", "reviewer": "Reviewer_7fmM", "summary": "This paper addresses the critical issue of backbone-classifier mismatch in Pre-trained Model (PTM) based Class-Incremental Learning (CIL), where an evolving backbone diverges from fixed classifiers of previous tasks. The proposed solution consists of two components:\n\nIncremental Merging (IM): Progressively merges PEFT modules (using LoRA) to create a unified backbone.\n\nLocal Classifier Alignment (LCA): A novel loss function applied after IM to align all classifiers (past and present) with the merged backbone. LCA approximates class features with Gaussian distributions and minimizes both classification loss and a local robustness term (sensitivity penalty) using sampled features.\n\nThe combined approach (IM+LCA) demonstrates strong performance, matching or exceeding state-of-the-art results on seven CIL benchmarks, with notable improvements in robustness evaluations (CIFAR100-C/P).", "questions": "Please see the weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "[S1] The paper clearly identifies and addresses a critical practical problem in PTM-based CIL methods: backbone-classifier misalignment that occurs when incrementally updating the backbone. This is a timely and important contribution to the CIL field.\n\n[S2] Robustness and Generality: Demonstrates enhanced model stability through robustness tests and shows LCA can be complementarily applied to improve other CIL methods, proving its general applicability.", "weaknesses": "[W1] Since the robustness penalty is a core contribution, this ablation is essential to distinguish its specific benefit from the classifier alignment component (first term), which SLCA also employs. Without this experiment, it remains unclear whether the improvements stem from the novel robustness penalty or merely from classifier alignment.\nI strongly encourage the authors to include this ablation, as it would clearly demonstrate the added value over existing approaches like SLCA and strengthen the paper's novelty claims.\n\n[W2] The provided Theorem 3.1 assumes fixed class distributions (prototypes), whereas the IM component actively modifies the backbone and thus the feature space. This gap needs better reconciliation or clearer framing (e.g., viewing alignment as optimizing classifiers for a newly fixed merged backbone).\n\n[W3] Cost Concerns (LCA Stage): Requires resampling features and retraining all classifiers after each task, leading to a computational cost that increases linearly with the number of tasks. \n\n[W4] Missing Related Work: While citing direct competitors (EASE, MOS), the paper misses relevant work on data-free replay for classifier calibration, such as: \nFeature Replay / Calibration: FeTrIL (WACV 2023), PASS (CVPR 2021). Classifier Calibration / Alignment: FOSTER (ECCV 2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RY2oYCvp5D", "reviewer": "Reviewer_MRmy", "summary": "This paper proposes a new loss function, called Local Classifier Alignment (LCA), which aims to address the potential mismatch between task-specific Gaussian-based classifiers and the continuously updated pre-trained models (PTMs, e.g., ViT) within the framework of PTM-based continual incremental learning (CIL). The paper adopts a model-merging approach for CIL, in which the PTM parameters fine-tuned on task-specific datasets are incrementally aggregated. Such incremental updates can lead to inconsistencies with previously trained (and typically frozen) task-specific classifiers — a problem that LCA is designed to mitigate. The key idea behind LCA is to minimize not only the class-wise loss but also the sensitivity of the loss to small perturbations in input samples. Through this simple yet effective regularization, LCA demonstrates clear improvements in CIL performance, along with enhanced robustness and stability across experiments.", "questions": "1. What is the reason behind the relatively low performance on OB or CUB datasets?\n2. Why is the OB graph missing in Figure 2?\n3. Are there any prior studies that apply model merging to continual learning? If not, do you explicitly clarify that this is the first attempt at incremental model merging for CIL?\n4. What exactly is the model architecture? Does it consist of a shared PTM backbone and multiple MLPs (one per task), where each MLP corresponds to multiple Gaussian components (one per class)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is a solid and well-motivated paper with a clear understanding of the underlying problem in PTM-based CIL.\n2. The application of incremental model merging to continual learning appears to be novel and is one of the key contributions of this work.\n3. The core idea of LCA is both insightful and elegant, achieving simplicity without unnecessary technical complexity.\n4. LCA is evaluated comprehensively, both in terms of performance and robustness, across seven benchmarks, and its theoretical foundation is supported by a probabilistic guarantee.", "weaknesses": "1. While the paper is generally well-written, certain parts require improvement. In particular, Section 3.3 should be rewritten in direct connection with Section 3.1. The current version introduces new notations (e.g., C_t, x') without prior definition and refers to Gaussian-based classifiers (e.g., N_i) without adequate explanation.\n2. Section 3.2 lacks novelty. Although applying model merging to continual learning may be a new attempt, the overall algorithmic formulation appears nearly identical to TIES-Merging, as also acknowledged by the authors.\n3. Model merging itself is not sufficiently described in Section 3.2, making the paper somewhat less self-contained.\n4. The paper does not provide an analysis of performance variation across different dataset characteristics. For example, model merging seems less effective on certain datasets (e.g., CUB, OB).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)