Unlocking [CLS] Features for Continual Post-Training
Authors: Murat Onur Yildirim
Elif Ceren Gok Yildirim
Joaquin Vanschoren
Venue: Accepted by TMLR ()

Abstract: Continual learning requires models to integrate new classes or domains over time while preserving previously acquired knowledge. Within this paradigm, foundation models often achieve strong performance, but they still remain subject to the stability–plasticity trade-off, where excessive plasticity leads to forgetting of prior knowledge, and excessive stability constrains the adaptation. This necessitates an effective post-training strategy that introduces minimal yet functional modifications. To address this challenge, we first introduce a new parameter-efficient fine-tuning module ‘Learn and Calibrate’, or LuCA, designed to acquire task-specific knowledge through an adapter-calibrator couple, enabling well-refined feature representations. Then, for each task, we deploy a sparse LuCA module on top of the last classification token [CLS] just before the classifier, which we refer to as ‘Token-level Sparse Calibration and Adaptation’, or TOSCA. By leaving the generalization capabilities of the foundation models intact and adapting exclusively via the last token, our approach achieves a harmonious balance between stability and plasticity while reducing both training and inference complexity. We demonstrate that TOSCA yields state-of-the-art performance while introducing 8 times fewer parameters compared to prior methods.

Published in Transactions on Machine Learning Research (02/2026)
Unlocking [CLS] Features for Continual Post-Training
Murat Onur Yildirim, Elif Ceren Gok Yildirim, Joaquin Vanschoren AMOR/e Lab, Findhoven University of Technology
Correspondence to: m. o. m.o.y.ldda.. yylddrimmtue..d ml
Reviewed on OpenReview: https: // openreview. net/ forun? id- OkfWy6krc
Abstract
Continual learning requires models to integrate new classes or domains over time while preserving previously acquired knowledge. Within this paradigm, foundation models often achieve strong performance, but they still remain subject to the stability-plastiiie trade- off, where excessive plasticity leads to forgetting of prior knowledge, and excessive stability constrains the adaptation. This necessitates an effective post-training strategy that intro- duces minimal yet functional modifications. To address this challenge, we first introduce a new parameter-effiee -fficcient Ane-tuning module 'Learn and Calibrate', or LuCA, designed to acquire task-specific, knowledge through an adapter-calibrator couple, enabling well-refined feature representations. Then, for each task, we deploy a sparse LuCA module on top of the last elassification Loken [CLS] just before the classifier, which we refer to 'Token-level Sparse Calibration and Adaptation', TOSCA. By leaving the generalization capabilities of the foundation models intact and adapting exclusively via the last token, our approach achieves harmonious balance between stability and plasticity while reducing both training and inference complexity. We demonstrate that TOSCA yields state-of-the-ar performance while introducing -8x fewer parameters compared to prior methods. The code is available at https: /ttp///gtttttttt wow///mraattouuryrrrrraaoooooooe
1 Introduction
Learning continuously from a series of concepts or classes using a unified model is a challenging problem due to catastrophic forgetting (1), a phenomenon where the model's performance on carlier concepts degrades as new classes or domains are observed. Class-incrementall learning (CIL), a branch of continual learning, addresses this issue by enabling models to acquire knowledge from new classes while preserving their ability correctly classify previously learned categories
(2). Until recently, most CIL methods have focused on relatively small networks such ResNets (3) and often trained them starting from random initialization (4; 5). With the rise of large Foundation Models (FMs) (6; 7; 8) such Vision Transformers (ViTs) (9), many CTT methods now capitalize on the robust representations provided by FMs, marking a significant paradigm shift in the field, showing that leveraging strong initial representations from large-scale pro-training significantly enhances incremental learning (10; 11; 12). However, sequential fine-tuning of FMs inevitably alters the pre-trained representations, leading substantial forgetting (13; 14; 15).
To tackle this, post-training adaptation strategics such learnable prompts (16; 17; 18; 19; 20; 21) and lightweight adapters (22; 23; 24) restrict updates to small subsets of parameters, atop large frozen pre-trained backbones. While this helps to mitigate forgetting, they introduce new tradc-offs. Learnable prompts aim to steer the FMs existing knowledge to new tasks by introducing small trainable embeddings, keeping the model's core parameters unchanged. This guides the model to activate relevant pre-existing knowledge for new tasks and offers great stability, but often limits task-specífic adaptability. Tn contrast, adapters inserts small trainable neural networks directly into the FM's layers to provide localized fcature refinement with high plasticity, but this flexibility often comes at the cost. of quadratic parameter growth with increasing model depth. This trade-off exemplifies the well-known stabilit-plasticiity dilemma (25) and motivates the central question of this work:
'How can we efficiently tackle the Slubility-plastiii dilemma in continual post-training?"

---

Published in Transactions on Machine Learning Research (02/2026)
To address this question, we take inspiration from neuroscience, where the brain achieves continual learning by forming invariant representations in the ventral visual stream (26; 27), while flexibly adapting and modu- lating them through task-specific circuits in the prefrontal cortex (28; 29; 30). In other words, the prefrontal cortex receives these stable visual representations and refines them through selective synaptic plasticity, enabling flexible adaptation to lask demands and effectively guiding appropriate behavioral responses.
Analogously, we aim to leverage a large pre-trained model to emulate the ventral visual stream, which extracts stable and invariant features. To adapt these general features for specific tasks, we insert small lightweight modules just before the decision layer, mirroring how cortical circuits flexibly refine representations based on task-specific demands. This design avoids redundant relcarning of low-level features and aligns with biological learning principles, enhancing both efficiency and adaptability.
To this end, we first introduce a new parameter-effcieee fine-tuning (PEFT) module 'Learn and Calibrate', or LuCA, which compriscs two components: (1) a residual adapter that applics task-specific feature trans- formations, and (2) a calibrator that reweighs and enhances the adapted features via attention-like gating.
Then, to enable post-training in CIL setup, train a single sparse LuCA module for each task, operating exclusively on the final [CLS] token represcntation of ViTs. We refer to this approach as 'Token-level Sparse Calibration and Adaptation,' or briefly TOSCA. By localizing adaptations at the final semantic aggregation point and preserving the low/mid-leve feature hierarchy, TOSCA mirrors the harmony between the ventral visual stream and the prefrontal cortex.
Specifically, task-sp information is residually acquired just before classification through a dedicated LuCA module, sparsified via f-rreuularization to promote parameter orthogonality, which improves the spe- cialization and distinctiveness across modules. This targeted injection in continual post-training protocol preserves the stability of the rich, generalizable features of the FMs, while providing the necessary plas- ticity through precise, task-specific adjustments at the point of makiin. The inference protocol leverages entropy minimization over task-specific prodictions, as correct modules produce low-entropy class dist ions. This approach removes the reliance on task identifiers or exemplar replay while achieving state-of ahet performance without complicated procedures.
Our contributions are three-fold:
I. We introduce a new PEFT module LuCA designed to learn task-spocific residual transformations while refining features through additional calibration gating.
II. We propose TOSCA, a neuro-inspired and theoretically grounded continual post-training approach that strategically integrates our LuCA module at the final semantic aggregation point in the network. This balances stability and plasticity while maintaining model-agnostic parameter count, unlike many prompt- and adapter-based methods that scale linearly with the number of layers.
III. We validate TOSCA's advantages with extensive experiments on six benchmarks. We find that TOSCA yields (i) 7 21% higher accuracy than prompt-based methods and 4-12% higher than adapter-based methods on out-of-distributiin datasets, (ii) ~2.5x faster overall runtime, and (iii) -8x fewer param- eters than layer-wise adapters.
2 Related Work
CIL with Randomly Initialized Models. Not long time ago, the focus in CIL was training deep neu- ral networks sequentially from scratch, and the strategies be categorized into four main approaches. Regularization--asee methods (31; 32; 33; 34) maintain the model by selectively stabilizing changes in parameters or predictions. Replay-based methods approximate and reconstruct previously learned data distributions by either storing (35; 36; 37; 38; 39; 40; 41; 42; 43) or generating (44; 45; 46; 47; 48) sam- ples from past experiences. Architecturo-based methods allocate distinct parameters and subspaces different sets of classes, whether expanding the architecture (49; 50; 51; 52; 53) or pruning the existing one (54; 55; 56; 57; 58; 59; 60; 61) to obtain and preserve key parameters.
2

---

Published in Transactions on Machine Learning Research (02/2026)
dassifier
classifier
claseifier
classifier
cls
cLS
g
Pretrained Transformer Layer
Transformer Layer
Pretrained Transformer Layer
Pretrained Transformet Layer
+
Pretrained formerrr Layer
Proteained Transformer Layer
Pretrained Transformer Layer
Pretrained Transformer Layer
Pretrained Tormer Layer
Preteaincd Transformer
Pretrained Transformer Layer
Pretrained Transformet Layer
el
Pectrainel Embollding Layer
Pestruined Embeelding Layer
Pretrained
Pretniinal Embelking
Embeediing Lever
(a) Prompt-Based
(b) Adapter-Baar -esperr-Basdd
(c) TOSCA (ours)
Figure 1: Overview of the FM-based continual post-training methods. Prompt-based methods influence the self- attention process of an FM, either from the input layer or across all layers. Adapter-based methods enable task- specific adaptations by inserting lightweight neural modules into the FM's layers. In contrast, we propose to train single module that operates exclusively on the final [CLS] token representation, efficiently adapting and calibrating features just before classification. This design offers streamlined and effective alternative existing methods.
CIL with Pre-Trained Models. In contrast, recent advancements in CIL research have shifted towards leveraging pre-trained FMs since representations derived from those models have proven to be effective not only in facilitating knowledge transfer but also in mitigating catastrophic forgetting during downstream con- tinual learning (10; 11) with minimal adaptations (12). Therefore, post-training methods in this context aim improve performance with minimal additions and modifications while freezing the FMs. L2P (18) borrows technique from NLP by introducing learnable prompt pool and selecting instance-specific prompts via key-query matching selection mechanism to guide the FMs response. DualPrompt (19) extends L,P by designing G-Prompt and E- E-Prompt, which encode task-invariant and task-specific instructions respectively, CODAPrompt (20) uses contrastive learning to decorrelate representations of the prompts to reduce interfer- ence and combine them by attention-based weighting method. HiDePrompt (21) leverages prompts together with class-level feature statistics to enable implicit featurc-space replay and improve class-wise alignment. APER (22) explores various PEFT methods and shows that a simple prototypical classifier called Simple- CIL serves a strong baseline. RanPAC (15) extends SimpleCIL by randomly projecting features into a high-dimensional space with analytical ridge classifier. EASE (23) attaches adapters to each layer of FMs to create expandable subspaces, and during inference, it concatenates all feature representations from different sets of adapters. MOS (24) adds replay generation for classifier alignment and an adapter merging over EASE to reduce mistakenly retrieving irrelevant modules during inference due to parameter drift.
3 Background
In this section, we first formally introduce the preliminaries of the CIL and describe how FMs are utilized to facilitate incremental learning. We then provide an overview of existing approaches, highlighting their strengths and the key limitations they face in effectively addressing the challenges of continual learning.
3.1 Class-Incremennt Learning (CIL)
Class-incromental lcarning is scenario where a model contimually learns to classify new classes to build a unified classifier (35). Formally, we train models sequentially on a series of tasks {T.) where each lask with a task-specific dataset D, = ((x,, H)) introduces a set of previously unseen classes ys e Y, with n, instances, ensuring Y. for any + E', i.c. non-overlapping classes for different datasets, while the underlying data distribution may evolve over time. During the t-th training stage, the model is updated using data exclusively from D..
3

---

Published in Transactions on Machine Learning Research (02/2026)
From the model perspective, following typical FM-based CIL works (18; 19; 20; 22; 23; 24), we assume that foundation model is available for the initialization of the model f(x) which we define with two components: f(x) = W (x),, where & : X' + R is the feature extractor and W E [ddx[>i is the classifier. For a vanilla ViT (9), the initial embodding layer converts the input image into a sequence of token embeddings, denoted as X = [xeo Xex.-...,x,, X E R((+1)xd, where L is the number of patch tokens and Xeo represents the prepended [CLS] token. The sequence X is then processed through subsequent transformer layers, including multi-head self-attention and feed-forward networks, to produce the final embeddings. Following the standard practice, we use [CLS] token Xe, the global image representation and consider $(x).
The effectiveness of the model is evaluated across all encountered classes, collectively can be represented as y = Yi U UY2 U... Y, after each learning stage. Specifically, we aim to find a model f(x) : X + ) that minimizes empirical risk across all test dataset without task indices by balancing between learning new classes and retaining information about old ones the replay-free setting (18; 19; 20; 22; 23).
3.2 Overview of Post- Dastnirann in CIL
In the era of FMs, the main idea of many works seeks to modify the pre-trained weights slightly with post-training, to maintain the gencralization strength and we can mainly divide these approaches into three.
Learning Prototypical Classifiers. These methods (15; 17; 22) focus on learning set of prototypical class representations, typically by computing class centroids or prototypes from the features of incremental classes. Given an imput instance with label y E Y,, let (xx) be its feature vector extracted by a pre-trained backbone. Then, the class prototype Py is defined as
Py
(x;;)
(1)
and instances are classified by measuring their distance to these protolypes in the feature space. is an efficient solution for simple dass-incremental learning tasks by training only a classifier. However, these methods tend to rely too heavily on pro-trained knowledge and often fail to sufficiently adapt to new classes. This limits their effectiveness in more complex learning scenarios requiring feature-space reorganization.
Learning Prompts. This body of works (18; 19; 20; 21) construct and train a learnable pool of prompts that can be shared across all tasks to influence the self-attention process either from the input layer alone or across all layers. This prompt pool with a size of M denoted as P = (P1, P2, ... PM), where P₁ E Xd represents single prompt with token length L.p and the same embedding size d as image patch embedding x.- Each prompt is paired with a trainable key vector k; E Rdd encodes task-specifie information while preserving the pre-trained backbone ((-), creating set of key-prompt pairs {(k,, (k2, P2).... (KM. PM)).. The training objective jointly optimizes prompts, keys, and classifier through
N
min T (((;;)) + A Σ ((((), ka.) P.W.W
(2)
where (((,)) is cross-entropy loss measuring the discrepancy between the prediction and ground truth, 7()) measures cosine similarity between keys and queries, 5; denotes the index of the i-th selected key from the key-prompt set, and À balances task performance against prompt selection elficacy. During inference, the model first extracts key features (xx) from the frozen FM without any prompts to solve the prompt retrieval objective
N
K = arg
Σ (o)) SUM
min
(
(3)
where the argmin operator returns the index set K; = {s:) ) which then used to access the correspond- ing prompts {Pa,, "......, Paw>. }. These prompts then condition the transformer's self-attention layers via concatenation with patch embeddings, yielding final predictions through an additional pass on the modified encoder P)x;
4

---

Published in Transactions on Machine Learning Research (02/2026)
Although they present relatively efficient adaptations, selecting the correct prompt for given task becomes challenging especially in long and complex scenarios, as the fixed key embedding space o(-) struggles to discriminate between semantically similar but task-distinct prompts, leading to retrieval conflicts when (ks,, ky) = for prompts P., Pj from incompatible tasks, resulting in forgetting.
Learning Adapters. These approaches (22; 23; 24) address catastrophic forgetting by inserting lightweight neural modules called adaplers into the FM's layers, enabling task-specific adaplations while preserving frozen base parameters. Each set of adapters, A, = A., AN), for task t operates via residual connections across N transformer layers. These adapters typically project features through a low-dimensional bottleneck, given an intermediate feature as defined in
A(z)=
dowzWdown)NDl W ups Wdown E Rdxr, Wa E R
(4)
where z represents the output of the MLP block in a transformer layer, vi denotes non-linear activation function, typically GELU, and the adapter's projection layers follow the constraint & d. Task-specific adapter sets then can be trained using either a feature concatenation strategy or a module merging strategy. Under the feature concatenation strategy (23), adapter sets are trained sequentially for each session and their outputs are concatenated with the FM features at the cost of quadratic scaling or a lincar increase in dimensionality. In contrast, the module merging strategy (24) builds on previous adapter sets where each new set A refines the representation produced by the preceding set Ar-1 to produce gradual and unified feature representation. This is more parameter-efficient compared to the feature concatenation strategy, but it risks accumulating feature drift over successive tasks, especially when new class distributions diverge significantly from those of earlier sessions.
Although they modify the pre-trained model's feature representations via residual additions, inserting adapters into all N transformer layers incurs substantial parameter overhead, requiring (T x N 2dr) additional parameters, where T' denotes the number of tasks, r is the bottleneck projection dimension, and d is the embedding size. Morcover, these residual modifications introduce subtle yet cumulative deviations from the original pre-trained feature space, which become particularly pronounced in deeper layers. Conse- quently, while individual adapters are lightweight, their pervasive placmennt across layers poses challenges for overall parameter efficiency during both training and inference.
4 Methodology
Here, we first introduce our pareros- PEFT module LuCA and then present the details of TOSCA, a specialized instantiation of LuCA designed for post-training in CIL, with theoretical insights into the stability-plastiiity trade-off of the feature manifolds.
d(x)'
LuCA Module. Our module is based on an adapter-style design, where it learns re- fined task-specific information. In particular, LuCA separates feature transformation from discrininative feature enhancement through a dual adapter-calibrant structure, enabling fine-grained control over parameter updates. LuCA can process any interme- diate representation z E Rd through two sequential operations:
Calibrator
Adapter
L(z) = C(A(z)),
(5)
where A(-) is residual adapter that applies bottlenecked feature modulation with Eq. (4) to preserve original semantics via skip connections while learning task-specific offsets. The calibrator C(.) then reweights the adapter's output features through an attention-like gating, and refines the more discriminative features with
((x)
Figure 2: LuCA.
C(z) = (zVaoun Vup. Vdoum E Rdxr, Vup Rxd
(6)
5

---

Published in Transactions on Machine Learning Research (02/2026)
where O denotes the Hadamard product and indicates a sigmoid activation function. We call this compo- nent a 'calibrator' because its gating mechanism performs task-dependent feature calibration: it produces a soft importance mask that rescales each feature dimension according to how informative it is for the current task. This operation corrects over-activated or noisy channels while amplifying stable and discriminative ones; thereby calibrating the representation before it is passed to the classifier. Compared to complete fine- tuning that scales with (dd)), LuCA provides an efficient and ficxible mechanism for task adaptation with only 4 d X r trainable parameters, leading to a significantly reduced (dr) complexity, where r < d.
TOSCA: Specialization for Continual Post-Training In this work, to enable continual post-training with neuroscientific inspirations, we instantiate the LuCA module as TOSCA which is a strategic implemen- tation operating exclusively the final [CLS] token just before the classifier. Given an input x, the frozen pre-trained backbone generates features ((x) of the last [CLS] token, and TOSCA refines them through
(xx) L(x(x)) C(A(((x))).
The design of placing at the last token a deliberate architectural choice with three advantages: First, by localizing adaptation to the final [CLS] token, TOSCA preserves the feature hierarchy: low- and mid-level representations remain stable, while only the high-level abstractions adapt to new tasks. This minimizes disruption to learned invariant features (stability) while still injecting flexible task-specific adjustments (plas- licily) at the final semantic aggregation point, mirroring the functional synergy between the ventral visual stream and the prefrontal cortex where stable representations from the ventral stream (26; 27) are integrated and modulated by task-spocific circuits in the prefrontal cortex (28; 29; 30) just before driving behavioral responses. Second, the last [CLS] token inherently aggregates all semnattic information, making an opti- mal locus for task-specific refinement, contrast to input-layer modifications of prompt-based approaches. which indirectly influence later representations through the self-attention mechanism. Third, this design avoids modifying multiple layers and ensures that the total parameter count remains architecture-agnostic with a fixed footprint of 4dr that does not scale with model depth. In contrast, layer-wise adapters scale linearly as N x 2dr for N layers. This significant reduction in parameters leads to decreased training and inference complexity.
Training Protocol. We completely freeze the parameters of FM and only train the TOSCA's parameters e- = (Woom,, Wup, Vdown, Vip} together with the protolypical classifier WT. We utilize a new TOSCA module for each incremental stage t with the parameters O,, which encodes task-specific information by optimizing composite objective function that combines cross-cntropy loss with l-regulaariztion as
min e, UWT
Σ (x.))DDe
Ic (WTT(x)) v) + A111/ +(x); = Lx(((x))
(8)
where X controls the regularization strength. The & term induces to use of only sparse subset of weights in the module, which encourages orthogonality. This orthogonal specialization enables each module to specialize on distinct feature dimensions, preventing interference between the tasks (62; 63). After training, we store O while keeping the pre-trained backbone +(-) intact.
We divide our inference protocol into a two-stage design for computational effi- ciency with minimal overhcad. In the first stage, single forward pass through the frozen backbone extracts a shared representation o(x) for the input batch, up to the classifier. In the second stage, each TOSCA module independently processes this representation to produce task-specific predictions. This design avoids redundant computation and enables efficient reuse of features across all modules. Each transformed feature *(x),, produces a task-specific probability distribution, and the module with the lowest output entropy is selected to make the final prediction over the union of all classes. This leverages the fact that an appropriate tpsk-specific module typically yields lower uncertainty due to its specialized feature calibration and orthog- onality, thereby enabling selection of the relevant module without access to task labels. The procodure can be formalized as in Eq. (9), where H--) denotes the Shannon entropy and T. represents task priors, assumed uniform by default.
Inference Protocol.
6

---

Published in Transactions on Machine Learning Research (02/2026)
T Σ (mpp(iis)) ). pe(v)x) softmar(W ((x))
9 - arg min H yEy
(9)
Summary. Although prompt-based methods are primarily better at stability by modulating the self- attention dynamics within frozen foundation models, adapter-based strategies focus more on promoting plasticity through lightweight residual adaptations that fine-tune task-specific knowledge. Despite the success of both directions in CIT, they often address only one side of the stability-llastictyy dilemma. To combine the complementary benefits of both, we introduce a new gencral-purpose PEFT module LuCA that integrates adapler with calibrator to yield more coherent and refined feature representations. Unlike exisling continual pest-training methods that place modules at every layer, we strategically position a sparse LuCA module to operate solely on the final [CLS] token just before the classifier, which we refer to as TOSCA. This neuro-inspired design encourages each module to specialize orthogonally in its feature subspace, prevents task interference during inference, and efficiently achieves an elegant balance between stability and plasticity, without relying on complex tricks, offering a principled step forward to post-training in CIL. Please refer to Appendix A.1 for the theoretical underpinnings and algorithmic flow.
5 Experiments
In this section, we describe our setup and present results on seven benchmarks, comparing our approach with sfate-of-the-art algorithms. Additionally, we compare against a joint training performance and provide task-wise accuracies to demonstrate the adaptivity capacity of each method. Finally, we share a parameter and run-time analysis, along with an ablation study and offer deeper insights with further discussion.
5.1 Experimental Setup
Datasets. Since FMs often exhibit substantial knowledge of upstream tasks, we adopt the evaluation framework proposed in (18; 19; 20; 22; 23; 24) to assess their performance across a diverse set of benchmarks. These include CIFAR-100 (64), CUB-200 (65), ImageNet-R (66) ImageNet-A (67), OmniBenchmark (68), and VTAB (69). These datasets encompass both standard CIL benchmarks and out-of-distributiin datasets which exhibit significant domain shifts relative to the dataset used for pre-training, c.g. ImageNet (70). Furthermore, to assess true out-of-distribution performance, we use the FuroSAT (71), a dataset with fundamentally different satellite imagery domain. CIFAR-100 has 100 classes of natural images, each with 500 training images. CUB-200 datasct consists of images from 200 bird classes, with about 30 images per class for training. ImageNet-R includes 24000 for training images from 200 classes with abstract forms. ImageNet-A consists of 200 classes and 7500 training samples that are usually misclassified by ResNet models. Ommibenchmarrk with 300 classes and VTAB with 50 classes are designed to evaluate the generalization of visual representations. Finally, EuroSAT contains 10 satellite imagery classes, each with roughly 2,000-3,000 samples, totaling 27,000 geo-referenced images. To perform class-ineremental learning, follow (22; 23; 24) and adopt the notation 'B-m Inc-n', where denotes the number of classes in the initial stage and n indicates the number of classes introduced al each incremental stage.
Comparison Methods. We select state-of-the-art continual post-training methods for comparison: Sim- pleCII. (22), RanPAC (15), L.2P (18), DualPrompt (19), CODAPrompt (20), HiDePrompt (21), APER (22), EASE (23) and MOS (24). All of them work under the FM-based replay-free CIL setting, except HiDePrompt and MOS which generate pseudo-replay from class stotistics for classifier aligument. We also include one lower-bound and one upper- upper-bound bound reference point: 'Finetune' sequentially fine-tunes the FM; and 'joint' trains the model with all classes the same time. All methods are implemented using the same FM.
Evaluation Metrics. We compare the methods with well-recognized continual learning metrics which are basod on the accuracy over all stages obtained after last stage, and the accuracy across all stages (35). We denote the Top-1 accuracy after the t-th stage A, and use Ar to represent the performance after the final stage. The average performance across all incremental stages is then measured by Á = + Ar
7

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Average accuracy (A) and last accuracy (Ar) on six datasets with ViT-B/16-IN21K 'IN-R/A" stands for "ImagNNetRR/A and 'OmniBenchh' stands for 'OmmiBenchmark? We report all compared methods with their source code and show the best performance in bold. '_' denotes non-applicability under the released implementation.
CIFAR BO Ins5
CUB BJ Incld
IN-R BU Inc20
IN-A Bo Ine20
OmnilBench BD Inc30
VTAB BU Incl0
Method
A
Ar
A
Ar
A
Ar
A
Ar
A
Ar
A
Ar
Joint
05.21
92.62
81,92
67.97
85.41 a.
94.96
Finetame
60.65 ... 48.12
65.78
33.13
59.09
49.46
30.58
19.86
63 71
45.45 vi
31.60
21.63
SimpleCIL
86.18 ... 81,25 am
91.38 am
80.73 aM
61.31 aM
51.55 Loa
38,92 aM 18.77
79.59 ..> 73.13 aM
90.65 ...
81.43 AM
RanPAC
93.41.. ... 90.02 ... 03.35 .*r
89.27 *x
81.31 ...
75.44 12
64:22 ..4 54.76 .13
83.38 .. 78.33 ..s 94.91 ... 91.93 4m
L2P
81.90 ... 80.06 a. 73:22 am
61.55 am
75.92 and
70.58 ...
50.13 ..
50 ..
73.96 ... 61.63 ..
78.61 .. 64.81 am
DralPrompt
85.6 ..
...
70.51
71.48 .0.
66 11
51.57 -..
75.58 ... 66.46 ...
.*.
75.86 .T
CODAPrompt
87.61 ... 81:46 .M
77.65
68.44
76.25 1.0
71:39 ..3
58.82 L..
47.18
73.73 ...
09:46 ...
87.60 a.
86.71
HiDePPompt
85.99 ... 82:95 xxr
83.86
79.56
79.63
74.06
APER-Viaptee
89.37
81.91
91,02
50.72
71.81
66.97
30.57
19.16
80.48
7101
90.59
81,28
EASE
90.79
85.97
02.51
56.49
80.33
75.74
61.00
51.90
81.11
74.16
90.26
82,07
MOS
93.45
90.04
03.42
90.07
82.20
77.02
63.57
54.60
84.73
79.97
92.75
92.74 Am
TOSCA (ours)
96.37
95.64
93.47
91.09
82.27
79.28
66.92
65.37
84.75
82.35
96.59
93.87
CIFAR BO Inc5
CUB BO Inc10
ImageNet-R BO Inc20
100
100
95
Acctracy [%]]
6.861
0.981
3.671
90
80
80
80
60
65
20
40
60
80
100
50
100
150
200
50
100
150
200
ImageNet-A B00 Inc20
Onnibenchmark B0 Inc30
VTAB B0 Inc10
95
100
1.25t
Accuracy [%]
12.021
1.581
65
80
85
45
65
70
50
100
150
200
100
200
300
20
30
40
50
Number of Classes
Number of Classes
Number of Classes
SimpleCIL HiDePrompt
RanPAC APER
L2P EASE
DralPrompt
CODAPrompt TOSCA (ours)
MOS
Figure 3: Last accuracy (Ar) after each learning session on different methods under different setrings. All methods are initialized with VIT-B/16-INIK. annolate relative improvement TOSCA above the runner-up method with numbers at the last incremental stage.
Implementation Details. We conduct experiments on NVIDIA A100, and reproduce the compared methods using Pyroh (72) and Pilot (73). Consistent with (22; 23; 24), we utilize two representative FMs: ViT-B/16-IN21K and VIT-B/I6-INIK. Both models are pre-trained on ImageNe121K, with the latter further fine-tuned on ImageNetIK. For TOSCA, we train the model with SGD using a batch size of 48 over epochs. The learning rate follows cosine anncaling schedule, starting at 2.5e The projection dimension set to 48 and the & contribution À is set - We perform multiple runs with five different random seeds and report mean and standard deviation for each method.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table Task accuracy (Ac) on fine- grained CUB Inc10 with VII-B/16-IN2LK, TOSCA returns the highest average task-wise accuracy, illustrating its superior plasticity against existing approaches.
Task
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
T11
T12
T13
T14
T15
T16
T17
T18
T19
T20
Avg
SimpleCIL
100
97.6
97.9
97.2
98.1
89.6
96.4
93.9
87.5
92.1
83.2
86.9
88.9
84.9
95.1
87.3
93.2
82.5
82.6
91.2
RanPAC
100
96.1
97.9
97.2
98.1
90.9
92.4
97.1
91.7
91.1
94.7
57.6
85.7
91.5
87.3
96.7
91.0
95.2
83.3
87.6
92.9
L2P
100
85.2
71.1
77.3
85.9
91.8
71.7
87.2
78.2
64.7
85.2
59.3
79.4
66.1
62.7
75.8
70.2
59.2
54.5
59.2
74.2
DualPrompt
100
96.9
97.4
90.9
95.3
81.9
76.4
95.0
77.2
86.1
81.1
58.5
59.2
75.8
76.8
69.9
62.1
58.2
80.2
CODAPrompt
97.3
98.3
72.4
87.4
97.6
85.7
59.6
94.4
93.9
89.3
80.8
09.1
50.8
83.8
46.4
40.9
72.9
61.9
94.3
86.9
78.1
IrDePrompt
99.5
92.6
77.2
86.2
82.9
90.9
70.3
90.8
92.9
92.3
88.5
79.1
67.8
87.0
65.7
58.5
87.5
80.7
90.6
88.6
84.9
APER-Adapter
97.6
97.3
98.1
89.6
96.1
87.5
92.2
86.9
88.9
84.9
95.1
87.3
93.2
82.6
91.3
EASE
100
97.6
97.9
97.2
97.2
88.3
93.4
96.5
95.5
88.2
92.2
87.6
87.8
88.9
86.5
94.3
88.1
95.2
85.6
86.7
92.2
MOS
98.3
97.7
97.9
98.1
98.1
95.5
94.3
97.8
94.7
91.1
95.6
91.1
89.0
93.2
89.6
95.9
91.0
90.9
88.5
92.8
94.1
TOSCA (ours)
97.7
98.9
99.1
100
96.4
100
100
97.1
99.1
99.1
90.6
91.5
96.8
99.2
99.0
94.7
93.9
97.6
100
TOSCA
Addeed (M) per tock
93.2
91.7
90.2
96
Accuracy [%]]
L1 strength (2)
Accuracy [%6]
1e.3 5e-4 1e-4
94
94.7
94.9
92.5
LiDeefompt
BASE
92
94.9
95.3
93..4
DualPrompt
90
80
Adapter
Calibrator TOSCA
93.6
93.4
91.0
88
TOSCA_r
CoDAPrompt
100
200
96
10 20 30 40
70 80 90 100 Number of Classes
Run time (mins)
Projection dimension (r)
(a) Memory computational cost
(b) Hyperparameter analysis
(c) Design and component ablation
Figure 4: Performance of TOSCA across different perspectives. (a) Memory & computational cost highlights TOSCA's efficiency, (b) Hyperparameter analysis illustrates effect & strength (X) and projection dimension (r) on accuracy, (c) Design and component ablation presents the impact of different components and flows accuracy.
5.2 State-of-the-Ae Art Comparison
We compare TOSCA with leading state-of-the-arl continual learning methods across six benchmark dalasets and multiple pre-trained foundation models. Table summarizes the final-stage accuracy using VfT-B/10 16 pre-trained on IN21K for cach mothod. TOSCA consistently achieves the highost performance across all six benchmarks, significantly surpassing existing approaches such as EASE and MOS. Notably, MOS relies on replay generation for classifier alignment and therefore not strictly replay-free, whereas TOSCA achieves superior results without requiring a replay, To further examine learning dynamics, we report the incremental performance trends over successive training sessions in Figure 3. Across datasets, TOSCA outperforms the next-best methods by 12% CIFAR100, ImageNet-R, and ImageNet-A, indicated by the annotations at the end of learning sessions. These improvements are particularly pronounced on out-of-distributin datasots, highlighting TOSCA's robustncss and capacity to genoralize under distributional shifts. Please see our Appendix A.2 for more results.
5.3 Comparison to Joint Training
To provide bascline, which represents an upper-bound for performance since it has access to all data at once. While joint training still holds the top spot, TOSCA demonsurates highly competitive performance despite only training a single, lightweight module for each new task. This crucial finding, as TOSCA delivers near- optimal performance while drastically reducing computational cost and training time. Unlike full joint training, it avoids the need to revisit all past data, making it a practical and scalable solution for real-world continual learning scenarios where memory and compute are limited.
complete picture, also compare TOSCA to traditional 100-epoch joint training
9

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Task-wise accuracy (Ai) on outtof-distribution scenario with EuroSAT BO Inc2 to show adaptation of each approach.
Figure 5: Last accuracy (Ar) performance after each learning session on EuroSAT BO Inc2.
Task
T1
T2
T3
T4
T5
Avg
100
SimpleCIL
99.7
85.1
85.8
70.0
88.8
85.9
Accurac [96]
90
RanPAC
100
98.6
98.2
95.2
98.8
98.1
L2P
99.9
89.6
56.2
46.0
91.7
76.7
80
DualPrompt
100
92.6
60.6
23.7
75.5
70.5
CODAPrompt
99.9
88.8
92.8
90.0
72.1
88.7
70
APER-Adapter
99.9
87.2
86.2
67.3
90.8
86.3
2
4
6
10
EASE
99.9
93.6
92.1
71.1
93.4
90.0
Number of Classes
MOS
100
96.1
96.0
93.6
98.2
96.8
SimpleCIL
RanPAC
TOSCA (ours)
99.9
99.6
99.3
98.8
98.8
99.3
DualPromge
CODAAsompe
EASE
MO3
TOSCA
5.4 Task-Wrr Wise Plasticity
Plasticity a critical property for continual learning models, as reflects the model's ability effectively adapt to new tasks. potential concern for the plasticity of TOSCA may be that its adaptation is restricted to single module that acts only on the final layer's [CLS] token, whereas other methods perform adjust- ments across multiple layers. To investigate this, give detailed breakdown of the task-wise accuracy performance, evaluating across all methods. In particular, to truly evaluate plasticity, we examine perfor- mance on the CUB BO Inc10 scenario which is highly fine-grained and presents long sequence of tasks, and stresses the model's s ability to distinguish subtle inter-class differences over many incremental learning stages. Our results in Table show that TOSCA consistently achieves high task-wise accuracy, often matching or surpassing mothods that adapt multiple layers. This indicates that, despite its singlc-module design, TOSCA is capable of strong lask-specific adaptation. We attribute this performance to the strategic placement. of the module just. before the classification layer, which allows it to modulate features directly relevant to the decision without disrupting the pre-trained, stable representations in earlier layers.
5.5 Parameter and Rime Analysis
We further investigate FM-based continual post-training approaches in terms of accuracy, computational cost, and parameter efficiency CIFAR BO Inc5 benchmark in Figure 4a. TOSCA achieves the top performance while maintaining low computational cost and parameter overhead per task. In contrast, methods like CODAPrompt and EASE require significantly more parameters and longer run times, making them less efficient. Notably, MOS also allains high accuracy, but it comes at a higher computational expense due to additional processes such as adapter merging and replay generation. Overall, TOSCA demonstrates its offectiveness in CIL with minimal parameter overhcad and shorter run-time, striking balance between efficiency and performance.
5.6 Out-of- Distribution Performance
Although existing continual learning benchmarks incorporate certain levels of domain shift, evaluating ro- bustness under more drastic distribution changes remains an open and compelling testbod. We therefore compare all methods on the EuroSAT benchmark which introduces a substantial domain shift to assess true out-of-distribution performance. Our results in Figure 5 reveal that all approaches except RanPAC, MOS and TOSCA exhibit substantial performance degradation tasks progress. While some methods achieve strong task-wise performance, they fail to retain this knowledge incrementally. For example, EASE attains high per-task accuracy with an average of 90% in Table 3, yet its overall incremental performance drops to 5%, highlighting pronounced forgetting under distribution shift. Despite the significant domain shift, TOSCA outperforms all compoting approaches, demonstrating more robust generalization beyond the pre-training distribution. Notably, this performance gain is achieved with minimal additional computation, reinforcing TOSCA a practical and scalable solution for continual learning under severe distribution shifts.
10

---

Published in Transactions on Machine Learning Research (02/2026)
1.00
0.81
0.81
0.72
0.71
1.00
0.55
0.57
0.49
0.62
1.00
0.37
0.49
0.47
0.45
0.81
1.00
0.77
0.88
0.69
0.55
1.00
0.53
0.41
0.46
0.37
1.00
0.42
0.34
0.20
0.77
0.92
0.53
0.63
0.42
0.44
0.72
0.88
0.71
1.00
0.77
0.49
0.41
0.69
1.00
0.51
0.47
0.34
0.47
1.00
0.22
0.71
0.69
1.00
0.62
0.46
0.63
0.51
1.00
0.20
0.44
1.00
1
4
1
4
(a) x=0
(b) A=le-4
(c) A=5e-4
Figure 6: Cosine similarity of modules with increasing & strength (A). It encourages modules to become more
orthogonal, leading to learn different representations while facilitating to select the most suitable module inference.
5.7 Ablation Study
We also perform an ablation study on CIFAR BO Inc10, evaluating the incremental performance across different learning settings. First, we analyze the impact of &-reegularization strength (A) and projection dimension (r) on performance, shown in Figure 4b. Our findings indicate that & regularization improves accuracy, with performance reaching peak at À = 5c This promotes orthogonality among different mod- ules, improving module selection during inference. However, excessive sparsily degrades performance by excessively constraining representations, thereby reducing expressiveness and learning capacity. Similarly, increasing the projection dimension (r) improves accuracy up to r = 48, beyond which performance dete- riorates due the larger bottleneck. Based on these observations, we identify the optimal configuration À 5e-1 and 48, achieving an accuracy of 95.3%. Additionally, we compare the performance of different components, alternative module designs and configurations against TOSCA in Figure 4c. This includes a reversed variant, TOSCA r, which integrates new information atop the calibrated pre-trained features, formulated as (xx)' = ((C(((x))) We attribute this behavior to the ordering of the modules: when calibration operates on unmodified features before adaptation, the flexibility to modify representations is reduced, whereas introducing task-specific adjustments first and selectively regularizing them after calibra- tion enables more effective adaptation on newly acquired features, explaining the performance gap between TOSCA and TOSCA r. Therefore, our results highlight the crucial role of the calibrator while TOSCA surpassos all variants by effectively harmonizing its two modules working together. We further discuss this behavior and provide slep-by-step analysis of the functioning of our approach below.
5.8 Further Analysis and Discussion
We first analyze the effect of the & on the learned module representations by computing pairwise cosine similarities wetween modules under different regularization rengths. We reveal in Figure that a monotonic drop in off-diagonal cosine similarity as increases, indicating substantial redundancy and overlap in their functional roles. The underlying mechanism is straightforward: L regularization induces sparsity, and sparsity reduces overlap in the active feature dimensions across modules. When two modules rely on fewer shared coordinates, their output directions exhibit lower cosine similarity. In other words, & pushes modules toward using disjoint subsets of features, which manifests as reduced correlation and hence more 'orthogonal' representations in practice. This explanation aligns precisely with the trends in Figure 4b: at A = 5e -4 the representations become substantially more decorrelated, with similarities dropping toward Rs 0.3-0.4.
We next analyze how architectural components influence representational structure using t-SNE embed- dings of [CLS] features under three configurations as presented in Figure 7 on the first learning session of CUB BO Inc10 with VIT-B BIT-B/16-IN21K. The pre-trained ViT exhibits diffuse and overlapping cluster geome- try, producing an inter /intra-claass distance ratio of 1.97. Adding the adapter expands and scparates clusters by performing Lask-sensitive residual transformations while geometrically shifting representations towards a new task-relevant subspace, yielding a saassseparation measure of 5.32, around 1.5x improvement over the pre-trained basclinc, but clusters remain clongated and partially entangled. Incorporating the calibrator after the adapter leads to compact, well-aligned, and cleanly separated clusters, achieving a class-scparation distance of 99, corresponding to an almost. 4x improvement.
11

---

Published in Transactions on Machine Learning Research (02/2026)
15
10
10-
-5-
-19-
-15
(a) Vanilla [CLS]
(b) [CLS] Adapter
(c) [CLS] | TOSCA
Figure 7: t-SNE visualization of different architectural choices after the first learning session CUB BO Inc10. (a) Vanilla pretrained VIT shows substantial overlap in the [CLS] token space. (b) Adding an adapter improves separability but still exhibits noticeable spread and mixing. (c) TOSCA module further calibrates representations, leading to clearly distinct clusters with minimal overlap.
This refinement occurs because the calibrator amplifies or suppresses the adapter's task-specific signal ac- cording to relevance, which in result creates shrunk and well-grouped clusters. Critically, this adapter-then- calibrator ordering ensures that first the adjustment is introduced, then selectively regularized; inverting this order weakens enforcement, as calibration would operate on unmodified features before task-specific adaptation occurs.
Together, these analyses clarify the behavior our approach: & regularization encourages modules to rely on distinct subsets of representation space, yielding decorrelated outpuls that function as orthogonal experts. Meanwhile, the adapter and calibrator produce complementary geometric effects by separating and then refining class structure and ultimately driving the gains observed in ablations and main results.
6 Conclusions
In this paper, we introduced LuCA, new parametor-cffiiien fino-tuning module that combines a lightweight adapler with calibrator to refine feature representations and facilitale the integralion of new information. Building upon this, we proposed TOSCA, a neuro-inspired continual post-training framework based on foundation models, which leverages a single sparse LuCA module applicd exclusively to the final [CLS] token before the classifier for each task. This targeted adaptation enables highly efficient, task-specific modulation while maintaining orthogonality between tasks, resulting in minimal memory and computational overhead. Extensive experiments across multiple benchmark datasets demonstrate that TOSCA consistently achieves state-of-the-art performance. It effectively balances the prior methods with substantially fewer additional parameters, while retaining scalable and model-agnostic design suitable for advancing continual learning with foundation models.
tradc-off, outperforming
Limitations and future works. Although TOSCA shows strong performance, relies on pre-trained foundation models and its effectiveness depends on the generalizability of these models when adapting through single token. Future work would explore extending TOSCA to additional modalitics or multimodal models and diverse continual learning scenarios, including few-shot, blurry, and class-revisiting incremental settings.
Broader Impact
This paper aims to advance the field of machine learning, with focus replay-free class-increnan insremmntteettll post-training. By introducing a lightweight, trainable module that enables continual adaptation without the need lo store past data, the proposed approach addresses key concerns related to privacy, memory elficiency, computational overhead, and scalability, These properties make particularly suitable for deployment in privacy-sonsitive or resourco-constrained settings, such as personalized models in healthcare, adaptive vision systems for assistive technologies. While the method does not raise immediate ethical concerns, its application in high-stakes domains may require additional safeguards to ensure responsible use.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Acknowledgements
This work is supported by the EU Horizon programme through SYNERGIES, project under GA No. 101 146542; and ELLIOT, project under GA No. 101214398; and Dutch reiture with the support of SURF Cooperative using GA no. EINF-10242; and Turkish MoNE scholarship.
References
[1] Michael McCloskey and Neal Cohen. Catastrophic interference connectionist networks: The se- quential learning problem. In Psychology of learning and motivation. Elsevier, 1989.
[2] Mare Masana, Xialci Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. inasemnncreltttl learning: survey and performance evaluation on image classification. TPAMT, 2022.
[3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[4] Wei Zhou, Qi- Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class- incremental learning: A survey. TPAMI, 2024.
[5] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. TPAMI, 2021.
[6] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[7] Stéphane d'Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In ICML, 2021.
[8] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale. In ICLR, 2021.
[10] Vinay Venkalesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on calastrophic forgetting in neural networks. In ICLR, 2022.
[11] Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. JMLR, 24(214), 2023.
[12] Tx- Ying Wu, Gurumurthy Swaminathan, Zhizhong Li, Avinash Ravichandran, Nuno Vasconeelos, Rahul Bhotika, and Stefano Soatto. Class-incremental learning with strong pro-trained models. In CVPR, 2022.
[33] Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: A strong replay-free baseline for class-incremental learning. In ICCV, 2023.
[14] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning pre-trained model. In ICCV, 2023.
[15] Mark D McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad, and Anton van den Hengel. Ranpac: Random projections and pre-trained models for continual learning. In NewrIPS, 2024.
116] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's 's razor for domain incremental learning. 2022.
13

---

Published in Transactions on Machine Learning Research (02/2026)
[17] Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny, A simple bascline that questions the use of pretrained-models in continual learning. NearIPS Workshop on Distribulion Shifls, 2022.
[18] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022.
[19] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, 2022.
[20] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, et al. Coda-prompt: Continual decomposed attention-based prompting for rehcarsal-free continual learning. In CVPR, 2023.
[21] Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decom- position of prompt-based continual learning: Rethinking obscured sub-optimality. NewrIPS, 2023.
[22] Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, Dc-Chuan Zhan, and Ziwei Liu. Revisiting dass-incremantal learning with pre-trained models: Generalizability and adaplivity are all you need. IJCV, 2024.
[23] Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, and De-Chuan Zhan. Expandable subspace ensemble for pre-trained model-based class-incremental learning. In CVPR, 2024.
[24] Hai-Long Sun, Da-Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, and Han-Jia Ye. Mos: Model surgery for pre-trained model-based daassincremental learning, In AAAI, 2024.
[25) Stephen Grossberg. Studies of mind and bruin: Neural principles of learning, perceplion, development, cognition, and motor control. Springer Science & Business Media, 2012.
[26] Irving Biederman and Peter C Gerhardstein. Recognizing depth-rotated objects: evidence and condi- tions for threo-dimensional viewpoint invariance. Journal of Experimental Psychology: Human percep- tion and performance, 19(6):1162, 1993.
[27] Irving Biederman. Recognition-byy-coopmeee a theory of human image understanding. Psychological review, 94(2):115, 1987.
[28] Ningyu Zhang and Ning-long Xu. Reshaping sensory representations by task-specific brain states: Toward cortical circuit mechanisms. Current Opinion in Newrobiology, 77:102628, 2022.
[29] Karl Friston and Stefan Kiebel. Cortical circuits for perceptual inference. Neural Networks, 22(8):1098- 1104, 2009.
300 Apoorva Bhandari, Haley Keglovits, and David Badre. Task structure tailors the geometry of neural representations in human lateral prefrontal cortex. bioRaiv, 2024.
[31] James Kirkpatrick et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
[32] Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMT, 2017.
[33] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
[34] Friodemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, 2017.
[35] Sylvestre-Alvis Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incre- mental classifier and representation learning. In CVPR, 2017.
[36] Ameya Prablu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning, In ECCV, 2020.
[37] David Lopez-Paz and Aurelio Ranzato. Gradient episodic memory for continual learning. In NewrIPS, 2017.
14

---

Published in Transactions on Machine Learning Research (02/2026)
[38] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, 2019.
[39] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness class incremental learning. In CVPR, 2020.
[40] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Wor Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, 2021.
[41] Yaoyao Liu, Qianru Sun, and Qianru Sun. Rmm: Reinforcod memory management for class-incremental learning. In NeurIPS, 2021.
[42] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. In JCLR, 2022.
[43] Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. Error sensitivity modulation based experience replay: Mitigating abrupt representation drift in continual learning. In ICLR, 2023.
[44] Decebal Constantin Mocanu, Maria Torres Vega, et al. Online contrastive divergence with generative replay: Experience replay without storing data. arXiv:1610.0555 05555, 2016.
[45] Hamul Shin, Jung Kwon Lec, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In NewrIPS, 2017.
[66] Chen He, Ruiping Wang, Shiguang Shan, and Xilin Chen. Exemplar-supporter generative reproduction for class incremental learning. In BMVC, 2018.
[47] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao Tao, Dongyan Zhao, Jinwen Ma, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In ICLR, 2019.
[48] Grégoire Potit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. Fetril: Feature translation for exemplar-free class-incremental learning. In WACV, 2023.
[49] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In CVPR, 2017.
50] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In CVPR, 2021.
[5]] Fu-Yun Wang, Da-Wer Wei Zhou, Han-Jia Ye, and De-Chnan Zhan. Foster: Feature boosting and compres- sion for class-incremental learning. In ECCV, 2022.
[52] Wenjin Wang, Yunqing Hu, Qianglong Chen, and Yin Zhang. Task difficulty aware parameter allocation & regularization for lifelong learning. In CVPR, 2023.
[33 Da- WeiWei Zhou, Qi- Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient class-incremental learning. Tn ICI.R, 2023.
[4] Arun Mallya and Svetlana. Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In CVPR, 2018.
[55 Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. In NewrIPS, 2019.
56] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In NewrIPS, 2020.
[57]
Haeyong Kang, Rusty John Lloyd Mina, et al. Forget-free continual learning with winning subnetworks. In ICML, 2022.
15

---

Published in Transactions on Machine Learning Research (02/2026)
[58] Zifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan, Wei Niu, Tong Jian, Bin Ren, Stratis Ioannidis, Yanzhi Wang, and Jennifer Dy. Sparcl: Sparse continual learning on the edge. In NewrIPS, 2022.
[59]
Haeyong Kang, Jachong Yoon, Sultan Rizky Hikmawan Madjid, Sung Ju Hwang, and Chang D. Yoo. On the soft-subnetwork for few-shot class incremental learning. In TCLR, 2023.
(60) Murat Onur Yildirim, Elif Ceren Gok Yildirim, Ghada Sokar, Decebal Constantin Mocanu, and Joaquin Vanschoren. Continual learning with dynamic sparse training: Exploring algorithms for effective model updates. In CPAL, 2024.
[61] Murat Onur Yildirim, Elif Ceren Gok Yildirim, Decebal Constantin Mocanu, and Joaquin Vanschoren. Self-regulated neurogenesis for online data-incremental learning. arXiv preprin1:2403. 14684, 2025.
[62] Kun-Peng Ning, Hai-Jian Ke, Yu-Yang Liu, Jia- Yu Yao, Yong- Hong-Hog Tian, and Li Yuan. Sparse orthog- onal parameters tuning for continual learning. 0rXiu:2411.022 11.02813, 2024.
[63] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv:2310. 1,152, 2023.
[64] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[65]
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd dataset. 2011.
(66) Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Fvan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of cutto--distribution generalization. In ICCV.
[67] Dan Hondrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
[68) Yuanhan Zhang, Zhenfei Yin, Jing Shao, and Ziwei Liu. Benchmarking omni-vision representation through the lens of visual realms. In ECCV, 2022.
[69] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXio:1910.0455 2019.
[70] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-soale hierar- chical image database. In CVPR, 2009.
[71] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep lcarning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remole Sensing, 2019.
[22] Adam Paszke, Sam Gross, Francisco Massa, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS. 2019.
73] Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Pilot: A pre-trained model-basee continual learning toolbox. arXio:2309.071 2023.
16

---

Published in Transactions on Machine Learning Research (02/2026)
A Appendix
A.1 Compared Methods and TOSCA
Here, we provide an overview of the methods evaluated in the main paper. To ensure a fair and consistent basis for comparison, all methods utilize the same FM. This standardization allows us to isolate the contri- butions of each method's unique approach and compare their performance more accurately. Additionally, we present the pseudocode for TOSCA, providing a clear and detailed description of its working algorithm. This helps to better understand how TOSCA operates, offering insights into its efficiency and functionality within the context of caass-nncremenal post-training.
Joint: This method adheres to the traditional supervised batch learning paradigm, where all classes are presented simultaneously and trained over multiple epochs. It serves as the upper bound for caasemnnremlntal learning methods, as it does not experience forgetting.
This method updates all parameters of the pre-trained model when continually trained on new tasks. While it can achieve strong performance, it is susceptible to catastrophic forgetting, where previous knowledge is lost when learning new tasks.
Finetune:
SimpleCIL (22): SimpleCIL uses the FM in its original form, combined with prototypical classifier. It constructs a prototype for each class and utilizes cosine classifier for classification, aiming for efficient task learning without additional adaplations.
RanPAC (15) extends SimpleCIL by randomly projecting the features into the high high-dimensional space and learning the analytical ridge classifier in the Gram space for final classification.
L2P (18): L,2 integrates visual prompt tuning into class-incremental learning with pre-trained Vision Transformer (ViT). The method places the prompt only in the initial embedding layer, ensuring that the prompt adjusts the features at the carly stage of the model while maintaining the frozen structure of the rest of the pre-trained model.
DualPrompt (19): DualPrompt builds on L2P by introducing two types prompts: general prompts (G-Prompt) and expert prompts (E-Prompt). The G-Prompts are applied the carlier transformer blocks, allowing for broad task-specific adaptation. E-Prompts, on the other hand, are used in the latter blocks of the transformer, providing more specialized tuning for later stages of task processing. This separation allows for more efficient adaptation.
CODAPrompt (20): It addresses the challenges of selecting instance-specific prompts by introducing prompt rewcighting. It enhances the selection process through an attention mechanism that dynamically weights prompts, improving lask-specific performance.
HiDePrompt (21): This method leverages task-specific prompts and class-level feature statistics to perform implicit feature-space replay, explicitly optimizing task-identity and task-adaptive prediction. While effective for classifier calibration under supersupe pre-training, it introduces additional optimization stages and bookkeeping of representation statistics.
APER (22): This approach builds on SimpleCIL by introducing an adapter to each transformer layer, but only for the initial task. This adapter helps the pre-trained model to extract task-specific features during the first incremental phase, ensuring better adaptation the new task while minimizing forgetting subsoquent tasks.
EASE (23): This method adds adapters to each layer of FM for every task. This approach leads to good performance by concatenating the feature representations of multiple task-specific backbones, but it comes with an increase in model complexity due to the addition of task-specific adapters at every stage.
MOS (24): also trains adapters for each FM layer for every task. However, MOS introduces the concept of adapter merging and replay generation for classifier correction. These processes incrcases computational complexity, particularly during training, as the model must handle the merging of multiple task-specific adapters with an increasing number of parameters.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Algorithm TOSCA for Continual Post-Training
Require: Incremental datasets: {D), (D',D?.... .?....,,',
Pre-trained embedding: ((x)
1: for t .., do
2: Get the incremental training set DD
3: Initialize a module L, with parameters θ on top of last [CLS] token
4: Optimize the paramcters O of the modulc L, and prototypical classifier WT via Eq. (8)
5: Test the model with all classes seen so far via Eq. (9)
end for
TOSCA (ours): It trains lightweight adapter calibrator pair per task on the final [CLS] token immedi- ately before classifier. This neuroscience-insprred formulation reduces computational and memory overhead without sacrificing expressiveness, leading to better-calibrated representations and improved performance. Please see our Algorithm 1 for formal and more detailed flow.
Theoretical underpinnings of our approach. The core theoretical strength of TOSCA lies in its ability to preserve these representations by avoiding broad modifications to the feature space and instead localizing adaptation to the final layer's [CLS] token, while existing methods alter the feature space throughout the network and risk compromising the high-quality represenlations carefully learned from FMs. Its surgical design not only safeguards representational fidelity, but also provides principled mechanism to maintain stability while enabling procise, task-specific plasticity. Specifically, the design of TOSCA guarantees that, for all layers n < N, the feature manifolds H, of the FM are preserved as Eq. (10), meaning the feature distributions remain identical to the FM's distributions up to the penultimate layer:
Vn N H. TOSCA PTM
HPTM
(10)
Adapting solely through the [CLS] token of the final layer N, TOSCA allows a small bounded deviation the feature manifold while maintaining backward compatibility of the pre-trained features. This bounded
deviation mechanism can be expressed Fq. (11), where is small constant controlled by the residual connection and constrains changes in the feature space, ensuring that the representations remain robust while providing task-specific ficxibility during post-traning, thus achieving a principled stability-paasticiy balance.
PHPTM HTOSCA) Sc
(11)
A.2 Additional Results
In this section, we provide Table A for pre-trained ViT-B/16-INI and Figure A that illustrate each incre- mental step with Vin-B/I6-IN2IK, showcasing the performance of our proposed method. The slight drop in average accuracy A observed in few benchmarks with ViT-B/16-INC stems from its relatively limited generalization capacity compared to ViT-B ViT-B/16-IN21) 11116K,,,,,, given that TOSCA introduces only a single trainable module per task. This design choice is intentional, prioritizing computa- tional and parameter efficiency while maintaining adaptability in continual learning settings. Notably, our approach consistently achieves superior last incremental accuracy A, across all evaluated datasets, under- scoring its effectiveness. Overall, TOSCA improves the incremental performance with a minimal overhead cost during both the training and inference phases, emphasizing the efficiency of our method, and making it highly suitable for real-world applications. Furthermore, to verify that TOSCA accurately selects the appropriate adapler calibrator pair at inference time, we evaluate the accuracy of its entropy-based module selection strategy across all incremental tasks on CUB B0 Inc10. Specifically, we report the percentage of samples for which the selected module matches the ground-truth task identity. The results, summarized in Table B, demonstrate the high reliability of the proposed selection mechanism, especially with higher strength.
compared to existing approaches,
18

---

Published in Transactions on Machine Learning Research (02/2026)
Table A: Average accuracy (A) and last accuracy (Ar) six datasets with VIT-B/16-INIE 'IN-R/A' stands for 'ImageNet- R/A A, and 'OmniBench'' stands for 'OmmiBenchmark" We report all compared methods with their source code and show the best performance in bold. denotes non-applicability under the released implementation.
CIFAR BO Iucã
CUB BO Inc10
IN-R BO luc20
IN-A BO Inc20
OamiBBmth Inc30
VTAB
Inc10
Method
A
Ar
Ar
A
Ar
A
Ar
A
Ap
A
Ar
Joint
96.85 am
90.19
83,87 AM4
74.05 ..
83.08 aM
93.21
Finetane
41. PM
39.7 am
57.27 am
31.76 am
66.96 am
53.61 L1
28.64 L..
11.20 ...
63.35 ...
45.70 ..
67.81 1.
51.12 AM
SimpleCIL
82.21
76.24
90.42
85.16
66.89
61.27
58.70
49.44 .r:
78.67
72.20
90.50
83.61
RanPAC
02.43 ... 87.92 xM.
91.05 .M
86.77
83.11 21
76.50 co1
69.60 .m 39.64 to
83. ...
77.80 .. 02.26 1..
00.51
L2P
83.37 w.r 78.64 a.
70.64
58.70
77.22 aea
72.35 -.a
32.32
44.30
72.76
63.10 .. 81:25 =.
06.71
DralPrompt
82.41 + 76.39
73.78
63.47
74.37
69.58
56,42 mm
46.99
73:21
63.63 ... 82.84
70.39
CODAProupt
86.67 .. 80.68 a.
70.75 aM
61.01 AM
78.37 4.0
73.07 ..a
63.61 a.a
32.32 axa
72:22 a.3
08.26 am
81.85 a..
82.91
HiDePromph
84.67
79.39
81,46
77.11
80.32
76.06
APER-Adapter
89.46 ..
83,16 am
87.61 iis
90.63
78.25 190
72.07 Lo3
86.56
58.83
77.66 2.2
70.72 amc
89.59 1.4
82.60 1M
EASE
89.94 -..
*x
81.00 .*
82.96 ..a
77.45 w0s
02.36
78.40 ... 71.60 ...
90.71 ...
MOS
92.71
88.82
92.24
88.02
83.53
78.94
60.14
61.22
85.33
78.28
91.81
91.77
TOSCA (ours)
96.03
95.37 ox>
91.55 ...
89.05 1/ 83.57
82.25 ...
74.48 .. 72.30 ...
82 1.
79.65
94.33 ... 91.80 10
CIFAR B0 Inc5
CUB 30 Inc10
ImageNet-R B0 Inc20
100
100
95
5.551
2.761
Accuracy [%]]
2.771
90
80
75
80
60
55
20
40
60
80
100
50
100
150
200
50
100
150
200
ImageNet-A BO Inc20
Omnibenchmark B0 Inc30
VTAB B0 Inc10
85
95
100
1.481
Accuracy [%]
11.81
2.401
65
80
45
65
65
50
100
150
200
100
200
300
20
30
40
50
Number of Classes
Number of Classes
Number of Classes
SimpleCII.
RanPAC APER
T.2P
DualPrompt
CODA TOSCA (ours)
HiDePrompt
EASE
MOS
Figure A: Last accuracy (Ar) after each learning session under different settings with VIT-B/16-IN21K. Relative improvement of TOSCA is annotated above the runner-up method at the last incremental stage.
Table B: Per-task module selection success rate [%] on CUB BO Inc10 across different regularization strengths. Increased strength leads to more distinct modules and better module selection performance.
Task
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
T11
T12
T13
T14
T15
T16
T17
T18
T19
T20
0
88.2
85.4
83.9
87.1
84.3
82.6
92.5
94.1
86.7
93.2
84.9
88.7
90.2
83.4
89.3
87.1
91.5
85.6
88.3
92.1
A
lo44
92.4
89.7
90.1
92.6
90.8
88.9
97.3
98.1
91.5
98.7
89.6
92.9
94.3
88.7
93.5
91.2
95.4
90.6
92.9
96.1
A
5c-4
95.3
94.1
96.3
94.2
92.9
92.5
100
100
93.4
100
94.7
96.5
97.8
100
96.9
95.1
98.6
94.3
96.8
99.2
19


Reviews:
[{"review_id": "07dsdHrKB0", "reviewer": "Reviewer_Qtdv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors propose **LuCA** (*Learn and Calibrate*), a Parameter-Efficient Fine-Tuning module that integrates LoRA-style adaptation weights with an attention gating mechanism. Furthermore, they extend the utility of LuCA to the Continual Learning setting through **TOSCA** (*Token-level Sparse Calibration and Adaptation*). The TOSCA framework trains a LuCA module directly on the $\\texttt{[CLS]}$ token, incorporating a sparsity loss to encourage task specificity across individual LuCA modules.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experimental evaluation is comprehensive in its dataset coverage, and the results demonstrate that the proposed method achieves state-of-the-art performance compared to the reported baselines. While the additional experiments and ablation studies provide valuable insights, their presentation could be refined for greater clarity.", "audience_yn": "Yes", "audience_explanation": "The well-established setting of Class-Incremental Learning remains a subject of significant interest to a broad research audience. Within this context, TOSCA positions itself as a contribution that enriches the already dense landscape of existing methodologies.", "requested_changes": "### **Major Comments**\n\n**1. Experimental Evaluation and Baselines**\n\nWhile the experimental section presents a robust evaluation across multiple datasets, the comparison would benefit significantly from the following additions:\n\n* **State-of-the-Art Baselines:** In the well-established Class-Incremental Learning (CIL) setting, the absence of **Hide-Prompt** [1] and **RANPAC** [2] represents a gap in the comparative analysis. Hide-Prompt is widely recognized as a state-of-the-art method in the prompt learning category and is cited by the authors; similarly, RANPAC offers a robust analytical approach. Including these methods --- both in the performance tables and the \"Memory & computational cost\" plot --- is essential for a complete assessment.\n* **Inclusion of EuroSAT** [3]: To rigorously assess Out-of-Distribution (OOD) performance, I recommend including EuroSAT, given its distinct nature as a satellite imagery dataset.\n\n**2. Metric Clarification (Table 2)**\n\nTable 2 requires greater clarity regarding the reported metrics. Specifically, explicitly link the results to the specific accuracy metric $\\mathcal{A}$ employed.\n\n**3. Ablation Studies (Figure 4c and TOSCA_r)**\n\nThe explanation of the ablation study presented in **Figure 4c**, as well as the definition and role of the variant **TOSCA_r**, needs to be expanded. The current description lacks the detail necessary for full interpretation.\n\n[1] Wang, Liyuan, et al. \"Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality.\" Advances in Neural Information Processing Systems (2023).\n\n[2] McDonnell, Mark D., et al. \"Ranpac: Random projections and pre-trained models for continual learning.\" Advances in Neural Information Processing Systems (2023).\n\n[3] Helber, Patrick, et al. \"Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.\" IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2019).\n\n---\n\n### **Minor Comments**\n\n**1. Terminology: \"Post-training\"**\n\nThe term \"post-training\" appears misleading in this context. Although TOSCA utilizes a pre-trained backbone --- a well-established practice --- it still involves a training phase for the final LuCA module and the classifier. It would be more accurate to describe this as \"adaptation\" or \"fine-tuning\" rather than implying training has ceased.\n\n**2. Notation and Nomenclature**\n\nSeveral mathematical notations require correction or clarification:\n\n* In $x_i \\in \\mathbb{R}^D$, the symbol $D$ creates a conflict, as it is previously used to denote Datasets.\n* The statement *\"Finally, the embedded [CLS] token is considered as $\\phi(x)$\"* is unclear, particularly following the introduction of $x_e$. While the $\\texttt{[CLS]}$ token is a standard concept, the specific mapping from $x_e$ to $\\phi(x)$ should be formalized.\n* The phrase *\"across all encountered classes\"* appears contradictory if followed immediately by $\\mathcal{Y}_b$ (which usually implies the current task batch). Ensure the scope of the classes (cumulative vs. current) matches the notation.\n* In the phrase *\"Given an input instance $x$ with label $y \\in \\mathcal{Y}_b$,\"* verify if the intended notation for the set of labels is $\\mathcal{Y}_b$ or $Y_b$ for consistency.\n* In the section \"Learning Prompts,\" the variable $k_{s_i}$ is introduced without definition. Please clarify the query mechanism used for key matching.\n\n**3. Structural Observations**\n\n* The \"Background\" section provides explicit equations for prompt learning. Since the proposed method does not utilize prompts, this level of detail is unnecessary and could be condensed to focus on relevant concepts.\n* There are multiple instances where equations are cited in the text before they are displayed. This disrupts the reading flow; ensure equations are introduced or displayed prior to, or immediately alongside, their textual reference.", "broader_impact_concerns": "The Broader Impact section raises no concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UF7kkl7BwZ", "reviewer": "Reviewer_thQQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces TOSCA, a lightweight PEFT module combining a low-rank adapter and a calibrator, and TOSCA, a continual post-training method that places a single sparse LuCA module only on the final [CLS] token for each task. The idea is to adapt the final semantic representation while keeping the entire pretrained model completely frozen.", "claims_and_evidence_yn": "Yes", "claims_explanation": "They run experiments on different datasets, compare against strong baselines, report multiple seeds, and provide runtime + parameter analyses. The improvements are large and consistent.", "audience_yn": "Yes", "audience_explanation": "The method is conceptually appealing, easy to implement, and empirically strong.", "requested_changes": "- Can you clarify the claims around L1 regularization promoting “orthogonality” ( 5.6 Ablation Study ) ? \n- for the “bounded deviation” (Eq. 11), can you explain it further by explaining what is meant by \"bounded deviation\"? It will be interesting to see how the residual/calibrator structure enforces this as claimed in the text.\n- Can you provide more intuition for the separate roles of the adapter and calibrator, and why the adapter/calibrator ordering performs best? A short conceptual or geometric explanation would make Section 5.6 more insightful\n- Can you add a “last-layer fine-tuning” baseline? This is the fairest baseline to compare against TOSCA I think\n- Can you explain the figure 4c further in the text ( It's an interesting figure but in the text you didn't analyse it enough ) \n- How does entropy-based task selection perform when the task boundries are blur ?", "broader_impact_concerns": "Because the backbone remains frozen, any biases present in the original model are fully preserved and potentially reinforced across tasks. Although each module is small, the accumulation of many per-task modules increases storage, inference cost, and environmental footprint over long task sequences.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "v3eH32s8pK", "reviewer": "Reviewer_x8Rr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a continual learning approach for pre-trained models based on adapters. The approach, called LuCA, adds an adapter-calibrator pair to any intermediate representation computed by each layer. The approach is then applied to a class-incremental setup by using a single adapter-calibrator pair on top of the [CLS] features on the last layer. This application is dubbed TOSCA. TOSCA keeps a separate adapter-calibrator pair per task (like a task-incremental setup). However, TOSCA does not use task labels and relies on confidence prediction among all pairs to select the final answer. The approach shows an effective performance on a range of image classification tasks.\n\nThe approach is an application of adapter-based strategies to the final layer of pre-trained models (kept frozen). While I do not particularly enjoy using models pre-trained on ImageNet to study the performance on datasets like CIFAR or ImageNet itself, I acknowledge that this is becoming a common practice in continual learning. The authors also made an effort to test TOSCA on other datasets as well, with good results. \nOverall, the main merit of the proposed approach is that it achieves a solid performance against state-of-the-art approaches.", "claims_and_evidence_yn": "No", "claims_explanation": "The empirical evaluation results show that TOSCA is effective on image classification tasks. I find this claim to be supported by the  discussion in the paper, even though some additional studies could make the point stronger (see comments below). \n\nHowever, the neuroscientific inspiration for TOSCA, claimed across the whole paper, is simply not present (again, see comments below).\n\nThe claim about task-wise plasticity is also debatable: since the pre-trained model is fixed, it would be surprising to observe a plasticity loss. I do not consider the task-wise plasticity a positive consequence of TOSCA.", "audience_yn": "Yes", "audience_explanation": "The topic of continual learning for pre-trained models is an interesting topic for the machine and deep learning community.", "requested_changes": "1. In the Introduction, the authors claim to \"take inspiration from neuroscience\" for the design of TOSCA, mentioning the invariant representations developed by the ventral visual stream and the task-specific circuitry of the prefrontal cortex. Subsequent claims of neuroscientific inspiration are spread out across the rest of the paper as well. \nI personally don't like such vague, neuroscience-inspired claims when the approach clearly does not really build on any neuroscientific-related knowledge. Properties such as \"forming invariant representations\"... \"while flexibly adapting and modulating them through task-specific circuits\" is a broadly applicable statement. I might be wrong on this, but it really looks like the source of inspiration has been devised a posteriori rather than used as a starting point to design the approach. If that is the case, the provided statements are not entirely honest. \nOtherwise, are the authors really convinced that a large pre-trained model equipped with an adapter mechanism emulates high-level phenomena exhibited by the ventral visual stream or the prefrontal cortex?\n1. In the methodology section, high-level statements like \"It decouples feature transformation from discriminative feature enhancement with the dual adapter-calibrator architecture, allowing precise control over parameter updates.\" are confusing to me. For example, it would be much more informative to start the same paragraph by framing the proposed approach as an adapter-based method.\n1. Why is the calibrator called as such? What is the evidence that there is any calibration going on?\n1. Is LuCA more efficient in terms of the number of trainable parameters with respect to the adapter approaches previously discussed? It looks like the only advantage is due to the fact that TOSCA only applies LuCA at the [CLS] level. Also, I do not really see the need to introduce two different names for the same approach.\n1. There is no analysis on the impact of the soft-attention gating function. Is some sort of attention really happening? \n1. There is no analysis on the task-inference performance. Are TOSCA correctly selecting the adapter-calibrator pair related to the inference task? What is the confidence of each module? \n1. The theoretical underpinnings do not seem to add much to the overall contribution. The pre-trained model is always frozen, so the feature manifolds cannot change. The second theoretical consideration is related to the change in the feature manifold of the [CLS] token, which is upper-bounded by a small constant, according to the authors. There is no proof or bound on the constant. I do not think it would be that useful even if there were one. Overall, if there are no theoretical considerations to be made, the authors could avoid them altogether. I don't think this would diminish the impact of the work. Trying to create a theoretical derivation out of nowhere might not always be the way to go.\n1. The ablation study in Figure 4c should be properly described, as it is very interesting. The authors could consider providing a more detailed experimental setup for each of the settings in the plot.\n1. Since the pre-trained model is entirely frozen, why were the authors expecting a task-wise plasticity loss?\n\n\nNotation:\n1. In Equation 2 shouldn't N be M? Also, why is $\\phi$ part of the optimization problem? Why is the key $k_{s_i}$ indexed like that if it was previously indexed by $i$ only? What is the difference?\n1. Equation 3 is difficult to read. The argmin returns a set of indices but if I understood its meaning, the equation should return a prompt, not a set of indices.\n1. What is the difference between equation 7 and eq 5 since $z=\\phi(x)$?\n\nMinor changes:\n1. Acronyms should be introduced before use: e.g., PEFT in the introduction, N in equation 2.\n1. I did not completely get the difference between architectural methods and parameter-isolation methods in the related work section. Aren't they using separate modules? Pruning is often part of architectural methods as well.\n1. Equations are detached from the text, and they just interleave the paragraphs. They should be part of the text as if they were words.\n8. Caption of Figure 1 should be expanded to be self-contained.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)