Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention
Authors: Vaibhav Singh
Rahaf Aljundi
Eugene Belilovsky
Venue: Accepted by TMLR ()

Abstract: Foundational vision-language models (VLMs) excel across diverse tasks, but adapting them to new domains without forgetting prior knowledge remains a critical challenge. Continual Learning (CL) addresses this challenge by enabling models to learn sequentially from new data while mitigating the forgetting of prior information, typically under supervised settings involving label shift. Nonetheless, abrupt distribution shifts can still cause substantial forgetting, potentially nullifying the benefits of supervised updates, especially when storing or replaying past data is infeasible. In this work, we propose leveraging unlabeled test-time data in an unsupervised manner to reinforce prior task performance without requiring replay or stored examples. Unlike traditional Test-Time Adaptation (TTA), which primarily focuses on domain shift or corruption, our method improves performance on earlier tasks by exploiting representative test samples encountered during deployment. We introduce a simple teacher-student framework with gradient-based sparse parameter updates, and show that it effectively mitigates forgetting in class-incremental CL for VLMs, offering a memory-free alternative to episodic replay with strong empirical results.

Published in Transactions on Machine Learning Research (01/2026)
Dual-Phase Continual Learning: Supervised Adaptation Meets Unsupervised Retention
Vaibhav Singh
waibaau.singh@@mille quebee
Deportment of Computer Science Mila, Concordia University
Rahaf Aljundi
mabeffaljuunddttoooleeerrrr...mo
Toyola Motor Europe
Eugene Belilovsky
eugen..bcloousk@@ooooonii..c
Mila, Concordia University
Reviewed OpenReview: https: //ppreeveeen pet/pdf? id= GFr
Abstract
Foundational LanguuLa Models (VLMs) excel across diverse tasks, but adapting them to new domains without forgetting prior knowledge remains a critical challenge. Continual Learning (CL) addresses this challenge by enabling models to learn sequentially from new data while mitigating the forgetting of prior information, typically under supervised settings involving label shift. Nenetheless, abrupt distribution shifts can still cause substant forgetting, potentially mullifying the benefits of supervised updates, especially when storing or replaying past data is infeasible. In this work, we propose leveraging unlabeled test- lime data in an unsupervised manner lo reinforce prior task performance without requiring replay or stored examples. Unlike traditional Test-Time Adaptation (TTA), which primarily focuses on domain shift or corruption, our method improves performance earlier tasks by exploiting representative test samples encountered during deployment. We introduce simple Teacher-Student framework with gradient-based sparse parameter updates, and show that it effectively mitigates forgetting in offering a memory-free alternative to episodic replay with strong empirical results.
CL for VLMs,
1 Introduction
Foundation models in computer vision have shown impressive performance on various downstream tasks and domains, which renders them key building block of various solutions, including generative vision language models (Li et al. 2022 Chen et al. 2023 Bommasani et al. 2021). However, naively adapting these pre-trained models to distribution shifts or new tasks often leads to catastrophic foryetting (McCloskey & Cohen 1989 where new learning sessions interfere with what model has previously acquired.
To address raphst forgetting, Continual Learning (CT.) enables models to adapt to evolving data distributions over time. Key approaches include regularization- based methods (Kirkpatrick al., 2017
Maltoni & Lomonaco 2019 Li & Hoiem 2017
Schwarz et al. 2018 Singh et al., 2024). external memory 2017 Rolnick et al. 2019: Shin et al. 2017). and dynamic model
approaches (Lopez- Paz & Ranzato
architecture techniques (Douillard et al. 2022 Pham et al. 2021). However, training from scratch often overlooks the rich representations learned by large pre-trained models. With the advent of foundation models,
there is a growing interest in integrating CT. with their representational power (Han et al. 2021 Radford
et al. 2021 Ridnik et al. 2021 Caron et al. 2021 Oquab et al. 2023
Significant efforts have been made to improve foundational models' adaptability to new data streams (Wang
et al.] 2022d Smith al. 2023 Zhou et al. 2023 Zhang et. al. 2023 Goyal et al.] 2023 Wang et al.

---

Published in Transactions on Machine Learning Research (01/2026)
Supervised Phase
Task
Task
Task
Test Time Learning
Task
Task Task 2
Task 1 Task 2.... Task N
Single
Single
Single
Pass
Pass
Pass
Figure 1: An illustration of our proposed setting of Continual Learning with Interleaved Test Time Learning. After each supervised training session, the model is deployed to adapt in unsupervised deployment phase, where il encounters data from both current and previously seen tasks. During this phase, the model adapts to the current task's classes while striving preserve performance on earlier tasks, thereby mitigating forgetting.
2022c3, primarily through supervised training. However, this focus often leaves models static and prone catastrophic forgetting (Wang et al. 2024a] Prabhu al.] 2023). In contrast, the diverse unsupervised data encountered during inference presents underexplored opportunity to mitigate forgetting without additional supervision. We consider a continual learning setup, especially the challenging scenario of class incremental learning (CIL), where supervised training phases are interleaved with unsupervised deployment, as shown in Figure [ To mitigate catastrophic forgetting, the model leverages unlabeled test-time data in an online manner processing each sample once and discarding it, thereby reducing both privacy risks (Verwimp et al, 2023 and computational overhead.
To the best of our knowledge, in continual learning, particularly through interleaved test-time learning stage. We propose novel Dual- Phase Continual Learning framework: DoSAPP (described in Section 3 that uniquely combines supervised adaptation with unsupervised retention, all without relying on replay buffers or explicit task boundaries. Built upon the CLIP foundation model [Radford et al., (021), which offers strong generalization and transfer (Rasheed et al., 2023 Pei et al. (202)), our approach introduces two dist learning phases: supervised sessions enable efficient task-specific adaptation through sparse parameter updates while unsupervised sessions promote long-term stability by reinforcing previously acquired knowledge. Central to this design is the Teacher- Student [Tarvainen & Valpola 2017b) framework governed by our novel dual-mone momennmmmn mechanism, which decouples the adaptation rates of the teacher and student to balance plasticity and stability. Additionally, cumulative mask consolidation across tasks ensures scalable memory retention without interference. This synergy enables robust continual learning realistic, dynamically evolving environments.
among the carly works that utilize test-time data for alleviating forgetting
Existing works such as Test-Time Adaptation (TTA) Sun et al., 2020b Wang et al. 2020 Zhang al.
2022; Niu al.) 2022 Sun al. 2020a) and Continual Test-Time Adaptation (CTTA) (Wang et al. 2022a Gong al. 2022 Niu et al. 2022 Song et al.) 2023 Tian & Lyu 2024 Wang al. 2024b similarly utilize unsupervised test-time data, but with different focus: adapting models to unknown distribution shiffts in data during deployment. These methods consider previously unscen domain shifts and corruptions in the test-time data itself, aiming to perform well on this unsupervised data, while not. degrading the performance of the model on past data. Further, these methods cannot address forgetting in a class incremental learning (CIL) setup. On the other hand, our sctting, which interleaves supervised and unsupervised sessions, considers leveraging test-time data that does not require domain chift/corruption with respect to previously seen data. Instead, it aims to the unsupervised data to combat forgetting in the CIL setting through the supervised
2

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Comparison of Continual Test-Time Adaptation (CTTA) methods with our approach, DoSAPP. Our work is fundamentally orthogonal to TTA and CTTA methods, which primarily tackle distribution shifts during test time. Moreover, existing CTTA approaches do not typically address forgetting of previous tasks (classes) upon the introduction of new ones. These methods are also incompatible with class incremental learning (CIL). In contrast, DoSAPP leverages test-time data enhance CIL performance and reduce forgetting through unsupervised learning lest samples.
Corrupted Data
Uncorrupted Data
Interleaved Supervision
Reduces Forgetting
Handles
Method
New Classes (CIL)
CoTTA (2022)
Wang et al. 2022a
X
NOTE (2022) Gong et ml.
2022
X
X
X
EATA (2022) (Ni et al.
2022
X
X
V
ToA (2023) Song et al. 21233
X
X
V
RMT (2023) (Dbbler et al.
2023
X
X
PSMT (2024) TTan & Lyu
2024
X
X
X
DSS (2024) Wang et al.
2024b
X
X
X
DoSAPP (ours)
V
sessions. Table ! highlights the key differences between our proposed approach DoSAPP and existing CTTA methods.
Our contributions are follows: 1) We propose new setting for continual learning where test-time data can be leveraged for forgetting without explicit knowledge of task boundaries, especially in the challenging scenario of CIL. 2) We investigate different baselines for this setting. 3) We propose a novel approach that illustrates the utility of test-time data supervised continual learning and the significant reduction in forgetting without relying on any external replay buffer.
2 Related Work
Continual Learning from Pre-trained Models: Continual learning with pre-trained models is gaining popularity due to the availability of powerful foundation models (Radford et al. 2021 Oquab et al. 2023 Brown et al. 2020). Recent approaches (Koh et al.) 2022 Boschini et al. [022] employ a Teacher-Studen framework with knowledge distillation but rely on memory buffers mitigate catastrophic forgetting, which can be memory-intensive (Zhou et al., 2022 Prabhu al., 2023). Additionally, stored logits become outdated, requiring updates with task boundary information, which may not always be available in task-free CL. Further recent works such (Chen et al. 2025) utilize OOD-based techniques to calibrate only the classifier layer of pretrained model using test-time samples, aiming to improve performance in CIL setting. While our mothod also operates on the test stream, it differs fundamentally from (Chen et al. 2025): their approach requires access to task boundaries, which may not be available at inference. In contrast, our method makes no such assumptions and remains applicable even when task boundaries are unknown.
To address those limitations, CLIP (Radford et al., 2021 | presents an attractive alternative as it inherently avoids classification head issues and retains a broad feature space, making it well-suited for continual learning. Inspired by SPU (Zhang et al.) 2024), which preserves generic knowledge by modifying sparse subset of parameters based on gradient scoring, we explore leveraging test data in continual learning. This opens avenues for self-supervised techniques to enhance feature representations while mitigating recency bias.
Test Time Adaptation (TTA): TTA methods primarily focus on handling domain shifts and adapting to data corruption in test data. These approaches aim to improve model performance on the adapted test data, often leveraging techniques such as self-supervised learning (Sun et al. 2020a), batch normalization Nado et al. 2020: Vianna et al. 2024). entropy minimization (Wang et al. 2020 Niu et al. 2023), pseudo labeling (Chen ct al. 2022 Liang ct al., 2021). , and continually adapting to varying test-time distribution
shifts Wang el al. 2022a Gong el al. 2022 Niu el al. 2022 Song et al. 2023 Tian & Lyu 2024 Wang el al., 2024b). In contrast, our approach leverages test-time data for a broader purpose: mitigating forgetting of past tasks. Unlike TTA methods, which leverage the test-time data to correct domain shifts, our method utilizes unsupervised test-time data from previous tasks to retain past knowledge obtained in the supervised training sessions.
3

---

Published in Transactions on Machine Learning Research (01/2026)
Supervised Continual Learning
Unsupervised Test Time Learning
Froemn paratetton of Studen)5)
porameters Tescher)) Selected permeters Teocher())
Sckentel panmeten of Stadem)5)
Test Time Data
Sparse Selection of parameters in S through:
Mr
Mg
dtop gradiento
(03.
9i (1EA)
NI
(o(z))yy(a)) > (Ms))
1-L 1-8
1- 1
0-' i+1 = p0 + 90 ++1
0 = p' p'02 + q
Gradient updates these selected parameters(m).
p = (> 8)m +
p' (A 8)m +
0==00 0m v/ [(MMssx)))
q = (6 - 7)m + 1
.........ggrbeeen gradients flow
d = (6 - A)m + 1-
Figure 2: DoSAPP employs Teacher-Student (MT, Ms) models respectively. the Supervised Continual Learning phase, MMs performs sparse parameter selection using gradient-baset scoring function F, followed by training the selected parameters 9 E 0 After each update, MM parameters l" are updated through weighted exponential smoothing based the affine projection of the boolean mask m, controlled by dual momentum terms 8 and for Mr and Ms, respectively. In the unsupervised test-time learning phase, My adapts using "pseudo-label" derived from Mr-Ms logits comparison. Mr then undergoes weighted smoothing again, with momentum terms and À for Mr and Ms (where < X < 8). This two-phase approach ensures generalization over previous knowledge while maintaining adaptability to new lasks.
3 Methodology
We propose & novel class incremental continual learning (CIL) setting using test-time data. As shown in Figure the model recovers lost knowledge from previously seen tasks after each supervised session, adapting to new classes while minimizing forgetting without relying an external replay buffer. We consider a setting where supervised datasets [Di, D?. drawn from different distributions arrive incrementally at training sessions ranging from 0 to T. Each session with N, instances includes dataset Dj = (x², yf) where an instance x E R belongs class y E Y,, with disjoint label spaces Y, & for t + t'. At session t, only the current dataset Dj is available for training the model M()).
After training on Dj, the model is deployed until Di+1 becomes available. During this phase, it encounters unsupervised test-time data Dj, drawn from all previously seen tasks shown in Figure We leverage this data for online unsupervised adaptation to mitigate forgetting. Evaluation is performed on distinct test datasets Dj to ensure proper generalization assessment.
We further note that although supervised phases may permit multiple passes through the data until convergence, it would be impractical collect unsupervised data in production and then perform adaptation on it. We thus restrict the unsupervised phase to be in the online setting (Sun et al. 2020a Jang et al. 2022 Cai et al. 2021 This is especially important in cases where data privacy is constraint, c.g., assistant robot in private smart home environment.
3.1 DoSAPP: Double Smoothing via Affine Projected Parameters
We propose simple yet effective method for continual test-time lcarning, Double Smoothing via Affine Projected Parameters, aka DoSAPP. Our approach combines two key components: 1) sparse and local updates: to reduce forgetting, maintain generalization, by constraining adaptation to a small set of parameters, and 2) Teacher-Student framework to promote stability in online updates and minimize forgetting. In the continual test time learning, we can identify two distinct phases of learning, as outlined below.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm 1 DoSAPP algorithm for continual and test time learning
Require: Ms(0*), CLIP loss: (( sparsity threshold c, learning rate 17.
0" 50
for t in tasks do
3: @m + lop-K (K=c) params from MLP layers of 0 based on JF
D Sparse Selection, Eq.
4:
for (II, in Dj do
5: 0T - mm (V v.) + + - p + g0
Take one SGD stop Dual momentum for teacher EMA update, Eq
6:
end for
Compute union of masks for all tasks seen far m. Select m. params in Ma for x2 in Dj do IT + max(Mr((a)) dim =) Is + max(M dim dim =
b Start of Unsupervised Phase
10:
11:
12:
13:
if lT Ig then
14:
t- arg max (MT(x:))
15:
else
16:
+ arg max
17:
end if
18:
mm + mm -VL((Ms())))))
Take one SGD step
19:
0 + O' + +
D Dual momentum for teacher EMA update, Eq 6
20: end for
21:
end for
Phase 1: Continual Learning Supervised Training with Sparse Selected Parameters
Our goal is to rapidly acquire new knowledge while preserving generic knowledge during both training and testing. To achieve this, we update only a small subset (sparsity threshold c = 10% of parameters. Wo choose c 10% for all main experiments because it offers a consistent efficiency sisiency-perfrmmance trade-off across tasks and ensures stable behavior during long horizon time adaptation. Tt has been demonstrated that updating only a small subset of relevant parameters pre-trained models like CLIP can significantly reduce forgetting (Zhang et al. 2024 Complementary findings indicate that the MLP blocks in transformers act key-value memory components, with the first MLP layer serving This implics that updating only the first MLP layer may suffice for retaining prior knowledge. Thus, we restrict updates to the first MLP layer of each transformer block in CLIP. From these candidates, we select the top-K (K=c) parameters, where c the sparsity threshold. This results in sparse, localized parameter updates, instead of broad model changes which helps preserve prior knowledge. For the sake of simplicity, model parameters in the paper refer to these candidate parameters.
pattern detector (Geva et al. 2020
the gradient magnitude of the loss with respect to the input data to assess parameter relevance, where higher magnitude indicates greater expected reduction in loss. We optimize the model Ms by first identifying the most relevant parameters from the candidate parameters of first. MLP layer of each transformer block, fm (0m E (S), using:
Following Zhang et al., 2024),
NY 9i (FF)
F F F(O D.)
(1)
N
where giy (Ix) is the gradient of the CLIP loss L(Ms, Øky yk) w.rtt. parameter S computed per data point (1) E D,. Iterating over the datasct once, we compute these scores and select the top-K (K= =) most relevant parameters based on sparsity threshold 0.1. This results in binary mask m, freezing all but the selected parameters.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Teacher Student Framework
To ensure stability later during online updates and reduce forgetting, we utilize a Teacher-Student frame- work (Taraainn & Valpola 2017a; Koh el al. 2022 Boschini el al. 2022 Döbler el al., 2023 Michel el al.,
2024 where the student model denoted by Ms(0 and the teacher model is denoted by Mr(0').
During both supervised and unsupervised phases, the teacher model My parameters 0 updated using the exponentially moving average (EMA) of the student model parameters 0 Typically, in the Teacher-Student framework, all teacher parameters move toward the student parameters with a single smoothing parameter (momentum). However, we demonstrate that single smoothing parameter is insufficient and leads to suboptimal performance, as shown in our ablation studies (Table 6 and in the Appendix (Table 7 In our setting, most of the student model parameters (candidate parameters of the first MLP block) remain frozen, with only a small subset (sparsity threshold, c c 10%) being updated. We propose that the teacher model parameters corresponding to frozen candidate parameters of the student model should adapt at different rate than those associated with active parameters. To address this, we introduce dual smoothing parameters (dual momentum), adjusting teacher parameter updates based on an affine transformation of the binary mask m. It should be noted that the non-candidale parameters (e.g,, attention blocks) a always remain frozen and are not updated at all.
Weighted exponential smoothing with dual momentum
After each gradient update step (i) for Ms, parameters of Mr are updated by EMA of the student model parameters. Typically, EMA is governed by
o 50 + (1 - 5)0 1+
where d is the smoothing parameter. Further, it has been shown in (Trvaainen & Valpola 2017a Oquab
et al., 2023 Koh et al. 2022) that setting ô to a high value (e.g., 0.998) maintains stable teacher model that can be considered as strong reference for past tasks f 1}. But updating the tcacher model with single smoothing parameter in cases where parameters are masked creates dissonance and increases forgetting because all the parameters are updated with equal importance, disregarding those parameters which are selected by the gradient scoring function (where [mij = 1)). To account for masking, we modify Eq. 2 as
0 = po? + 9
(3)
where p and q denote the smoothing parameters for the teacher and student model, respectively, and can be computed as
p (y 6)m + 8, (5 - m + --8
(4)
where y < 8. This implies that the selected parameters of the teacher model (Imy = 1]) move slightly faster towards the student model as compared to the frozen candidate parameters (where [maj 0)). As such, parameters where [m:j = 0] will move at slow rate of 8, and unmasked parameters would be updated with 7. When 7 = 6, the weighted scheme becomes EMA with a single smoothing parameter. A detailed proof is
given in Appendix 8.1
Phase 2: Unsupervised Test Time Learning (TTL)
After supervised training is completed, both Mr and Ms are deployed for Test Time Learning (TTL). We consider teacher (My) and student (Ms) models as two experts on different data distributions, the Ms on the most recent and the Mr on previous sessions distributions.
We take inspiration from Out-of- Distr--Distritution (OOD) literature (Hendrycks & Gimpel 2016), where a prodictor assigns high scores In-Distiibution (ID) samples. Recent work (Hendrycks et al. (2019) shows that using the unnormalized maximum logit as an ID score is more robust than softmax probability, which can overconfidently
6

---

Published in Transactions on Machine Learning Research (01/2026)
classify unknown samples (Yang et al. 2021 p. For CLIP, this logit corresponds to the cosine similarity between the image batch and text features. Following (Hendrycks et al.) 2019), we use the maximum logit value of each expert as an ID score and select for each test sample the expert with the highest ID score, indicating that the sample is likely to be better represented by said expert. We then accept the pscudo-label of the selected expert. Formally, the pseudo label can be calculated as follows:
vr ys
if IT > Is
(5)
otherwise'
where û is the accepted pscudo label and 4r max(Mr(x)) and Is max(Ms(x)) are the maximum logil score for teacher and student. model respectively, and similarly v/r arg max (Mr(x)) and yis arg max(Ms(x)) are the pseudo labels by teacher and student models respectively. During test-time training, the student model Ms is updated by minimizing CLIP contrastive loss given a pseudo label p. In realistic settings, multiple iterations on test data are often not always possible, for example, in a streaming data pipeline. We too mimic this setting, where the entire data is processed only once during the TTL, phase.
Similar the above-mentioedd supervised phase, we also apply sparse local updates to Ms. However, the estimation of masks based on the online data might be noisy and largely reduce the efficiency, gradients of all parameters must be estimated for each mini-batch of test samples. To overcome this, and following the assumption that test data are drawn from the distributions of all previous tasks, we leverage the masks estimated for previous I We accunulate a union of the binary masks (m.) over all the previously seen tasks such that m. = m, U m2 U m.....m... To maintain the same sparsity level (c = 0.1) of performed updates, we further select the same top-K (K=e) most relevant paramcters from these new masked m., parameters based on their previously computed gradient scores.
Finally, Mr is updated using the same dual momentum scheme, but with different smoothing vectors p'. q as:
(6) where p' = (A- 11--6j 6)m + d and q' - (6 A)m + 1- In the TTL phase, the momentum parameter À is kept. such that < A < 8. Similar to phase 1, the selected parameters [m.. 1] of Teacher Model (0")) move slightly faster towards the student model @ as compared to frozen candidate parameters "m, = 0j, ensuring stability in case of frequent and possibly noisy online updates. We discuss the effect of different momentum values The algorithm can be fully understood as given in algorithm block
04, = ='0 + + ++
4 Experiments
4.1 Setup
Architecture: We apply DoSAPP to vision-language classification tasks, given their relatively robust knowledge mcasurement in such tasks. To ensure consistency across experiments, CLIP-VIT-B/16 (Radford et al. 2021) is used as the backbone in DoSAPP as well as in all the baselines. We report the accuracies recorded by the Teacher model. We refer to (Zang et al. 2024 for hyperparameter selection other than
dual momentum, which are given in Appendix 8.3
Datasets: We consider five different vision datasets, three fine-grained (Aireraft (Maji et al. 2013 (Wah et al.] 2011), Stanford Cars Krause et al. 2013). one coarse dataset (CIFAR100 (Krizhessyy 2012 and one -fittr-iiitiitttion dataset (GTSRB (Stallkamp et al.) 2012)). These datasets are chosen primarily based on their initially low zero-shot performance with CLIP pre-trained models. To form the continual learning sequences, we split each dataset into 10 subsets with disjoint classes, composing 10 tasks. For all the datasets, the training data is used in the supervised learning phase. The test data is divided into two disjoint splits, D" and De, where Du is used for unsupervised Lest-Lime learning and De is reserved for evaluation. This separation ensures a fair assessment of the method's generalization performance.
CUB
Evaluation Metrics: After cach supervised session t, and the following test-time adaptation session, we evaluate the model's test, performance on holdout datasets from all T tasks. To do this, we construct the
Parameters not relevant to the current stream of tasks will remain frozen, maintaining models' unrelated generic knowledge.
7

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Acc. (Average Accuracy, f) and F. (Forgetting, 4) of different methods all using CLIP ViT-B/16 backbone with trainable vision and text encoders, without any Replay Buffer in CIL scenario. DoSAPP can achieve positive backward transfer: lorgetting is negalive the Cars data (without ER).
Aircraft
Cars
CIFARIO0
CUB
GTSRB
Method
Acc. (t) F. (1)
Acc. (t) F. (1)
Acc. (1) F. (1) Acc. (f) F. (L)
Acc. (1) F. (1)
CLIP- LerP--erohhot [Radford et al.
2021
24.45
64.63
68.25
55.13
43.38
FLYP
(fie-tuning)
Goval et sl.
2023
18.63
39.93
51.64
25.65
46.26
37.78
45.74
26.62
21.76
55.48
MAS
Aljundi
2018
33.69
27.50
69.43
9.18
63.88
21.16
61.72
12.05
42.04
25.38
ZSCL
Zheng et al.
30.96
15.65
67.79
8.27
80.50
1.05
61.09
7.69
62.92
13.54
SPU
Zhang et al.
2024
30.94
28.36
69.41
16.91
58.80
26.37
62.31
7.2
43.06
19.16
DP
39.14
12.55
74.87
-0.74
79.16
7.73
68.17
2.15
72.33
1.02
ER methods (ER=1000)
FLYP (finettnning) ER
French
1999
41.42
31.38
69.08
16.42
82.86
3.41
64.07
17.72
96.28
-7.48
LWF ER LLi Hoiem
36.08
18.12
72.56
4.04
74.32
8.16
65.11
5.90
53.56
11.86
PRD + ER Asadi et al.
2023
37.11
17.35
74.08
3.75
79.66
3.10
65.92
6.55
63.00
12.44
L2P + ER WWan ot al
2022d
32.20
21.73
67.04
11.22
67.71
18.81
64.04
6.82
75.45
2.68
DualPrompt + ER (Wang
2022c
26.61
17.20
63.30
18.67
61.72
19.87
64.38
12.94
69.65
8.43
SparseCI. ER (Wang al.
20226
31.95
19.77
71.57
5.38
69.35
15.23
62.50
9.66
48.99
24.91
S1.CA + ER (Zhang et al.
2023
29.40
11.45
62.65
4.42
70.03
0.19
53.87
7.75
46.01
0.83
SPU + BR
42.89
15.55
73.69
5.84
79.65
7.36
71.92
4.67
87.64
2.18
DoSAPP + ER-200
47.32
8.10
79.17
3.92
88.41
-1.96
74.39
2.77
83.67
1.92
matrix R E RTXT, where Rag is the test classification accuracy of the model on task łj after observing the last sample from task 4,- Thus, we compute Average Accuracy (Acc. = E Rr,,), Average Forgetting (F. Σ T-1 fyr, fúr (I... (a. ar.))) (Lopez-Paz & Ranzato 2017 Wang et al. 2024a
Zhang et al. 2024). These metrics allow us to assess how well a continual learner solves. a classilication
problem while overcoming forgetting.
Baselines: We comprehensively compare our method against various baselines. Firstly, we evaluate our approach against. the best, fine-tuning method of CLIP, FLYP [Goyal al. 2023). We further integrate FLYP classical continual learning components evaluate their performance on the CLIP backbone, including ER [Rolnick al. 2019), weight regularization method, MAS (Aljundi et al.] 22018), and functional regularization methods LwF Li & Hoiem
2017) and PRD (Asadi et al., 2023). We combine these functional regularization methods with replay buffer (ER). We further consider the latest pre-trained model-based continual learning
techniques L2P (Wang et al. 2022d). DualPrompt (Wang et al. 2022c), and SLCA (Zhang et al. 2023
Finally, we compare to recent methods that (arget knowledge retention of foundation models ZSCL
ZZengg
et al. 2023 SparseCL (Wang et al. 2022b), and SPU (Zhang et al. 2024
5
Results
5.1
Comparison with CL Methods
We compare our method (DoSAPP) with recent and diverse approaches in the challenging scenario of class incremental learning (CII.), as shown in Table 2 DoSAPP achieves the largest improvement over simple fine-tuning (FLYP) Goyal al. 2002). Compared to MAS (Aljundi al., 2018 DoSAPP increases performance by at least 16% Cifar100, highlighting that not all parameters require updating. DoSAPP also outperforms prompt-based methods L2P (Wang et al. 2022d) and DualPrompt (Wang et al. 2022c by 12% and 17% respectively on Cifar100. Even against dynamic network approaches like SLCA (Zhang et al. 2023 SparceCL [Wang el al. 2022b). ZSCL (Zheng et al. 2023), and SPU (Zhang el al. 2024), DoSAPP achieves state-of the-art state-of-the-rrt or comparable results across all datasets. This demonstrates that test-time data can enhance transfcrability and preserve learned knowledge. Notably, DoSAPP achieves strong performance without a replay buffer, and when provided a small buffer of just samples per task, it significantly outperforms methods requiring large buffers.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Acc. (Average Accuracy, f) and F. (Forgetting, 4) in evaluating DoSAPP on imbalanced test data (referred as imb, in the table below), demonstrating its effectiveness in mitigating forgetting while maintaining high accuracy.
Aircraft
Cars
CIFAR100
CUB
GTSRB
Method
Acc. (t) F. (L)
Acc. (t) F. (L)
Acc. (1)
F. (4)
Acc. (1) F. (4)
Acc. (1) F. (L)
FLYP (finctune)
16.20
42.85
47.92
29.40
43.30
40.12
41.63
29.61
18.52
58.70
ZSCL
26.89
23.80
64.58
10.85
62.10
12.67
57.77
10.54
58.45
17.91
SLCAA
27.44
19.23
56.83
14.17
59.92
18.01
48.66
10.82
39.67
21.56
SPU
30.94
28.36
69.41
16.91
58.80
26.37
62.31
7.2
43.06
19.16
DoSAPP
35.99
15.26
72.68
6.38
75.70
9.81
64.84
3.73
68.17
5.63
Table 4: Acc. (Average Accuracy, 1) and F. (Forgetting, J) comparing DoSAPP with CIL methods like SPU (Zhang et al.
2024), SparsCL (Wang et al. 2022b) integrated with one of the most recent CTTA method: RMT Dōbler et al. 2023 can be observed that fusing the lypical CTTA method in the CIL pipeline exacerbates the catastrophic
forgetting, DoSAPP, the other hand, outperforus all of them by a significant margin all the datasets.
Aircraft
Cars
CIFAR100
CUB
GTSRB
Method
Acc. (t) F. (L)
Acc. (t)
(4)
Ace. (t) F. (1)
Acc. (t)
(1)
Acc. (t) F. (L)
SPU SPU + D
30.94
28.36
69.41
16.91
58.80
26.37
62.31
7.2
43.06
19.16
27.72
24.86
68.91
7.34
74.09
10.43
61.21
4.01
60.17
6.94
SparsCL + RMT
27.11
16.29
69.81
17.22
70.82
12.25
60.03
10.58
51.98
11.40
SPU + RMT
29.33
15.10
62.32
21.95
63.06
23.28
63.87
6.34
54.13
17.56
DoSAPP
39.14
12.55
74.87
-0.74
79.16
7.73
68.17
2.15
72.33
1.02
Imbalanced test data: We evaluate DoSAPP under a more realistic and challenging setting where D. (unsupervised data) is clasa-imbalanced as shown in Table 3 Each task is sampled from a symmetric Dirichlet distribution with concentration parameter equal to the task length, leading to severe class imbalance, and at times, the complete absence of certain classes. This setup mirrors real-world scenarios where test-time data is often skewed. For fair evaluation, we retain balanced evaluation set. Remarkably, despite the imbalance, DoSAPP still yields significant gains over purely supervised CL methods. While its performance is lower than in the balancod setting, this is expected: the model naturally adapts more to frequently observed classes during TTL. In practice, such behavior is desirable, as performance on the test distribution, rather than on rarely observed classes, often the priority at deployment.
5.2 Comparison with TTA+CL Methods
We highlight the key innovation of our approach: leveraging unsupertised test-time data - readily available in production, to enhonce continual learning. In contrast, most existing CL methods are not designed to incorporate such data, limiting their adaptability in this setting. Here, we investigate whether approaches that leverage unsupervised data in online setting are capable of fully benefiting from test-Lime data in our setting. To examine this, we combine the best-performing CL method, SPU, with simple self-training mechanism (SPU + D where pseudo-labels are assigned based on the model's max logit output. We also integrate RMT Döbler et al. 2023), with SPU (Zhang et al.) 2024 and SparsCL (Wang et al, 2022b). Although more recent CTTA methods like DSS (Wang et al. (024b) and PSMT (Tian Lyu (0024) have been proposed, we select RMT due to its consistently lower mean classification error across all the benchmark corruption datasets, making it stronger baseline. We find that our proposed method, DoSAPP, consistently outperforms all variants as shown in Table 4 This highlights critical limitation: Continual Test Time Adaptation (CTTA) methods, when combined with ongoing supervised learning, suffer from substantial forgetting due to their inabilily lo adapl across a long sequence of shifling lasks. DoSAPP addresses this with a principled use of dual momentum over masked parameters. Furthermore, RMT performs worse than basic aseudo-labeling in nearly all cases, confirming that TTA methods are ill-suited for continual learning with expanding task distributions. We also demonstrate in Appendix [8.6 and 8.8 that DoSAPP remains robust to noise in rest-time data and scales effectively with varying amounts of unlabeled input.
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Average Accuracy (Avg Acc.), First Task Accuracy (FTA), Current Task Accuracy (CTA), and Forgetting
(F.) measured for long sequence tasks made by the concatenation of the Aircraft (Maji et al. et 2013 datasets.
2013 and Cars Krause
Method (CIIP)
Avg Acc. (t)
FTA (1)
CTA (1)
F. (4)
Finetuning (uo TTL)
35.24
5.90
75.44
16.87
SPU
39.62
24.31
74.94
7.32
DoSAPP
45.01
30.63
71.13
2.34
Table 6: Acc. (Average Accuracy, 1) and F. (Porgetling, 4) when different components of DoSAPP incrementally
added to the Teacher-Student framework referred to as A1. A2 denoles the sparse parameter selection added to A1.
EMA (6) represents single momentumm updales, while EMA (6, 7) refers dual momentum updales. m. denotes the union of mask technique described in section 3.1
Aireraft Acc. (1) F. (L)
Cars
CIFARI00
CUB
GTSRB
ID
Description
Acc. (1) F. ()) Acc. (1) F. (C) Acc. (1) F. (L) Acc. (t) F. (U)
A1
Teacher-Student (EMA (6))
30.12
13.50
67.72
3.66
77.82
5.17
62.67
4.11
53.57
5.38
A2
A1 + sparse params
34.16
18.61
69.12
3.41
71.93
8.21
66.32
3.98
55.32
5.81
A3
+ m.
31.79
10.42
70.99
72.66
8.86
66.98
61.54
4.01
A4
A2 + EMA (6, 7)
35.49
11.53
72.14
3.58
75.93
8.02
67.28
3.41
61.15
3:20
DoSAPP* A4 +
39.14
12.55
74.87
-0.74
79.16
7.73
68.17
2.15
72.33
1.02
5.3
Class Incremental Long Sequence scenario with domain shift
We consider a long sequence of tasks trained in a iasssinnrrmmetall manner. For these experiments, we combined tasks from Aircraft (Maji al., 2013) and 10 from Cars (Krause et al.] 2013), introducing domain shift after the first 10 tasks. Table 5 shows that our method, DoSAPP, outperforms SPU and Finetuning. Unlike SPU and Finetuning, which exhibit recency bias, DoSAPP achieves better overall and first-task accuracy, with only 3.8% drop in current task accuracy (CTA). This demonstrates its ability to retain carlier knowledge while adapting effectively to new tasks. We also report per task forgetting for Finetuning, SPU, and DoSAPP highlighting the effectiveness of our proposed method in subsection 8.7
6 Ablation Study
In this section, we quantitatively analyze the effect of different components of our proposed method, DoSAPP. We evaluate the effects of each component incrementally, as seen in Table g We begin with baseline Teacher-Student setup using single momentum (A1), and compare it with a variant that applies localized sparse updates to the first MLP layer of each transformer block (A2). This yields performance gains in 4 out of 5 datasets. Next, we introduce the union of supervised task masks (m,) during the TTL phase (A3), which leads to marginal improvements across most datasets, except for Aircraft where applying a single momentum to both masked and unmasked parameters causes mismatch and accuracy drop. To isolate the effect of dual momentum, we incorporate it into A2 without m., (A4), which shows consistent improvements over both A2 and A3. Finally, combining both dual momentum and mask union yields our complete method, DoSAPP, which achieves the best overall performance. These results underline the individual value of each component and their synorgistic effect when integrated. Additional ablations on the sparsity threshold c, importance of pseudo label selection from both
and the role of TTL phases are provided
Appendices 8.9 8.10 and 8.11
7 Discussion and Conclusion
This work explores how test-time data can leveraged to improve the retention of previous tasks, drawing inspiration from human learning build more adaptive models. We show that when usod effectively, test-time data is a valuable resource. Our method, DoSAPP, enhances CLIP's zero-shot performance in the challenging inss-inncementll setting by combining sparse parameter updates with dual-momentum EMA across supervised
10

---

Published in Transactions on Machine Learning Research (01/2026)
and unsupervised phases. Although our experiments focus exclusively CIL classification, future work could investigate whether DoSAPP's mechanisms generalize to generative other transformer-based tasks.
References
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In European Conference on Computer Vision (ECCV), 2018.
Nader Asadi, MohammadReza Davari, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Prototype- sample relation distillation: towards replay-free continual learning. In International Conference on Machine Learning, pp. 1093 1106. PMLR, 2023.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emna Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2188.00 07205888 2021.
Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Simone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In European Conference Computer Vision, pp. 692-709. Springer, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Noelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference Neural Information Processing Systems, N1PS'20, 2020.
Zhipong Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribution shifts: An empirical study with visual data. In Proceedings of the IEEE/CVP international conference on computer vision, pp. 8281-8290, 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 ieee. In CVF International Conference on Computer Vision (ICCV), volume 3, 2021.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEF/CVP Conference on Compuler Vision and Paltern Recognilion, pp. 295-305, 2022.
Haoran Chen, Micah Goldblum, Zuxuan Wu, and Yu-Gang Jiang, Adaptive retention & correction: Test-time training for continual learning. ICLR, 2025.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision- language multi-task learning. arXic preprint orx::23310.094 0977,, 2023.
Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7704-7714, 2023.
Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for contimual learning with dynamic token expansion. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9285 9295, 2022.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3:128-123 1999.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012. 14913, 2020.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test-time adaptation against temporal correlation. Advances in Newral Information Processing Systems, 35:27253-27266 2022.
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuming of zero-shot vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognilion, pp.
2023.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, 2021.
Liang Zhang, et al. Pre-trained models: Past, present and future. Al Open,
Dan Hendrycks and Kevin Gimpel. A baseline for intecting misclassified and out-of-distribation examples in neural networks. arXi preprint 1610::1610.022 02100....6 2016.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Stcinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132, 2019.
Minguk Jang, Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest. neighbor information. arXiv preprint 1xXiv:2207.10192, 2022.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, et al. Overcoming catastrophic
Kieran Milan, John Quan, Tiago Ramalho, Agnieszka forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 3526, 2017.
Hyunsco Koh, Minhyuk Seo, Jihwan Bang, Hwanjun Song, Deokki Hong, Seulki Park, Jung Woo Ha, and Jonghyun Choi. Online boundary-free continual learning by scheduled data prior. In The Eleventh International Conference on Learning Represenlations, 2022.
Jonathan Krause, Michael Stark, Jin Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEFE international conference on computer vision workshops, pp- 554-561, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888 12900. PMLR, 2022.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE tramsactions on pattern analysis and machine intelligence, 40(12):2935-290) 2017.
Jian Liang, Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pottern Analysis and Machine Intelligence, 44(11):6002-801 2021.
David Lopez-Paz. and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthow Blaschko, and Andrea Vedaldi. Fino-grained visual classification of aircraft, 2013. URL https://arxiv: wrgeliiwworraaaae wttg/abs/1306.55 5151
Davide Maltoni and Vincenzo Lomonaco. Continuous learning in single-incremental-tass scenarios. Newral Networks, 116:56-73, 2019.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and molivation, volume 24, pp. 109-165. Flsevier, 1989.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Nicolas Michel, Maorong Wang, Ling Xiao, and Toshihiko Yamasaki. Rethinking momentum knowledge distillation in online continual learning. In Proceedings of the 4 41st International Conference on Machine Learning, 1CML'24, 2024.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' A'Amour Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prodiction-time batch normalization for robustness under covariate shift. ar preprint arXiv:2006.10900 2020.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shifian Zheng, Peilin Zhao, and Minglui Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pp. 16888 16905. PMLR, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint wxXi:2332.12240, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Xiv preprint WE5:23344071900 2023.
Renjing Pei, Jianzhuang Liu, Weimian Li, Bin Shao, Songcen Xu, Peng Dai, Juwei Lu, and Youliang Yan. Clipping: Distilling clip-based models with student base for video-language retrieval. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition, pp. 18983 18992, 2023.
Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. In Advances in Neural Information Processing Systems, volume 34, pp. 16131-16144, 2021.
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3698-3707, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp.
PMLR, 2021.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6545-6554, 2023.
Tal Ridnik, Emanuel Ben- Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.108 10972, 2021.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska-Barwind Barwinska, Yee Whye Teh, Razvan Pascamu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pp. 4528-4531 PMLR, 2018
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
Vaibhav Singh, Anna Choromanska, Shnang Li, and Yilun Du Wake-sleop energy based models for continual learning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pullern Recognition, pp. 2024.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bouile Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-basod prompting for rehearsal-fie free contimual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 11909-11911 2023.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficiente continual test-time adaptation via self-distileed regularization. Tn Proceedings of the TEEE/CVR Conference on Computer Vision and Pattern Recognition, pp. 11920 11929, 2023.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian lgel. Man vs. computer: Benchmarking machine learning algorithms for traflic sign recognition. Newral networks, 32:323 -32, 2012.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for gencralization under distribution shifts. In International conference on machine leurning, pp. 9229 9248. PMLR, 2020a.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229 9248. PMLR, 2020b.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017a.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in newrol information processing systems, 30, 2017b.
Jiaxu Tian and Fan Lyu. Parameter-selective continual test-time adaptation. In Proceedings of the Asian Conference on Compuler Vision, pp. 1384 1400, 2024.
Eli Verwimp, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L. Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, Christoph H Lampert, et al. Continual learning: Applications and the road forward. Xiv preprint arXiv:2311.11198, 2023.
Pedro Vianna, Muawiz Chaudhary, Paria Mehrbod, An Tang, Guy Cloutier, Guy Wolf, Michael Eickenberg, and Eugene Belilovsky. Channel-selective normalization for label-shift robust test-time adaptation. arXiv preprint Xrx::24402.0 04958, 2024.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-2011 dataset. Technical report, California Institute of Technology, 2011.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy mininization. arXiv preprint ar arXio:2006.10726, 2020.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEFE Transactions on Paltern Analysis and Machine Intelligenc, 2024a.
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201-7211, 2022a.
Yanshuo Wang, Jie Hong, Ali Cheraghian, Shafin Rahman, David
Lars Petersson, and Mehrtash Harandi. Continual test-time domain adaptation via dynamic sample selection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1701-1710, 2024b.
Zifeng Wang, Zheng Zhan, Yifan Gong, Geng Yuan, Wei Niu, Tong Jian, Bin Ren, Stratis Ioannidis, Yanzhi Wang, and Jennifer Dy. Sparcl: Sparse continual learning on the edge. Advances in Neurol Information Processing Systems, 35:20366 2--0666000300,,, 2022b.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In Earopean Conference Computer Vision, pp. 631-648 Springer, 2022c.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 139 149, 2022d.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint ar11::210011394,, 2021.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning on pre-trained model. In 2023 IEEE/CVF International Conference Compuler Vision (ICCV), 2023.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmenta- tion. Advances in neural information processing systems, 35:38629 38642, 2022.
Wenxuan Zhang, Paul Janson, Rahal Aljundi, and Mohamed Elhoseiny. Overcoming generic knowledge loss with selective parameter update. In Proceedings of the IEEE/CVF Conference on Computer Vision and Poltern Recognition, pp. 24046 24056, 2024.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yuc, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVV International Conference on Computer Vision, pp. 19125-19136, 2023.
Da- Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient class-incremental learning. arXiv preprint 2205:: 13218, 2022.
Da- Wei Zhou, Han-Jia Ye, De- De-Chuan Zhan, and Ziwei Liu. Revisiting inaseincrementll learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338 2023.
15

---

Published in Transactions on Machine Learning Research (01/2026)
8
Appendix / supplemental material
8.1
Derivation for dual momentum
In section
3
the teacher model parameters of undergo exponential moving average as
0 =p + q0 ++1
(7)
where and q denote the smoothing parameters for the teacher and student model, respectively, and can be computed as
p = 01 m + 81, a2m + B2
(8)
where a, and 3, for E {1,2} are the coefficients for the affine transformation the boolean mask vector m. To account for masked parameters, two momentum values y are introduced for teacher and student models respectively, such that for the teacher model, affine coefficients 01, 31 computed by solving the equations:
a +] 81 =Y,
a1 m 0] B1 =
and 012, 32 computed by solving the equations
02 m = + 82 =
02 m.y
0] + B2 8
(10)
This gives
8,
81 =
(11)
022 = 8 7, 32 = 1 - 6
This gives
p (? 8) +
(12)
(6 - 7 m +
8.2
Effect of Momentum A) Average Accuracy
Table 7 shows the sensitivity of our method the choice of momentum values A, 8 in Table 7
A high has
been chosen to keep the Teacher model stable, as shown in (Tavaiine & Valpola 2017a: Oquab et al. 2023
Koh et al. 2022 It can be seen that when X (single momentum EMA), the performance significantly drops. DoSAPP is less sensitive to the choice y, but it highly depends on A. We can also see that as < y, the performance again drops.
8.3 Hyperparameters
Table g shows different hyperparameters that have been used for all the experiments using CLIP backbones. The lyperparameters were selected based on the performance the first task of the Cars (Krause al., 2013 dataset. All the results have been gathered over experiments running on Nvidia V100 GPU, averaged over random seeds.
8.4 Limitation
DoSAPP is robust algorithm which can be potentially applied to any CL technique for unsupervised adaptation of Test Time Data. However, since it utilises the test data, its primary bottleneck becomes the quality of test data especially if its highly skewed. Another limitation is the increase in the computational
16

---

Published in Transactions on Machine Learning Research (01/2026)
Table 7: Effect of Momentum (y, A) Average Accuracy (Ace in % ) Average Forgetting (F.), and First Task Accuracy (FTA.) *0.9999, 0.8, 0.9 have been used in the main results.
Aircraft F. (4)
Momentum (%, X)
Acc. (t)
FTA. (1)
0.9999, 0.9999
23.99
18.36
12.15
0.5, 0.9
38.41
13.27
37.64
0.7, 0.9
37.22
13.05
37.72
0.8, 0.9*
39.14
12.55
38.13
0.8, 0.6
37.06
15.12
29.63
0.8, 0.5
32.95
13.40
26.33
Table 8: Hyper Parameters for all the experiments using the CLIP VIT-B/16 model.
Hparams
CLIP model
Batch Size
64
Optimizer
AdamW
Learning Rate
7.5e
CL Epochs
10
Buffer
0
TTL batch size
64
Momentum-EMA (6. 7,
0.9999, 0.8, 0.9
sparsity (c)
0.1
budget due to two deployed models: Teacher-Student framework. We address this by leveraging efficient sparse parameter selection method. While DoSAPP remains effective under class imbalance Table extreme skew or partial coverage in the test-time stream may still bias updates toward overrepresented classes. To improve robustness in such settings, several practical mitigations can be incorporated without modifying the core framework. First, iafidence-bbssed update gating can prevent updates when both teacher and student produce low-confidence predictions, reducing drift caused by unreliable pseudo-labels. Second, a balanced mask reactivation strategy can periodically reintroduce a small portion of past-task mask entries based on earlier gradient scores, stabilizing parameters corresponding to tasks that are temporarily absent. Third, slower EMA updates (using a À closer to can reduce teacher drift when the observed stream is highly imbalanced. Finally, adaptive learning-rate scaling based on prediction entropy can temper the inffuence of cajority-class samples during TTL. These mechanisms are lightweight, compatible with DoSAPP's sparse-update design, and improve applicability in real-world settings where test-Lime data is often skewed.
8.5 Broader Impact
This work proposes a general algorithm DoSAPP which can be implemented in any of the existing continual learning settings. Further it can be also integrated with other CL algorithms. Therefore, the potential negative societal impacts of our method are similar to those of the Continual Learning algorithms. Generally, DoSAPP would greatly improve performance especially in Test Time Learning. However, with most. CL algorithms, DoSAPP cannot guarantee to take safe and effective actions in all kinds of scenarios. We advocate that the users of DoSAPP should be aware of the potential consequences and utilize DoSAPP safely, especially in online environments such as self-driving, robotics, and healthcare.
8.6 Dependence on quality of test data used for unsupervised learning
We want to highlight that the trained model is expected generalize the distribution of the test data. We also assume that any quality degradation will be consistent across time steps. For instance, if the data is corrupted with noise, our method would generalize and adapt the model to this corruption well. To
17

---

Published in Transactions on Machine Learning Research (01/2020)
Airoraft
Cars
Alferaff
Care
Alrorstt
Cars
Arunn
000
Tal
TER
(a) DoSAPP
SPU
Finetuning
Figure 3: Per-task forgetting matrices for the long-se CIL setting (Aircraft - Cars). Each healmap shows
Fi, RA, a ie., how much performance on task is lost after learning later tasks. The vertical and horizontal black lines denote the domain shift from Aircraft (left/top) to Cars (right/ hight/bottom). DoSAPP achieves the lowest forgetting across the entire task sequence, with several tasks even exhibiting negative forgetting, indicating improved retention training progresses. In contrast, SPU shows moderate degradation, while standard finetuning undergoes severe catastrophic forgetring in both domains.
illustrate this, we conducted small experiment by adding random Gaussian noise (mean 0, std 0.1) to different combinations of the test and evaluation suite (referred to as GN in Table 9 The results are shown below, with average accuracy (Acc.) followed by lorgetting (F.). We observe that when corruption is present in the test-ting time-timeee data, the model is still able to leverage these data and improve on clean evaluation data compared to the test-time baseline by a significant margin of 17% (SPU alone). Interestingly, the model adapted test-time data with Gaussian noise performs better on evaluation data with Gaussian noise than the case when the test-time data clean. This is evidence of our method's ability to adapt and generalize the present test-time conditions.
Table 9: Performance of DoSAPP with noise added to D and D for CIFAR100 Data
Test Time Data (D") Evaluation Data (D*)
Acc. (t)
F. (L)
Clean
Clean
79.16
7.73
GN
Clean
75.67
9.93
Clean
GN
69.50
12.86
GN
GN
73.42
6.86
8.7 Forgetting Long Sequence scenario with domain shift
To analyze forgetting across the long sequence setting (Aircraft-+Cars), we compute the per-task forgetting matrix gTxT from the full accuracy trajectory R, where RG, denotes the test accuracy on task i after learning task Following standard continual learning analysis, we define the forgetting of task i after learning task > i as
Fif Ry. Rr
(13)
where R2 is the accuracy of task the moment first learnod. Thus, Fe, quantifics the absolute loss performance on task caused by subsequent tasks. We visualize F using heatmaps in Fig. with block boundary at the Aircraft Cars transition.
Across all methods, the domain shift from Aircraft (tasks 1-10) to Cars (tasks 11-20) induces substantial forgetting, but the magnit varies sharply by algorithm. DoSAPP exhibits markedly lower forgetting throughout both domains, with only mild degradation the upper-right block corresponding to cross-domain interference. SPU shows moderate forgetting, particularly within the Aircraft block, but remains substantially more stable than naive fine-tuning. In contrast, Finetuning catastrophically forgets earlier fasks, especially after the domain shift, reflected by large negative values across most of the forgetting
18

---

Published in Transactions on Machine Learning Research (01/2026)
matrix. These visualizations reinforce the quantitative results in Table 5: DoSAPP most effectively mitigates long-sequence and cross-domain forgetting.
8.8 Ablation study about the size of test-time data D
In method, we divided the evaluation data into two halves. One half is for unsupervised learning (D"), and the other half is for evaluation (D*). In the table below, we feed the fraction of D for test time learning. 0.25 means that 25% of the original D is fed to the model for unsupervised learning. We notice that when the fraction is below 0.75, there is an appreciable difference between the performance of our proposed model. However, at 0.75, the performance is quite close to that of the whole D
Table 10: Dependence of the performance of DoSAPP with different proportions of the testing data D on the CTFAR100 dataset.
Fraction of Du
Acc. (t)
F. (1)
0.25
73.97
14.23
0.5
76.83
9.44
0.75
79.02
8.16
79.16
7.73
8.9 Ablation on sparsity threshold (c)
We conduct an ablation study on the sparsity threshold (c) on different datasets, as shown in Table 11 We observe that increasing the sparsity threshold (c) beyond 0.1 leads to modest gains, indicating parameter redundancy as the number of updated parameters increases. Conversely, drastically reducing to 0.01 results in significant drop in accuracy. This is because the small number of updated parameters do not have enough capacity to accommodate the learned tasks.
Table 11: Effect of Sparsity Threshold (c) on Average Accuracy Across Dalasets
Sparsity (c)
Aircraft
Cars
CIFAR100
CUB
GTSRB
0.01
35.52
70.12
72.31
63.04
68.25
0.05
37.98
72.84
77.42
65.91
70.43
0.10
39.14
74.87
79.16
68.17
72.33
0.30
40.48
74.12
80.91
67.54
71.88
0.50
41.49
73.65
78.44
66.82
70.02
0.90
41.34
72.97
77.85
65.93
70.31
8.10 Pseudo label only from teacher (Mr) expert
It must be emphasized that pseudo-labeling is not solely dependent the student model. We further perform experiment where pseudo labels are given only by the teacher, and it can be observed that the performance across all the datasets deteriorates as shown in the Table 12] This is because the teacher model is purposefully updated with very high EMA momentum and adapts slowly to the most recent task. Its logits remain biased toward earlier tasks, leading to outdated pseudo-labels during TTL for most recent tasks at hand. The student, in contrast, plastic and calibrated for the current lask, making logits critical for accurate pseudo-label selection. Our max-logit expert selection therefore balances teacher stability with student plasticity.
8.11 Ablation on TTL phase:
Below, we provide the ablation showing average accuracy on previously seen tasks (excluding the current task to avoid bias from supervised training) before and after the unsupervised test-time learning (TTL.) phase,
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 12: Acc. (Average Accuracy, 1) and F. (Forgetting, 4) comparing DoSAPP with configuration when only Teacher Expert is used in providing pseudo-label at TTL phase.
Aireraft
Cars
CIFAR100
CUB
GTSRB
Description
Acc. (t) F. (4)
Acc. (t) F. (4) Acc. (1) F. (L) Acc. (t) F. (4) Acc. (t) F. (4)
Pseudo-Label..
36.92
13.84
71.04
0.15
75.48
8.22
65.21
2.97
68.09
1.56
DoSAPP
39.14
12.55
74.87
-0.74
79.16
7.73
68.17
2.15
72.33
1.02
using the Aireraft dataset. The results show consistent improvement on previous lasks, demonstrating the usefulness of unsupervised TTL as observed in Table 13
Table 13: Average Test Accuracy (referred Acc. below) Before and After TTL
Task
Acc.
(Before TTL) Acc. (After TTL)
78.03
78.03
66.55
68.21
57.32
60.13
50.67
54.09
45.13
46.22
6
43.39
47.58
7
39.21
45.47
8
41.84
45.08
9
41.97
44.14
10
37.38
39.80
We further provide an ablation Table 14 showing average accuracy after training complete on all the tasks using DoSAPP, with and without TTL phase. We can see that without the TTL phase, the model significantly deteriorates in performance, highlighting the usefulness of test time data.
Table 14: DoSAPP Ablation Study: With vs Without TTL
Dataset
DoSAPP without TTL DoSAPP with TTL
Aircraft CIFAR100
35.42
39.14
67.30
79.16
8.12 Comparing Evaluation on full test set
Here we compared one of the SOTA CL baselines (SPU [Zhang et al., 2024] evaluated on the complete test set, instead of the subset De- From table (aable 15 we can observe that DoSAPP when evaluated on the De: outperforms model trained via SPU method and evaluated on full test set (Da U Dz).
Table 15: Ace. (Average Accuracy, t) and F. (Forgetting, 4) where the latest CL method SPU is evaluated the complete test set, i.e, without splitting the test set into Du and De- DoSAPP is still evaluated on D. since it utilizes Da unsupervised training, and including this subset for evaluation will be unfair. These results show that although the baseline performance of SPU has increased slightly compared to the performance mentioned in seccion 5)
DoSAPP outperforms both with and without buffer.
Aircraft
Cars
CIFAR100
CUB
GTSRB
Method
Acc. (t) F. (L)
Acc. (t) F. (4)
Acc. (t)
F. (1)
Acc. (t) F. (1)
Acc. (t) F. (L)
SPU
32.51
24.74
70.59
14.26
64.98
19.74
62.43
6.89
48.97
15.51
DoSAPP
39.14
12.55
74.87
-0.74
79.16
7.73
68.17
2.15
72.33
1.02
SPU + ER0
44.43
14.42
77.51
3.26
83.99
-0.39
71.51
4.84
94.25
-7.87
DoSAPP
ER=200
47.32
8.10
79.17
3.92
88.41
-1.96
74.39
2.77
83.67
1.92
20

---

Published in Transactions on Machine Learning Research (01/2026)
8.13 Computation time:
We include comparison of wall-clock training time (in seconds) between the full model (all parameters trainable) and the sparse variant (c = 0.1) across 10 tasks and one of the latest SOTA Continual Test Time Adaptation method (RMT) that we have utilized Table 5.2 to compare with our proposed algorithm. As shown in [abll 16, the sparse variant yields consistent but modest reductions in training time per task. While these improvements are incremental rather than substantial, they demonstrate that restricting updates to a small set of parameters does not introduce additional computational overhead.
Table 16: Training Time Comparison Across Methods
Task
All Params
DoSAPP (c 0.1)
SPU SPU+RMT
46.61
45.69
47.13
70.83
68.96
71.27
77.75
75.90
78.14
89.55
86.92
90.07
101.60
98.81
102.18
112.19
108.93
112.86
118.28
114.71
119.03
130.49
127.40
131.21
142.59
139.38
143.22
10
154.01
149.65
154.77
21


Reviews:
[{"review_id": "MDYiaio5xZ", "reviewer": "Reviewer_bPWk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a dual-phase continual learning framework (DoSAPP) that combines supervised adaptation with unsupervised retention. The paper clearly describes the proposed simple yet effective framework and empirically validates its general performance across various scenarios. I only have a few concerns as listed in the weaknesses below.\n\nStrengths:\n\n1. The key contribution is the idea of leveraging test-time data to reduce catastrophic forgetting without requiring replay data. The authors restrict the test phase to the online setting to ensure practicality and data privacy.\n\n2. The proposed framework is composed of sparse parameter updates, a teacher-student framework, and dual momentum. The authors verify the effectiveness of each component using prior works and ablation study results.\n\n3. The experimental results are extensive, considering both continual learning and test time adaptation baselines on five benchmark datasets. The proposed framework outperforms the baselines, and the authors provide additional experimental results demonstrating its robustness under imbalanced data, domain shifted data, and different hyperparameters.\n\nWeaknesses:\n\n1. The paper claims to be the first work to leverage test-time data in continual learning to reduce forgetting, but the reviewer believes it needs comparison with related work [1].\n\n2. The proposed framework appears to be a combination or extension of existing methods. Although the framework is simple yet effective and outperforms all the baselines, the reviewer believes it lacks novelty.\n\n3. The proposed framework is dependent on hyperparameters (sparsity and momentum) and test-time data, and the reviewer wonders whether these hyperparameters can be tuned during the test phase.  \n\n4. The paper does not include large-scale benchmark datasets such as Tiny ImageNet or ImageNet in the experiments, which are widely used benchmark datasets in the continual learning literature.\n\n5. Since the benefit of the framework comes from leveraging test-time data, the performance may depend on the quality of the test-time data. Since test-time data is generally more challenging, the reviewer believes the proposed framework may fail when adversarial or noisy data appears at test time.\n\n[1] Chen et al., \"Adaptive Retention & Correction: Test-Time Training for Continual Learning\", ICLR 2025.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper supports its claims using prior works, theoretical proofs, and experimental results.", "audience_yn": "Yes", "audience_explanation": "Continual learning is an important and well-known research area for the audience. The exploration of using test-time data to reduce catastrophic forgetting is a new finding in this field.", "requested_changes": "Please address the weaknesses.", "broader_impact_concerns": "The paper describes its broader impact in the appendix, and there is no concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gB3YXvR9IE", "reviewer": "Reviewer_QmV7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary\nThis paper focuses on solving the \"catastrophic forgetting\" problem of foundational Vision-Language Models (VLMs) like CLIP in class-incremental learning (CIL) scenarios. It proposes a dual-phase continual learning framework called DoSAPP (Double Smoothing via Affine Projected Parameters).\n\n\n## Contributions  \n1. Proposing a Novel Continual Learning Setting. The paper introduces a new learning paradigm where **supervised training phases and unsupervised deployment phases are interleaved**. This avoids privacy risks from data storage and reduces computational overhead.\n\n2. Designing the DoSAPP Framework for Forgetting Mitigation. Built on a Teacher-Student architecture, DoSAPP balances \"plasticity\" (adapting to new tasks) and \"stability\" (retaining old knowledge) through three core mechanisms:  **Sparse parameter updates**, **Dual-momentum EMA**, and **Mask union in unsupervised phases**.  \n\n2. Validating Effectiveness and Generalization. The paper conducts comprehensive experiments across 5 datasets and compares DoSAPP with over 10 baselines. DoSAPP outperforms all baselines without replay buffers and remains robust in challenging scenarios, such as class-imbalanced test data, long-sequence tasks with domain shifts, and noisy test data. \n\n\n## Strengths  \n1. Innovation in Learning Paradigm. By integrating unsupervised test-time data into CIL, DoSAPP eliminates the need for external replay buffers—addressing privacy risks (no historical data storage) and computational costs (no data replaying) that plague traditional CL methods. \n\n2. Strong Performance and Robustness. DoSAPP achieves SOTA results across diverse datasets and scenarios. It outperforms both traditional CL methods and Test-Time Adaptation (TTA)/Continual TTA (CTTA) methods. It also maintains high performance under class imbalance, domain shifts, and noisy test data—proving its adaptability to real-world data irregularities.  \n\n3. Efficiency and Generalizability.  Sparse parameter updates reduce training time by 3–5% compared to full-parameter updates. The Teacher-Student architecture, though dual-model, avoids excessive overhead via sparse updates. \n\n\n## Weaknesses  \n1. Dependence on Test Data Quality. DoSAPP’s performance is highly tied to the quality and distribution of unsupervised test-time data. If test data is severely skewed (e.g., missing entire classes) or low-quality (e.g., heavy noise), its ability to retain old knowledge degrades. \n\n2. Computational Overhead of Dual-Model Architecture. While sparse updates mitigate costs, the Teacher-Student dual-model design still increases deployment overhead compared to single-model CL methods. This may be a bottleneck for resource-constrained devices where model size and latency are critical.  \n\n3. Limited Exploration of Generative Tasks. The paper only validates DoSAPP on classification tasks. Though the authors suggest extendability to generative tasks, no experiments or detailed designs are provided for scenarios like image-text generation—leaving uncertainty about its performance in non-classification VL tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission’s core claims—interleaving supervised training with unsupervised test-time learning mitigates forgetting, DoSAPP’s components (sparse updates/dual-momentum/mask union) balance plasticity/stability, and robustness to imbalance/domain shifts—are strongly supported via empirical studies.", "audience_yn": "Yes", "audience_explanation": "Multiple subgroups within TMLR’s audience will find the paper’s findings highly relevant, as its contributions align with core research areas of the journal and address practical challenges that resonate with both academic researchers and practitioners.", "requested_changes": "1. Clarify Pseudo-Label Selection Robustness. The ablation showing performance drops with teacher-only pseudo-labels lacks analysis of why student input is critical.  \n\n2. Address Test Data Quality Limitation with Concrete Mitigations. The paper acknowledges DoSAPP’s sensitivity to skewed test data but provides no actionable mitigations. This is critical for real-world applicability, as test data imbalance is common. \n\n\n3. Validate Long-Sequence Domain Shift Results with Additional Metrics. Table 5 evaluates long-sequence CIL (Aircraft + Cars) using Avg Acc., FTA, and CTA—but does not report forgetting per task (e.g., does forgetting increase sharply after the domain shift from Aircraft to Cars?).", "broader_impact_concerns": "The paper includes a brief Broader Impact section (Section 8.5) that acknowledges high-level risks of DoSAPP aligning with general Continual Learning (CL) algorithms.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FebzxguKcs", "reviewer": "Reviewer_5vJu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses the challenge of catastrophic forgetting in VLMs when adapted to new tasks in a Class-Incremental Learning setting. The authors propose that standard supervised continual learning is insufficient and can be supplemented by leveraging unlabeled test-time data, which is often available during deployment. They introduce a novel \"Dual-Phase Continual Learning\" framework, DoSAPP, which interleaves a standard supervised adaptation phase for learning new tasks with an unsupervised retention phase for reinforcing old tasks. The DoSAPP method is built on a Teacher-Student framework that uses sparse parameter updates via a dual-momentum mechanism and a logit-based pseudo-labeling strategy. Core contribution includes:\n- The paper proposes and formalizes a \"Continual Learning with Interleaved Test Time Learning\" setting. This setting leverages unlabeled test-time data not for adapting to domain shifts as in TTA, but specifically for retention and mitigating forgetting in a CIL context.\n- The authors introduce DoSAPP, a novel dual-phase method to address this new setting. The authors further propose a dual-momentum EMA update mechanism, a Teacher-Student pseudo-labeling strategy, and a sparse update strategy.\n- The paper provides a comprehensive experimental evaluation across five datasets. DoSAPP significantly outperforms other methods in both w/ replay buffer and w/o replay buffer settings. The authors also provide a thorough ablation study to validate each component of their proposed method.\n\n**Strengths**\n- The paper introduces a well-motivated interleaved learning scenario. The idea of using unlabeled test-time data for retention rather than for adaptation is a novel perspective. This setting is practical, as unlabeled data is often abundant during deployment.\n- The dual-momentum mechanism is a key technical contribution. It is shown to be empirically critical in Table 7. \n- The main results in Table 2 are strong. DoSAPP (no replay) achieves state-of-the-art results compared to all other non-replay baselines on all five datasets.\n\n**Weaknesses**\n- The method's sparse update strategy is central, but key choices feel arbitrary. The method only updates parameters in the first MLP layer of each transformer block. This is a strong assumption based on (Geva et al., 2020) and (Zhang et al., 2024), but it is not ablated. It is unclear if including sparse parameters from other layers would be better or worse. Moreover, the sparsity threshold $c=0.1$ is used for all main experiments. The paper's own ablation in Table 11 shows that $c=0.5$ achieves higher accuracy while $c=0.1$ provides a marginal computation time gain, as shown in Table 16. The choice of $c=0.1$ seems suboptimal just to support the claim of sparsity.\n- The paper claims \"efficient updates\" and uses training time as evidence. However, the wall-clock time saved is marginal. This minimal time saving does not strongly support the efficiency claim, especially given the method's deployment limitation of requiring two models.\n- The assumption that the unsupervised data is being \"drawn from all previously seen tasks\" seems unrealistic. For example, what if the unsupervised data encountered after task 5 only contains samples from tasks 1 and 2, but not 3 and 4? This potential data drift in the unsupervised stream is more realistic yet not explored.", "claims_and_evidence_yn": "No", "claims_explanation": "Overall, the central claims of the paper are well-supported by evidence.\n\nThe main claim that leveraging unlabeled test-time data via a dual-phase unsupervised retention mechanism can significantly mitigate CIL forgetting is backed by the main table and ablation studies. The claim that the proposed DoSAPP method is an effective and novel way to implement this idea is also well-supported. Tables 6 and 7 clearly show that the specific technical components of DoSAPP are crucial and superior to simpler alternatives.\n\nHowever, there is one gap that the claims of \"efficient updates\" and training efficiency are weak. The evidence in Table 16 shows only a marginal reduction in wall-clock time. This evidence does not strongly support the claim of computational efficiency. The authors should adjust this claim to focus on parameter efficiency (sparse updates) or memory efficiency (buffer-free) rather than computational speed.", "audience_yn": "Yes", "audience_explanation": "The paper will be of high interest to several communities within TMLR's audience, such as the continual learning community and the TTA community.", "requested_changes": "- A brief ablation or a stronger justification for the decision to only update the first MLP layer, as opposed to other sparse parameters, will be appreciated. A justification of the use of $c=0.1$ for all main experiments, when $c=0.5$ appears to perform better, would be helpful as well. If this is an efficiency trade-off, please state that. The claim of efficient updates should be reframed, as this is not supported by current evidence.\n- The paper's primary claims should be confined to the domain where evidence is provided, e.g., CIL for vision classification. The last sentence about generative tasks in the conclusion section also lacks supporting evidence. Consider rephrasing it like \"Future work could explore adapting this work to ...\"", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)