DPCore: Dynamic Prompt Coreset for Continual Test-Time Adaptation
Authors: Yunbei Zhang
Akshay Mehra
Shuaicheng Niu
Jihun Hamm
Venue: ICML 2025 poster (2025)
Keywords: Test-Time Adaptation
Continual Test-Time Adaptation
Visual Prompt
Transfer Learning
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We propose a dynamic Continual Test-Time Adaptation (CTTA) setup and a novel CTTA method DPCore.

Abstract: Continual Test-Time Adaptation (CTTA) seeks to adapt source pre-trained models to continually changing, unseen target domains. While existing CTTA methods assume structured domain changes with uniform durations, real-world environments often exhibit dynamic patterns where domains recur with varying frequencies and durations. Current approaches, which adapt the same parameters across different domains, struggle in such dynamic conditions—they face convergence issues with brief domain exposures, risk forgetting previously learned knowledge, or misapplying it to irrelevant domains. To remedy this, we propose **DPCore**, a method designed for robust performance across diverse domain change patterns while ensuring computational efficiency. DPCore integrates three key components: Visual Prompt Adaptation for efficient domain alignment, a Prompt Coreset for knowledge preservation, and a Dynamic Update mechanism that intelligently adjusts existing prompts for similar domains while creating new ones for substantially different domains. Extensive experiments on four benchmarks demonstrate that DPCore consistently outperforms various CTTA methods, achieving state-of-the-art performance in both structured and dynamic settings while reducing trainable parameters by 99% and computation time by 64% compared to previous approaches.

DPCore: Dynamic Prompt Coreset for Continual Test- Time Adaptation
Yunbei Zhang'
Akshay Mehra 2
Shuaicheng Niu
Jihun Hamm'
Abstract
ViDA error
DPCore erfor (ours)
Sunny
Fog
Tunnel
Continual Test-Time Adaptation (CTTA) seeks to adapt source pre-trained models to continually changing, unseen target domains. While exist- ing CTTA methods assume structured domain changes with uniform durations, real-world en- vironments often exhibit dynamic patterns where domains recur with varying frequencies and du- rations. Current approaches, which adapt the same parameters across different domains, strug- gle in such dynamic conditions- they face con- vergence issues with brief domain exposures, risk forgetting previously learned knowledge, or mis- applying it to irrelevant domains. To remedy this, we propose DPCore, a method designed for robust performance across diverse domain change patterns while ensuring computational ef- ficiency. DPCore integrates three key compo- nents: Visual Prompt Adaptation for efficient domain alignment, Prompt Coreset for knowl- edge preservation, and a Dynamic Update mecha- nism that intelligently adjusts existing prompts for similar domains while creating new ones for substantially different domains. Extensive experiments on four benchmarks demonstrate that DPCore consistently outperforms various CTTA methods, achieving state-of-the-art per- formance in both structured and dynamic set- tings while reducing trainable parameters by 99% and computation time by 64% compared to previous approaches. Code available
43.4%
39.9%
time
Continual Structured Change (CSC)
52.1%
42.1%
Proposed CTTA: Continual Dynamic Change (CDC) time
Figure I. Illustrated through autonomous driving scenario where a vehicle encounters varying weather and lighting condi- tions. The top panel shows the conventional CSC setting with struc- tured, uniform-length domain tramsitions, while the bottom panel illustrates our proposed CDC setting where domains varying frequencies and durations Derattons-bbettr better reflecting real-works world challenges. When evaluated on ImageNet-to-mmaggNeer- with ViT base model, previous SOTA ViDA's error rate increases sig- nificantly from 43.4% to 1% when moving from CSC CDC, while DPCore maintains robust performance (39.9% 42.
with
degradation when encountering domain discrepancies be- tween training and test environments (Recht et al., 2019; Hendrycks & Dietterich, 2019; Koh al., 2021). This chal- lenge, ubiquitous in real-world applications, has spurred the development of Test-Time Adaptation (TTA) (Wang et al., 2021; Iwasawa & Matsuo, 2021; Liang et al., 2020; Gao et al., 2023: Schneider et al., 2020; Mirza et al., 2022: 2023; Niu et al., 2024; Liang et al., 2023; Sun et al., 2020). TTA adapts pre-trained models to unseen target domains at test time without altering the original training process, making it particularly suitable for practical applications.
Real-world scenarios, however, present an even more chal- lenging problem: continuously changing target domains. Consider an autonomous driving system that must rapidly adapt to varying weather conditions, lighting changes, and road conditions. Continual Test- Adaptation (CTTA) (Wang al., 2022a: Niu et al., 2022) has emerged ad- dress this challenge. The initial CTTA setting (Wang et al., 2022a), which term Continual Structured Change (CSC), assumed domains change in a structured manner with uni- form durations. However, real-world scenarios exhibit far more dynamic patterns where domains may recur multiple times with varying durations, from brief encounters to ex- tended periods. We term this more realistic scenario Contin-
1. Introduction
Despite remarkable advances in deep learning, Deep Neu- ral Networks (DNNs) often face significant performance
'Tulane University "Dolby Laboratories 'Nanyang Technological University. Correspondence to: Yun- bei Zhang <yzhangll1 @ttulanecdu>>, Jihun Hamm <jhamm3@tulll @ttamm3.uulaaneduuu.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DPCore for Continual Test-Time Adaptation
ual Dynamic Change (CDC). For instance, consider an au- tonomous vehicle driving through a mountainous highway, where it frequently transitions between sunny conditions, dark tunnels, dense fog in valleys, and sudden rain showers due to high humidity, as illustrated in the bottom of Fig. 1. Our experiments show that state-of-the-art (SOTA) CTTA methods like ViDA (Liu et al., 2023b) suffer substantial degradation in performance, with error rates increasing by 8.7% (from 43.4% 52. 1%) on MaageNet-to-ImageNee when moving from CSC to CDC settings.
ing from homogeneous domain groups while preventing negative knowledge transfer between dissimilar domains. Notably, DPCore requires only 1% of ViDA's parameters and merely 300 unlabeled source examples for computing statistics, eliminating the need for expensive warm-up proce- dures on the entire source dataset (see Table 4 and Fig. 5c).
Our extensive experiments across four CTTA bench- marks demonstrate DPCore's superiority. On ImageNet- to-ImageNet-C, DPCore achieves +15.9% improvement over the source model, surpassing ViDA by 3.5% in the conventional CSC setting. Moreover, in the challenging CDC setting, DPCore maintains robust performance with an error rate of 42.1% %, outperforming ViDA by 10.0%. This effectiveness extends to semantic segmentation, where DP- Core improves mIoU scores over ViDA by 0.6% and 1.8% in CSC and CDC settings respectively. Our contributions are summarized as follows:
When encountering CDC environments, existing CTTA methods, which typically adapt the same parameters across different domains (Wang et al., 2021; Liu et al., 2023b; Song et al., 2023), face three critical limitations. First, Conver- gence Issue: these methods inject numerous parameters that require substantial data to converge. While this is achievable in CSC where each domain persists for long duration, it becomes problematic in CDC where certain domains ap- pear only briefly. As shown in Fig. 4b, the increasing error rates during initial adaptation demonstrate that these meth- ods struggle to achieve proper convergence with limited domain exposure. Second, Catastrophic Forgetting: since these methods continuously update the same large parameter space across all domains, knowledge learned from previous domains is progressively overwritten through successive adaptations, resulting in long-term performance degradation on previously encountered domains. Third, Negative Trans- fer: knowledge learned from one domain can adversely affect adaptation to substantially different domains, causing significant performance degradation (see in Sec. 4.3). While these issues exist in CSC, they are amplified in the CDC setting where domains change rapidly and unpredictably.
We introduce a new CTTA Continal Dynamic Change (CDC)-thee better reflects real-world scenar- ios through frequent domain shifts and varying dura- tions. We highlight that previous SOTA CTTA methods suffer significantly in CDC setup due to convergence issues, catastrophic forgetting, and negative transfer.
We propose DPCore, dynamic CTTA approach that effectively manages domain knowledge through a dy- namic prompt coreset: preserving knowledge from seen domains, updating prompts for similar domains, and incorporating new knowledge for unseen domains.
We show that DPCore is theoretically sound and achieves SOTA results for classification and segmenta- tion tasks in both CSC and CDC settings while being computationally efficient.
Furthermore, the practical deployment of CTTA models de- mands computational and memory efficiency, particularly for edge devices. Yet existing approaches overlook this re- quirement. Some methods rely on computationally intensive teacher-student architectures and extensive data augmenta- tion (Wang et al., 2022a; Liu et al., 2023b), while others introduce large numbers of additional parameters that re- quire pre-adaptation warm-up using source data (Liu et al., 2023b; Lee et al., 2024; Song et al., 2023; Gan et al., 2023).
2. Related Works
Test-Time Adaptation (TTA) (Wang et al., 2021; Iwasawa
& Matsuo, 2021; Liang et al., 2020; Gao et al., 2023; Schnei- der et al., 2020; Mirza et al., 2022; 2023; Niu al., 2024; Liang al., 2023; Sun al., 2020; Xiao & Snoek, 2024; Xiao et al., 2023; Mehra et al., 2024b) enhances pre-trained model performance using unlabeled data at test time, with- out access to the original training phase. TTA techniques fall into two main categories based on their use of source data. The first adjusts models through self-supervised losses like entropy minimization (Wang et al., 2021: Niu et al., 2023) and consistency maximization (Wang et al., 2022a; Liu et al., 2023b). The second involves preliminary steps using source data: either by extracting source characteristics such as statistics or features (Mirza et al., 2023; Niu et al., 2024; Zhang et al., 2024)) or by warming up injected pa- rameters on source data before adaptation (Lee et al., 2024; Song et al., 2023; Liu et al., 2023b; Gao et al., 2023).
To address these challenges, we introduce Dynamic Prompt Coresct (DPCore). a novel CTTA method designed for ro- bust performance across varying domain change patterns while maintaining computational efficiency. As shown in Fig. 2, DPCore combines three key components: (1) Visual Prompt Adaptation that aligns source and target domains with minimal parameters and no warm-up requirements, (2) a Prompt Coreset that preserves knowledge from previous domains while accommodating new ones, and (3) a Dynamic Update mechanism that intelligently adjusts prompts based on domain similarities. This design enables efficient learn-
2

---

DPCore for Continual Test-Time Adaptation
At time stept
PromptCorese
Dynamic Update
Visual Prompt (VPA)
P
Evaluate P. on B
I compurad
Test Batch Statstics
Test Batch
via Weighted Prompt
d(r (T)
offine win
B
P
Σ w'p'
uniaeeed source
Pu
P.X
Transformer Layer
Ex
examples
...
At time
Update P. P41
Transformer Layer
L2
PEi
step
Refine
VPA learned p*
Forward
Transfomer Layer
(p' r')
Add (p" , coreset
Backward
based on w
Test Betch Embeddings]
Frozen
Care Element
P (PromotCorese
Trainable
Figure 2. Overview of DPCore with three key components. At time step f, Prompt Coresct P. (upper left) maintains core elements consisting of learned prompts p' and statistics 1d from previous domains. Dynamic Update (middle) evaluates new test batch B, by computing a weighted prompt p.. based distances between batch statistics I's and core elements 1 If p performs well, existing core elements (p' ]T] are refined using weights w²; otherwise, Visual Prompt Adaptation (right) learns a new prompt by aligning the test batch with source statisties (computed offline from 300 examples) and adds to the prompt coreset. The updated coreset Pr+1 is then used for the next time step +++ 1.
Crucially, both approaches are source-free adaptations since they operate without source data during adaptation. While our method falls into the second category, it significantly reduces source dependency compared to existing methods: DPCore requires only 300 unlabeled source examples to compute statistics, in contrast to DePT (Gao et al., 2023), ViDA (Liu et al., 2023b), VDP (Gan et al., 2023) and Be- CoTTA (Lee et al., 2024) which need the entire source dataset for warm-up. Moreover, as demonstrated in Fig. 5c, our approach maintains effectiveness even when source data is completely unavailable.
mental limitations in current CTTA approaches (see Fig. 4b). Most existing methods employ a single-model strategy, up- dating the same set of parameters across all encountered domains ( (Wang et al., 2022a; Liu et al., 2023b; Hoang et al., 2024; Niu et al., 2022). While this approach works reason- ably well in structured settings, it struggles significantly in dynamic environments due to three critical issues: conver- gence difficulties with brief domain exposures, catastrophic forgetting of previously learned knowledge, and negative transfer between dissimilar domains. Although some alter- natives exist such as PeTTA (Hoang et al., 2024), which uses memory components, or RDumb (Press et al., 2023), which employs periodic resets these still fundamentally rely single-model adaptation. In contrast, our DPCore addresses these limitations through a multi modliin strat- egy via dynamic prompt coreset. This approach learns and manages distinct prompts for different domain groups, en- abling DPCore to preserve and reuse knowledge for similar domains while isolating learning for novel ones, thus pro- viding robust adaptation across the varied domain changes characteristic of CDC settings.
Continual Test-Time Adaptation (CTTA) (Wang et al., 2022a; Niu al., 2022; Niloy et al., 2024; Liu et al., 2023a; Boudiaf et al., 2022; Yuan al., 2023; Hoang et al., 2024; Press et al. 2023) tackles the challenge of continually chang- ing target domains. This paradigm was first introduced by COTTA (Wang et al., 2022a), which we refer to as Continual Structured Change (CSC), where domains change in a struc- tured manner with clear boundaries and uniform lengths. While several subsequent works (Niu et al., 2023; Boudiaf al., 2022; Gong al., 2022; Yuan et al., 2023; Lee al.,, 2024) explore TTA challenges such as mixed domains, label imbalance, temporal correlation, and gradual shifts, they still primarily focus on structured domain progressions or static domain scenarios. However, real-world environments often exhibit more dynamic patterns where domains recur with varying frequencies and durations critical aspect that existing setups often overlook. To address this gap, we introduce the Continual Dynamic Change (CDC) setting, where domains can appear multiple times with irregular durations, better reflecting realistic deployment scenarios.
3. Preliminaries and Problem Formulation
In this section, we introduce Vision Transformers (ViT') and formally define the CTTA problem across both CSC and our proposed CDC settings.
3.1. Vision Transformers (ViTs)
We focus primarily on Vision Transformers (ViTs) due to their exceptional representation learning capabilities (Doso- vitskiy et al., 2021: Liu et al., 2021). A VIT (Dosovitskiy
The dynamic nature of CDC environments exposes funda-
3

---

DPCore for Continual Test-Time Adaptation
et al., 2021; Liu al., 2021) f with N transformer lay- ers {C.) can be decomposed into feature extractor 6 X' + Z with parameter 0g and classifier h 2 + y with parameter d%. such that f o Φ. Let = = {f denote the input sequence to the (i+ 1-th layer L where k is the number of image patches and eş represents the classification token [CLS] from layer L,. The standard prediction process follows:
tively manage domain knowledge by preserving knowledge from previously visited domains, updating learned prompts when encountering similar domains, and incorporating new knowledge when facing new domains.
4.1. Visual Prompt Adaptation (VPA)
We leverage visual prompts (Jia et al.. 2022; Ge et al., 2023) for efficient adaptation at test time by introduc- ing L. learnable tokens p := {[Prompt] 1 where (2) [Prompt] J E IR 768 for ViT-Base models. These prompts augment the input sequence as E = {ee, p, et ) modifying Eq. (1) to:
E. = ((E,1)), i= N,
(1)
3.2. CTTA Problem Formulation
Given a model fo with parameters 0 pre-trained on source domain DS (Xx, )s the goal of Continual Test- Teme Adaptation (CTTA) is to adapt this model to a sequence of unlabeled target domains (D' , .... Dz }. where M denotes the number of potential target domains and can be unknown or infinite in wollw scenarios. The model f' operates in an online setting where it processes a sequence of test data batches {B, encountering one batch B, at each time step t. Following standard assumptions in the literature (Wang et al., 2022a; Yang et al., 2024), we con- sider all samples within batch B, belong to the same target domain, though the domain identity remains unknown. At each time step t, CTTA aims to adapt the model parameters from O, to d++1 by learning from the current batch B,, to improve prediction performance on future batches.
E ò ( i i= =1,1,,.. N.
(3)
As illustrated on the right of Fig. 2, we first extract source features 2 = = 00). i.e., exy, in one-time offline computation before adaptation. For the current batch B at time step t from unknown target domain T, we initialize prompt tokens from a Gaussian distribution (Jia et al., 2022) without any warm-up. Denote the extracted features of the current batch with prompt p by 2 (p) = <(B,', 9 p). We align the statistics (mean and standard deviation) of B IT (p) == {nL (p). o. (p)) == {u(2, (p)), (2 (p)) with the source r (p) = (p . } = (u(2)),0(22
through the following distance
d(rs, (TT,, = || - (p)lla + ||o - a, (p)ll2- (4)
The prompt then learned from scratch by minimizing this distance
3.3. Continual Dynamic Change: new CTTA setup
p arg min d(rS, (T()),
(5)
The CTTA setting in previous works (Wang et al., 2022a; Liu et al., 2023b; 2024) typically assumes that target do- mains change in structured manner: each domain has uniform length and changes occur at regular intervals (Fig. Top). However, real-world scenarios present more dynamic patterns where the same domain may appear multiple times with varying durations, (Fig. Bottom). To systematically study this setting, we simulate dynamic environments using a Dirichlet distribution with parameter 8. As demonstrated in Fig. 4a, smaller values of ó result in domain transitions more closely resembling the conventional CSC setting. As 5 increases, domain changes become more frequent and unpredictable. For our main experiments, we set = to keep a balance between structured and completely ran- dom domain changes. Various 8 and other distributions are discussed in Appendix D.
where we omit the superscript T for simplicity, and the subscript 1 denotes the test batch B. Note that while this distance requires source data, labels are unnecessary since we only perform marginal distribution alignment. Our em- pirical results in Fig. 5c show that merely 300 unlabeled source examples are enough to achieve stable performance.
4.2. Prompt Coreset
To address catastrophic forgetting in continually changing environments (Niu et al., 2022; Kirkpatrick et al., 2017; Li & Hoiem, 2017), we introduce a Prompt Coreset mechanism inspired by Online K-Means (Duda & Hart, 1973). The coreset is initialized as an empty set Po as adaptation begins and undergoes adaptive updates each time step t: P, + P.+1. Each core element the coreset consists of a learned prompt and its corresponding feature statisties, illustrated on the left of Fig.2. For example, when the first batch Bj arrives, we begin by extracting features using the pre- trained model fa without any prompt, denoted as 2,, and compute its statistics Iı = {M) σ1} = (p(Zz), o (Z))). We use Zi (p) and 2, to distinguish features extracted with
4. Method
In this section, we present Dynamic Prompt Coreset (DPCore), illustrated in Fig. 2, consisting of three com- Visual Prompt Adaptation, Prompt Coreset, and Dynamic Update. These components work together to effec-
ponents:
4

---

DPCore for Continual Test-Time Adaptation
Domain
50
40-
Source
Gauss bnisedeamed Prompt
Size of Coreset
Nolse
Blur
Weather:
Digital
Error Rate
Size
30
DPCon
25
8
Error Rata
15
Distncce
16
20
10
stup
Insoltoe
Defecor
Glay
Moston
Zonti
Snow
Rosl
Fog
Bright
Contratt
Blastic
Pleat
IPSd
Greees
Stor
impoien
Glads
Mation
Stowy
Freat
Feg
Brgge
Contrat Blasti Farel
peg
2
9 10
Round
(a) Distance
(b) Growth of Coreset
(c) Ten Different Orders
Figure 3. Analysis of DPCore on MongeNe-ttoImmaeNeee in the CSC setting. (a) Distance between test batches and source domain Eq. (4) using source model (baseline), DPCore, and a prompt trained only Gaussian Noise. DPCore consistently reduces domain gaps across all corruptions. (b) Evolution coreset size across corruption types, showing strategic grouping within the four main corruption categories (Noise, Blur, Weather, Digital). (c) Performance stability across ten different domain orders.
and without prompts respectively. Subsequently, we learn prompt P1 from scratch for B1 through Eq. (5). The learned prompt and its corresponding statistics form pair (PI.Ta) I1) that is stored in the coreset as its first element. Unlike existing approaches such as (Wang et al., 2022b) that require maintaining a bulfer of test data, our method stores only feature statistics, significantly reducing memory requirements while enhancing data privacy protection.
once. DPCore computes distances d(T,, I*) between the current batch statistics and all existing core elements, then generates weighted composition of all K prompts pw as
Pu :=
p'
(6)
where expl (op(-d(T,,T)]1
Σ
4.3. Dynamic Update to the Prompt Coreset
and is temperature parameter. The weighted prompt approach offers more flexibility than using the nearest neigh- bor. When a new domain shares characteristics with multi- ple visited domains, it can be effectively decomposed into a weighted combination of existing prompts, providing more accurate adaptation than forcing alignment with a single most similar domain. Notably, all statisties used here are based on features extracted without prompts, as the source pre- trained model can effectively evaluate sample similarity (Yang et al., 2024) while requiring less computation.
Motivations. VPA has demonstrated effectiveness in single- domain TTA (Niu et al., 2024; Zhang al., 2024). Our empirical studies in Appendix Table 6 show that prompts learned from one domain can significantly benefit adap- tation to similar domains (e.g., a Gaussian Noise-learned prompt achieves 40.4% error rate on Shot Noise, compa- rable to a Shot Noise-learned prompt's 39.9%). However, this transferability is not universal as the same prompts can be ineffective or even harmful for substantially different domains (e.g., increasing error rate from 91.4% to 95.7% Contrast). These findings suggest that existing prompts can be effective for similar domains while new ones should be learned for significantly different domains. To achieve this, we employ a dynamic update strategy which first evaluates the current coreset on the test batch and updates the prompt coreset based on the evaluation result.
Updating P. to P.+1- We evaluate the effectiveness of Pw by comparing the distances without and with weighted prompts d(r T2) and d(rs,, (P)))) If the distance re- duction is insufficient, indicating B, likely comes from a new domain, we learn new prompt from scratch using Eq. (5) and add the pair (Tt,P:) to the coreset as new element, following the same procedure used for the first batch B1. Conversely, if the distance decreases significantly, indicating similarity to previously seen domains, we refine Pw with a single update step to obtain the final prompt p: for B. The core elements are then adaptively updated based their weights wo and an updating weight a as
Evaluating P. B& via Weighted Prompt. We employ distance based method to evaluate each new batch after the first batch against existing core elements. Specifically, at time step > 1, given K > 0 elements in the coreset (()) 1 we first extract features Z, from B, without any prompt, i.e., Z, = d> B,; 0)). and compute its statistics Tr. Here, we abuse the superscript to note the element in the coreset. Rather than evaluating each core element individually, which would require processing the same batch K times, we employ a distance-base based weighted prompt approach and process B, with the weighted prompt only
p' + p¹ + awi(p. p²), 1 - 12 + and (T: - 12). (7)
To ensure valid statistics, we use max (o, o')} in the update to prevent negative standard deviations. We compute ratio of the two distances d((=))((pp.)) and compare it "II".. with a predefined threshold ratio p to evaluate whether a
5

---

DPCore for Continual Test-Time Adaptation
quencies-a crucial advantage in dynamic environments.
Detailed analysis is provided in Appendix B.4.
Bil
Diriched Parametter
Emar Rate
13-
5. Experiments
CHOEE
In this section, we evaluate the effectiveness of DPCore on classification and segmentation tasks across CSC and CDC settings. Our evaluation focuses on three aspects: 1) its ability handle (dynamically) changing domains, 2) the growth of coreset, and 3) its hyperparameter sensitivity.
Time
(a) Different 5
(b) Error Rate in CDC
Figure 4, Analysis of CDC setting on ImageNet-to- ImageNet=to-ImagNee-- (a) Domain change patterns with varying Dirichlet distribution pa- rameter ó (colors represent different domains): :: smaller ò produces CSC-like structured changes while larger é leads to more frequent and unpredictable transitions. (b) Performance comparison in CDC setting (6- = 1). where previous methods show significant degrada- tion while DPCore maintains robust performance.
5.1. Experimental Setup
Datasets. We evaluate our proposed method three classi- fication datasets: ImageNer-oo-ImageNeee. CIFAR 100-to- CIFAR 100C, and 10-0-RO---- IOC (Hendrycks & Dietterich, 2019). Each dataset comprises corrupted images in 15 types of corruption, categorized into four main groups: Noise, Blur, Weather, and Digital. Our experiments use the highest level of corruption (i.e., severity 5). Moreover, we validate DPCore on CTTA segmentation task: Cityscapes- ACDC (Sakaridis et al., 2021; Cordts et al., 2016).
batch represents a potential new domain. The ratio is used for applying a nomainn-deendee
distance threshold. This approach allows the coreset to grow dynamically based on the complexity of the domain space, demonstrated in Fig. 3b. Given that the total number of domains is typically unknown in real-world scenarios, we do not impose strict constraints on the coreset size, allowing it to adapt naturally to the complexity of the domain space.
Methods Compared. We compare our method against strong CTTA baselines including Tent (Wang et al., 2021), SAR (Niu et al., 2023), CoTTA (Wang et al., 2022a), VDP (Gan et al., 2023), DePT (Gao et al., 2023), RoTTA (Yuan et al., 2023), ECOTTA (Song et al., 2023), ViDA (Liu et al., 2023b), BDG (Yang et al., 2024), and C-MAE (Liu et al., 2024). Following their protocols,
The strength of our algorithm lies in two key aspects. First, learns prompts from homogeneous domain groups rather than individual batches, enabling more comprehensive adap- tation. This ability is particularly valuable when domains change rapidly with short durations- wheeee previous CTTA methods might struggle with convergence, our method can effectively learn and utilize prompts even across discon- tinuous domain appearances. Second, mitigates error accumulation by avoiding updates existing prompts when encountering significantly different domains. The complete algorithm is presented in Appendix Alg. 1.
conduct several warm-
up iterations on labeled source data (e.g., ImageNet) for VDP, DePT, ViDA, and EсoTTA to ensure optimal perfor- mance. Implementation details are in Appendix A.
CSC and CDC Settings. We evaluate DPCore across the CSC and CDC settings. The CSC setting follows the imple- mentation by (Wang et al., 2022a). In the experiment under CDC setting, employ Dirichlet Distribution with 6 = 1 to generate the data stream from all domains dynamically. Other distributions are discussed in Appendix D.
4.4. Theoretical Analysis
Implementation Details. To ensure consistency and com- parability in our experiments, we follow the implementation details specified in CoTTA (Wang et al., 2022a) and ViDA (Liu et al., 2023b). For classification tasks, we use the ViT-Base model (Wightman, 2019), while for segmentation tasks, we employ the pre-trained Segformer-Bi B5 model (Xie et al., 2021). Following the protocol established in (Zhang et al., 2022), we determine hyperparameters using the four disjoint validation corruptions provided with ImageNet-C and CIFARIO-C (Hendrycks & Dietterich, 2019). We uti- lize the AdamW optimizer (Loshchilov & Hutter, 2017; Kingma & Ba, 2014) with learning rate of 0.01, and select the number of prompt tokens at I. 8 and the threshold at p 0.8. The source statistics are computed using 300 examples from the corresponding source domain. For other
We further provide theoretical insight into why DPCore maintains consistent performance across domain changes. Inspired by online K-Means (Duda & Hart, 1973), we ana- lyze a simplified version of DPCore where batches naturally cluster into distinct groups based on their distances. Under the assumption that these clusters are well-separated, we prove three key propositions: 1) (Prop B.1) DPCore cor- rectly assigns batches to their respective clusters regardless of the sequence length, 2) (Prop B.2) these assignments re- main correct regardless of the order in which batches arrive, and 3) (Prop B.3) the learned prompts are independent of batch order. Together, these properties demonstrate that DPCore can effectively manage domain knowledge even when domains appear in arbitrary orders with varying fre-
6

---

DPCore for Continual Test-Time Adaptation
Table 1. Classification error rate (%) for ImageNectto--maaeNeer in previous CSC setting, evaluated on ViT-Base with corruption severity
level 5. Bold and underline indicate best and second-bess performance respectively.
Algorithm
Gauss.
Shot
Impulse [Defocus Glass
Motion
Zoom
Snow
Frost
Fog
Bright
Contrast
Elastic
Pixel
JPEG
Gaint
Source
53.0
51.8
52.1
68.5
78.8
58.5
63.3
49.9
54.2
57.7
26.4
91.4
57.5
38.0
36.2
55.8
0.0
Psendo ce. 2013)
45.2
40.4
41.6
51.3
53.9
45.6
47.7
40.4
45.7
93,8
98.5
99.9
99.9
98.9
99.6
61.2
-5.4
Tent (Wang et al. 2022a)
52.2
48.9
49.2
65.8
73.0
54.5
58.4
44.0
47.7
50.3
23,9
72.8
55.7
34,4
33.9
51.0
+4.8
CoTTA (Wang al., 2022a)
52.9
51.6
51,4
68.3
78.1
57.1
62.0
48.2
52.7
55.3
25.9
90.0
56.4
36,4
35.2
54,8
+1.0
VDP (Gan et al., 2023)
52.7
51.6
50.1
58.1
70.2
56.1
58.1
42.1
46.1
45.8
23.6
70.4
54.9
34.5
36.1
50.0
+5.8
SAR (Niu et al. 20233)
45.8
45.9
47.7
$2.3
63.7
46.2
50.9
40.3
42.4
41.8
24.4
53.4
53.6
38.4
36.6
45.6
+10.2
RoTTA (Yuan et ul., 2023)
$1.5
50.3
51.7
60.4
58.7
$2.6
54.8
47.2
43.5
42.8
25.9
49.1
48.8
46.3
39.7
48.2
+7.6
RDumb (Press et al., 2023)
46.4
42.1
43.7
56.1
53.1
47.2
49.7
43.0
41.1
46.8
27.7
52.3
49.9
35.3
34,6
44.6
+11.2
EcoTTA (Song et al., 2023)
48.1
45.6
46,3
56.5
67.1
50.4
57.1
41.3
44,5
43.8
24.1
716
54.8
34.1
34.8
48.0
+7,8
ViDA (Liu al., 20236)
47.7
42.5
42,9
56.9
45.5
48.9
38.9
40.7
24,3
52.8
49.1
33.5
33.1
+12,4
BGD (Yang et al., 2024)
47.5
42.1
41.6
55.5
55.4
44.5
47.9
38.8
37,8
39.6
23.6
57.0
44.4
31.5
32.3
42.8
+13.0
C-MAE (Liu et al.. 2024)
46.3
41.9
42.5
514
54.9
43.3
40.7
34.2
35.8
64.3
23,4
60.3
37.5
29.2
3144
42.5
+13.3
DPCore (Proposed)
42.2
38.7
39.3
47.2
51.4
47.7
46.9
39.3
36.9
37.4
22.0
44.4
45.1
30.9
29.6
39,9
+15.9
datasets that lack validation sets, we apply these same hyper- parameters without additional tuning to align with practical testing conditions, where hyperparameters are selected be- fore accessing any target data. More details of DPCore's implementation are available in Appendix B.3. We ensure fair comparison by maintaining the same hyperparameters across both the CSC and CDC settings for all methods.
strategically varied based on both the complexity of indi- vidual domains and their similarities to other corruptions, reflecting a sophisticated approach to domain management.
To further validate DPCore's robustness, we conducted addi- tional experiments using 10 random domain orders, follow- ing the evaluation protocol of CoTTA (Wang al., 2022a). DPCore demonstrates consistent performance, achieving an average error rate of 40.2% with an average coreset size of 13.9 elements in Fig. 3c. Details are available in Sec. C.2.
5.2. Main Results
We present experimental results ImageNet-to-Imageee C and Cityscapes-o--ACDCC across both CSC and CDC settings. Results for CCFAR110100-0--CAAARRI0 provided in Appendix C.3 due to space limit.
in CDC. We benchmark CTTA methods in the more challenging CDC setting. Results in Fig. 4b reveal significant challenges for existing CTTA meth- ods in this scenario. Although these methods perform well in CSC, their gains diminish substantially in the more realis- tic CDC environment. Strikingly, previous SOTA methods suffer severe degradation: ViDA's error rate increases dra- matically from 43.4% to 52. 1%, while Tent not only loses its advantage but performs worse than the source model, with error rates increasing from 51.0% to 57.1% 1%, compared to the source model's 's 55.8%. These substantial performance drops highlight the limitations of existing methods when handling varying domain lengths and rapid domain shifts.
AmageNet-to-ImmageNee
are
in CSC. We first evaluated DP- Core in the conventional CSC setting, where 15 types of corrpption occur sequentially, from Gaussian Noise to JPEG compression during test time. Results, as shown in Ta- ble 1, indicate that DPCore achieves a SOTA improvement of +15.9% over the source model, surpassing the second- best method, C-MAF, by 2.6% The coreset's evolution during this process, illustrated in Fig. 3b, reveals that DP- Core does not treat each corruption type independently but rather groups similar corruptions to optimize adaptation. For instance, examining the Noise group (Gaussian, Shot, Impulse) in Fig. 3a, we observe that these domains share similar distances to the source domain. Recognizing this similarity, DPCore efficiently handles the entire group with a single core element. In addition, DPCore shows sophis- tication in handling domains that exhibit significant differ- ences. A notable example occurs in the Weather group: when transitioning from Fog Brightness, the distance metrics change dramatically despite both corruptions be- longing to the same category. In response, DPCore adapts by learning two separate core elements for Brightness, lead- ing to superior performance compared to other baselines on the Brightness domain. Overall, DPCore maintains ef- ficiency by limiting the total number of core elements to 14 and demonstrates remarkable fiexibility in distributing these elements across corruption types. This distribution is
SorgeeNet-to-Immgeeee
In contrast, DPCore shows remarkable resilience, experi- encing only a modest increase in error rate from 39.9% to 42. 1 %, outperforming the second-best method by sig- nificant margin of 10.0%. To handle the dynamic domain changes, DPCore adapts its coreset size from 14 to 25 ele- ments between CSC and CDC settings. This allows DPCore to maintain performance stability across diverse scenarios.
Cityscapes-t-AACDC in CSC and CDC. Beyond classi- fication, we evaluate DPCore on real-world semantic seg- mentation using the Cityscapesto-AACDC dataset, where performance is measured by mean Intersection over Union (mloU, %). As shown in Table 2, in the CSC setting follow- ing (Liu et al., 2023b; 2024) where domain sequences repeat three times, DPCore shows consistent improvement in mloU across cycles (62.1 + 62.6 + 62.8), outperforming previ- ous SOTA ViDA by 0.6%. In the more challenging CDC
7

---

DPCore for Continual Test-Time Adaptation
Table 2. Average mloU score (%) for Citseapeesto-ACDDC across both CSC and CDC settings. The same target domains are repeated three rounds. Full results in Appendix C.4.
Table 3. Effect of DPCore's three components: VPA (Visual
Prompt Adaptation), PC (Prompt Coreset), and DU (Dynamic
Update). Time shows relative computation time (Tent=1.0), Err Mean shows classification error rate (%).
Setting Algorithm
Round
Round
Round 3
Mean"
Gaint
Source
56.7
56.7
56.7
56.7
0.0
VPA
PC
DU
Time
Err Mean!
Gaint
Tent
56.7
55.9
55.0
55.7
-1.0
Source Tent
55.8
CoTTA
58.6
58.6
1.0
51,0
+4.8
TooTTA
56.0
55.9
55.8
Exp 1
DPCore
C-MAB
VIDA
G188
02.1
61.1
61.6
62.6
62.2
62.0
62.3
62.8
62.5
61.8
61.9
+5.2
+
+5.8
DPCore
Exp-2
Exp 3
11.4
1.6
45.1
48.3
39.9
+10.7
+15.9
+7,5
Tent
56.6
54.8
53.4
54.9
-188
CoTTA
58.0
57.3
$6.5
57.3
+0.5
CDC
EOOTTA
55.6
55.3
55.1
55.3
-1.4
VIDA
60.1
59.1
58.6
59.2
+2.5
C-MAB
39.6
58.8
38.7
to learn its own prompt. In contrast, high p values lead all batches after the first one being considered as from visited domains, potentially reducing the coreset to a single element. However, even a high ratio p such as 1 is likely to learn a new core element if the weighted prompt Pu: increases the domain gap significantly for different unseen domains, i.e., d(r (Pw)) > d(r d(r). We examine p within a range of 0.1 to 1.0. Experimental results, shown in Fig. 5b, demonstrate stable performance for p values from 0.6 to 0.9. To ensure consistency and avoid dataset-specific tuning, we fix 0.8 across all our main experiments before any exposure to target data.
DPCore
61.4
60.8
60.7
61.0
+4.3
setting, while Tent and EcoTTA perform worse than the source model and ViDA's improvement drops from +5.2% to +2.5%, DPCore maintains robust performance (from +5.8% to +4.3%) while adapting its coreset size from 5 elements in CSC to in CDC to handle increased domain complexity. Fine-grained results are available in Appendix C.4.
5.3. Ablation Studies
We provide comprehensive analyses and ablation studies for DPCore on ImageNer-to-Imagger ImageNet-C in the CSC setting. Additional results are available in Appendix E.
Effect of test batch size. To comprehensively assess the impact of test batch size, we evaluate various CTTA meth- ods across batch sizes ranging from 1 to 256. The results illustrated in Fig. 5d reveal consistent pattern across meth- ods: regardless of their objective functions (entropy, con- sistency, or distribution alignment), they maintain stable performance with sufficiently large batch sizes but exhibit increased error rates as batch sizes decrease. In the extreme case of single-sample adaptation (batch size one), all meth- ods demonstrate significantly degraded performance, falling below the source model's 's baseline. This observation aligns with findings reported in recent literature (Yuan et al., 2023; Song et al., 2023). Based on these empirical findings and consistent with prior work, we standardize the batch size to 64 for fair comparison across methods. Moreover, DPCore maintains exceptional stability for batch sizes above 16 and shows minimal degradation for smaller sizes (8 and 4) while outperforming other methods. For extremely small batch sizes (e.g. 1), we introduce DPCore-B, which uses buffer zone to accumulate samples before updates. With a buffer size of 64, DPCore-B achieves a 41.2% error rate even with single-sample batches, demonstrating its practical utility (details in Appendix 1.
Effect of Each Component. Table 3 evaluates the con- tributions of DPCore's key components: Visual Prompt Adaptation (VPA), Prompt Coreset (PC), and Dynamic Up- date DU). In the first experiment (Exp-1), employing only VPA and DU without PC, we see a 4.8% reduction in error rate compared to the source pre-trained model, with com- putation time similar to Tent as it involves learning a single prompt from scratch for all domains. Exp-2 introduces PC but omits DU, requiring new prompts for each batch, enhancing performance to +7.5% but increasing computa- tion time tenfold compared to Tent. In Exp-3, we replace VPA with NormLayer parameters while retaining PC and DU, outperforming Tent by +5.9%, illustrating the method's adaptability beyond ViT architectures CNNs by adapting LormLayer parameters. The full DPCore setup, integrat- ing VPA, PC, and DU, achieves SOTA improvement of +15.9% 5.5 while maintaining computational efficiency.
Effect of prompt length L. We assess the impact of prompt size by varying the number of prompt tokens L within the range of {1, 2, ..., 10}. As depicted in Fig. 5a, DPCore's performance exhibits strong stability across different values of L, demonstrating low sensitivity this parameter. We fix L= 8 for all main experiments. DPCore's sensitivity to threshold p. The threshold p in Alg. I controls DPCore's 's sensitivity to domain changes. Lower p values result in more batches being classified as samples from unseen do- mains. For example, setting P near zero causes each batch
Effect of number of source examples. We evaluate DP- Core's 's sensitivity to source data volume by varying the num- ber of source examples used for computing statistics from O to 10k. As shown in Fig. 5c, DPCore maintains effective adaptation even with 50 examples. For all main experiments, we randomly select 300 unlabeled source examples, notably different from methods like VDP, DePT, EcoTTA, and ViDA
8

---

DPCore for Continual Test-Time Adaptation
60
60
60
Esrer Rate (%)
Error Rate (%)
Errer Rate (16)
50
Source DPCore
150
Source DPCore
Error Rate
Source
70
40
40
50-
35
1
9
10
0.1
0.3
0.5
0.7
0.9
5. 10 50 100 200 500 1k Sk 10k
2
16
32
64
120256
(a) Prompt Length L
(b) Threshold Ratio p
(c) Number of Source Examples
(d) Batch Size
Figure Ablation studies on ImageNet-to-Imme Mor---to--mggeee-CCC impact of (a) prompt token length, (b) threshold ratio for core element evaluation. (c) number of source examples used for statistics computation, and (d) hatch size classification error rate.
Table Computational analysis
SeggeNeettt-ImagNNeee #Param. shows learnable parameters (Extra Param. indicates injected parameters beyond model), #FP/#BP shows propagation counts, Time shows relative computation (Tent=1 0), and Err Mcan
Table S. Memory management strategies for fixed-size coreset
(K=15) on ImageNet-to--mageNeee across 10 different orders.
DPCore
Algo.
Source
(flexible K)
(K=15, discard) (K=15, merge)
shows classification error rate (%).
Err. Mean 1
55.8
40.2
42.3
41.1
Algo.
PParam.
Extra Param.
VFP
ABP
Time
Err. Mean
Tent 0.03M
1.0
$1.0
CoTTA
86.57M
11.7
54.8
VDP
1800
2
1.5
50.0
prompts exceeds K, we either 1) discard the oldest prompt or 2) merge the most similar prompts by averaging those with the smallest statistics distance. As shown in Table 5, both strategies significantly improve upon the source model, with merging performing slightly better than discarding. Since the number of domains is typically unknown in real-world scenarios and the coreset grows only when encountering truly unseen domains, our default approach allows K to evolve naturally.
BcOTTA 3,46M
1.9
48.0
VIDA
43.4
Ours
0.08M
3.1
1.8
39.9
that require the entire labeled source dataset for parameter initialization. We further explore an extreme scenario where source data is completely inaccessible (shown at source data =0 0 in Fig. 5c). For ImageNet-C experiments, using 300 un- labeled STL10 (Coates et al., 201 images as reference still achieves an error rate of 45.6%, improving over the source model by 10.2%. Details are provided in Appendix E.2.
6. Conclusion
DPCore's computation and memory efficiency. We ana- lyze computational complexity across methods in Table 4, comparing learnable parameters, forward/backward propa- gation counts, and relative computation time (normalized to Tent). DPCore achieves remarkable efficiency by in- troducing only 0.08M parameters (0.1% of model parame- ters), whereas ViDA requires 93. 93. parameters (7.13M additional parameters, 89x more) and 55.6% more com- putation time. While this analysis primarily focuses test-time computation, DPCore also maintains efficiency in pre-ataptt unlike previous methods which require ex- tensive source data warm-up (e.g., EcoTTA needs epochs of ImageNet training for parameter initialization), DPCore requires minimal preparation, involving only forwarding 300 source examples and computing their statisties. This shows DPCore's 's efficiency in both the preparation and adap- tation phases. Detailed analysis is in Appendix E.3.
We introduce a new CTTA setup: Continual Dynamic Change (CDC) setting that better reflects real-world scenar- ios where domains recur with varying frequencies and du- rations. Through extensive benchmarking on four datasets, we demonstrate that previous methods for CTTA struggle in CDC due to convergence issues, catastrophic forgetting, and negative transfer. To remedy this, we propose DPCore, a simple yet effective approach to CTTA that achieves ro- bust performance across diverse domain change patterns while maintaining computational efficiency. DPCore inte- grates three complementary components: Visual Prompt Adaplation for efficient domain alignment with minimal parameters, Prompt Coreset for strategic knowledge preser- vation, and Dynamic Update for intelligently managing domain knowledge- updating existing prompts for similar domains while creating new ones for substantially different ones. Through our experiments, we show that DPCore is effective and achieves SOTA performance for classification and segmentation tasks for the standard CSC and the new CDC setting. Our theoretical analysis and comprehensive ablation studies further validate DPCore's 's effectiveness.
knowledge-updating
Fixed-Size Coreset Management. We explore fixed-size coreset strategies for scenarios with strict memory con- straints. We evaluate two approaches to maintain a fixed- size coreset on ImageNet-C with K=15: once the number of

---

DPCore for Continual Test-Time Adaptation
Acknowledgment
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. A continual learning survey: Defying forgetting in classifi- cation tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3385, 2021.
We thank the anonymous reviewers for their insightful com- ments and suggestions. We are also grateful to Evan Shel- hamer for his thoughtful feedback on connecting our work to the broader continual test-time adaptation literature. This work was supported by the NSF EPSCoR- Materi- als Design Alliance (LAMDA) program #01A-1946231.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M.. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, An image is worth 16x16 words: Transformers for image recognition scale. In International Conference Leamning Representations, 2021. URL https:// openreview net Eorumrididiiooo..ooo -ien-idd--YiidTT....
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Duda, R. O. and Hart, P. E. Pattern recognition and scene analysis, 1973.
References
Farajtabar, M., Azizan, N., Mott, A., and Li, A. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773, PMLR, 2020.
Ben-David, S., Blitzer, J., Crammer, K., and Pereira, F. Analysis of representations for domain adap- tation. In Schölkopf, B., Platt, J., and Hoffman, T. (eds.), Advances in Neural Information Pro- cessing Systems, volume 19. MIT Press, 2006. URL
Gan, Y., Bai, Y., Lou, Y., Ma, X., Zhang, R., Shi, N., and Luo, L. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7595-5557777 759957763, 2023.
https::/proceeednng..c neurips. cc/paper_fill hhappe__fhhee//pper////////// 1l10432ccafb00ee1142261131111189 pdf.
Paper.
Bobu, A., Tzeng, E., Hoffman, J., and Darrell, Adapting to continuously shifting domains. 2018.
Gao, Y., Shi, X., Zhu, Y., Wang, H., Tang, Z., Zhou, X., Li, M., and Metaxas, D. N. Visual prompt tuning for test-time domain adaptation. arXiv preprint arXiv:2210004831, 2022.
Boudiaf, M., Mueller, R., Ayed, I. B., and Bertinetto, L. Parameter-free online test-time adaptation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8334-8343, 2022. URL /aaps::///p.... sttps://api.semmntiics. semanticscholar: org/ org/CorppssD::24001583
Gao, Y., Shi, X., Zhu, Y., Wang, H., Tang, Z., Zhou, X., Li, M., and Metaxas, D. N. Visual prompt tuning for test-time domain adaptation, 2023. URL https:// openreview net/ forum Iid=33IIBjIIITT.
Ge, C., Huang, R., Xie, M., Lai, Z., Song, S., Li, S., and Huang, G. Domain adaptation via prompt learning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Coates, A., Ng. A., and Lee, H. An analysis of single- layer networks in unsupervised feature learning. In Gordon, G., Dunson, D., and Dudík, M. (eds.), Pro- ceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Pro- ceedings Machine Learning Research, pp. 115-223,, Fort Lauderdale, FI., USA, 11-13 Apr 2011. PMLR. URL wwwphttps/oc http///prceeenng.eeee/// mlr press/v15. coatesila. html.
Gong, T., Jeong. J., Kim, T., Kim, Y.. Shin, J., and Lee, S.- J. NOTE: Robust continual test-time adaptation against temporal correlation. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. arXiv preprint arXiv: /903. /2261, 2019.
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. The cityscapes dataset for semantic urban scene un- derstanding. 2016 IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 3213-3223, 2016. URL https: : api. www.eeetincooool.... org/CorpusID::500 502946.
Hoang, T.-H., Vo, D. M., and Do. M. N. Persistent test-time adaptation in recurring testing scenarios. In Thirty-eighth Conference on Neural Information Processing Systems (NeurIPS), 2024. 2024.
10

---

DPCore for Continual Test-Time Adaptation
Iwasawa, Y. and Matsuo, Y. Test-time classifier adjustment module for model-agnostic domain generalization. In Ranzato, M. Beygelzimer, A., Dauphin, Y.. Liang, P. and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 2427-2440. Curran Associates, Inc., 2021. URL https: https://prooceedings. meuriiis cc apaapeeeiieee/pppprr///////////e #455efffaoaale55odeff5522222222 pdf.
Liang, J., He, R., and Tan, T. A comprehensive survey test-time adaptation under distribution shifts. arXiv preprint 230.v::333.13366. 2023.
Liu, C.,, Wang, L., Lyu, L., Sun, C., Wang, X., and Zhu, Q. Deja vu: Continual model generalization for unseen domains. arXiv preprint arXiv:2301. 10418, 2023a.
Liu, J., Yang, S., Jia, P., Lu, M., Guo, Y. Xue, W., and Zhang, S. Vida: Homeostatic visual domain adapter for continual test time adaptation. arXiv preprint arXiv:2306.04344 2023b.
Jia, M., Tang, L.,, Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Liu, J., Xu, R., Yang. S., Zhang, R., Zhang, Q. Chen, Z.. Guo, Y., and Zhang, S. Continual-mac: Adaptive distribution masked autoencoders for continual test-time adaptation. pp. 28653-265222 11653-2866,, 2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Liu, 7., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13)3351-33526 2017.
Trabska-Barwingg
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv:
2017.
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. Wilds: A benchmark of in-the- wild distribution shifts. In International conference on machine learning, pp. 5637-5664. PMLR, 2021.
Mai, Z., Li, R., Kim, H., and Sanner, Supervised con- trastive replay: Revisiting the nearest class mean clas- sifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3589-3599, June 2021.
Le, T., Nguyen, T., Ho, N., Bui, H., and Phung, D. Lamda: Label matching deep domain adaptation. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th In- ternational Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6043-6054. PMLR, 18-24 Jul 2021. URL https: proceedings ngs. ml r. press 1ttm
Mai, 7., Li, R., Jeong, J., Quispe, D., Kim, H., and Sanner, S. Online continual learning in image classification; An empirical survey. Neurocomputing, 469:28-51, 2022.
Marsden, R. A., Döbler, M., and Yang, B. Uni- versal test-time adaptation through weight ensem- bling, diversity weighting, and prior correction. 2024 IEEF/CVF Winter Conference on Applica- tions of Computer Vision (WACV), pp. 2543-2553, 2023, URL https: hoss://app //ppi. semanticscholar. org / C/rp/orrrussaa// 2589997344
Lee, D., Yoon, J., and Hwang. S. J. Becotta: Input- dependent online blending of experts for continual test- time adaptation. In International Conference on Machine Learning, 2024.
Lee, D.-H. Pseudo-label : The simple and efficient semi- supervised learning method for deep neural networks. 2013.
Mehra, A., Zhang, Y., and Hamm, J. Understanding the transferability of representations via task-relatedness. The Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024a. URL https: / //openreviee ew. net wt:mm/ormmm/// d=ccdMMkxXN..
Li Z. and Hoiem, D. Learning without forgetting. IEEE Iransactions patlern analysis and machine intelligence, 40(12):2935-290 2017.
Mehra, A., Zhang, Y., Kailkhura, B., and Hamm, J. On the fly neural style smoothing for risk-averse domain generalization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3800-3811, 1, January 2024b.
Liang, J., Hu, D., and Feng, J. Do we really need to access the source data? source hypothesis transfer for unsuper- vised domain adaptation. In International conference on machine learning, pp. 6028 00899 PMLR, 2020.

---

DPCore for Continual Test-Time Adaptation
Mirza, M. J., Micorek, J., Possegger, H., and Bischof, H. The norm go on: Dynamic unsupervised do- main adaptation by normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14765-14775, 2022.
Sakaridis, C., Dai, D., and Gool, L. V. Acde: The adverse conditions dataset with correspondences for semantic driving scene understanding. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10745-10755, 2021. URL. https: org CCoppusI:
Mirza, M. J., Soneira, J.J Lin, W., Kozinski, M., Possegger, H.. and Bischof, H. Actmad: Activation matching to align distributions for test-tmme-traiiin. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 24152-24161, 2023.
233423465.
Schneider, S., Rusak, E., Eck, L., Bringmann, O., Bren- del, W., and Bethge, M. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in neural information processing systems, 33: 11539-11551, 2020.
Niloy, F., Ahmed, M., Raychaudhuri, D. S., Oymak, S., and hoy-Chowdhuuy, A. K. Effective restoration of source knowledge in continual test time adaptation. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2091-2100, 2024.
Shen, J., Qu, Y., Zhang, W., and Yu, Y. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI conference on artificial intelli- gence, volume 32, 2018.
Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng. S., Zhao, P., and Tan, M. Efficient test-time model adaptation with- out forgetting. In International conference on machine learning, pp. 16888-16905, PMLR, 2022.
Shim, D., Mai, Z., Jeong, J., Sanner, S., Kim, H., and Jang, J. Online class-incremental continual learning with adver- sarial shapley value. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pp. 9630-9638, 2021.
Niu, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., and Tan, M. Towards stable test-time adaptation in dynamic wild world. arXiv preprint arXiv:2302.12400 2023.
Silver, D. and Mercer, R. The task rehearsal method of life-long learning: Overcoming impoverished data. In Advances in Artificial Intelligence: 15th Conference of the Canadian Society for Computational Studies of Intelligence, Al 2002 Calgary, Canada, May 27-29, 2002 Proceedings 15, pp. 90-101. Springer, 2002.
Niu, S., Miao, C., Chen, G., Wu, P., and Zhao, P. Test- time model adaptation with only forward passes. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, (eds.), Pro- ceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 30238-8288888 38315. PMLR, 21-27 Jul 2024. URL https: /ttpp//poceennssseeee/ mlr press/ 0235/niu244: html.
Song. J., Lee, J., Kweon, S., and Choi, S. Ecotta: Memory- efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 1 11920-11929, 2023.
Press, O., Schneider, S., Kuemmerer, M., and Bethge, M. RDumb: A simple approach that questions our progress in continual test-time adaptation. In Thirty- seventh Conference on Neural Information Processing /
Sun, J., Ibrahim, M., Hall, M., Evtimov, I., Mao, Z. M.. Ferrer, C. C., and Hazirbas, C. Vpa: Fully test-time visual prompt adaptation. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 5796- 5806, 2023.
Systems, 2023. URL
Forum?id=ver 6VTV
Rebuffi, S.-A., Kolesnikov, A., Sperl, G.. and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Sun, Y., Wang. X., Liu, Z., Miller, J., Efros, A., and Hardt, M. Test-time training with self-supervision for generaliza- tion under distribution shifts. In International conference on machine learning, pp. 9229-92222 9248. PMLR, 2020.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In Interna- tional conference on machine learning, pp. 5389-5400. PMLR, 2019.
Wang, D.,, Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimiza- tion. In International Conference on Learning Represen- tations, 2021. URL https: oppprevieeeeeeeee// nnt/ forum? i d= www.kubbbb.k.rr.cc
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., and Wayne, G. Experience replay for continual learning. Advances in neural information processing systems, 32, 2019.
Wang, Q., Fink, O.,, Van Gool, L., and Dai, D. Continual test- time domain adaptation. In Proceedings of the IEEF/CVF
12

---

DPCore for Continual Test-Time Adaptation
Conference on Compuler Vision and Puttern Recognition, pp. 7201-7211, 2022a.
Wang, Z., Zhang, Z.,, Lee, C.-Y., Zhang, H.,, Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognition, pp. 139-149,200 2022b.
Wightman, Pytorch image models. https: / com/ wwwpwwwwwwgggtteeeeeeeeeee pptpoccttooohoooooo mnnage-modelle 2019.
Xiao, Z. and Snoek, C. G. M. Beyond model adaptation at test time: survey, 2024. URL httpss://aaxttt org/ aaaaab//224/1 03687.
Xiao, Z., Zhen. X., Liao. S., and Snoek, C. G. Energy- based test sample adaptation for domain generalization. In The Eleventh International Conference on Learning Representations. 2023.
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., and Luo, Segformer: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems (NeurIPS). 2021.
Yang, X., Chen, X., Li, M., Wei, K., and Deng, C. A versa- tile framework for continual test-time domain adaptation: Pro-
Balancing discriminability and generalizability. ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR), pp. 23731-23740, June 2024.
Yuan, L., Xie, B., and Li, S. Robust test-time adaptation in dynamic scenarios. Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15922-15932. 2023.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995, PMLR, 2017.
Zhang, M. M., Levine, S., and Finn, C. MEMO: Test time robustness via adaptation and augmentation. Oh, A. H., Agarwal, A., Belgrave, D., and Cho, (eds.), Advances in Neural Information Processing Systems, httpt::s//peeeeeiie...
2022. URL httpt::///// wpt.reveee.eeeeeeee/ppeee ner/forum? 1d=XrGEkCOREX2.
Zhang, Y.. Mehra, A., and Hamm, J. OT-VP: Optimal transport-guided visual prompting for test-time adapta- tion. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), 2024.
13

---

DPCore: Dynamic Prompt Coreset for Continual Test- Adaptation Supplementary Materials
In the Appendix, provide detailed supplementary materi- als to enhance understanding of our work: First, we present background materials: related works and CTTA baselines and implementations (Sec. A), and additional details about our method and CDC setting, including hyperparameter se- lection and theoretical analysis (Sec. B). The experimental aspects are covered in several sections: additional experi- mental results (Sec. C), discussion of our CDC setting with other CTTA settings (Sec. D), and extended ablation studies (Sec. E). Finally, we discuss limitations of our method in Sec. F.
in ViT- Base..
DePT (Gao et al., 2023) integrates visual prompts with Vision Transformers to adapt to target domains through memory bank-based pseudo-labeling This method utilizes significant number of prompts, which are initialized by warming up on source data.
SAR (Niu et al., 2023) introduces sharpness aware and awwre reliable entropy minimization method. This approach se- lectively filters noisy samples and optimizes model weights
towards stable regions in the parameter space.
VDP (Gan et al., 2023) employs visual domain prompts in pixel space to adapt to continually changing domains. dynamically updates lightweight prompts to facilitate do- main adaptation, modifying input images instead of model parameters. The pixel prompts are initialized on source data.
A. Related Works and Baselines
A.1. Continual Learning (CL)
CL (Rebuffi et al., 2017: De Lange et al., 2021; Farajtabar et al., 2020; Rolnick et al., 2019; Li & Hoiem, 2017; Kirk- patrick et al., 2017: Zenke et al., 2017; Silver & Mercer, 2002; Bobu et al., 2018; Mai et al., 2022; 2021; Shim et al., 2021) addresses catastrophic forgetting by enabling models to retain learned information while acquiring new knowl- edge. This challenge is relevant to TTA, especially in Continual Dynamic Change (CDC) environments, where adapting new target domains often degrades performance on previously seen domains. To address these limitations, our approach integrates principles from CL into the TTA framework. We employ a dynamically updating coreset that preserves knowledge across visited domains, alongside a weighted updating mechanism to reduce error accumulation.
RDumb 4 (Press et al., 2023) employs periodic resets to the original pretrained weights at regular intervals to prevent the accumulation of adaptation errors that lead to performance collapse in continual test-time adaptation.
TTO (Song et al., 2023) utilizes lightweight meta net- works and self-distilled regularization to maintain memory efficiency and ensure long-term adaptation stability. This method emphasizes consistency between the outputs of meta networks and the original frozen network, with meta- networks warmed up on source data for several epochs.
ViDA (Liu et al., 2023b) uses high-rank and low-rank do- main adapters to manage domain- and shared knowl- edge. Designed to address continual test-time adaptation, it dynamically responds to changing domain conditions. The adapters are warmed up on source data.
A.2. Baselines
In this section, we provide the details of the CTTA basclines we use in our paper.
Tent (Wang et al., 2021) updates the Norm Layer param- eters through prediction entropy minimization. We follow the same hyperparameters that are set in Tent.
BDG (Yang al., 2024) is a framework designed to bal- ance discriminability and generalization in CTTA by gen- erating high-quality supervisory signals. This framework uses adaptive thresholds for pseudo-label reliability, lever- ages knowledge from pre-trained source model to adjust unreliable signals, and calculates a diversity score to ensure future domain generalization.
2 (Wang et al., 2022a) is the first to perform TTA on continually changing domains and propose teacher-student learning scheme with augmentation based consistency max- imization. We follow the same hyperparameters that are set in CoTTA. The trainable parameters are all the parameters
http://ghhabbccommm/mggggg/////
http:://gghaaacco////mmnr//CCCC
httpss//gittubb.oom/eququ Wang/tent httpps///ituub.com/qneeeeeet
hhtps:/ghhibboommyygggennaaaaa
14

---

DPCore for Continual Test-Time Adaptation
C-MAE' (Liu et al., 2024) introduces Adaptive Distri- bution Masked Autoencoders for CTTA, which employ Distribution-aware Masking mechanism to adaptively sam- ple masked positions in images.
Algorithm 1 The proposed algorithm DPCore
Input: A source pre-trained model [(x),, source statistics r test
batches {Be \T=1
Initialization: An empty coresel, pre-defined ratio threshold p. and random prompt tokens.
We utilize official implementations of the method where available; otherwise, we implement it ourselves using the hyperparameters reported in the original paper. To ensure fair comparison, we maintain consistent hyperparameters across both CSC and CDC settings.
1:
for the first batch Bı do
2: Compute the statistics of batch Bi without prompt.
3: Learn prompt from scratch (Eqq. 8).
4: Add the batch statisties and prompt to the empty coreset.
5: end for
6: for t 2,..., T do
7: Compute the statistics of batch B, without prompt.
B. Additional Details of DPCore
8: Compute the weights and weighted prompts (Eq. 6).
9: Compute batch statistics with the weighted prompt.
B.1. Visual Prompt Adaptation in TTA
10:
Compute ratio of distances with/w.o. weighted prompt.
11:
if ratio S then
In the context of TTA where labels are unavailable, prompts
12:
Update weighted prompt on B, by step.
can be optimized through consistency maximization, en- tropy minimization, or unsupervised distribution alignment (Niu et al., 2024; Sun al., 2023; Gao et al., 2023; Zhang et al., 2024). We employ distribution alignment due to its proven effectiveness and computational simplicity (Ben- David et al., 2006; Mchra et al., 2024a).
13:
Update all coreset elements (Eq. 7).
14:
else
15:
Learn prompt from scratch for B, (Eq. 8). Add the batch statistics and learned prompt the coresct new element.
16:
17:
end if
18:
end for
Output: Prediction for all batches; The learned coreset.
For the current batch BT at time step from unknown target domain T, a prompt optimized by minimizing the distribution alignment between source and target features:
compute the statistics and set the number of prompt tokens to 8 with a test batch size of 64. The updating weight is set as a = 0.999. We learn the prompt from scratch for 50 steps and refine the existing prompt for only step. The model is optimized using AdamW with a learning rate of 0.01, and the threshold p is set to 0.8. These hyperparameters were determined using four disjoint validation corruptions from ImageNet-C and CIFAR 10-C (Hendrycks & Dietterich, 2019): [Speckle Noise, Gaussian Blur, Spatter, Saturate), following (Zhang et al., 2022).
p arg min U(25,27' (T)())), (p))..
P
where d represents distribution distance and Z (p) p) denotes the extracted features of the current batch with prompt p. shown on the right of Fig. 2. We employ marginal distribution alignment
et al., 2006; Le et al., 2021; Shen et al., 2018) to avoid potential error accumulation from pscudo labels in continual learning scenarios, though other sophisticated distance metrics are also applicable.
For datasets without validation sets (e.g., CIFAR100-to CIFARI00C and Cityscapesto-AACDC) we apply these same hyperparameters without additional tuning align with practical testing conditions, where hyperparameters must be selected prior to accessing target data. Moreover, we employ identical hyperparameters across both CSC and CDC settings.
Unlike previous prompt-based TTA methods such DePT (Gao et al., 2023) or VDP (Gan et al., 2023) that require prompt warm-up on source data for optimal initialization, we initialize prompt tokens from a Gaussian distribution without any warm-up, following Visual Prompt Tuning in supervised settings (Jia al., 2022). This approach makes our method more practical and efficient.
B.4. Comprehensive Analysis of DPCore
We elaborate on the theoretical analysis introduced in Sec. 4.4. For simplicity, we consider a version of Alg. where we use one-hot weights instead of general weights-only the closest core element a given test batch is considered, and only the mean is stored in the coreset. The simplified algorithm is presented in Alg. 2, denoted as A
B.2. Algorithm of DPCore
To illustrate how DPCore operates as shown in Fig. 2, we present its complete algorithm in Alg. I.
B.3. Implementation Details
Here we detail the hyperparameters used in our main ex- periments. We randomly sample 300 source examples to
Consider batches Bi,..,, B, naturally clustered into M mutually exclusive clusters {G based on their dis- tances. Let Conv(G") be the convex hull of batches in
15

---

DPCore for Continual Test-Time Adaptation
G¹. diam(S) 2''s d(ar,=") be the diameter of set
Proposition B.3. For any 1 > 1, Alg. 2 learns prompi Pt for batch Be independent of batch order Bx())..... Br(t-1y VT E [1(t=1),
S, and d() = infres.r'ES' d(x,, x²) be the set dis- lance. We denote by (1(t-1) the set of permutations (1...,t-11, 1), by G2 the set of batches in cluster at time i, and by {Pi, }4} the i-th core element at time L. Each core element consists of a core prompt pi and a core mean uz. Abstractly, the algorithm A generates the current prompt Pt = A({B,-1,... B1}) using the batch history (Bi,... ..., } as imput.
Proof. From Proposition B.2, cluster assignment is inde- pendent of batch order. For batch B,, two cases arise: 1) If B, belongs an existing cluster i, P. is learned starting from existing core prompt p'. Since p' is order-independent Pt is also order-independent. 2) If B, creates new clus- ter, Pt is learned from scratch, making it inherently order- independent.
Assumption (Well-separated clusters); There exists 0 >>0 0 such that:
diam(Conv(G')) < dd(onv(G')) Conv Vi $.
Our analysis demonstrates three key properties of DPCore under the well-separatedness assumption: 1) Correct Clus- tering: DPCore correctly assigns all batches to their re- spective clusters regardless of sequence length. 2) Order Independence: Cluster assignments remain correct regard- less of the order in which batches arrive. 3) Prompt Stability: The learned prompts are independent of batch order.
This implies batches within the same cluster are closer to each other than to batches from different clusters.
Proposition B.1. For any 2 1, Alg. 2 assigns all batches .i.... By to correct clusters.
Proof. We prove by induction. At since no clusters exist, Bi creates the first cluster with μ = p( Suppose cluster assignments are correct for batches B1. B2-1- Since each core mean fu² is updated through interpolation a) + appl Be), it remains within the convex hull of its cluster: p4; E Co)v
These properties together show that DPCore can effectively manage domain knowledge even when domains appear in arbitrary orders a crucial advantage in dynamic environ- ments. While this analysis uses simplified assumptions, it provides valuable insight into DPCore's robustness to random domain changes. The full analysis under general conditions remains an interesting direction for future work.
When new batch B, belonging to cluster j arrives, we con- sider four cases based on whether cluster j or other clusters contain batches: 1) If no clusters exist ([G²/_l - |G
Algorithm A simplified version of the proposed algorithm
0, Vk # j). B, creates a cluster. 2) If only other clusters exist (|G/_1== 0, and 3h # |G > 0), well-separatedness ensures ((p((B), > 0, so B, cre- ates a new cluster. 3) If only cluster j exists (G'/1 > 0, and |G = O, Vk # j), well-separatedness ensures d(u((,) B.), p²) < so B, joins cluster j. 4) If multiple clus- ters exist (G > O, and 3k fj: |Gt-1l > 0), well- separatedness ensures d(u((B), (e) < 8 < (((u(Bi). ) so B, correctly joins cluster j.
1: for / = 1,2,,... T do
Compute the batch mean μu and distances di ((u(B.), p²)), i == 1,... ., K against all existing core elements.
3:
if coreset is empty or miny d > 0 then
4:
Learn prompt from scratch for B, (Eq. 8). Add the batch mean and learned prompt to the coreset as a new element. else
5:
Assign B, to the closest core element j" arg min, d'.
Proposition B.2. For any t > 1, Alg. 2 assigns B, to the correct cluster independent of batch order Ba(I)...., Bx(t-1); Yn E IT(--)) Furthermore, if a = G then the core mean is the cluster mean: pà = Enery mBB,, also independent of batch order.
8:
Update the prompt in j"tth core element by step. Update j" core element (Eq. 7).
9:
10: end if
II: end for
Output: Prediction for all batches; The learned coreset.
Proof. From Proposition 1. correct assignment at time t depends only on well-separatedness and correct assignments at 1- 1, not on batch order. For the second claim, consider batches B(1), ,(2), are the batches assigned to cluster j in the order they are processed. With a |c-| updates yield: -¹ ((B(1),, μd¹ = (((B()) + ((B())), μ¹= (((B())+ + (((()) + μu (B))))), and so on. This running average is independent of batch order.
C. Additional Results
C.1. Performance of Single Domain-learned Prompt
To validate the core intuition behind DPCore, we conduct a preliminary experiment examining how a prompt learned from one domain transfers to others. Specifically, we learn a
16

---

DPCore for Continual Test-Time Adaptation
prompt from scratch on the Gaussian Noise domain and eval- uate its effectiveness across the remaining 14 domains. For comparison, we also learn domain-specific prompts inde- pendently with known domain boundaries, where the model resets each time it encounters a new domain. As shown in Table 6, the Gaussian Noise-learned prompt yields interest- ing transfer patterns: it significantly improves performance on similar domains (Shot Noise and Impulse Noise), shows limited effectiveness others (Defocus Blur and Elastic), and even degrades performance on substantially different do- mains (Motion and Contrast). These performance variations strongly align with the domain distance measurements illus- trated in Fig. 3a, suggesting that the effectiveness of prompt transfer directly related to the similarity of domains in feature space.
60
Source Tent
CoTTA SAR
VIDA RoTTA
C-MAE DPCore
Error Rate (%)
55-
50-
45
40
38
CSC
5=0.01
6= 0.1
CDC (6 =) 1) 5 10
Figure 6. Performance comparison across different domain change patterns on SoageNee-tt--mmgeNeeee The x-axis shows settings from CSC to CDC with increasing Dirichlet parameter 6, where larger 6 indicates more frequent domain changes. DPCore main- tains stable performance even with rapid domain transitions, while other methods degrade significantly.
Notably, DPCore surprisingly outperforms the domain- specific prompts despite requiring no domain boundary information. This superior performance stems from DP- Core's ' ability to learn a unified prompt for similar domains (e.g., the entire Noise group including Gaussian. Shot, and Impulse), achieving better adaptation than learning from individual domains in isolation.
trained model and surpassing the previous SOTA (ViDA) by 5.3%. This improvement is consistent across all corruption types, with particularly strong performance on noise corrup- tions (reducing error rates from 60. % to 22.0% on Gaus- sian Noise). In the more challenging CDC setting, while previous methods show substantial degradation (ViDA's im- provement drops from +7.5% to +4.6%)) DPCore maintains robust performance with an 11.1% improvement over the source model.
These empirical findings reveal two crucial insights that motivate DPCore's ' design: (1) prompts can transfer effec- tively between similar domains, potentially enabling more efficient adaptation, and (2) for fundamentally different do- mains, learning new Nomain-specitee prompts is essential to prevent negative transfer.
C.2. Results for 10 Random Orders in CSC
This section provides details for the results presented in Fig. 3c. We strictly follow the ten random orders used in CoTTA (Wang et al., 2022a) and evaluate DPCore on MoageNet-to-ImageNNee
Similarly, for CETAR100-OOCIFAARIOO task (results in Ta- bles 13 and 14), DPCore demonstrates strong performance across both settings. In CSC, DPCore achieves 10.3% improvement over the source model and outperforms ViDA by 2.2%, with notable improvements across all corruption types, particularly on Glass (44.1 1% vs 60.5% source error rate) and Contrast corruptions (13.2% vs 34.8% source error rate). The CDC setting presents a greater challenge, where several methods (CoTTA, VDP) perform worse than the source model, and ViDA's improvement drops from +8. 1% to +5.6%. In contrast, DPCore maintains robust perfor- mance with an 8.0% improvement over the source model, demonstrating its effectiveness.
across all ten domain orders. It is important note that even though the domain order is changed, the domain change frequency and domain length remain fixed, resulting in the same domain change pattern in CSC setting. DPCore demonstrates robust performance in this setting, achieving an average error rate of 40.2% with an average coreset size of 13.9 elements. These re- sults highlight the consistency and effectiveness of DPCore across various domain orders, further validating its applica- bility in real-world scenarios where the order of encountered domains may vary.
C.4. Comprehensive Results for Cityscapes-to-ACDC
C.3. Results for CIEARIO/IO-0O-CCIARROO000
In this section, we provide fine-grained results on Cityseapes-t-ACCDD presented in Table 15 and Table 16 for CSC and CDC settings respectively. We evaluate per- formance across three rounds and report the average mloU scores for each domain and round, offering a comprehen- sive view of adaptation effectiveness over repeated domain sequences.
In this section, we present the results on CIFAR 10/100C across both CSC and CDC settings. The results for CIFAR 10----00OAAARR ble 8 for CSC and CDC settings respectively. In the CSC setting, DPCore significantly outperforms all existing meth- ods, achieving + 12.8% improvement over the source pre-
are in Table 7 and Ta-
17

---

DPCore for Continual Test-Time Adaptation
Table Classification error rate (%) ) for ImageNet-to-ImageNerr in the CSC setting, Gaussian Noise-leared Prompt denotes the prompt learned solely on Gaussian Noise and evaluated across other domains. Domain Domain-specific prompt denote the prompt learned on each domain separately with known domain boundary.
Algoritlun
Gauss, Shot Impulse Defocus Glass
Motion
Zoom
(noo
Frost
Fog
Bright
Contrast
Elastic
Pixel
JPEG
Mean Gaint
Source
53.0
51.8
52.1
68.5
78.8
58.5
63.3
49.9
54.2
57.7
26.4
91.4
57.5
38.0
36.2
55.8
0.0
Tent
52.2
48,9
49.2
65.8
73.0
54.5
58.4
44,0
47.7
50.3
23.9
72.8
55.7
34.4
33,9
51.0
44.8
Gaussian Noise-lcarned Prompt
42.2
40.4
42.1
67.5
65.3
60.4
66.8
50.0
41.2
56.8
27.6
95.7
56.3
37.7
32.6
52.2
+3.6
Domain-spen specific Prompt
42.2
39.9
41.6
58.8
58.1
48.6
44.0
35.0
40.6
41.5
21.1
42.9
45.6
29.6
28.0
41.2
+14.6
DPCore (Proposed)
42.2
38.7
39.3
51.4
47.7
46.9
39.3
36.9
22.0
44.4
30.9
29.6
39,9
+15.9
Table 7. Classification error rate (%) for CIEARIO-tO- -IFA--OOOO- in the CSC setting,
Algorithm
Gauss.
Shot
Impulse Defocus Glass
Motion Zoom
Snow
Frost
Fog
Bright
Contrast Elastic Pixel
JPEG
MMan.]
Gaint
Source
60.1
53.2
38.3
19.9
35,5
22.6
18.6
12.1
12.7
22.8
5.3
49.7
23.6
24.7
23.1
28.2
0.0
Tent (Wang et al., 2021)
51.7
56.3
29.4
16.2
35.3
16.2
12.4
11.0
11.6
14.9
4.7
22.5
15.9
29.1
19.5
23.5
+4.7
COTTA (Wang al., 2022a)
58.7
51.3
33.0
20.1
34.8
20
15.2
11.1
18.5
4.0
34.7
18.8
19.0
17.9
24.6
+3.6
VDP et al., 2023)
57.5
49.5
31.7
21.3
35.1
19.6
15.1
10.8
10.3
18.1
4
27.5
18.4
22.5
19.9
24.1
+4.1
VIDA (Liu al., 2023b)
52.9
47.9
19.4
11.4
31.3
13.3
7.6
7.6
9.9
12.5
3.8
26.3
14.4
33.9
18.2
20.7
+7.5
DPCore(Proposed)
22.0
18.2
14.9
14.3
24.4
13.9
12.0
11.6
10.7
15.0
5.7
21.8
15.6
12.7
18.0
15.4
+12.8
D. Discussion on CTTA Setting.
termine the number of consecutive batches to assign to this domain, also using uniform distribution. If selected do- main has no remaining batches, we perform reselection until finding an available domain. This process continues until all batches across all domains have been assigned, naturally en- suring unpredictable domain changes with varying durations while maintaining complete coverage of all domains.
D.I. CDC with Different Dirichlet Distributions
We analyze how different domain change patterns affect CTTA methods by varying the Dirichlet distribution param- eter 5 that controls our CDC simulation. As shown in Fig. 6, smaller values (0.01, 0.1) produce domain changes more similar to CSC, while larger values (1, 10) lead to increas- ingly frequent transitions. DPCore demonstrates remarkable stability across all settings, with error rates increasing only modestly from 39.9% (CSC) to 43.1% (6 10), consis- tently outperforming previous SOTA methods. The stability of other methods, however, deteriorates significantly as ō increases. In the most challenging scenario (6 = 10) where domains change most rapidly, most previous methods per- form similarly to or worse than the source model (55.8%). While SAR maintains some improvement with 50.2% error rate, DPCore significantly outperforms all methods with 43.1% ,, demonstrating its robust adaptation capability even under extreme domain variation.
This uniform distribution- CDC-2D, offers an alternative perspective on dynamic do- main changes compared the Dirichlet approach in the main paper. As shown in Table 10, we observe two signifi- cant trends similar to our main results: First, previous SOTA methods suffer substantial performance degradation when moving from CSC to this CDC-2D setting, with several methods showing minimal improvement over the source model's 55.8% error rate and even the best baseline achiev- ing only moderate gains. This degradation is particularly evident in challenging corruption types where most meth- ods struggle to significantly improve upon the source model. Second, DPCore maintains robust performance with a 43. 43.2% error rate (comparable to 42.1% achieved with Dirichlet distribution) and consistently outperforms other methods, achieving a significant improvement of +12.6% over the source model while the second-best method only achieves +8.1
generation, which we term
D.2. CDC with Other Distributions
To provide a comprehensive understanding of our CDC setting beyond the Dirichlet distribution, we present a se- quence generation method based on two independent distri- butions: one for domain selection and another for domain duration (number of batches), termed as CDC-2D. This two-distribution approach allows to flexibly control both which domains appear and how long they persist, creating realistic scenarios of domain changes.
To further validate DPCore's 's effectiveness across diverse do- main change patterns, we explore CDC settings with differ- ent probability distributions. Taking TmageNNe-to--ImageNe C as an example with its 15 corruption domains, we consider probability vectors [PI, Pz..., Pis] where each P. repre- sents the probability of selecting the i-th domain. These vectors must satisfy P, > 0 for all (ensuring every domain has some probability of being selected) and E P; (making it valid probability distribution). The uniform
Taking uniform distributions as simple case, we iteratively construct the domain sequence through the following pro- cess: at each step, we randomly select domain from the domain pool using uniform probability, then randomly de-
115
18

---

DPCore for Continual Test-Time Adaptation
Table Classification error rate (%) ) for CIEARIO-to-
the CDC setting.
Algorithm
Gauss.
Shot Impulsc Defocus Glass Motion Zoom
Snow
Frost
Fog
Bright
Contrast
Elastic Pixel
JPEG
Mcan] Gain
Source
60.1
53.2
38.3
19.9
35.5
22.6
18.6
12.1
12.7
22.8
5.3
49.7
23.6
24.7
23.1
28.2
0.0
Tent (Wang al., 2021)
57.4
51.7
34.2
18.2
35.1
22.4
16.9
11.8
12.6
21.6
5.1
36.7
22.1
24.3
23.2
26.2
+2.0
CoTTA (Wang al,, 2022a)
59.2
54.8
45.1
20.5
36.1
21.9
19.1
11.6
12.1
21.2
6.1
35.1
22.5
21.7
19.7
27.1
+1.1
GDn et 2023)
58.1
51.2
32.2
22.4
34.8
20.4
15.6
11.2
11.4
20.1
4.9
30.4
20.4
22.7
19.9
25.0
+3.2
VIDA (Liu et al., 2023b)
56.4
50.8
28.4
16.7
32.5
15.2
13.7
10.1
10.4
13.5
4.2
26.5
19.4
35.1
21.2
23.6
+4.6
DPCove(Propose))
24.5
26.5
14.2
13.2
13.4
22.5
16.8
20.1
18.5
Table 9. Classification error rate (%) ) for SoangeNet-to--mmeNNeee in the CDC-2D setting,
Algorithm
Gauss.
Shot Impulse Defocws Glass
Motion
Zoom
(Snw
Frost
Fog
Bright
Contrust Elastic Pixel
JPEG
Mcan]
Gain
Source
$3.0
51.8
52.1
68.5
78.8
58.5
63.3
49.9
54.2
57.7
26.4
91.4
57.5
38.0
36.2
55.8
0.0
Tent (Wang et al., 2022a)
54.1
53.2
52,9
66.7
74.6
56.6
61.2
49.2
52.3
57.5
52.1
90.4
56.1
37.7
37.1
55.4
+0,4
CoTTA (Wang al., 2022a)
52.4
52.2
51.1
68.4
76.3
57.9
62.1
48.7
53.1
55.1
26.1
88.9
57.1
37.6
38.2
55.0
+0.8
VDP (Gan et al., 2023)
53.1
53.3
51.5
62.4
73.1
53.4
60.1
43.5
54.3
58.2
25.1
82.2
56.6
35.7
36.3
53.3
2.5
SAR (Niu al., 2023)
47.0
46.1
46.1
56.1
66.4
49.9
55.1
41.1
44,8
50.9
23.6
65,3
54.2
32.9
31.4
47.4
+8.1
EcOTTA (Song et al., 2023)
48.9
47.4
49.3
59.2
71.2
54.1
59.8
46.2
44.3
57.2
24.0
84,1
55.2
37.2
35.3
51.6
+4.2
VIDA (Liu et al., 2023b)
48.8
49.2
47.3
56.6
71.6
55.3
59.6
41.2
49.4
60.5
27.1
83.9
57.9
34.6
34.4
51.8
+4.0
DPCore (Proposed)
42.7
40.4
42.2
57.4
61.0
51.1
52.4
35.8
41.0
38.8
22.1
55.4
48.3
31.1
28.1
43.2
+12.6
distribution represents a special case where P; = 1 for all We randomly generate ten different such probability vectors, each defining unique pattern of domain changes. The results in Table 10 demonstrate DPCore's remarkable consistency across these varied settings, achieving mean error rate of 43.5% with standard deviation of 0.7. This stability across different domain change patterns further val- idates DPCore's 's robustness and its ability to handle diverse scenarios of distribution shift.
(Yuan et al., 2023) assumes label-balanced batches with local class correlation (e.g., single-class batches) in CSC. This contrasts with our setting, which follows the common CTTA assumption (Wang et al., 2022a; Niu et al., 2022; Liu et al., 2023b; 2024) of uniform sampling and label bal- ance. Interestingly, despite not being designed for PTTA, DPCore shows substantial improvement in this setting (re- sults in Table 11). Meanwhile, RoTTA (Yuan et al., 2023), though specifically designed for PTTA, demonstrates lim- ited effectiveness on ImageNet-C with ViTs compared to its stronger performance smaller datasets like CIFAR1O. consistent with findings in (Marsden et al., 2023). Further- more, RoTTA's performance degrades significantly in our CDC setting. In contrast, DPCore maintains robust perfor- mance across CSC, CDC, and PTTA settings, demonstrating its versatility across different test-time scenarios.
T5
D.3. Comparison with Other CTTA Settings
In this section, we discuss how our proposed CDC setting differs from other CTTA variants. First, we consider the re- peating setting introduced in (Wang al., 2022a), where 15 corruption domains repeated for 10 rounds. While this setting eventually accumulates numerous domain changes, it differs fundamentally from CDC: each domain maintains uniform duration (782 batches), and changes occur gradu- ally. This structured nature makes it more similar to CSC than our CDC setting, where domain durations vary signifi- cantly and changes occur more frequently.
Our CDC setting introduces unique challenges through its combination of frequent domain changes and varying do- main durations, distinguishing it from previous CTTA vari- ants. This more dynamic and unpredictable environment better reflects real-world scenarios while posing significant challenges for existing CTTA methods.
Second, the gradual domain change setting proposed in (Lee et al., 2024) assumes blurred domain boundaries, where batches near transitions contain mixed-domain data. While this realistic assumption differs from traditional CTTA meth- ods, it still follows the CSC pattern where mixed batches constitute only a small portion of all data due to long do- main durations. Although we haven't 't directly evaluated DPCore in this setting, our proposed variant DPCore-B B (de- tailed in Section E.I) demonstrates effective handling of mixed-domain batches, suggesting potential applicability to gradual domain changes.
E. Additional Ablation Studies
E.1. DPCore with Small Batch Size
While DPCore performs effectively with batch sizes of four or larger (as shown in the Fig. 5d), significant challenges emerge with extremely small batches, particularly in single- sample scenarios. To address this limitation, we propose DPCore-B (DPCore with Buffer Zone), which accumulates a predefined number of samples (D) in the buffer zone before performing any updates. During this accumulation period, temporarily store the [CLS] features from each
Third, the Practical Test-Time Adaptation (PTTA) setting
19

---

DPCore for Continual Test-Time Adaptation
Table 10. Average error rate (%) of DPCore across different CDC-2D settings (R1-10) on ImageNet-to-lmageNeeee
Source
Uniform
R1
R2
R3
R4
R5
R6
R7
R8
R9
R10
Average
Std
Err Mean..
55.8
43.2
42.1
42.8
44.2
43.5
43.5
42.7
43.9
44.4
44.7
43.0
43.5
0.7
Core's practicality in real-world applications. Following the insights from (Mehra et al., 2024a), we propose using public datasets that share similar or identical label spaces. For instance, in ImageNet-tor wrageNet-to-lmagNNee- or CIFARIO-to- CIFARIOC tasks, we leverage STL10 (Coates et al., 201 I), which shares label space with CIFAR 10 despite being dis- tinct from ImageNet.
Table 11. Average enror rate (%) of DPCore and RoTTA across various CTTA settings on LongeeNe-to-ImageNee
The improve-
ment over the souree model is shown in parentheses.
Algorithm CSC
PTTA
CDC
Average
Source
55.8
55.8
55,8
RoTTA
48.2 (+7.6)
49.5 (+6.35)
53.1 (+2.7)
50,3 (+5.6)
DPCore
39.9 (+15.9)
43.9 (+12.0)
(13.7)
(+13.9)
To ensure quality reference data, we filter the public dataset using the source model with an entropy- -hased threshold. Specifically, following (Niu et al., 2022), we set the thresh- old as 0.4 In C. where C' is the number of classes in the task (e.g., 1000 for ImageNet-to -ii---tt--mmeNN---C) We maintain consistency with our main experiments by selecting 300 un- labeled samples from STLI0 using this filtering strategy and compute statistics to replace the source statistics. Remark- ably, as shown in Fig. 5c, using these 300 filtered STL110 images as reference data for ImageNet-C experiments still achieves an error rate of 45.6% yielding a substantial 10.2% improvement over the source model. These results demon- strate that DPCore without access to source data, further establishing its practi- cal utility in real-world scenarios where source data might be unavailable due to privacy, security, or storage constraints.
Table 12. Average error rate (%) of DPCore-B across various huffer zone size D on Mo-geeer-o--mmggeNe-CCCC in CSC setting when batch size is one.
Source D= 8
D= 16
D= 32
D= 64
Err Mean
55.8
43.0
42.1
41.6
41.2
batch.
Our evaluation of DPCore-B in the CSC setting with various values of D shows promising results (Table 12), effectively maintaining DPCore's performance even with small batches. However, this success is limited the CSC setting where domain changes are structured and predictable. Even when accumulated samples span domain boundaries (violating the single-domain batch assumption shared by most CTTA methods, including ours), the impact remains minimal in CSC due to the limited number of domain transitions. For instance, on SrageNet-tt--Imageeee each domain has 782 batches, totaling 11,730 batches across 15 domains. Only 14 transition points potentially contain mixed-domain data, causing negligible impact overall performance.
maintain effective adaptation even
E.3. Details in Computational Complexity
We provide a detailed analysis of the computation and mem- ory efficiency of DPCore. In Table 4, we list the total number of trainable parameters required for ImageNet-to- ImageNet-C tasks. The parameter count for DPCore not static; increases with the addition of more core elements. The number of core elements depends on the number of unseen domains encountered. However, we demonstrate in Fig. 3c that the core set size remains stable across different domain orders of 15 corruptions in the CSC setting.
with batch size 64,
In contrast, CDC presents a fundamentally more challeng- ing scenario where domain transitions occur frequently and unpredictably. The substantially higher number of poten- tial mixed-domain batches in CDC violates our method's 's assumptions more severely, making DPCore-B ineffective. This limitation further demonstrates why CDC, with its fre- quent domain changes and varying durations, better reflects real-world challenges compared to conventional CSC set- tings. The development of effective small-batch adaptation strategies for CDC remains an important direction for future research.
For ImageNet- BongeNeettt--mmageee to-ImageNet-C tasks, the total number of parameters computed as the number of core elements (14) multiplied by the prompt length (8) and the dimen- sion of each prompt token (768). The counts for forward and backward propagations are averaged per batch. In DP- Core, new prompt is learned from scratch over 50 steps only when a test batch is evaluated as originating from a potential new domain. This results in the forward and backward operations being repeated for 50 steps for these batches. Notably, only 14 prompts undergo this extensive learning process out of 11,730 batches. For the remaining batches, DPCore updates the weighted prompt in a sin-
E.2. DPCore without Access to Source Data
In this section, we explore an extreme scenario where source data is completely inaccessible, further demonstrating DP-
20

---

DPCore for Continual Test-Time Adaptation
gle step (one forward and one backward operation) and then uses the updated weighted prompt refine all exist- ing prompts without additional forward or backward passes. Therefore, the average number of backward propagations per batch is 14x50+
F.2. Limitations
While our method significantly advances CTTA, it has certain limitations that may affect its broader application. Firstly, DPCore requires access to source statistics or ref- erence data, which may not always be available, especially in scenarios where the source data is proprietary or sensi- tive. Secondly, DPCore assumes that each batch of data comes from the same domain, which may not hold true in mixed-domain environments. In real-world applications, data streams could contain mix of samples from various domains, and the method's performance in such scenarios remains to be investigated. Furthermore, while DPCore demonstrates impressive efficiency compared to other meth- ods, it still introduces additional computational overhead during the adaptation phase, which may pose challenges in resource-conee conotraec environments or applications with strict latency requirements.
R: 1.06. For forward prop- agation, each batch requires two additional forward passes (one without prompt, one with weighted prompt) for eval-
11730
uation, leading to an average of 1.06 + 2 = 3.06 forward passes.
The computational overhead of learning new prompts be- comes negligible given the large number of total batches. Additionally, the parameters introduced by the prompts are minimal compared to the parameter increases required by other CTTA methods. Consequently, our prompt parameters do not require any warm-up, which conserves memory and reduces computation both before and during adaptation.
F. Discussions and Limitations
F.1. Potential Failure Cases
Our method, DPCore, is designed be robust against sce- narios such as Overlapping Distributions, Noisy or Small Domain Shifts, and Inconsistent Weighting Effects being inherent failure cases. The core objective of DPCore is not to identify an identical number of prompts as domains or to pinpoint every distinct domain. Instead, aims to up- date the same prompt for groups of similar domains. This is supported by our findings (Sec. 4.3 and Appendix C.1) that prompt learned for one domain (e.g., Gaussian Noise) can be effective for a similar domain (e.g., Shot Noise) and can be further improved with minor updates. Due to the use of distance-induced weights, a new domain can decomposition of existing do- mains, allowing the weighted prompt perform well even if derived from different domains, domain similarity is assessed by distance and prompts are learned through dis- tance minimization. A more significant potential failure case arises when each test batch contains data from multiple domains. In this situation, batch statistics would become unstable, potentially leading DPCore to treat each batch as new, distinct domain. This would significantly reduce efficiency, the algorithm would need learn prompts from scratch for each batch, a scenario analogous to our experimental setup in Table 3 Exp-2 (where prompts were learned from scratch for each batch, though each batch was from a single domain). Such multi-domain batches pose a challenge to most CTTA methods, which typically assume single-domain test batches. We identify this as an avenue for future investigation.
effectively represented
21

---

DPCore for Continual Test-Time Adaptation
Table 13.
Classification error rate (%) for SETAR100-O-CIFARRII in the CSC setting.
Algorithm
Gauss.
Shot Impulse
Defocus Glass Motion Zoom
Snow
Frost
Fog
Bright
Contrast
Elastic Pixel
JPEG
Mean]
Gaint
Source
55.0
51.5
26,9
24.0
60.5
29.0
21.4
21.1
25.0
35.2
11.8
34.8
43.2
$6.0
35.9
35.4
0.0
Tent (Wang al., 2021)
22.3
26.5
19.0
21.0
30.1
11.8
25.2
39.0
47.1
COTTA (Wang
2022a)
25.8
59.2
28.9
11.7
31.7
40.4
55.7
+0.6
VDP (Gan et al., 2023)
54.8
51.2
25.6
24.2
59.1
28.8
21.2
20.5
23.3
33,8
7.5
11.7
32.0
51.7
35.2
32.0
+3.4
ViDA (Liu al., 2023b)
50.1
40.7
22.0
21.2
45.2
21.6
16.5
17.9
16.6
25.6
11.5
29.0
29.6
34.7
27.1
27.3
+8.1
DPCore (Proposed)
48.2
40.2
21.3
20.2
44.1
21.1
16.2
18.1
15.2
22.3
9.4
13.2
28.6
32.8
25.5
25.1
+10.3
Table 14.
Classification error rate ( (%) for CIFAR100-to- CTFARTOOC C in the CDC setting.
Algorithm
Gauss.
Shot
Impulse
Defocus
Glass
Motion Zoom
Snow
Frost
Fog
Bright
Contrast
Elastic Pixel
JPEG
Mean.
Gain
Source
55.0
31.5
26.9
24.0
60.5
29.0
2144
21.1
25.0
35.2
11.8
34.8
43.2
56.0
35.9
35.4
0.0
Tent (Wang al., 2021)
53.5
26.2
25.8
61.0
28.1
23.5
21.7
22.6
11.0
24.4
41.3
49.1
36.7
+1.9
CoTTA (Wang al., 2022a)
27.1
60.5
29.2
22.5
35.6
13.7
33.7
41.6
59.1
29.4
35.9
-0.5
VDP (Gan al., 2023)
56.6
31.8
29.1
63.9
33.9
23.5
25.7
29.9
38.5
12.1
15.5
34.0
53.8
39.6
36.1
-0.7
ViDA (Liu al., 2023b)
50.1
42.1
23.9
23.3
48.1
23.7
19.5
18.7
18.5
29.6
11.6
36.1
32.6
37.1
32.8
29.8
+5.6
DPCore (Proposed)
54.0
42.5
23.5
22.8
45.3
21.4
18.6
21.2
16.8
23.2
10.0
15.1
35.7
34.6
25.9
27.4
+8.0
Table 15. mIoU score for Cityscapes-t-AACDCC in the CSC setting. The same target domains are repeated three times.
Round
Round
Round 3
Algorithm
Meant
Gaint
Fog
Night Rain
Snow
Mean
Fog
Night
Rain Snow
Mean
Fog
Night
Rain Snow
Mean
Source
69.1
40.3
59.7
56.7
69.1
40.3
59.7
57.8
56.7
40.3
57.8
$6.7
Tent (Wang et al., 2021)
40.2
60.1
57.3
56.7
39.0
60.1
56.3
55.9
67.5
37.8
55.0
55.7
-1.0
COTTA (Wang et al., 2022a)
70.9
41.2
62.4
59.7
58.6
70.9
41.1
62.6
59.7
58.6
70.9
41.0
62.7
59.7
58.6
58.6
+1.9
DePT (Gao et al., 2022)
71.0
40.8
58.2
56.8
56.5
68.2
40.0
55.4
53.7
54.3
66.4
38.0
47.3
47.2
49.7
53.4
-3.3
VDP (Gan et al., 2023)
70.5
41.1
62.1
59.5
58.3
70.4
41.1
62.2
59.4
58.2
70.4
41.0
62.2
59.4
58.2
58.2
+1.5
SAR (Niu et al., 2023)
69.0
40.2
60.1
57.3
56.7
69.0
40.3
60.0
67.8
59.3
67.5
37.8
59.6
55.0
$5.0
$7.0
+0.3
EcoTTA (Song al., 2023)
68.5
35.8
62.1
57.4
56.0
68.3
35.5
62.3
57.4
55.9
68.1
35.3
62.3
57.3
55.8
55.8
-0.9
VIDA (Liu et al., 2023h)
71.6
43.2
66.0
63.4
61.1
73.2
44.5
67.0
63.9
62.2
73.2
44.6
67.2
64.2
62.3
61.9
+5.2
C-MAE (Liu al., 2024)
71.9
44.6
67.4
63.2
61.8
71.7
44.9
66.5
63.1
61.6
72.3
45.4
67.1
63.1
62.0
61.8
+5.1
DPCore (Proposed)
71.7
47.2
63.3
62.1
47.8
66.5
63.1
62.6
73.4
47.8
67.1
62.7
62.8
62.5
+5.8
Table 16. mloU score for Cityccapes-o--ACDC in the CDC setting. The same target domains repeated three times.
Round
Round Rain
Round Rain Snow Mean"
Algorithm
Mean
Gain
Fog
Night
Rain
Snow
Mean
Fog
Night
Snow
Mean
Fog
Night
Source
69.1
40.3
59.7
57.8
56.7
69.1
40.3
59.7
57.8
56.7
69.1
40.3
59.7
57.8
56.7
56.7
0.0
Tent (Wang et al., 2021)
68.5
39.8
59.7
58.3
56.6
67.0
38.1
58.9
55.3
54.8
66.1
36.4
58.4
52.6
53.4
$4.9
-1.8
CoTTA (Wang et al., 2022a)
70.1
41.2
61.9
58.6
58.0
69.7
41.0
61.4
57.2
57.3
68.7
40.5
60.3
56.4
56.5
57.3
+0.5
DcPT (Gan et al., 2023)
69.5
40.2
58.7
56.2
56.2
68.4
39.8
58.3
55.7
55.6
66.7
39.1
57.9
54.3
54.5
55.4
-1.3
VDP (Gan al., 2023)
70.5
40.8
61.9
59.2
58.1
70.1
40.3
61.2
57.8
57.4
70.0
39,2
59.9
57.6
56.7
57.4
+0.7
SAR (Niu al., 2023)
69.1
41.6
57.5
57.0
41.2
59.7
57.4
56.8
40.8
57.0
56.5
56.8
+0.1
RcoTTA (Song al., 2023)
68.4
34.6
61.8
57.4
55.6
68.1
34.3
61.6
57.2
55.3
33.4
61.7
57.0
55.1
55.3
-1.4
ViDA (Liu et al., 2023b)
71.0
42.1
64.2
62.9
60.1
41.5
62.1
62.1
59.1
70.2
40.9
61.5
61.6
59.2
+2.5
DPCore (Proposed)
71.9
46.3
64.1
63.3
61.4
71.6
45.1
63.4
63.1
60.8
71.6
44,2
63.9
63.2
60.7
61.0
+4.3
22


Reviews:
[{"review_id": "n79XuBsk7Z", "reviewer": "Reviewer_92up", "summary": "The paper focuses on continual test-time adaptation. Under a more complex setting where domains recur with varying frequencies and durations, the paper proposes DPCore with a dynamically updated prompt coreset for the adaptation on different distributions. The experiments on both common continual test-time adaptation and the proposed dynamically continual test-time adaptation settings demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer and address each point:\n## Q1. Value of Proposed CDC Setting\nOur CDC setting models real-world scenarios with irregular distribution shifts where domains recur unpredictably (Fig.1), unlike existing CTTA approaches that assume uniform changes. For example, an autonomous vehicle driving through a mountainous area would encounter brief tunnels, extended sunny stretches, and intermittent fog or rain with unpredictable patterns and durations. SOTA methods degrade significantly in CDC (ViDA's error rate increases from 43.4% to 52.1%), highlighting that CDC presents new challenges not adequately addressed by existing settings. CDC amplifies three critical limitations in existing settings: convergence issues with brief domains, catastrophic forgetting with irregular patterns, and negative transfer between dissimilar domains.\n\nThese findings suggest CDC better reflects real-world challenges and provides a rigorous testbed for evaluating CTTA methods. Multiple reviewers recognized our approach: te3N noted CDC is \"**practical and aligns well with real-world dynamic changes**\" and \"**the proposed new setting is valuable**\" while jKst called it \"**novel and interesting**.\" These independent assessments validate our motivation. Differences between CDC and existing settings (e.g., SAR) are discussed in Appendix E.2.\n## Q2. Visual Prompt (VP) Implementation Details\nWe follow standard shallow VP implementation from [1], prepending prompt tokens to image tokens after the CLS token (Fig.2). The CLS token is invariant to the location of prompts since they are inserted after positional encoding ([1]). We explored using entropy minimization to learn VP at test time before settling on distribution alignment, but it led to degenerate solutions (increasing error rate by ~10% over the source model), which aligns with findings in recent work ([2]).\n\nIt's important to note that our main contribution is not how to learn VP, but rather DPCore's ability to manage domain knowledge through a dynamic coreset for CTTA. Our method remains effective when VP is replaced with NormLayer parameters (Table 3 Exp-3). For additional details, please see our response to Reviewer jKst's Q2. \n## Q3. Relationship between Feature Statistics and Prompts\nWhile feature statistics and prompts exist in different spaces, our objective function (Eq.5) creates a meaningful mapping between them through optimization. When we learn a prompt to minimize the distance between source and target statistics, we're effectively creating a prompt space that aligns with the statistical relationships in feature space. Our empirical evidence supports this approach: Table 5 shows prompts learned for similar domains transfer effectively between them; Fig.3a demonstrates our prompts consistently reduce domain gaps across all corruption types; and Fig.3c confirms stability across diverse domain orders. Essentially, the optimization creates a reliable bridge between statistical relationships and prompt functionality.\n## Q4. Comparison with SAR\nThere's a misunderstanding in the comparison. SAR addresses **single** domain (evaluating each domain independently) while we focus on **continually changing** domains (15 domains as one sequential task). SAR isn't designed for CTTA, so its performance drops. We use consistent hyperparameters (e.g., batch size=64) across all methods for fair comparison (detailed in Appendix B).\n\nThe three challenging settings in SAR are originally designed for **single** domain, not **changing** CTTA scenarios. But we further address these for CTTA in Appendix E.3 and F.1:\n1. **Batch size=1**: Fig.5d shows all CTTA methods struggle with small batches in changing domains. Our DPCore-B variant (Appendix F.1) addresses this by using a negligible buffer to accumulate sample features, achieving 41.2% error rate with single-sample batches (vs. 39.9% with batch size=64).\n2. **Imbalanced label shifts**: Though our method wasn't specifically designed for this challenge, it achieves better performance in this setting (improving the error rate to 43.9% as shown in Table 10).\n3. **Mixed domain**: In the CTTA setting, batches containing data from different domains usually occur near domain boundaries. We verified our method in this case with DPCore-B, which can still achieve an improvement of +14.6% over the source model (Table 11).\n## Q5. Computational Efficiency Concerns\nWe have carefully analyzed our method's computational efficiency (Table 4 and Appendix F.3). While DPCore does require additional computation compared to some simpler methods (e.g. Tent), it remains significantly more efficient than many SOTA approaches (e.g., CoTTA, ViDA). Our method requires a similar number of backpropagation operations (~1) but significantly fewer forward propagations (3.1) than CoTTA (11.7) and ViDA (11.0), which require forwarding extra augmented test data.\n\n[1] \"Visual Prompt Tuning\" ECCV2022\n\n[2] \"Test-Time Model Adaptation with Only Forward Passes\" ICML2024", "claims_and_evidence": "Most of the claims made in the submission are clear with evidence. \n\nHowever, as there are already many different settings for continual test-time adaptation [1, 2], it is not clear why the newly proposed setting (CDC) is unique, more challenging, and more like real-world applications. More discussions and evidence are required for the setting.", "methods_and_evaluation_criteria": "Overall, the proposed method and evaluation criteria sound good. \n\nHowever, some details of the method are not clear and are confusing.\n\n1. Visual prompt tuning has already been investigated by several papers and there are different strategies to introduce visual prompt into the transformer. How does the method decide the position and layers of the learnable prompt? Is there any experiment to demonstrate the strategy? How to decide the positional embedding of the learnable prompt?\n\n2. Typically test-time adaptation use entropy minimization for optimization, while the proposed method aligning the adapted target features with source features? What's the motivation and benefits to use such objective function? \n\n3. The proposed method use the feature statistics to generate prompts for new domains. However, the output features (statistics) and the input prompts are in different feature spaces. How can the method guarantee that the relationship (or similarities) between statistics can reflect the relationships between prompts?", "theoretical_claims": "The paper provides some theoretical analyses on how DPCore correctly assigns batches to respective clusters.", "experimental_designs_or_analyses": "The experiments include both the common continual test-time adaptation setting and the newly proposed setting. Results and ablation studies demonstrate the effectiveness of the proposed method. \n\nHowever, there are still some questions about the experiments.\n\n1. One question is whether the hyperparameters of the baseline methods in Table 4 are the same as the proposed method (e.g., batchsize = 64, 300 available source samples at test time). Because SAR (Niu et al., 2023) can achieve 43.7% error rate with batchsize=1 in their table 4, which is even lower than the number reported in this paper (45.6%). I assume that with a larger batch size, the results of SAR will be better.\n\n2. The previous methods also propose some different challenging settings for continual test-time adaptation, for example, the \"imbalanced label shifts\", \"mixture of different corruption types\", and \"with batch size=1\" in SAR (Niu et al., 2023). It looks like the proposed method doesn't perform well with \"batchsize=1\". Is there any analysis for the failure case? Can the proposed method handle the other two challenging settings?", "relation_to_broader_scientific_literature": "The paper is related to test-time adaptation and generalization problems, as well as continual learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow.\n\n2. The experiments demonstrate the effectiveness of the proposed method\n\nWeaknesses:\n\n1. It is not clear why the newly proposed settings for continual test-time adaptation is unique, challenging, and practical compared with the previous methods or settings.\n\n2. Some details of the method are not clearly stated and analyzed. Please refer to \"Methods And Evaluation Criteria\"\n\n3.  The comparisons and evaluations also need more details and analyses. Please refer to \"Experimental Designs Or Analyses\"\n\n4. Since the similarities are calculated between statistics without learnable prompts, each test batch requires two feedforward passes of the model for prompt generation and one more backpropagation for optimization. This will introduce much more computational costs, which is not so efficient.", "other_comments_or_suggestions": "Commonly, the related works should be included in the main paper for the readers to understand the task and previous methods.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "00SWUz0XS1", "reviewer": "Reviewer_koCY", "summary": "This paper utilizes a dynamic prompt coreset (DPCore) for continual test-time adaptation (CTTA). DPCore involves three components: visual prompt adaptation, prompt coreset, and a dynamic update mechanism for either updating the existing or, creating new prompts based on how similar the prompt is to the ones in the corset.\nExperiments on benchmark datasets have been reported, showing better performance compared to existing approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their careful reading of our paper and constructive feedback. Below, we address the specific concerns and questions raised:\n## Q1. Source Data Requirement\nWe appreciate this concern and would like to clarify:\n\n1. DPCore requires source data only before adaptation starts. In practice, feature statistics could be provided by the model publisher alongside the pre-trained model. Moreover, several SOTA methods (e.g., ViDA, EcoTTA, VDP, DePT, BeCoTTA) also require source data for preparation. The key difference is that these methods need to warm-up their parameters on the **entire** source dataset for several epochs, which demands significantly more data and computation than our approach. They require both forward and backward propagation on the entire labeled dataset, whereas our method needs only a single round of forward passes on 300 unlabeled examples to extract features.\n2. When no statistics are provided during preparation, they could be computed from the test data stream. In real-world scenarios, ID data is usually mixed with OOD data in the test batch stream. Filtering out 300 ID samples is straightforward using simple metric thresholds such as entropy.\n3. As demonstrated in Fig.5c, our method's performance remains stable even with as few as 50 unlabeled source examples.\n4. Most importantly, we show in Fig.5c and Appendix F.2 that DPCore can function effectively without **ANY** source data by using public datasets (e.g., STL10) as proxy reference data. Even in this extreme scenario, DPCore still outperforms the source model by +10.2% on ImageNet-C.\n\nTherefore, while we technically use source examples during preparation, our approach is source-data free during test time and can even operate without source data entirely, making it practical for real-world scenarios with limited or no access to source data.\n## Q2. Performance on CNNs\nWhile our paper primarily focused on ViTs due to their strong representation capabilities, we have conducted additional experiments on ResNet for the datasets mentioned. The choice of models for each dataset follows EcoTTA. For CNNs, instead of learning visual prompts, the coreset stores NormLayer parameters (same approach as Table 3 Exp-3):\n|Dataset|Model|Source|Tent|CoTTA|EcoTTA|PETAL|Ours|\n|-|-|-|-|-|-|-|-|\n|ImageNet-C|ResNet50|82.4|66.5|63.2|63.4|62.7|**61.0**|\n|CIFAR10-C|WideResNet-28|43.5|20.7|16.3|16.8|16.0|**15.7**|\n|CIFAR100-C|WideResNet-40|69.7|38.3|38.1|36.4|36.8|**35.4**|\n\nThese results demonstrate that DPCore consistently outperforms existing methods across all CNN architectures and datasets, confirming our approach's effectiveness beyond ViT architectures.\n## Q3. Novelty of Dynamic Prompt Coreset\nThe dynamic prompt coreset approach in our paper is indeed novel for test-time adaptation. While coresets have been explored in various machine learning tasks (e.g., continual learning), their application to managing prompts in TTA scenarios has not been previously investigated. The key novelty lies in:\n1. The design of a coreset specifically for storing and managing visual prompts in dynamic CTTA. Prior methods continuously update the same parameters across different domains, which leads to convergence issues with brief domain exposures, risks forgetting previously learned knowledge, or misapplies it to irrelevant domains. DPCore manages domain knowledge through a dynamic prompt coreset to mitigate these issues.\n2. The dynamic coreset approach is not restricted to prompts but can also be used for other types of parameters such as NormLayer parameters. It still improves the performance of the source model by +10.7% when applied to NormLayer parameters (Table 3 Exp-3). These results show that our dynamic coreset can be applied to other model architectures beyond ViT and integrated with other TTA methods.\n3. Our novel decision mechanism determines whether to update existing prompts or create new ones based on domain similarity, mitigating negative transfer between dissimilar domains while allowing coreset prompts to be updated on similar ones. This approach leverages domain similarities through weighted prompt generation, computing efficient combinations that ensure constant evaluation time regardless of coreset size.\n\nRelated work such as [1] explores prompt management for supervised continual learning, but requires labeled data and addresses a fundamentally different problem (discussed in lines 216–219). Similarly, online K-means clustering has been used in various applications, but our adaptation of these ideas to prompt-based CTTA with a dynamic update mechanism is novel.\n## Others\nWe will correct the Table 3 error (gain for Exp-3 should be +10.7, not +8.7) in the camera-ready version. The code will be published upon acceptance.\n\nWe appreciate the thorough review and constructive feedback and hope these clarifications address all concerns.\n\n[1] \"Learning to Prompt for Continual Learning\" CVPR2022", "claims_and_evidence": "Mostly yes. Refer to Weaknesses and Questions.", "methods_and_evaluation_criteria": "Mostly yes. Refer to Weaknesses and Questions.", "theoretical_claims": "I have read the theorems in the Appendix, but not checked their proof in detail.", "experimental_designs_or_analyses": "Refer to Weaknesses and Questions.", "relation_to_broader_scientific_literature": "The key contribution of the paper is that it enhances the TTA ability of a model. However, the contributions are limited to continual TTA and do not generalize much beyond it.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths**\n* An interesting idea of updating an existing prompt or adding a new prompt to the coreset based on the ratio of distances with or without the weighted prompt.\n* Experimental results show improvement over existing approaches over compared benchmarks.\n\n**Weaknesses**\n* The requirement of unlabeled source example data (even if only 300), violates the source data-free assumption.\n* CCN architecture experimental results on widely used benchmark is lacking (refer to Questions)", "other_comments_or_suggestions": "In Table 3, gain for Exp-3 should be +10.7 instead of +8.7.", "questions_for_authors": "1. How does the proposed approach perform on CIFAR10C, CIFAR100C, and ImageNetC on CNN-based architecture as discussed in the continual test-time adaptation line of work such as [1], [2], [3]?\n2. Is the idea of dynamically updating coresets for prompt novel? Or is the paper utilizing this idea from an existing work that explored it for a different problem?\n\n**References**\n1. Wang, Qin, et al. \"Continual test-time domain adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n2. Song, Junha, et al. \"Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n3. Brahma, Dhanajit, and Piyush Rai. \"A probabilistic framework for lifelong test-time adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xohtcM7b3q", "reviewer": "Reviewer_te3N", "summary": "This paper introduces DPCore, a novel approach to Continual Test-Time Adaptation (CTTA) that addresses challenges in dynamically changing environments where domains recur with varying frequencies and durations. DPCore employs Visual Prompt Adaptation for efficient domain alignment, a Prompt Coreset for knowledge retention, and a Dynamic Update Mechanism to intelligently adjust or create prompts based on domain similarity. Extensive experiments on four benchmarks show that DPCore achieves state-of-the-art.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their positive feedback and insightful questions. We appreciate the thorough review and the recommendation to accept our paper. Below, we address the specific questions and concerns raised:\n## Q1. Memory Efficiency Analysis\nIn Lines 216–219, we compare with [1] where test samples are directly stored in a buffer. Our approach stores only test batch statistics, requiring significantly less space (e.g., from 64×3×224×224 per batch to just 2×768, about 0.02% of the original size). Additionally, our memory analysis on ImageNet-C using a single RTX 6000 Ada GPU:\n|Algo.|Memory (MB)|Err Rate(%)|\n|-|-|-|\n|Tent|3,877|51.0|\n|CoTTA|8,256|54.8|\n|ViDA|5,728|43.4|\n|Ours|3,879|39.9|\n\nDPCore achieves minimal memory footprint by 1) storing only learned prompts (0.34 MB total) and compact statistics, 2) requiring no test sample storage, and 3) sharing core elements across similar domains.\n## Q2. Sensitivity Analysis of ρ Across Datasets\nWe've analyzed ρ sensitivity across datasets, with results in this [figure](https://anonymous.4open.science/r/DPCore-Supp-8D17/ablation_rho_all.png). DPCore maintains stable performance on ImageNet-C, CIFAR10-C, and CIFAR100-C for ρ values between 0.7 and 0.9. We fix ρ=0.8 for all main experiments.\n## Q3. Potential Failure Cases\nThis is an excellent question that helps us better understand the limitations of our approach. However, \"Overlapping Distributions\", \"Noisy or Small Domain Shifts\" and \"Inconsistent Weighting Effects\" are not necessarily failure cases for our method. The goal of DPCore is not to learn an identical number of prompts as the number of domains or to identify all different domains. Instead, DPCore aims to update the same prompt for similar domain groups. Specifically:\n1. This design is motivated by our findings in Sec 3.3 and Appendix D.1, where we observed that a prompt learned on one domain (e.g., Gaussian Noise) could work effectively on a similar domain (e.g., Shot Noise), and performance could be further improved by slightly updating the prompt on the second domain. \n2. Since distance-induced weights are used to generate the weighted prompt, a new domain can be effectively represented as a decomposition of existing domains. The weighted prompt might perform well on a new domain even when derived from different domains, which is acceptable in our framework since domain similarity is evaluated by distance and prompts are learned through distance minimization.\n\nOne potential failure case for our method would be when each test batch contains data from multiple domains. In this scenario, the statistics of each batch would not be stable, and our method might treat each batch as a different domain. This would reduce efficiency since the algorithm would need to learn prompts for each test batch from scratch, similar to what we show in Table 3 Exp-2 (learning prompts from scratch for each batch but the entire batch comes from the same domain). This scenario would pose challenges to most CTTA methods, as most assume each test batch contains data from a single domain. We plan to explore this direction in future work.\n## Q4. Memory Optimization for Prompt Coreset\nRegarding memory overhead concerns, maintaining prompts in memory is quite negligible in practice. For example, the 14 prompts learned for ImageNet-C domains only require 0.08M parameters (0.34MB). If we were to allocate the same number of parameters as ViDA (7.13M), we could store approximately 1,247 prompts, making DPCore highly memory-efficient in practice.\n\nWhile our current implementation has modest memory requirements, exploring potential optimizations aligns well with practical deployment considerations. We have explored two approaches to maintain a fixed-size coreset on ImageNet-C with K=15 (matching the number of domains). Once the number of prompts exceeds K, we either 1) discard the oldest prompt or 2) merge the most similar prompts (computing the average of the two prompts whose statistics distance is smallest). We evaluated these strategies in the 10-different-order setting (Fig.3c) and report the average error rates:\n|Algo.|Err Rate(%)|\n|-|-|\n|Source|55.8|\n|DPCore (flexible K)|40.2|\n|DPCore (K=15, discard)|42.3|\n|DPCore (K=15, merge)|41.1|\n\nBoth strategies still significantly improve upon the source model, though merging existing prompts performs slightly better than simply discarding one of them.\n\nIn real-world scenarios, the number of domains is typically unknown, and fixing K at a small value could lead to suboptimal performance. Since the coreset grows only when encountering unseen domains and the memory overhead is negligible, our default approach is to allow K to evolve naturally as adaptation progresses.\n\nWe appreciate the reviewer's thoughtful questions, which have helped us articulate our approach's strengths and limitations. We believe addressing these points will further strengthen the paper.\n\n[1] \"Learning to Prompt for Continual Learning\" CVPR 2022", "claims_and_evidence": "The authors mention that storing prompts and statistics is more memory-efficient (Lines 216–219). However, the authors do not provide a quantitative comparison. Can the authors present a memory consumption analysis, specifically comparing the storage requirements of their approach to storing coreset representations? This would help validate the claimed efficiency advantage.", "methods_and_evaluation_criteria": "1. The proposed new setting is practical and aligns well with real-world dynamic changes. The authors have also conducted studies to demonstrate its relevance and value.\n2.  The proposed method is novel and effectively addresses the challenges posed by the new setting.\n3.  The benchmark used in this paper is appropriate and provides a meaningful evaluation of the proposed method's performance.", "theoretical_claims": "NO theoretical claim", "experimental_designs_or_analyses": "The evaluation of the proposed method is comprehensive, and the authors have conducted extensive empirical studies to validate its effectiveness.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "The related works are adequately discussed.", "other_strengths_and_weaknesses": "The paper is well-written, and the proposed new setting is valuable. The method is novel and demonstrates strong performance in this setting. The authors have conducted extensive analysis and experiments, making the empirical results highly compelling.\n\nWeaknesses:\n\n1.\tSensitivity of the Hyperparameter \\rho: The new prompt is trained when a batch of data is identified as belonging to a new domain. However, the hyperparameter \\rho may be sensitive to the dataset. Can the authors provide a sensitivity analysis of  \\rho across different datasets and generate figures similar to Figure 5(b) to illustrate its impact?\n2.\tMemory Overhead of Storing Prompt Coresets: The proposed method requires retaining a prompt coreset, and as the number of domains increases, the coreset size may grow linearly. To improve memory efficiency, can the authors explore merging similar domain prompts to reduce storage requirements? An alternative approach could be maintaining a fixed-size coreset while ensuring that it remains representative, which would better align with practical memory constraints.", "other_comments_or_suggestions": "No additional comments", "questions_for_authors": "The authors use the distance between prompts with and without weighting to identify new domains. However, it is important to understand under what circumstances this method might fail to correctly identify a new domain.\n\nCan the authors explain the potential failure cases? Specifically, are there scenarios where:\n\n1.\tOverlapping Distributions: The prompt distances for new and existing domains are too similar, making it difficult to distinguish between them?\n2.\tNoisy or Small Domain Shifts: If the domain shift is subtle or gradual, could the method fail to detect a new domain?\n3.\tInconsistent Weighting Effects: Are there cases where the weighting mechanism leads to misleading prompt distances, potentially misclassifying an existing domain as new (or vice versa)?\n\nA discussion on these failure cases, along with possible mitigation strategies, would strengthen the paper’s robustness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ksA0FIBN2Z", "reviewer": "Reviewer_jKst", "summary": "This paper proposes DPCore for Continual Test-Time Adaptation. It integrates a Visual Prompt Adaptation for efficient domain alignment, a Prompt Coreset for knowledge preservation, and a Dynamic Update mechanism. Extensive experiments on four benchmarks demonstrate that DPCore outperforms existing CTTA methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback and address the specific concerns raised:\n## Q1. Hyperparameter Analysis\nWe determine hyperparameters using four disjoint validation corruptions from ImageNet-C and CIFAR10-C. The same hyperparameters (detailed in Sec 4.1, Appendix C.3) are used across all experiments. Additional results on ImageNet-C:\n### 1. Impact of Temperature τ\nThe temperature τ in Eq.6 controls weight assignment softness. We evaluated τ in [0.1, 10.0]:\n|τ|0.1|0.3|0.5|1.0|3.0|5.0|10.0|\n|-|-|-|-|-|-|-|-|\n|Err Rate(%)|45.1|44.7|42.9|40.2|39.9|40.3|43.5|\n\nDPCore is stable for τ between 1.0 and 5.0. We used τ=3.0 for all experiments.\n### 2. Impact of Update Weight α\nWe use exponential moving average in updating mechanism (Eq.7). We didn't tune α but set α=0.999 reported in [1] (comparable to ViDA: 0.999, CoTTA: 0.99). We evaluated α in [0.7, 0.999] and found stable performance when α≥0.9:\n|α|0.7|0.8|0.9|0.95|0.99|0.999|\n|-|-|-|-|-|-|-|\n|Err Rate(%)|42.1|41.3|40.2|40.0|39.6|39.9|\n### 3. Impact of Coreset Size K\nThe parameter K is not fixed but grows dynamically as new domains are encountered, better aligning with real-world scenarios where the number of unseen domains is usually **unknown**. It is not a hyperparameter that needs to be specified but rather an outcome of the algorithm. More results and discussion can be found in Fig.3b and our response to Reviewer te3N Q4.\n## Q2. Contribution of Visual Prompt Adaptation (VPA)\nWe agree that VPA itself is not novel, as noted in our paper with citations. Our key contribution is not the VPA component in isolation but how it is integrated within the **dynamic coreset** framework:\n1. **VPA as a Practical Component**: While not new, VPA offers a lightweight, efficient adaptation mechanism well-suited for our dynamic coreset approach. As shown in Table 3 (Exp-1), using VPA alone achieves only +5.0% improvement, but when combined with our dynamic coreset, it achieves +15.9%.\n2. **VPA is Replaceable**: Table 3 (Exp-3) shows our dynamic coreset approach isn't dependent on VPA. Replacing VPA with NormLayer parameters still achieves strong performance (+10.7%), highlighting that our main contribution is the dynamic coreset approach rather than VPA. \n3. **Efficient Domain Alignment**: Our VPA implementation offers computational advantages in changing environments. It requires minimal parameters (0.1% of model parameters) and few source examples (300 unlabeled), making it practical for real-world deployment.\n\nIn summary, while VPA itself isn't our core contribution, its integration in our dynamic coreset framework is key to DPCore's effectiveness. Our novelty lies in how we manage domain knowledge through the dynamic coreset, not in the specific adaptation method (e.g., VPA).\n## Q3. Updating Mechanism Concerns\nOur updating mechanism is novel and effective for several reasons:\n1. It works in tandem with our objective function (Eq.5), which creates a meaningful mapping between feature statistics and prompts, effectively reflecting the domain gaps. Empirical results support this: Table 5 shows that prompts learned for similar domains (with similar distances) transfer effectively between them; Fig.3a shows our prompts consistently reduce domain gaps; Fig.3c confirms stability across diverse domain orders.\n2. The mechanism only activates when the current batch belongs to a seen domain or is similar to seen domains. For completely new domains (with large domain gaps), it is not used. Instead, existing core elements stay unchanged while a new prompt is learned from scratch and added to the coreset to prevent negative transfer.\n3. Evaluating each prompt separately and selecting the closest one poses challenges (Sec 3.3): 1) It processes same test batch multiple times (linear to the coreset size), increasing computation. Linear combination reduces computation to constant time (only once); 2) Nearest neighbor selection might diminish coreset power, particularly for unseen domains which could be viewed as decompositions of known domains; 3) Increasing temperature τ to prioritize the nearest neighbor does yield heavier weighting, but performance drops (see Q1.1 τ table).\n4. The linear combination used in Eq.7, while simplifying domain relationships, is highly effective. Table 1 and Fig.4b show DPCore consistently outperforms more complex methods across both CSC and CDC settings. We chose this approach for its efficiency in handling not only visual prompts but also other parameters (e.g., NormLayer parameters from Q2.2), balancing effectiveness and simplicity, and achieving SOTA performance with computational efficiency.\n\nWe appreciate the reviewer's feedback on our work. We believe DPCore makes a significant contribution to continual test-time adaptation, particularly for the challenging CDC setting that better reflects real-world scenarios.\n\n[1] \"Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results\" NeurIPS2017", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "The proposed DPCore presents a novel solution for the Continual Dynamic Change (CDC) setup.", "theoretical_claims": "The Comprehensive Analysis of DPCore in section C.4 has been checked.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The Continual Dynamic Change (CDC) setup is novel and interesting, this paper explores to address this new problem in a memory efficient way.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths:\n1. The idea that addressing Continual Dynamic Change (CDC) setup is interesting;\n2. Using Online K-Means to generate Prompt Coreset is reasonable.\n\nWeaknesses:\n1. The visual prompt adaptation is not a novel solution for domain alignment;\n2. This paper requires extra hyperparameters, e.g. K, the temperature \\tau and \\alpha, while the authors do not clarify their settings.\n3. The updating mechanism is not very persuasive, as the linear combination of the coreset element in Eq. 7 can not depict the domain gaps.", "other_comments_or_suggestions": "no", "questions_for_authors": "1. Theoretically, the hyperparameter K, the temperature \\tau and \\alpha will affect the final results, why don’t you provide detailed ablation analyses?\n2. What’s the major contribution of the visual prompt adaptation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new TTA framework called DPCore that managed domain knowledge through a coreset of prompts updated dynamically during test, as well as a more challenging test-time adaptation (TTA) scenario, dubbed continual dynamic change (CDC). The proposed method demonstrated outstanding performance on both conventional and CDC settings. 

The reviewers recognized the intriguing idea of managing and exploiting the dynamic prompt coreset, extensive experiments, strong performance, and clarity of the paper. However, they at the same time raised concerns with unclear motivation and contribution of the CDC setting (92up), lack of details, motivation, and justification for the proposed method (92up), insufficient details and analysis in evaluations of baseline models (92up), violating the fully test-time adaptation assumption (koCY), missing analysis on memory-efficiency (te3N), potential sensitivity to extra hyperparameters (te3N, jKst), and questionable effectiveness of the updating mechanism (jKst). The rebuttal and subsequent discussions addressed most of these concerns and in the post-rebuttal phase, all reviewers except one turned towards acceptance. The negative reviewer did not come back, but the AC found that his/her remaining concern on the updating scheme has been addressed successfully by the last comment by the authors. 

The AC found that the strengths of the paper and the rebuttal outweigh the remaining concerns, and thus recommend acceptance of the paper. The authors are strongly encouraged to carefully revise the paper to reflect the valuable comments by the reviewers and to add new results brought up in the rebuttal and discussions.
Decision: Accept (poster)