ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
Authors: Jiaming Liu
Senqiao Yang
Peidong Jia
Renrui Zhang
Ming Lu
Yandong Guo
Wei Xue
Shanghang Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Continual test-time adaptation
Visual Adapter
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We design homeostatic low-rank and high-rank Visual Domain Adapters (ViDA) for Continual Test-Time Adaptation, addressing error accumulation and catastrophic forgetting simultaneously.

Abstract: Since real-world machine systems are running in non-stationary environments, Continual Test-Time Adaptation (CTTA) task is proposed to adapt the pre-trained model to continually changing target domains. Recently, existing methods mainly focus on model-based adaptation, which aims to leverage a self-training manner to extract the target domain knowledge. However, pseudo labels can be noisy and the updated model parameters are unreliable under dynamic data distributions, leading to error accumulation and catastrophic forgetting in the continual adaptation process. To tackle these challenges and maintain the model plasticity, we design a Visual Domain Adapter (ViDA) for CTTA, explicitly handling both domain-specific and domain-shared knowledge. Specifically, we first comprehensively explore the different domain representations of the adapters with trainable high-rank or low-rank embedding spaces. Then we inject ViDAs into the pre-trained model, which leverages high-rank and low-rank features to adapt the current domain distribution and maintain the continual domain-shared knowledge, respectively. To exploit the low-rank and high-rank ViDAs more effectively, we further propose a Homeostatic Knowledge Allotment (HKA) strategy, which adaptively combines different knowledge from each ViDA. Extensive experiments conducted on four widely used benchmarks demonstrate that our proposed method achieves state-of-the-art performance in both classification and segmentation CTTA tasks. Note that, our method can be regarded as a novel transfer paradigm for large-scale models, delivering promising results in adaptation to continually changing distributions.

Published as a conference paper at ICLR 2024
VIDA: HOMEOSTATIC VISUAL DOMAIN ADAPTER FOR CONTINUAL TEST TIME ADAPTATION
Jiaming Liu', Senqiao Yang', Peidong Jia 11 Renrui Zhang :
Ming Lu', Yandong Guo?, Wei Xue*, Shanghang Zhang'
NNtiooaa Key Laboratory for Multimedia Information Processing,
School of Computer Science, Peking University 2 RooRobosscs
3 'The Chinese University of Hong Kong Hong Kong University of Science and Technology jiamingli @ stu.pku. stu.pku.edu.cn, yangsenqiao. ai gmail.com, shanghang @ww...pk.u.......
ABSTRACT
Since real-world machine systems are running in non-stationary environments, Continual Test-Time Adaptation (CTTA) task is proposed to adapt the pre-trained model to continually changing target domains. Recently, existing methods mainly focus model-based adaptation, which aims to leverage self-training manner to extract the target domain knowledge. However, pseudo labels can be noisy and the updated model parameters unreliable under dynamic data distributions, leading to error accumulation and catastrophic forgetting in the continual adaptation process. To tackle these challenges and maintain the model plasticity, we design a Visual Domain Adapter (ViDA) for CTTA, explicitly handling both domain- specific and gomain-shaard knowledge. Specifically, we first comprehensively explore the different domain representations of the adapters with trainable high- rank or low-rank embedding spaces. Then we inject ViDAs into the pre-trained model, which leverages high-rank and low-rank features to adapt the current domain distribution and maintain the continual Nomain-sharce knowledge, respectively. To exploit the low-rank and high-rank ViDAs more effectively, we further propose Homeostatic Knowledge Allotment (HKA) strategy, which adaptively combines different knowledge from each ViDA. Extensive experiments conducted on four widely used benchmarks demonstrate that our proposed method achieves state-of- the-art performance in both classification and segmentation CTTA tasks. Note that, our method can be regarded as a novel transfer paradigm for large-scale models, delivering promising results in adaptation continually changing distributions.
INTRODUCTION
Deep Neural Networks (DNN) have achieved remarkable performance in various computer vision tasks, such as classification (He et al. a 2016), object detection (Zhu et al., (2020), and segmentation (Xie) al. (021) when the test data distribution is similar to the training data. However, weal-world machine perception systems (Yang et al. 2023a Li et al. 2023) Arnold et al. 2019) operate in non-stationary and constantly changing environments, which contain heterogeneous and dynamic domain distribution shifts. Applying a pre-trained model in these world tasks (Sakaridis et al. (02T)) can lead to significant degradation in perception ability on target domains, especially when the target distribution changes unexpectedly over time. Therefore, the development of continual domain adaptation (DA) methods is essential for enhancing the generalization capability of DNNs and improving the reliability of machine perception systems in dynamic environments.
classical source-free DA task, Test-Time Adaptation (Liang al., 2023) (TTA), eases the distri-
bution shift between a source domain and a fixed target domain. This is typically achieved through the utilization of self-training mechanisms (Mummadi et al., 2021 Wang al. 22020). However, when adapting to continually changing target domains, pseudo labels are noisy and the updated
Equal contribution. Equal technical contribution, e Corresponding author, Project page: http:://wweeeoooo.ccmmmmr///ooer

---

Published as a conference paper at ICLR 2024
Domaiin 1
Domain (Night)
Pinmsiae (Kain)
Ours: Low-rank Branch
The Pre-trained Parumeter
Ours: Hligh-rank Branch
Continually Changing Target Favirenments
Domain shared
Domain sprcifie
100
Low-Rank
Pre-trained
Parameter
High-Ronk Konk
Online
Demain-Shored
Demain-qpeeiffe
Network
Adapted
[ Demain (Fog)
Domoii (Night)
DDomain (Rain)
Demain 4 (Smon))
(1) Continual Teme-Timee Adaptation
(b) Feature representations of different branches
The problem and motivation. (a) Our goal is to effectively adapt the source pre-trained model to continually changing target domains. We propose Visual Domain Adapters with high- rank and low-rank embedding spaces tackle the error accumulation and catastrophic forgetting challenges during the continual adaptation process. (b) we conduct a t-SNE (Van der Maaten &
Figure 1:
Hinton 2008) distribution analysis for the different adapter representations across four target domains (ACDC). The low-rank branch exhibits a consistent distribution across the target domains, suggesting that it can effectively disregard the impact of dynamic distribution shifts. The high-rank branch demonstrates noticeable distribution discrepancies between the various target domains, suggesting that it primarily focuses on extracting domain- specific knowledge.
model parameters become uncertain, leading to error accumulation and catastrophic forgetting. To tackle this problem, Continual Test-Time Adaptation (CTTA) has been proposed (Wang et al.) 2022). which addresses sequence of different distribution shifts over time rather than single shift in TTA. Furthermore, CTTA also encompasses the efficient continual adaptation of foundation models
(Kiiillov al., 2023 to continual downstream tasks or distributions (Bahng et 2022).
Existing CTTA works have primarily employed model-based or prompt-based approaches to extract target domain-specific and domain-shared knowledge simultaneously. However, for model-based methods (Wang et al. 2022 Chakrabarty et al. (2023), the noisy pseudo labels are still unreliable and play limited role in avoiding error accumulation, particularly in scenarios with significant distribution gaps. Meanwhile, prompt-based methods (Gan et al. 2023 Yang et al. (023b) face difficulties in leveraging soft prompts with limited trainable parameters to learn long-term domain- shared knowledge and prevent catastrophic forgetting.
To tackle these limitations and maintain the model plasticity, we design a homeostatic Visual Domain Adapter ( (ViDA), shown in Fig (a), which explicitly manages domain-specific and domain-shared knowledge in the continual adaptation process. Specifically, we first carefully explore the different domain representations of ViDAs with trainable high or low-dimension embedding space in the middle layer. As shown in Fig. (bb, our observations reveal that ViDA with a low-rank embedding space focuses on task-relevant feature representation, showing trivial distribution distance in different domains and neglecting the influence of dynamic distribution shifts. Conversely, ViDA with a high-rank feature concentrates more on extracting domain-specifie knowledge, as evidenced by the feature distribution in different target domains showing an obvious discrepancy. We provide a detailed explanation of the motivations in Section 3.1 and Appendix B
This observation motivates us to inject ViDAs into the pre-trained model, which leverages different domain representations of high and low-dimension features to avoid error accumulation and catas- trophic forgetting simultancously. To better extract different domain knowledge, we further propose a Homeostatic Knowledge Allotment (HKA) strategy to dynamically fuse the knowledge from low-rank and high-rank ViDA. Based on the data distribution, HKA adaptively regularizes the balance of different feature representations, including original model, domain-specific, and rele-atl features. During inference, the low-rank and high-rank ViDAs can be projected into the pre-trained model by re- parameterizzaion (Ding et al., 2021). which ensures no extra parameter increase and maintains the model plasticity. In summary, our contributions are follows:
1) We study the different domain representations of the adapters with high-rank and low-rank features. Then we design Visual Domain Adapter (ViDA), explicitly managing domain-specifft and task- relevant knowledge to tackle the error accumulation and catastrophic forgetting problem, respectively.
2) Considering the various distribution shifts for each target sample, we further propose a Homeostatic Knowledge Allotment (HKA) strategy to dynamically merge knowledge from low-rank and high-rank ViDAs, thus enhancing ViDAs' distinct domain representations.
2

---

Published as a conference paper at ICLR 2024
3) Our CTTA method provides novel transfer paradigm for large-scale models, delivering promising results in adaptation to continually changing distributions. Meanwhile, we empower the source model with domain generalization ability through the proposed homeostatic ViDAs, achieving significant improvement on the unseen target domains.
2 RELATED WORK
Test-time adaptation (TTA), also referred to as source-fiee freee domain adaptation (Boudiaf et al. 2023 Kundu et al. 2020; Yang et al., 202T), aims to adapt a source model to an unknown target domain distribution without relying any source domain data. Recent research has explored self-training and entropy regularization techniques to fine-tune the source model (Liang et al. 2020 Chen et al., 2022). Tent (Wang et al.) 202I) updates the training parameters in batch normalization layers by minimizing entropy. This approach has prompted subsequent exploration in recent works (Niu et al.)
2023
Yuan et al. 2023), which continue to investigate the robustness of normalization layers.
Continual Test-Time Adaptation (CTTA) refers to a scenario where the target domain not static, presenting additional challenges for traditional TTA methods. The first approach address this challenging task is introduced in (Wang et al. 2022), which combines bi-average pseudo labels and stochastic weight reset. For addressing error accumulation, Ecotta (Song al. 2023) introduces meta-network to regularize the outputs from both the meta-network. and the frozen network. And
RMT (Döbler et al., 2023) introduces a symmetric cross-entropy loss. While these works tackle the
CTTA problem at the model level, (Gan et al., 2023 Yang et al., 2023b Ni et al.) (023) utilize visual domain prompts or small fraction of parameters to extract continual target domain knowledge.
Parameter-Efficien Tunnng (PEFT) has gained significant traction within the field of natural
language processing (NLP) (Hu et al.] 2021 Houlsby et al. 2019 Zaken et al., [021] Hu et al.) 2022
Gao et al. 2021: He et al. 2021 Vu et al. 2022 Qin et al. 2021). Adapter-based models, a form of
PEFT, have gained popularity in NLP. They employ bottleneck architecture adapter modules inserted between layers in pre-trained models. Inspired by NLP, adapters in visual tasks have also received widespread attention. In the initial phases adapter development, residual adapter modules (Rebulfi et al. 2017 2018) are proposed to aid in the effective adaptation of convolutional neural networks across multiple downstream tasks. AdaptFormer (Chen et al.) enhances the VIT (Dosovitskiy al. 2020) model by replacing the original multi-layer perceptron (MLP) block with a down-to-up bottleneck module in a parallel manner. VL VL-Adapter (Sung et al.) (022) improves the efficiency and performance of adapters by sharing low-dimensional layer weights to attain knowledge across tasks.
3 METHOD
Preliminary. In Continual Test-Time Adaptation (CTTA), we pre-train the model ( x)) on the source domain Dg = (Ys Xs) and adapt it multiple target domains Dr; ((Xx;)) where n represents the scale of the continual target datasets. The entire process can not access any source domain data and can only access target domain data once. The distributions of the target domains (i.e., Dr, Dr,,..., Dr.,) ) are constantly changing over time. Our goal is to adapt the pre-trained model to target domains and maintain the perception ability of the model the seen domain distributions.
Overall Framework. Drawing from the insight that mean teacher predictions are often more robust than standard models (Tarvainen & Valpolal 2017b Döbler al., 2023), we utilize a teacher- student framework to ensure stability during continual domain adaptation, which also presents fair
comparison with previous CTTA works (Wang et al. 2022 Gan al. 2022). The overall framework and the details of our method are shown in Fig
3.1 MOTIVATION
The CTTA encounters significant challenges, primarily due to error accumulation and catastrophic forgetting (Wang et al. 2022). Meanwhile, adapters with different dimensional middle-layer layer features demonstrate effectiveness in addressing these challenges. This encourages us to take a step further and justify the principles underlying the use of low-rank adapter and high-rank adapter in the CTTA.
3

---

Published as a conference paper at ICLR 2024
Earget Domaia Toput
Pre-trained Source Model
Prediction
funetion
Homesotatic Knowksdge Allicmest
Teacher model
(8)
Distribution
Augment
Uncertuinty Value Equ(4)
Reflect
Guide 4. Eq.(5)
Eq.(a)
A
2
shared
specifie (2)
Student
Consistency Loss Fq.(6)
Eq.(2)
down
ViDAs
Back-prop
Low- Rank
Original Linear or Conv
High-Rank
up
EMA
Feature:
Figure 2: The framework of Visual Domain Adapter (ViDA). (a) We inject low-rank and high-rank ViDAs into either linear or Conv layers of the pre-trained source model. The student model processes the original image, while the teacher model processes an augmented version of the same image. To update the ViDAs, we construct a teacher-student framework and use consistency loss (Eq. (6) as the optimization objective. In addition, the teacher model calculates an uncertainty value (Eq. 4). reflecting the distribution shift of each sample target domains. (b) Based on the degree of distribution shift, we introduce the Homeostatic Knowledge Allotment (HKA) strategy, which aims to dynamically fuse the knowledge from each ViDA with different domain representation.
Low-rank adapter: Our hypothesis regarding the effectiveness of adapters in mitigating catastrophic forgetting is that their low-rank embedding space representation plays crucial role. To explore this further, we conduct a t-SNE distribution study (Van der Maaten & Hinton, 2008) on the third trans- former block to analyze the feature distributions across four target domains (ACDC dataset (Sakaridis) et al. )) The results are depicted in Fig. [(b), Our analysis reveals that the low-rank adapter exhibits relatively consistent distribution across the different target domains, suggesting that its low-rank embedding space can effectively disregard the impact of dynamic distribution shifts and prioritize the extraction of domain-shared knowledge.
Furthermore, we adopt the domain distance definition proposed by Ben-David (Ben-David et al. 2006 20T0) and build upon previous domain transfer research (Ganin et al. 2016) by employing the H-diverreenc metric to evaluate the domain representations of adapters across different target domains. The discrepancy distance between two distributions Ds and Dr, can be calculated as:
(D, Dr.) sup Pr (D(x) = 1] Pr [D(x)= 1]]
D> where H denotes hypothetical space and denotes discriminator. Similar to (Ruder Plank 2017 Allaway et al. 2021), we adopt the Tensen-Shannon domains as an approximation of H-ddivergence because it has been shown to successfully distinguish domains. If the inter-domain divergence is relatively small, it can be demonstrated that the feature representation is consistent and less influenced by cross-domain shifts (Ganin et al.) 2016). We compare the JS values obtained by using the source model alone, injecting low-rank adapter, injecting high-rank adapter, and combining low- and high-rank adapters, as illustrated in Fig. 3 (a). Our results indicate that the feature representation generated by the low-rank adapter exhibits lower divergence compared to both the original source model and the high-rank adapter, especially when dealing with later target domains or significant domain shifts between adjacent domains (i.e.. target domains 9-13). This result simultancously demonstrates the low-rank adapter's 's ability to learn long-term camainn-saared knowledge in a continually changing environment.
z~Dr
(JS) divergence between two adjacent
To provide clearer evidence for the intuition, we extend our analysis by incorporating the qualitative analysis of Class Activation Mapping (CAM) on the ImageNet- Mir-Net-to-Imann to-ImmgeNet-CC CTTA. As shown in Fig 4 we showcase the feature representations from different target domains, including the noise of Gaussian and Snow. We observe that the low-rank ViDA is inclined to put more weight on the foreground sample while tending to disregard background noise shifts. This indicates that the low-rank ViDA attends to locations with more general and task-relevant information.
High-rank adapter. Regarding the domain representation of the adapter with high-rank feature, we propose that is better suited to address error accumulation in the continual adaptation process. We verify this by t-SNE analyzing the feature distributions between different domains, shown in Fig. [ (b), and observe that there is a clear discrepancy between domains. The distribution achieves better aggregation in single domain. This suggests that high-ranking adapters have
4

---

Published as a conference paper at ICLR 2024
0.6
(4) Inter-Domann Divergonce
(b) Intra-clax Divergence
0.5
0.-
0.3
04-
02-
02-
Domain
0.1
c9ec10
012xet3
Figure 3: cl to c15 represent the 15 corruption domains in CIFAR ICC listed in sequential order. (a) Low-rank adapter based model effectively mitigates inter-domain divergence than the source model across all 14 domain shifts. (b) High-rank adapter based model significantly enhances the intra-class feature aggregation, yielding results that closely approximate those achieved by our ViDA method.
Input Image
Original model
Loweronk VIDA
High-rnnk VIDA
Inpui image
Original model
Lowerak VIDA
High VIDA
Suise
Figure 4: The qualitative analysis of the CAM. We adopt CAM compare the attention of the low-rank branch, high-rank branch, and the original model during the continual adaptation process.
better grasp of target domain data distribution. Inspired by intra-class dissimilarity proposed by k-means (MacQueen) 1967), we use normalized intra-class divergence to further verify the domain representations of high-rank adapters in CIFARIOC, In given domain, if the intra-class divergence for each category smaller, it demonstrates that the model has better understanding of the current distribution (Li al. 2020). As illustrated in Fig. 3 (b), the high-rank adapter is found drive down the intra-class divergence within almost all domains, indicating that it can better adapt to current domain distribution and extract domain-specifet knowledge in continual target domains. For more straightforward verification, we conduct qualitative analysis by incorporating the visualization of CAM. Conversely, the high-rank ViDA exhibits an inverse pattern, as illustrated Fig 4 allocates more attention to locations characterized by substantial domain shift, encompassing the entirety of input images. This behavior aligns with the high-rank branch's 's tendency to global information and predominantly extract domain-speciffe knowledge from the target domain data.
In conclusion, the structure of low-rank ViDA reduces feature redundancy, which leads to an underfit state during CTTA. Consequently, it tends to acquire general information across continuous target domains, extracting tesk-relevant knowledge to mitigate catastrophic forgetting. contrast, high-rank ViDA employs higher-dimensional feature representation that better aligns with the target data distribution, thereby focusing on learning domain-specific knowledge to prevent error accumulation. We offer additional justifications and specifically designed experiments in Appendix B
3.2 VISUAL DOMAIN ADAPTER
The above observation motivates to introduce high-rank and low-rank Visual Domain Adapters (ViDAs) into the source pre-trained model, aiming to simultaneously adapt current domain distribution and maintain the continual domain-shared knowledge CTTA.
The architecture. The design principle of injecting ViDAs into the pre-trained model is simple yet effective, which illustrated in Fig 2 (b). As we can see there are three sub-branches, the linear (or Conv) layer in the middle branch originated from the original network, while the right branch and left branch are bottleneck structures and separately indicate the high-rank ViDA and low- rank ViDA. Specifically, the right branch (high-rank) contains up-projection layer with parameters Wh up E Rdxdd, doww-projecttoo layer with parameters W E Rdh x. where d,, (c.g., d, = 128) down
the middle dimension of high-rank feature and satisfies d, d. There is not any non-linear layer
5

---

Published as a conference paper at ICLR 2024
in the ViDA. And utilize the linear layer the projection layer when the original model is transformer architecture and adopt 1 x Conv as the projection layer when the original model is convolution network. In contrast, the left low-rank branch first injects down- nown-projection layer with parameters W down E Rdxd', then place an up- up-rrojection layer with parameters W up E Rdixd, where di (e.g.. di = 1) stand for the middle dimension of the low-rank feature (d: & d). For input feature S. the produced features of high-rank (Sr) and low-rank ViDA (ft) are formulated as:
fn W down (W upp f); fi W (W drown" -f)
The two-branch bottleneck is connected to the output feature of the original network (fo) through the residual connection via scale factors (A,, and At). The fusion knowledge (fy) can be described as:
Jf = So An x Sn + \ x Jt
The domain knowledge scale factors (An and X²) are adaptively obtained through the homeostatic knowledge allotment strategy, which is shown in Section 3.3 During inference, the different domain- represented ViDAs (linear relation) can be projected into the pre-trained model by re- parameterization (Ding et al. 2021), which ensures no extra model parameter increase of the original model.
3.3
HOMEOSTATIC KNOWLEDGE ALLOTMENT
Method motivation. In CTTA, target domain data can only be accessed once and exhibits different distribution shifts, which underscores the importance of efficient domain transfer. Moreover, to effectively address error accumulation and catastrophic forgetting, it becomes necessary to extract different knowledge and manage it separately. Although the specialized structures of low-rank and high-rank ViDAs contribute to distinct domain representation learning, the continual adaptation process also needs to regularize the knowledge fusion weight to ensure the efficient capture of relevant Nomain-specific knowledge without compromising the retention of long-term domain-sharce knowledge. HKA design. As depicted in Fig (b). we draw inspiration from (Ovadia et al. 2019) Roy et al. 2022) and introduce uncertainty value to quantify the degree of distribution shift for each sample. While the confidence score is a common measure to assess prediction reliability, it tends to fluctuate irregularly and becomes unreliable in continual changing environment. To address this limitation, we employ the MC Dropout technique (Gal & Ghahramani] 2016) on linear layers, enabling multiple forward propagations to obtain m sets of probabilities for each sample. Subsequently, we calculate the uncertainty value (xx)
for given input which are formulated as:
m m Σ
1
u(x)
(p((vlz) - -ul2?
Where pa Pa(yla .r) is the predicted probability of the input x in the jth forward propagation and J the average value of m times prediction. To dynamically adjust the scale factors (λ, and Xr) based on the uncertainty score, the formulation is as follows:
{ A 1 U(x) A= 1=11-4 u(x), u(x) 2 θ
(5)
1
\ =11=x((2) u(x) <
The threshold value of uncertainty is denoted as O, where Q = 0.2. To realize the homeostasis of different domain knowledge, when facing the sample with large uncertainty value, adaptively increase the fusion weight of domain-speciis knowledge (A,)) Conversely, if the input has a low uncertainty value, the fusion weight of domain- Nomain--hared knowledge (Ax) will be increased.
3.4 OPTIMIZATION OBJECTIVE
Following previous CTTA work (Wang et al. 2022), we leverage the teacher model T to generate the pseudo labels ý for updating ViDAs. And the consistency loss Lee the optimization objective.
1 Σ x(c) log ((c)
(x) -
(6)
Where is the output of our student model S C' means the number of categories. Same as previous (an et al., 2023), we load the source pre-trained parameters to initialize the weight of both models and adopt the exponential moving average (EMA) to update the teacher model with ViDAs.
T = a + (1 - a)s{
Where is the time step. And we set the updating weight 0.999 (arvainen & Valpola 2017a

---

Published as a conference paper at ICLR 2024
4 EXPERIMENT
In Section 4.2 and 4.3 we compare our method with other SOTA methods on classification and semantic segmentation CTTA. In Section 4.4 we employ the foundation models (DINOv2 (Kirillvv et al., 2023) and SAM (Oquab et al., 2023)) ) as the backbone and evaluate the efficacy of our method. In Section we further evaluate the domain generalization ability of the proposed method on unseen target domains. Comprehensive ablation studies are conducted in Section 4.6 More quantitative
comparisons and qualitative analyses are shown in the Appendix and D respectively.
4.1 TASK SETTINGS AND DATASETS
Dataset. We evaluate our method on three classification CTTA benchmarks, including CIFARIO- Io-CIFARIOC, CIFAR 100-to to-C 100C (Krizhevsky et al. 2009) and
(Hendryeks & Dietterich 2019p. For segmentation CTTA Yang et al., (023b), we evaluate our method on Cityscapes ACDC, where the Cityscapes dataset (Cordts et al. 2016) serves as the source domain, and the ACDC dataset (Sakaridis et al. 2021) represents the target domains.
CTTA Task setting. Following (Wang al.) 2022). in classification CTTA tasks, we sequentially adapt the pre-trained source model to the fifteen target domains with the largest corruption severity (level 5). The online prediction results data. Regarding segmentation CTTA (Yang et al., 2023b), the source model is an off-the-shelf pre-trained the Cityscapes dataset. As for the continual target domains, utilize the ACDC dataset, which consists of images collected in four unseen visual conditions: Fog, Night, Rain, and Snow. To simulate continual environmental changes real-life scenarios, cyclically repeat the same sequence of target domains
evaluated immediately after encountering the input
multiple times.
Implementation Details. In our CTTA experiments, we follow the implementation details specified in previous works (Wang et al. (022) to ensure consistency and comparability. We adopt ViT-base (Dosovitskiy al. 2020) and ResNet (He al., 2016) as the backbone the classification CTTA. In the case of ViT-base, we resize the input images to 224x224, while maintaining the original image resolution for other backbones. For segmentation CTTA, we adopt the pre-trained Segformer-B5 model (Xie et al. 2021) as the source model. We down-sample the input size from 1920x 1080 to 960x540 for target domain data. The optimizer is performed using Adam (Kingma & Ba 2014) with (81, (2) (0.9, 0.999). We set the learning rates to specific values for each task: Ic-4 for CIFARIOC, 5e-7 7 for ImageNetC, and 3e-4 for ACDC. To initialize our visual domain adapters, we train adapters for several iterations on classification datasets (e.g.. ImageNet). We apply range of image resolution scale factors [0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0] for the augmentation method and construct the teacher model inputs (Wang et al. 2022).
4.2 THE EFFECTIVENESS ON CLASSIFICATION CTTA
Table 1: Classification eиror rate(%) for ImageNet-too AangeNet-t-ImmagNNeeoC online CTTA task. Gain(%)
represents the percentage of improvement in model accuracy compared with the source method.
Backbone Method
REF
Ganssion
Aboot
imgodde
modton
ZOOm
froot
fog
brightouses
ContrAnt
Alottic trane
Piselate
i0g
Mean] Gain
Source
He
2015
97.8
97.1
2
81.7
77.1
75.9
41.3
82.5
79.3
82
ResNet50
TENT
Wany et m
2021
81.6
74.6
72.7
77.6
73.8
65.5
55.3
61.6
63
51.7
38.2
72.1
50.8
47.4
53.3
62.6
+19.4
CoTTA
82.1
80.6
48.3
62.7
Eco
Spag et
2023
63.4
+18.6
Ours
Proposed
79.3
74.7
73
76.9
74.5
65.0
56.4
59.8
62.6
49.6
38.2
66.8
49.6
43.1
46.2
61.2
+20.8
Source
Dosovissky.coo
2020
53.0
51.8
5
68.5
78.8
58.5
63.3
49.9
54
57.7
26.4
91.4
57.5
38.0
36.2
55.8
0.0
Pscudo
1120 2015
45.2
40,4
41.6
$1.3
53.9
45.6
47.7
40.4
45.7
93.8
98.5
99.9
99.9
98.9
99.6
61.2
-5.4
TENT
48.9
49.2
65.8
4
50.3
23.9
33.9
51.0
base
CoTTA
Wing 2t ml 2022
52.9
51.6
51.4
68.3
78.1
57.1
62.0
48.2
52.7
55.3
25.9
90.0
564
36.4
35.2
5488
+1.0
VDP
52.7
51.6
50.1
58.1
70.2
56
58.1
46.1
45.8
23.6
549
34.5
36.1
50.0
Ours
Proposed
47.7
42,5
42.9
52.2
56.9
45.5
48.9
38.9
42.7
40.7
24.3
52.8
49.1
335
33.1
43.4
+12,4
SoaageNet-to-ImmgeNeee Given the source model pre-trained on ImageNet, we conduct CTTA on ImageNet-C, which consists of fifteen corruption types that occur sequentially during the test time. In Table methods that utilize the ViT backbone achieve lower classification errors compared to those using the ResNet50 backbone, demonstrating VIT's superior generalization capability in
7

---

Published as a conference paper at ICLR 2024
Table Average error rate (%) for the stan- dard CIFAR -O-AL-A and CIFAR 10F- CICAAARIOOC CTTA. All results are evalu- ated the ViT-base, and the fine- ain-ggrined perfor- mances are shown in Appendix E
3: Average error rate (%) for the CIFARIO-
to-CIFARIOC CTTA task. All results are eval- uated on the ViT-Base, which uses the pre- trained encoder parameter of foundation large- scale models (DINOv2 and SAM).
Target
Method
Source Tent CoTTA
VDP Ours
Backbone
Method
Source
Tent
CoTTA
Ours
Ci
Mcan
28.2
23.5
24.6
24.1
20.7
DINOv2
Mean
25.0
21.7
29.3
20.2
Gaint
0.0
44.7
43.6
+4.1
+7.5
Gaint
0.0
43.2
-4.3
+4.8
10fc
35.4
32.1
34.8
35.0
27.3
SAM
Mean
39.3
37.5
39,4
34.1
Gaint
0.0
+3.3
+0.7
+0,4
+8.1
Gaint
0.0
+1.8
-0.1
+5.2
the continually changing environment. For ViT-base, the average classification error is up to 55.8% when we directly test the source model on target domains. Our method can outperform all previous methods, achieving a 12.4% and 6.6% improvement over the source model and previous SOTA method, respectively. Moreover, our method showcases remarkable performance across the majority of corruption types, highlighting its effective mitigation of error accumulation and catastrophic forgetting. In addition, we conduct a 10-round CTTA experiment in Appendix [.I which repeat 10 rounds of 15 corruption sequences in ImageNet-C. The performance of our method consistently improves over time, demonstrating its enduring robustmess in the long-term adaptation process.
To further validate the effectiveness of method, we conduct experiments CIFAR10-to- CIFARIOC and SEAAR100-t-CIFARRIOO
As illustrated in Table in CIFARIOC, our approach achieved 2.8% improvement compared to the previous SOTA model. We extend our evaluation to CIFAR100C, which comprises a larger number of categories in each domain. Our approach surpasses all previous methods, which show the same trend as the above CTTA experiments. Therefore, the results prove that our method mitigates the challenges posed by continual distribution shifts, regardless of the number of categories present in each domain. In addition, we provide supplementary CTTA experiments utilizing convolutional backbones in the Appendix C.5
4.3 THE EFFECTIVENESS ON SEGMENTATION CTTA
Table 4: Performance comparison for Cityscape-to-ACDD CTTA. We sequentially repeat the
same sequence of target domains three times. Mean is the average score of mloU.
Time
Round
Meant
Gain
Method
REF
Fog
Night
Rain
Snow
Meanf
Fog
Night
Rain
Snow
Moan
Fog
Night
Rain
Snow
Meant
Source
[Xee et
2021
69.1
40.3
59.7
57.8
56.7
69.1
40.3
59.7
57.8
36.7
69.1
40.3
59.7
57.8
56.7
56.7
TENT
Wang al
2020
69.0
40.2
60.1
57.3
56.7
68.3
39.0
60.1
56.3
55.9
67.5
37.8
59.6
55.0
55.0
55.7
-1.0
CoTTA
70.9
41.2
62.4
59.7
58.6
70.9
41.1
62.6
59.7
58.6
70,9
41,0
62.7
59.7
58.6
58.6
+1,9
DePT
Gan
al. 2022
71.0
40.8
58.2
56.8
56.5
68.2
40.0
55.4
53.7
54.3
66.4
38.0
47.3
47.2
49.7
53.4
-3.3
VDP
al.
70.5
41.1
62.1
59.5
58.3
70.4
41.1
62.2
59.4
58.2
70.4
41,0
62.2
59.4
58.2
58.2
+1.5
Ours
Proposed
71.6
43.2
66.0
63.4
61.1
73.2
44.5
67.0
63.9
62.2
73.2
44.6
67.2
64.2
62.3
61.9
+5.2
COCSSC. As presented in Table 4 observed a gradual decrease in the mloUs of TENT and DePT over time, indicating the occurrence of catastrophic forgetting. In contrast, our method has continual improvement of average mloU of target domains is repeated. Significantly, the proposed method surpasses the previous SOTA CTTA method (Wang et al. 2022) by achieving 3.3% increase in mloU. This notable improvement showcases our method's s ability to adapt continuously to dynamic target domains in the pixel-level task. The 10 rounds semantic segmentation CTTA experiments are shown in Appendix C.6
when the same sequence
4.4 CONTINUAL ADAPTING FOR FOUNDATION MODELS
Foundation models (Bommasani et al. 2021 are trained on large-scald scale datasets, endowing them with powerful generalization capabilities and the ability to capture representations of common features. However, performing full fine-tuning on the foundation model is time-consuming and economically impractical. Hence, our adaptation method proves valuable by enhancing the continual transfer performance of foundation models. As indicated in Table S we introduce foundation models as the pre-trained model and adapt them to continual target domains (CIFARIOC). Our approach achieved performance improvement of 4.8% on the representative image-level foundation model DINOv2 (Qquab et al. 2023) and 5.2% pixel-level foundation model SAM (Kirillov et al.. 2023).
8

---

Published as a conference paper at ICLR 2024
Table 6: Average error rate (%) for the MoageNNettto-Imageee represent the high-rank and low-rank ViDAs.
The domain generalization compar- isons on ImageNet-C. Results are evaluated on ViT-base. Mean and Gain %) represent the per- formance unseen target domains.
Table
ViDA,, and ViDA
IHKA means inversed HKA strategy.
ViDAA ViDA,
HKA IHKA Mean.
Directly test unseen domains
Unseen Mean..
55.8
Method
bri.
contrast elastic pixelate jpeg
E22
50.7
Source
26,4
91.4
57.5
38.0
36.2
49.9
Far
51.2
Tent
25.8
91.9
57.0
37.2
35.7
49.5
E2a
V
45.6
CoTTA
25.3
88.1
55.7
36.4
34.6
48.0
Ews
43.4
Ours
24.6
68.2
49.8
34.7
34.1
42.3
Ews
/
46.3
Our method consistently and reliably improves the performance of the foundation model on the continually changing environment. Note that, we only use the pre-trained encoder of SAM and add classification head, which is fine-tuned on the source domain. Our approach empowers the large- scale model with the capability of continuous transfer learning, without undermining its plasticity. Additional CTTA experiments of foundation models are shown in Appendix and
4.5 DOMAIN GENERALIZATION ON UNSEEN CONTINUAL DOMAINS
To investigate the domain generalization (DG) ability of our method, we follow the leave-one-domain- out rule (Zhou et al. 2021: Li et al. 2017 to leverage 10/15 domains of ImageNet-C as source domains for model training while the rest (5/15 domains) are treated as target domains without any form of adaptation. Specifically, we first use our proposed method to continually adapt the source pre-trained model to 10/15 domains of ImageNet-C without any supervision. Then we directly test on the 5/15 unseen domains. Surprisingly, our method reduces 7.6% on the average error on unseen domains (Table (5), which has a significant improvement over other methods. The promising results demonstrate that our method possesses DG ability by effectively extracting domain-sharee knowledge. More DG experiments are provided in the supplementary Appendix C.3
4.6 ABLATION STUDY
Effectiveness of each component. We conduct the ablation study on MaageeNettt--mmaeNNee CTTA scenario and evaluate the contribution of each component in our method, including high-rank ViDA (V)DA)), low-rank ViDA (ViDA),, and Homeostatic Knowledge Allotment (HKA) strategy. As shown in Table by introducing the high-ranh ViDA, the error decreases by 5.1% compared knowledge for
to Ex,, demonstrating that high-rank features can extract more domain-spectne adaptation in target domains. As shown in Fa low-rank ViDA gains 4.6% improvement compared to Ex1. The result proves that the domain-share knowledge extracted from low-rank feature can also improve the classification ability on continual target domains. Ex₁ has a remarkable improvement of 10.2% overall, demonstrating that the two types of ViDA can compensate for each other in the continual adaptation process. Ex; achieves a 12.4% improvement, demonstrating the effectiveness of the HKA strategy in enhancing the distinct domain representations of each type of ViDA. To further assess the effectiveness of HKA, we perform an additional experiment, denoted as Exg- by inverting the scale factors within the HKA strategy. Specifically, for samples exhibiting high uncertainty, we reduced A while increase Aq. This results in marginal increase 0.7% error compared to Er, and 2.9% error compared to Exs. The additional ablation studies are shown in Appendix C.4 C.4
CONCLUSION
In this paper, we propose homeostatic Visual Domain Adapter (ViDA) to address enror accumulation and catastrophic forgetting problems in Continual Test-Time Adaptation (CTTA) tasks. And we investigate that the low-rank ViDA can disregard the impact of dynamic distribution shifts and prioritize the extraction of domain-shared knowledge, and the high-rank ViDA can extract more reliable domain-speciffe knowledge. Meanwhile, we further propose Homeostatic Knowledge Allotment (HKA) strategy to dynamically fuse the knowledge from low-rank and high-rank ViDAs, thus enhancing ViDAs' distinct domain representations.

---

Published as a conference paper at ICLR 2024
Acknowledgements. Shanghang Zhang is supported by the National Science and Technology Major Project of China (No. 20222D0117801).
REFERENCES
Emily Allaway, Malavika Srikanth, and Kathleen McKeown, Adversarial learning for zero-shot stance detection social media. arXiv preprint arXiv:2/05.06603, 2021.
Eduardo Aroold, Omar Y Al-Jarrah, Mehrdad Dianati, Saber Fallah, David Oxtoby. and Alex Mouzakitis. A survey on 3d object detection methods for autonomous driving applications. IEEE Transactions on Intelligent Transportution Systems, 20(10):382--3795, 2019.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale scale models. 2022.
Shai Ben-David John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
Shai Ben- David. John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.072:0 2021.
Malik Boudiaf, Tom Denton, Bart van Merriënboer, Vincent Dumoulin, and Eleni Triantafillou. In search for a generalizable method for source free domain adaptation. 2023.
Goirik Chakrabarty, Manogna Sreenivas, and Soma Biswas. Sata: Source anchoring and target alignment network for continual test time adaptation. arXiv preprint 2304::2300.01133 2023.
Dian Chen, Dequan Wang. Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. ArXiv, abs/2204.10377 2022.
Shoufa Chen, GE Chongjian, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In Advances in Neural Information Processing Systems.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vtg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13733-13742, 2021.
Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 7704-7714, 2023.
2023.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11997 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yulu Gan, Xianzheng Ma, Yihang Lou, Yan Bai, Renrui Zhang, Nian Shi, and Lin Luo. Deco- rate the newcomers: Visual domain prompt for continual test time adaptation. arXiv preprint arXiv:2212.00 041145 2022.
10

---

Published as a conference paper at ICLR 2024
Yulu Gan, Yan Bai, Yihang Lou, Xianzheng Ma, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 7595-7603, 2023.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research,
2016.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACL- IJCNLP 2021, pp. 3816-3830. Association for Computational Linguistics (ACL), 2021.
Yunhe Gao, Xingjian Shi, Yi Zhu, Hao Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, and Dimitris N Metaxas. Visual prompt tuning for test-time domain adaptation. arXiv preprint arXiv:2210.04831, 2022.
Taesik Gong, Jongheon Jeong. Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test-time adaptation against temporal correlation. Advances in Neural Information Processing Systems, 35:27253-27266 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. international conference on computer vision, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Wei Low, Lidong Bing, and Luo Si. On the effectiveness of adapter-based based tuning for pretrained language model adaptation. arXiv preprint arXiv:2106.03 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp.
PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0940 09685, 2021.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. Knowledgeable prompt- tuning: Incorporating knowledge into prompt verbalizer for text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2225-2240, 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything, arXiv preprint 024::22304.02643, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jogendra Nath Kundu, Naveen Venkat, Rahul M, and R. Venkatesh Babu. Universal source-free domain adaptation. 2020.
Dong-Hyun Lee. Pseudo-label: : The simple and efficient semi-supervised learning method for deep neural networks. 2013.
11

---

Published as a conference paper at ICLR 2024
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Jingjing Li, Erpeng Chen, Zhengming Ding, Lei Zhu, Ke Lu, and Heng Tao Shen. Maximum density divergence for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, (((11):3918-9300 2020.
Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. arXiv preprint arXiv:2312.162177 2023.
Jian Liang, D. Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, 2020.
Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribu- tion shifts. arXiv preprint arXiv:2303. 15361, 2023.
J MacQueen. Classification and analysis multivariate observations. 5th Berkeley Symp. Math. Statist. Probubility, pp. 281-297. University of California Los Angeles LA USA, 1967.
Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen. Test-time adaptation to distribution shift by confidence maximization and input transformation. arXiv preprint arXiv:2106. 14999, 2021.
Jiayi Ni, Senqiao Yang. Jiaming Liu, Xiaoqi Li, Wenyu Jiao, Ran Xu, Zehui Chen, Yi Liu, and Shanghang Zhang. Distribution-wware continual test time adaptation for semantic segmentation. arXiv preprint 1XXi:2330913604 2023.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. arXiv preprint (rXiv:2302.124000 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv22304.07/93, 2023.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, et al. Exploring low-dimensional intrinsic task subspace via prompt tuning. arXiv preprint arXiv:21 2021.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi- domain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 81 19-8777, 2018
Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, and Arno Solin. Einertainty-guider source-free domain adaptation. In Computer Mxion-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXV, pp. 537-555. Springer, 2022.
Sebastian Ruder and Barbara Plank. Learning to seleet data for transfer learning with bayesian optimization. arXiv preprint arXiv:1707.055: 2017.
Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acde: The adverse conditions dataset with correspondences for semantic driving scene understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10765-10775, 2021.
12

---

Published as a conference paper at ICLR 2024
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann. Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. ArXiv, abs/2006.16971,, 2020.
Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattem Recognition, pp-
2023.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vi-adapter: Parameter-efficient transfer learning for laiion-and-aanguage tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227-5237, 2022.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Learning. 2017a
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Bigght-average consistency targets improve supervisd deep learning results. Advances in neural information processing systems, 30, 201
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
To Vu, Brian Lester, Noah Constant, Rami AI-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5039-5059, 2022.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprini arXiv:2006. 10726, 2020.
Dequan Wang. Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
Qin Wang. Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7201-7211, 2022.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 14:12077-12000, 2021.
Senqiao Yang. Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312. 140774 2023a.
Senqiao Yang, Jiarui Wu, Jiaming Liu, Xiaoqi Li, Qizhe Zhang, Mingjie Pan, and Shanghang Zhang. Exploring sparse visual prompt for cross-domain semantic segmentation. arXiv preprint arXiv:2303.0979) 09799, 2023b.
Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. international conference on computer vision, 2021.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15922-15999 2023.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. british machine vision conference, 2016.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efffiieet fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2 10/99, 2021.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from saatennceto-sequenee perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6881 16890, 2021.
13

---

Published as a conference paper at ICLR 2024
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization in vision: A survey, arXiv preprint arXiv:2103.02503, 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection, arXiv preprint arXiv:2010.04159, 2020.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
The supplementary materials presented in this paper offer a comprehensive quantitative and qualitative analysis of the proposed method. In Appendix B we provide additional empirical observations and justifications for our motivation, including specifically designed quantitative analysis, qualitative analysis of the distribution, and justification for distribution divergence. Additionally, we present extra continual adaptation experiments for Foundation Models in Appendices C.1 and C.2 which are
conducted on ImageNet-to- Metetto-ImageNet--CC and Cityscape-t-ACCDC scenarios. To assess the domain generalization ability of our method, we conducted additional experiments directly testing varying number of unseen domains in Appendix C.3) The ablation study on middle-layer dimension described in Appendix C.4 Furthermore, Appendix C.5 presents additional CTTA classification experiments utilizing the convolutional backbone, while Appendix(C.ooulineee 10 rounds of semantic segmentation CTTA experiments. We provide additional qualitative analysis in Appendix D Moreover, we extend the classification results of our submission to include fine- mme-grained performance in Appendix [ showcasing the error rates across fifteen corruption types.
B SUPPLEMENTARY JUSTIFICATIONS FOR MOTIVATION
The study of Continual Test-Time Adaptation (CTTA) poses significant challenges, particularly in addressing error accumulation and catastrophic forgetting (Wang et al.) 2022] Gan et al., (023). Notably, the use of adapters with low-rank and high-rank features have demonstrated promising results in mitigating these challenges in our submission. In this section, we aim to provide comprehensive implementation details regarding the evidence supporting our motivation. Furthermore, we have introduced two new specially designed quantitative experiments in Section B. The first one is 10-round CTTA experiment aimed at investigating the different domain representations of low-rank and high-rank ViDA during the long-term adaptation process. The second experiment explores the performance when all adapters adopt the same structures, such as using two high-rank adapters or two low-rank adapters. This experiment is conducted to validate that low-rank ViDA and high-rank ViDA complement each other in adapting to continually changing environments.
B. SPECIALLY DESIGNED QUANTITATIVE ANALYSIS
To provide stronger evidence for our assumption, we have developed two evaluation approaches for both low-rank and high-rank adapters, which directly reflect their ability to extract domain-shared and domain-specifet knowledge on ImageNee--o-ImageNeeec
First, as shown in Figure 5 (b), we execute 10 rounds CTTA experiment on ImageNet-to ImageNet-o-ImageNett C. In this comprehensive experiment, we simulate a long-term adaptation scenario by repeating 10 rounds of 15 corruption sequences in the ImageNet-C. Remarkably, the high-rank ViDA achieves competitive results over other methods during the initial I to 3 rounds. This result demonstrates the high-rank feature's 's capacity to efficiently learn target domain speeiii-ppeifii knowledge. However, an increment in error rates becomes obvious during the later rounds (rounds 5 to 10). The results validate the potential for encountering catastrophic forgetting when focusing exclusively domain-specific knowledge. In contrast, the performance of the low-rank ViDA remains consistently robust throughout the continual adaptation process, verifying concentrates more on extracting task-relevant knowledge and effectively prevents the catastrophic forgetting problem. And our proposed method consistently improves over time, demonstrating its robustness the long-term adaptation process.
Second, we execute an SaaggeNeettt--mageeeer CTTA experiment using a combination of two high- rank adapters or two low-rank adapters, as shown in Table 7 To ensure fairness, we conducted these experiments without implementing the homeostatic knowledge allotment (HKA) strategy. Notably, the two low-rank adapters (Ex2) demonstrated consistently lower long-term error rates compared to the source model and two high-rank adapters. The above results can be attributed to the fact that the two low-rank ViDAs tend to learn general information and domain-shared knowledge during continual adaptation. However, corruption types. This indicates that solely relying on low-rank adapters without the involvement of high-rank adapters is insufficient to fit target domains and match their data distribution. On the other hand, the performance of the two high-rank adapters initially surpasses our approach (Ex4) in the early stages, covering the first few target domains. Nevertheless, noticeable performance degradation
method outperforms the two low-rank adapters across 14 out of 15
15

---

Published as a conference paper at ICLR 2024
Table 7: Classification error rate(%) for ImageNee-to--lmaeeNeee online CTTA task. Gain(%)
represents the percentage of improvement in model accuracy compared with the source method. 2x
means using two same structures of adapters.
Method
Gaussian
shot
impulse
defocus
glass
motion
200m
Snow
frost
fog
brightness
contrast
elastic
pixelate
ipeg
Mean
Source
53.0 51.8
52.1
68.5
78.8 58.5 63.3 49.9 54.2
57.7
26.4
91.4
57.5
38.0
36.2
55.8
Ex2
2x Low-rank
51.2
48.3
47.8
56.9
66.5
49.3
54.4
42.1
47.0
45.2
23.2
65.6
52.0
33.4
33.5
47.7
Ex3
2x High-rank
50.1
47.9
45.3
54.8
66.7
51.4
56.1
44.0
49.2
48.3
25.7
69.7
56.3
34.6
33.7
48.9
Ex4
Ours
50.3
45.9
45.5
55.1
62.3
46.6
51.7
39.7
44.0
42.2
23.0
62.4
50.1
33.4
32.5
45.6
Transformer Blockl
Transformer Block?
Transformer Blocki
55.0
Low-rank Branch
525-
Saerag 6007 rate
50
45 0-
Highrrank Branch
Source
40.0
High rank
CoTTE
Lonsraae
Round
(a) t-SNE distribution analysis
(b) 10 Rounds CTTA on ImaggeNe--t--ImageNe..C
Figure 5: (a) We conduct more t-SNE results for the low-rank adapter and high-rank adapter on the ACDC dataset. The first to third columns illustrate the feature distributions of transformer blocks 1, and 4, respectively.
(b) The rounds CTTA experiment on 10 rounds of 15 corruption sequences.
repeating
becomes apparent in later target domains. This observation underscores a crucial finding: while increasing the number of high-rank ViDAs might enhance Nomain-specific knowledge acquisition during the initial phases of CTTA, it simultaneously exacerbates catastrophic forgetting throughout the entire adaptation process. In contrast, the fusion of both low-rank and high-rank ViDAs (Ex4) yields the most substantial improvement when compared to other configurations. Our collaborative approach leverages the distinct domain representations these adapters to compensate for each other's advantages and achieve more robust and effective continual adaptation.
B.2 ADDITIONAL DISTRIBUTION QUALITATIVE ANALYSIS
We employed t-distributed stochastic neighbor embedding (t-SNE) (Van der Maaten & Hinton 2008
to visualize the distribution of adapters across four continual target domains. This visualization was specifically conducted in the context of the Cityscapes-t--ACDDC experiment, representing a scenario with continually changing real-world environments.nn In our submission, we perform t- SNE analysis on the outputs of the third transformer block in the Segformer- model (Xie et al.) 2021). The objective was to qualitatively compare the feature distributions of ViDAs with different dimension features. Furthermore, our findings revealed that the qualitative results obtained from different layers (i.e., transformer block 1, 2, and 4) of the Segformer-B5 model exhibited similar distribution representations. As illustrated in Figure 5 (a), there is a noticeable distribution gap due to the significant domain shift between the night domain and other domains. Interestingly, the low-rank ViDA effectively reduces the distribution distance across different target domains, indicating its focus extracting task-relevar relevant knowledge. On the other hand, the high-rank ViDA exhibits notable distribution discrepancies among the various target domains, indicating its focus on extracting domain-specific knowledge.
16

---

Published as a conference paper at ICLR 2024
B.3 DISTRIBUTION DISTANCE
To provide clearer evidence for our assumption, we directly calculate the distribution distance to represent different domain representation of adapters. We adopt the domain distance definition proposed by Ben-David (Ben-David et al., 2006 2010) and build upon previous domain transfer research (Ganin et al. 2016) by employing the H-divergence metric to further evaluate the domain representations of adapters across different target domains. H-ddvergence between Dy and Dr, can be calculated as
dx Dr.)) = 2 sup Pr (D(x) D-H z~De
= Pr Z>Dr
(D(x) I]
(8)
where H. denotes hypothetical space and denotes discriminator. Similar (Ruder & Plank 2017 Allaway et al. 2021), we adopt the Jensen-Shannon (JS) divergence between two adjacent domains an approximation of H-divergence because it has been shown to successfully distinguish domains. If the inter-domain divergence is relatively small, it can be demonstrated that the feature representation consistent and less influenced by cross-domain shifts (Ganin et al., 2016
PDs + +Dr,,
1 + KL(Ppn,
PDs -DD.
JS(P = KL(Ppsl
(9)
2
2
Where K Kullback-Leibe LK) divergence between two domain is
Pi KL(P) = Σ (as)loog P2(x;)
(10)
Where P denotes probability distribution of model output features. We split the output feature space into mutually disjoint intervals Z₃. range from 0 to 1000. To investigate the effectiveness of adapters in adapting to continual target domains, we compare the JS values obtained by using the source model alone, injecting low-rank adapter, injecting high-rank adapter, and combining low-high adapters, as illustrated in Figure 3(a) of our submission. The low-rank adapter exhibits notably lower divergence values compared to the others, demonstrating robust task-relevant feature representation in various cross-domain phases, For high-rank adapter, we use normalized intra-class divergence to further verify the domain representations of high-rank adapters in CIFAR 1OC, which is inspired by intra-cluster dissimilarity proposed by k-means (MacQueen 1967). We first calculate the Euclidean distance clustering center for each category:
Σ 0s
(11)
where e; stands for output feature in class C. Then following (MacQueen) 1967). we introduce normalized intra-class divergence E by
E P C Σ ||e (le3)113))
(12)
(6-) denotes for normlization function. In a given domain, if the intra-class divergence for each category is smaller, it demonstrates that the model has better understanding of the current distribution (Li et al., 2020). As illustrated in Figure 3(b) of the submission, the high-rank adapter is found to drive down divergence within almost all domains and can better extract domain-specifet knowledge in target domains.
C ADDITIONAL EXPERIMENT
C. ADDITIONAL CLASSIFICATION CTTA EXPERIMENTS FOR FOUNDATION MODELS
To demonstrate the effectiveness of our proposed method enhancing the continual adaptation ability of foundation models such as DINOv2 (Oquab et al. 2023) and SAM (Kirillov et al. 2023), we conduct additional experiments on a more extensive dataset, namely ImagNet-to-Image.. Net-C.
Our approach involve loading the weight parameters of the foundation model and fine-tuning
17

---

Published as a conference paper at ICLR 2024
Table 8: Average error rate (%) for the on the ViT-Base, which uses the pre-trained encoder parameter of DINOv2 and SAM.
CTTA task. All results are evaluated
clasiic Eono
Backhone
Method
RHH
Gatte State
aboot
imgulee
defoccn
monion
Fross
Rog
hrightoeee
contaM
pisclate
Mean]; Gain
Scurce
52.3
50.5
51.2
57.3
83.8
60.1
62.
47.1
56.9
58.1
22.5
88.4
60.3
32.4
35.0
54.6
0.0
DINOv2
CoTTA
Wang
Waig et
2021
2022
CVPR2022
ICLR2021
51.4
62.1
50.4
78.3
75.2
62.8
60.3
46
48.4
59.0
58.8
31.6
90.7
49.2
29.5
39
36.5
56.9
-2.3
OBrs
Proposed
49,0 49.8
50.7
614
60.2
49.7
42.6
47
51.9
45.3
27.1
49.7
47.4
32.0
29.4
46.2
18.4
67.9
47.3
SAM
Tent CoTTA
Wane al
M021
67.2 59.1
48.8
56.2
72.5
49
61
49
979
63.7
33.
770
514
39.5
55.2
55.5
+8.1
CVPR2022
68.1 64.5
50.4
67.1
80.1
68.9
67.0
63.1
69
614
40.6
88.2
58.3
43.5
68.4
63.9
-0.3
Proposed
59,9
84.3
49.6
41.7
50.3
on ImagNet, thus constructing our source model. It is important to note that we solely utilize the pre-trained encoder of SAM and incorporated a classification head, which is fine-tuned on the source domain. Subsequently, we adapt the source model to continual target domains (ImageNet-C) comprising fifteen corruption types. The results, as depicted in Table demonstrate that our approach achieved a significant performance improvement of 8.4% on the representative image-level foundation model DINOv2 and 10.6% the pixel-level foundation model SAM. These outcomes underscore the effectiveness of our method for large-scale models, consistently and reliably improving performance across target domains. Combining Table 1-3 from the submission, we were surprised to discover a significant decrease in model performance for the classification CTTA task when using the pre-trained encoder parameters of SAM. As SAM pixel-level foundation model, we then attempted to investigate the effectiveness of SAM's pretrained parameters in the segmentation CTTA task.
C.2 ADDITIONAL. SEGMENTATION CTTA EXPERIMENTS FOR FOUNDATION MODELS
As shown in Table we conducted segmentation CTTA using SAM's pre-trained parameters on the Cityscapes-to-ACDD scenario. However, it's worth noting that the Segformer model (Xie et 2021). which we employed in our main experiments, does not incorporate positional encoding. Therefore, adopted the SETR model (Zheng et al. (021) as new baseline for loading SAM's pre-trained parameters. As shown in the table, our approach with SAM's pre-trained parameters outperforms others on the ACDC target domains. This aligns with the assumption that SAM, being pixel-level foundational model, excels in capturing fine ine-grained feature representations in dense CTTA tasks.
Table 9: Performance comparison for Cityscape to-eeettoAArree ACDC CTTA. All results are evaluated on the SETR, which uses the pre-trained parameter of source model or SAM.
Method
Pre-trained
Fog
Night
Rain
Snow
Mean mIoU
Source Xic
2021
Source mode]
72.6
43.1
63.0
64.3
60.8
Source
Xie al.
202T
SAM Kirillov et al.
2023
74.8
44.1
66.7
66.6
63.0
CoTTA
Wang et al.
2022
SAM
Kirillov et al.
2023
75.4
45.9
67.3
68.7
64.3
Ours
SAM
Kinilov et al.
2023
76.5
47.2
68.1
70.7
65.6
C.3 DOMAIN GENERALIZATION ON A DIFFERENT NUMBER OF UNSEEN TARGET DOMAINS
Similar to our previous submission, we follow the deave-one-oomain-out principle (Zhou et al. 2021 Li al., 2017). where we utilize subset of ImageNet-C domains as new source domains for model training. while leaving the remaining domains as target domains without any adaptation. However, in contrast to previous domain generalization experiments, we adopt an unsupervised continual test-time adaptation (CTTA) approach for training the model on these unlabeled source domains. We solely utilize the ImageNet pre-trained parameters as the initial weights of the model. In the supplementary material, we utilize out of 15 and 7 out of 15 domains from ImageNet-C as the source domains, leaving the remaining 10 out of 15 and 8 out of 15 domains unseen target domains. Surprisingly, the results presented in Table 10 and II demonstrate that our method achieves a reduction of 9.6% and 9.1% 111 in the average the DG ability of our method, as it effectively extracts domain -harre knowledge and provides a new perspective for enhancing DG performance within unsupervised paradigm.
on these unseen domains, respectively. These promising outcomes validate
18

---

Published as a conference paper at ICLR 2024
Table 10: The domain generalization experiments on ImageNet-C, where the source model was continually adapted on the first 5 domains and directly tested on 10 unseen domains. The evaluation of the results was conducted using ViT-base.
Directly test 10 unseen domains
Unseen
clastic clastic.trans trans
brightness
Method
motion
z00m
snow
frost
fog
contrast
pixelate
jpeg
Mean
Source Tent (Wang et al. 2021
58.5
63.3
49.9
54.2
57.7
26.4
91.4
57.5
38.0
36.2
53.3
56.0
61.3
45.7
49.6
56.6
24.8
94.0
55.6
37.1
35.1
51.6
CoTTA (Wang al. 2022
57.3
62.1
49.1
52.0
57.1
26.4
91.9
57.1
37.6
35.3
Ours
46.4
52.7
39.8
43.7
42.2
23.5
71.5
49.6
33,9
33.3
43.7
Table 11:
The domain generalization experiments on ImageNet-C, where the source model was continually adapted the first 7 domains and directly tested 8 unseen domains. The evaluation of the results was conducted using ViT-base.
Directly test on unseen domains
Unseen
clastic... clastic.trans trans
brightness
contrast
pixelate
Method
show
frost
fog
ipeg
Mean
Source Tent (Wang et al. 2021)
26.4
91.4
57.5
38.0
36.2
$1.4
44.3
48.8
51.8
24.9
83.7
55.2
35.4
34.7
47.4
CoTTA Wang et al. 2022
48.8
52.2
56.7
26.1
91.1
57.0
37.3
35.3
50.6
Ours
39.6
43.7
41.7
23.7
63.7
51.7
33.3
33.6
42.3
C.4 ADDITIONAL ABLATION STUDY
How does the middle-layer dimension influence the performance?
52.0
51.5
51.0
Error (%)
50.5
50.0
49.5
49.0
64 128 512
Dimension
Figure 6: The middle-layer dimension influence of the performance
According to Figure 6 we observe that as the dimension decreases, the error rate concurrently drops. This trend suggests that lower-dimension middle layer more effectively extract the domain-shared knowledge, leading to an improved model performance. However, an opposite trend emerges when dimension surpasses 16, with performance enhancements accompanying increased dimension. This correlation implies that middle layers with higher dimension excel extracting domain-specific knowledge. And find that when the dimension larger than 128, the performance improvement
19

---

Published as a conference paper at ICLR 2024
is limited but brings larger number of parameters. Therefore, set the dimension of the high- dimension middle layer to 128 in our study.
How do different adapter initialization methods impact ViDA performance?
Pre-training the low-rank and high-rank ViDAs using source data is an unnecessary step and does not compromise the effectiveness of approach. ViDAs demonstrate comparable CTTA performance when they have relatively stable initial parameter. As illustrated in the Table T2 we conduct an additional experiment the Citscape-to-ACCI scenario. ViDAs with random initial parameters and ViDAs with parameters pre-trained on ImageNet achieved 60.5 and 61. ..4 mloU in target domains, respectively, exhibiting notable improvements compared to previous methods.
Table 12: The ablation study examines adapter initialization methods on the Cityscape-t-AACDG CTTA scenario.
Adapter pre-train
Fog
Night
Rain
Snow
Mean (loU)
Source CoTTA
69.1
40.3
59.7
57.8
56.7
70.9
41.2
62.4
59.7
58.6
Ours Ours
Source
71.6
43.2
66.0
63.4
61.1
Random initial
71.6
43.6
64,9
61.9
60.5
Ours
ImageNet
71.6
44.3
66.0
63.5
61.4
C.5 EXPERIMENTS ON CLASSIFICATION CTTA WITH CONVOLUTIONAL, BACKBONES
Table 13: Classification error rate(%) for standard CIFARR1-1O-CIAAARII
online CTTA task.
Results are evaluated on WideResNet-28. Mean is the average value of the error rate. Gain(%)
represents the percentage of improvement model accuracy compared with the source method.
Method
REF
Conference
Mcan
Gain
Source
Zagoruyko & Komodakis
2016
BMVC2016
43.5
0.0
BN Stats Adapt
Schnerder al.
2020
NeurIPS2020
20.4
+23.1
TENT
Wang ct al. 2021
ICLR2021
20.7
+22.8
COTTA
Wang ct al 2022
CVPR2022
16.2
427.3
RoTTA
Yuan ct al. 2023)
CVPR2023
17.5
+26.0
NOTE
Gong et
20222
NeurIPS2022
20.2
+23.3
ΕсоTTA
Song el al.
20:23
ICCV2023
16.8
+26.7
SATA
Chakrabarty el al. 2023
2023.4.20
16.1
427.4
Ours
Proposed
2023.5.18
15.8
+27.7
CIFARIO-to-CIFARIG standard task. In contrast to the experiments conducted in our submission, introduce change in the backbone of the classification model to WideResNNt-28., which is consistent with previous works (Wang et al. 2022). Specifically, we modify the up-projection layer and dooo-prroectton layer utilize 1 x convolutions, while the adapters placed alongside the original 3 x 3 convolutions. For ViDA, we maintain a low-rank dimension of and a high-rank dimension of 128. As depicted in Table 13 our method achieves 27.7% improvement over the source model. These findings demonstrate that our method successfully address error accumulation and catastrophic forgetting problem, regardless of the network backbone employed.
C.6 ADDITIONAL. EXPERIMENTS ON SEGMENTATION CTTA
We further present the segmentation CTTA experiment with 10 rounds on Table 14 Notably, it demonstrates a consistent enhancement in mean mloU during the initial rounds (rounds 1-3) while maintaining stable performance in subsequent rounds (rounds 4-10). After averaging over 10 rounds. our method achieved 3.0% mloU improvement compared to the previous SOTA method. As shown in Table 14 (CoTTA"). ), we adjust the hyperparameters of the CoTTA method by raising the learning rate to 3e-4, which aligns with our implementation details. The impact of this adjustment is evident in the initial three rounds of segmentation, where performance notably improves. However, progress to subsequent CTTA rounds, we observe a noticeable decline in segmentation accuracy and encounter the problem of catastrophic forgetting.
20

---

Published as a conference paper at ICLR 2024
Table 14: 10 rounds segmentation CTTA on Cityscape-o-AACDC We sequentially repeat the
same sequence of target domains 10 times. Mean is the average score of mloU.
Kound
Mesn
Fog Night Kain
Fog
Night
Fog
Night
Snow
Fog
Nipht
Snow
Fog
Nipht
Source
89.1 40.3 59.7
57.8
56.7
69.1
40.3
59.7
57.8
56.7
59.1
40.3
59.7
57.8
56.7
56.7
40.3
59.7
57.8
56.7
56.7
40.3
59.7
57.8
56.7
cont
CoTTA
70.9 62.4
59.7
58.6
70.9
62.6
59.7
58.6
70.9
62.7
59.7
58.6
70.9
41.0
62.7
99.7
58.6
70.9
410
62.8
59.7
58.6
11.9
68.3
Ours
21.6 43.2 660 63.4
61.1
75.2
44.5
67.0
63.9
62.2
73.2
44.6
67.2
64.2
62.3
70.9
44.0
6600
63.2
610
720
43.7
66.3
63.1
61.3
COM
Round
10
Mean
Method
Fog Nighi Kain Snow
Mean
Fog
Nightaain Snow
POg
Night
Snow
Moan
Pog
Night
Rai
Snow
Mcan
Fog
Nipha
Snin Mcan
Source
59.1 40.3 59.7 57.8 36.7
59.1 10.3
59.7 57.8 56.7
59.1
10.3 59.7 57.8
56.7
56.7
10.3
59.7 57.8
56.7
56.7
10.3
59.7
57.8
56.7
56.7
CoTTA
70.9 62.8 59.7
70.9
62.6
59.7
70.9
41.1 62.6
59.7
58.6
70.8
41.1 62.6
59.7
58.6
70.8
62.6
59.7
58.6
58.6
67.7
67.3
67.6
65.0
58.0
Ours
72.2 41.0
66.6
62.9
61.4
72.3
44.8
66.5
62.9
61.6
72.1
45.1
66.2
62.9
61.5
71.9
45.3
66.3
62.9
61.5
72.2
45.2
66.5
62.9
61.6
61.6
D ADDITIONAL QUALITATIVE ANALYSIS
To further validate the effectiveness of our proposed method, we present additional qualitative comparisons on the Cityscapes-to-ACC model (Xie et al., (021) the source domain and subsequently adapt it to four target domains in ACDC. order to assess the performance of our approach, we conduct a qualitative comparison with two leading methods, namely CoTTA (Wang et al., 2022) and VDP (Gan et al., 2023). The visualizations of the segmentation outputs, obtained through the CTTA process, are depicted in Figure 7 Our method exhibits better segmentation map compared to CoTTA and VDP across all four target domains, as it effectively distinguishes the sidewalk from the road (shown in white box). This demonstrates the capability of method to achieve more accurate segmentation results while mitigating the impact of dynamic domain shifts. Moreover, in the other categories, our method's segmentation maps closely resemble the Ground Truth, leading visual enhancements. Lastly, have included a video visualization in the supplementary material that showcases comprehensive comparison of segmentation performance. This video provides dynamic and visual representation of the results obtained from our experiments.
CTTA scenario. Initially, we pre-train the Segformer-Bi
Image
CoTTA
VDP
ViDA
Ground Truth
Fog
Night
Rain
Snow
Haget
Figure 7: Qualitative comparison of our method with previous SOTA methods on the ACDC dataset.
Our method could better segment different pixel-wise classes such as shown in the white box.
21

---

Published as a conference paper at ICLR 2024
Table 15: A fine-grained Classification error rate(%) for standard CIFARIO0+-CIAFARIIC online
CTTA task. Results are evaluated on ViT-base.
clastic trant
Method
Eausssion
shoot
impalse
defocus
gloss
modion
Z0000
STOW
frust
foy
bri.
contrast
pixellte
ipeg
Mean,, Gain
Source
60.1
53.2
38.3
19.9
35.5
22.6
18.6
12.1
12.7
22.8
5.3
49.7
23.6
24.7
23.1
28.2
0.0
Pseudo-label
2013
59.8
37.2
17.6
11.6
12.3
20.7
41.7
21.
25.2
26,9
TENT-contincal Wane èt al. 2021
57.7
56.3
29.4
16.2
35.3
16.2
12.4
11.0
11.6
14.9
4.7
22.5
15.9
29.1
19.5
23.5
+4.7
CoTTA [Wang al 2022
58.7
51.3
33.0
20.1
34
152
11.1
11.3
18.5
4.0
34.7
18.8
19.0
17.9
24.6
+3.6
GDP
57.5
31.7
22.5
Ours (proposed)
52,9
47.9
19.4
11.4
31.3
13.3
7.6
7.6
9.9
12.5
3.8
26.3
14.4
33.9
18.2
20.7
+7.5
Table 16: A fine-graines Classification error rate( for standard DUTARI00-O-CIATARIIOO online
CTTA task. Results are evaluated on ViT-base.
elastic Irans
Method
gaussion
shot
impulsee
defocus
glass
monioo
200m
500W
Fmoot
fog
Pixelate
jpeg
Mean Gain
Source
$5.0
51.5
26.9
24.0
60.5
29.0
21.4
21
25.0
35.2
11.8
34,8
43.2
56.0
35.9
35.4
0.0
Pseudo-label Loe
2013
53.8
48.9
23.0
58.7
27.3
19.6
20.6
23,4
31.3
11.8
39.6
52.3
33.9
33.2
+2.2
TENT-comtmuaT
202
47.0
22.3
58.5
26.5
23.0
47.1
333
32.1
+3.3
CoITA (Wang et m.
2022
$5.0
51.3
25.8
24.1
59.2
28.9
21.4
21.
24.7
34.9
11.7
31.7
40.4
55.7
35.6
34.8
+0.6
(Gan al. 2023
51.2
24.2
59.1
28.8
21.2
20.5
23.3
32.0
51.7
32.0
Ours (prposed)
50.1
40.7
22.0
21.2
45.2
21.6
16.5
17.9
16.6
25.6
11.
29.0
29.6
34.7
27.1
27.3
+8.1
E FINE-GRAINED PERFORMANCE
In this section, we expand upon the classification results presented in submission by providing details of fine-grained performance. We assess the error rates across fifteen corruption types to gain deeper insights. To be specific, augment the information provided in Table of our submission with the additional details presented Table 15 and [6] These tables offer a comprehensive view of the performance of our approach in addressing the CIFAR- 10FAR-10-to-- C-B-tt-CIFAR---- R0C and CIFAR-100-to- CIFAR- 1000 CTTA scenarios, respectively.
22


Reviews:
[{"review_id": "VzKQXyoMoQ", "reviewer": "Reviewer_8EgD", "summary": "The paper introduces a Continual Test-Time Adaptation (CTTA) method called Visual Domain Adapter (ViDA), which addresses the challenges of error accumulation and catastrophic forgetting in models operating in dynamic environments. ViDA differentiates itself by leveraging both high-rank and low-rank feature spaces to maintain domain-specific and shared knowledge. The HKA(Homeostatic Knowledge Allotment) is also proposed to balance the integration of these features dynamically. This proposed method is evaluated on multiple benchmarks, showing good performance in classification and segmentation tasks.", "questions": "- Regarding Figure 1, it would be insightful to include the t-SNE results for each transformer block. Such detailed visualizations would help to understand feature spaces across the different layers of the network.\n\n- Since t-SNE visualizations represent relative rather than absolute distances, I am curious about the quantitative differences in domain distribution distances within the ACDC dataset when comparing low-rank to high-rank representations. Quantifying this difference could bolster the demonstration of your method's efficacy in managing domain shifts.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents an intriguing exploration of the differential capabilities of low-rank and high-rank features in domain adaptation, suggesting potential for a wide range of applications. The analysis substantiates the underlying mechanisms of the proposed method's operation. Also, the proposed method is validated through extensive experiments. The experiments include several benchmarks in classification and segmentation tasks.", "weaknesses": "- Observing the t-SNE plots in the low-rank branch (Figure 1), it appears that the nighttime representation in the ACDC dataset is not aligning as a domain-shared characteristic. I think nighttime data is not clustering in the low-rank space, which may indicate unique domain characteristics. Quantitative analysis with clustering metrics could clarify this.\n\n- In Figure 3(a), the inter-domain divergence does not seem to exhibit significant differences between high-rank and low-rank features across conditions c1 to c9. Does this imply that the effectiveness of the rank-based approach is sensitive to the type of corruption present in the data? Understanding this could inform the robustness of the proposed method across a wider array of scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EhpDO6IeGt", "reviewer": "Reviewer_RZSt", "summary": "This paper studies the problem of continual test-time model adaptation, where training data is not accessible and only continually changing target domains are available. To address error accumulation and catastrophic forgetting problems, they propose a homeostatic Visual Domain Adapter (ViDA) which explicitly manages domain-specific and task-relevant knowledge. Moreover, a Homeostatic Knowledge Allotment (HKA) strategy is introduced to dynamically merge knowledge from low-rank and high-rank ViDAs. Extensive experiments on standard TTA benchmarks demonstrate the effectiveness of the proposed approach.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The problem of tackling continuously shifting target domains represents a substantial challenge and is an area that has not been sufficiently explored in existing literature.\n\n- The paper is articulately composed and comprehensible, effectively communicating the core ideas of the study to the reader. I am confident that I have acquired a solid understanding of the authors’ work through my examination of the manuscript.\n\n- Experiments are comprehensive with a variety of benchmark tasks.", "weaknesses": "- My main concern is the novelty. Separating the extracted features into domain-invariant and domain-specific components has been extensively explored in previous domain adaptation and domain generalization methods. It lacks clear pieces of evidence to show why the proposed method is preferable in the context of test-time adaptation.\n\n- The experimental results, as illustrated in Table 1, indicate that the improvements achieved by the proposed method in comparison to previous approaches, such as CoTTA, are marginal.\n\n- The section of the paper discussing large models, as well as the associated experiments, appear incomplete and somewhat abrupt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NfKNYW5Ptp", "reviewer": "Reviewer_4MXU", "summary": "The paper proposes to add two branches of low-rank and high-rank adapters to handle the problem of continuous test time adaption. The paper suggests that the low-rank adapter acquires domain-agnostic information, whereas the high-rank adapter acquires domain-specific knowledge. In addition, the paper proposes a Homeostatic Knowledge Allotment (HKA) technique for determining the contribution of the two branches. Experiments are carried out for the image classification task on the ImagenetC, CIFAR10C, and CIFAR100C benchmarks, as well as for semantic segmentation on the Cityscapes-to-ACDC benchmark. The experimental findings show that the proposed mechanism is successful, producing state-of-the-art outcomes for some pre-trained deep neural network architectures.", "questions": "1. There is no theoretical justification, nor even an intuitive explanation, for why the low-rank adapter acquires domain-agnostic information whereas the high-rank adapter collects domain-specific knowledge other than experimental observation.\n2. The backbone architecture in some experiments differs from prior developments, such as CoTTA.\n3. What about the performance of the proposed approach CIFAR100-to-CIFAR100C method for ResNeXt-29 architecture?\n4. Using the source data, retrain the \"model added with low/high-rank adapters\" for a few steps. Without access to the source domain data, it is impossible to employ off-the-shelf pre-trained models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper employs two distinct adapters for acquiring domain-specific and domain-agnostic knowledge: low-rank and high-rank adapters.\n2. The HKA technique for updating the weight contribution of these adapters depending on the prediction uncertainty value is a good idea.\n3. The contribution of several components is demonstrated in the ablation analysis.\n4. In the TTA scenario, experimental results for zero-shot generalization are novel.", "weaknesses": "1. There is no theoretical justification or an intuitive explanation for why the low-rank adapter acquires domain-agnostic information, whereas the high-rank adapter collects domain-specific knowledge other than experimental observation.\n2. The backbone architecture in some experiments differs from prior developments, such as CoTTA.\n3. What about the performance of the proposed approach CIFAR100-to-CIFAR100C method for ResNeXt-29 architecture?\n4. Using the source data, retrain the \"model added with low/high-rank adapters\" for a few stages. Without access to the source domain data, it is impossible to employ off-the-shelf pre-trained models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PALu2IbKtv", "reviewer": "Reviewer_Ut2z", "summary": "The authors have proposed a method for continual test time adaptation and are improving previous methods significantly. They do so by incorporating a low-rank and a high-rank embedding scheme where high-rank scheme is shown to be sensitive towards domain specific features whereas low-rank scheme is shown to be sensitive towards domain-invariant features. Additionally, to weigh these features during adaptation, they introduced a homeostatic knowledge allotment strategy where they exploit the probabilistic nature of multiple forward passes to get the best set of parameters. Overall, the authors are achieving reasonably well results.", "questions": "I recommend clarifying some confusions from the weakness section about the non-linearity of the model. \n- In what sense the is not any non-linear layer in ViDA? Is there no activation? How are you implementing this network? Is there a direct linear projection of ViDA? It would be important to understand the relationship of the layers within the model.\n- How was the parameter 𝜭 in equation 5 set?\n- Additionally, I suggest incorporating details such as the ablation study on homeostatic knowledge allotment from the appendix to the main paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Continual Test Time Adaptation is a relatively new and challenging problem in the vision community. The authors have incorporated a paradigm that does just that. \n\nOriginality\n- This particular challenge is still at its infancy and the proposed ideas in this paper can help mature the challenge. In terms of originality, the authors have incorporated similar to [1] with their domain specific and domain invariant learning scheme; however, these proposed ideas have shown to contribute significantly, and they have shown it in the results. Additionally, their homeostatic knowledge allotment scheme is also quite useful in terms of what features should weigh more than the other. Therefore, I believe the work is original and is contributing to the field.\n\nQuality\n- Since the challenge has a very big  real world applicability, and the approach has shown to perform very well in this regard.\n- Additionally, the authors have shown extensive experiments on various standard datasets where they outperform other SOTA approaches including CoTTA [2] and VDP [1].\n- The authors have shown extensive ablation study and motivation to use the different components of the approach in the paper.\n\nClarity\n- The paper is well written and easy to follow.\n- The authors have clarified their statements with equations and experiments. \n\nSignificance\n- As mentioned previously, the CTTA has a lot of real-world applicability in IID and non-IID scenarios, and recently, works have been tackling both of them in different scenarios. This work has shown perform boost in this regard, and the authors have solifdified their claims with extensive experiments. \n- Additionally, as the paper mentions, it can be integrated with any networks very easily further boosting the applicability of the method.\n- This work has potential to contribute significantly to the community.\n\nReferences:\n1. Gan, Y., Bai, Y., Lou, Y., Ma, X., Zhang, R., Shi, N. and Luo, L., 2023, June. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 6, pp. 7595-7603).\n2. Wang, Q., Fink, O., Van Gool, L. and Dai, D., 2022. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7201-7211).", "weaknesses": "Concerns\n- The authors mentioned that there is not any non-linear layer in ViDA. It is not clear what they mean by that. There is no activation? How are they implementing this network? The authors do mention that there is a direct linear projection of ViDA; however, I would like to understand the relationship of the layers within the model.\n- How did the authors decide parameter 𝜭 in equation 5? Was it decided empirically?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)