Title: ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
Venue: NeurIPS 2023
Accepted: no
Keywords: Domain Adapter

Continual Test Time Adaptation

Efficient Fine-tuning
Abstract: Since real-world machine systems are running in non-stationary and continually changing environments, Continual Test-Time Adaptation (CTTA) task is proposed to adapt the pre-trained model to continually changing target domains. Recently, existing methods mainly focus on model-based adaptation, which aims to leverage a self-training manner to extract the target domain knowledge. However, pseudo labels can be noisy and the updated model parameters are uncertain under dynamic data distributions, leading to error accumulation and catastrophic forgetting in the continual adaptation process. To tackle these challenges and maintain the model plasticity, we tactfully design a Visual Domain Adapter (ViDA) for CTTA,  explicitly handling both domain-specific and domain-agnostic knowledge. Specifically, we first comprehensively explore the different domain representations of the adapters with trainable high and low-rank embedding space. Then we inject ViDAs into the pre-trained model, which leverages high-rank and low-rank prototypes to adapt the current domain distribution and maintain the continual domain-shared knowledge, respectively. To adapt to the various distribution shifts of each sample in target domains, we further propose a Homeostatic Knowledge Allotment (HKA) strategy, which adaptively merges knowledge from each ViDA with different rank prototypes. Extensive experiments conducted on four widely-used benchmarks demonstrate that our proposed method achieves state-of-the-art performance in both classification and segmentation CTTA tasks. In addition, our method can be regarded as a novel transfer paradigm and showcases promising results in zero-shot adaptation of foundation models to continual downstream tasks and distributions.

1

Abstract

Since real-world machine systems are running in non-stationary and continually changing environments, Continual Test-Time Adaptation (CTTA) task is proposed to adapt the pre-trained model to continually changing target domains. Recently, existing methods mainly focus on model-based adaptation, which aims to leverage a self-training manner to extract the target domain knowledge. However, pseudo labels can be noisy and the updated model parameters are uncertain under dynamic data distributions, leading to error accumulation and catastrophic forgetting in the continual adaptation process. To tackle these challenges and maintain the model plasticity, we tactfully design a Visual Domain Adapter (ViDA) for CTTA, explicitly handling both domain-specific and domain-agnostic knowledge. Specifically, we first comprehensively explore the different domain representations of the adapters with trainable high and low-rank embedding space. Then we inject ViDAs into the pre-trained model, which leverages high-rank and low-rank prototypes to adapt the current domain distribution and maintain the continual domain-shared knowledge, respectively. To adapt to the various distribution shifts of each sample in target domains, we further propose a Homeostatic Knowledge Allotment (HKA) strategy, which adaptively merges knowledge from each ViDA with different rank prototypes. Extensive experiments conducted on four widely-used benchmarks demonstrate that our proposed method achieves state-of-the-art performance in both classification and segmentation CTTA tasks. In addition, our method can be regarded as a novel transfer paradigm and showcases promising results in zero-shot adaptation of foundation models to continual downstream tasks and distributions.

1 Introduction

Deep Neural Networks (DNN) have achieved remarkable performance in various computer vision tasks, such as classification [22, 14], object detection [48, 63], and segmentation [9, 58], when the test data distribution is similar to the training data. However, real-world machine perception systems (i.e., autonomous driving [1, 28]) operate in non-stationary and constantly changing environments, which contain heterogeneous and dynamic domain distribution shifts. Applying a pre-trained model in these real-world tasks [50] can lead to significant degradation in perception ability on target domains, especially when the target distribution changes unexpectedly over time. Therefore, developing continual domain adaptation (DA) methods that can enhance the generalization capability of DNNs and improve the reliability of machine perception systems in dynamic environments.

Aclassical source-free DA task, Test-Time Adaptation [39] (TTA), eases the distribution shift between 33 a source domain and a fixed target domain. This is typically achieved through the utilization of 34 self-training mechanisms [42, 55]. However, when adapting to continually changing target domains, 35

pseudo labels are noisy and the updated model parameters become uncertain, leading to error 36

(a) Continual Test-Time Adaptation

(b) Feature representations of different branches

Domain 1 (Fog)

Domain 2 (Night)

Domain 3 (Rain)

Domain 4 (Snow)

Ours: Low-rank Branch

The Pre-trained Parameter

Ours: High-rank Branch

Domain-agnostic

Domain-specific

Pre-trained

Parameter

Low-Rank

High-Rank

Domain

agnostic

Domain

specific

Online

Adapted

Network

Continually Changing Target Environments

Domain 1

(Fog)

Domain 2

(Night)

Domain 3

(Rain)

Domain 4

(Snow)

accumulation and catastrophic forgetting. To tackle this problem, Continual Test-Time Adaptation 37 (CTTA) has been proposed [57], which addresses a sequence of different distribution shifts over time 38 rather than a single shift as in TTA. Furthermore, CTTA also encompasses the efficient zero-shot 39 adaptation of foundation models to continual downstream tasks or distributions [2, 29]. 40

Existing CTTA works [57, 7, 16, 59] have primarily employed model-based and prompt-based ap41 proaches to extract target domain-specific and domain-invariant knowledge simultaneously. However, 42 for model-based methods [57, 7], the noisy pseudo labels are still unreliable and play a limited role in 43 avoiding error accumulation, particularly in scenarios with significant distribution gaps. Meanwhile, 44 prompt-based methods [16, 59] face difficulties in leveraging soft prompts with limited trainable 45 parameters to learn long-term domain-shared knowledge and prevent catastrophic forgetting. 46

To tackle these limitations and maintain the model plasticity, we tactfully design a homeostatic 47 Visual Domain Adapter (ViDA), shown in Fig. 1 (a), which explicitly manages domain-specific 48 and domain-agnostic knowledge in the continual adaptation process. Specifically, we first carefully 49 explore the different domain representations of ViDAs with trainable high and low-rank embedding 50 space. Our observations reveal that ViDA with a low-rank prototype focuses on domain-agnostic 51 feature representation in different domains. As shown in Fig. 1 (b), the prototype distribution of the 52 adapter neglects the influence of dynamic distribution shifts. Conversely, ViDA with a high-rank 53 prototype concentrates more on extracting domain-specific knowledge, as evidenced by the prototype 54 distribution in different target domains showing an obvious discrepancy. We provide a detailed 55 explanation of the motivations in Section 3.1. 56

This observation motivates us to inject ViDAs into the pre-trained model, which leverages high and 57 low-dimension prototype to adapt current domain distribution and maintain the continual domain58 shared knowledge, respectively. According to the various distribution shift of each sample, we further 59 propose a Homeostatic Knowledge Allotment (HKA) strategy to dynamically fuse the knowledge 60 from each ViDA with different dimension prototypes. In Fig. 1 (b), HKA adaptively regularizes the 61 balance of different feature representations, including original model, domain-specific, and domain62 agnostic features. During inference, the different domain-represented ViDAs can be projected into 63 the pre-trained model by re-parameterization [13], which ensures no extra parameter increase and 64 maintain the model plasticity. In addition, through the proposed homeostatic ViDAs, we empower 65 the model with domain generalization ability, which achieves a significant improvement (+7.6%) on 66 the five unseen target domains of ImageNet-C. In summary, our contributions are as follows: 67

68

69

70

71

We carefully study the different domain representations of the adapters with high and lowrank prototypes. And we tactfully design a Visual Domain Adapter (ViDA) for CTTA, explicitly managing domain-specific and domain-shared knowledge to tackle the error accumulation and catastrophic forgetting problem, respectively.

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

According to the various distribution shift of each sample in the target domains, we further propose a Homeostatic Knowledge Allotment (HKA) strategy to dynamically fuse the knowledge from each ViDA with different rank prototypes.

Our proposed approach outperforms most state-of-the-art methods according to the experiments on four benchmark datasets, covering classification and segmentation tasks.

Our CTTA method provides a novel transfer paradigm and achieves a promising result in zero-shot adapting of foundation models to continual downstream distributions. Meanwhile, we empower the source model with domain generalization ability through the proposed homeostatic ViDAs, achieving a significant improvement on the unseen target domains.

2 Related work

2.1 Continual Test-Time Adaptation

Test-time adaptation (TTA) , also referred to as source-free domain adaptation [6, 34, 40, 60], aims to adapt a source model to an unknown target domain distribution without relying on any source domain data. Recent research has explored self-training and entropy regularization techniques to fine-tune the source model [35, 56, 40, 8]. Tent [56] updates the training parameters in batch normalization layers by minimizing entropy. Recently, there has been a surge of interest in performing Transformer-based TTA works [57, 20, 20]. Continual Test-Time Adaptation (CTTA) refers to a scenario where the target domain is not static, presenting additional challenges for traditional TTA methods. The first approach to address this challenging task is introduced in [57], which combines bi-average pseudo labels and stochastic weight reset. While [57, 7] tackles the problem in both classification and segmentation tasks at the model level, [16] introduces the use of visual domain prompts to address the issue at the input level specifically for the classification task. In this paper, we simultaneously focus on both classification tasks and dense prediction tasks.

2.2 Parameter-Efficient Fine-Tuning

Recently, Parameter-Efficient Fine-Tuning (PEFT) has gained significant traction within the field of natural language processing (NLP) [30, 26, 25, 61, 37, 27, 19, 23, 54, 45]. Adapter-based models, a form of PEFT, have gained popularity in NLP. They employ bottleneck architecture adapter modules inserted between layers in pre-trained models. During fine-tuning, only these modules are updated. Adapter-based models demonstrate dominant performance over other methods in certain tasks, sometimes surpassing standard fine-tuning [12]. Inspired by NLP, adapters in visual tasks have also received widespread attention. In the initial phases of adapter development, residual adapter modules [46, 47] are proposed to aid in the effective adaptation of convolutional neural networks across multiple downstream tasks. AdaptFormer [10] enhances the ViT [14] model by replacing the original multi-layer perceptron (MLP) block with AdaptMLP. AdaptMLP introduces a trainable down-to-up bottleneck module in a parallel manner, effectively mitigating catastrophic interference between tasks. VL-Adapter [51] improves the efficiency and performance of adapters by sharing low-dimensional layers weights to attain knowledge across tasks. Existing methods, as mentioned, have not addressed the challenges of long-term preservation of domain-agnostic knowledge and timely exploration of domain-specific knowledge amidst continuous unknown domain variations. Consequently, there is an urgent demand for an adapter with different domain representations that can simultaneously tackle the challenges of error accumulation and catastrophic forgetting.

3 Method

In Continual Test-Time Adaptation (CTTA), we pre-train the model q θ ( y | x ) on the source domain D S = ( Y S , X S ) and adapt it on multiple target domains D T i = { ( X T i ) } n i =1 , where n represents the scale of the continual target datasets. The entire process can not access any source domain data and can only access target domain data once. The distributions of the target domains (i.e., D T 1 , D T 2 , ..., D T n ) are constantly changing over time. Our goal is to adapt the pre-trained model to target domains and maintain the perception ability of the model on the seen domain distribution.

Our approach proposes a novel Visual Domain Adapter (ViDA) that contains both high and lowdimensional prototypes. This design allows us to explicitly manage domain-specific and domain-

Augment

Target Domain Input

Pre-trained Source Model

Consistency Loss

Back-prop

Prediction

Loss function

Student model

Homeostatic Knowledge Allotment &

1

2

3

4

5

6

7

8

9

10

Uncertainty

Value

1

2

3

4

5

6

7

8

9

10

Data Flow

Teacher model

EMA

ViDAs

Original

Linear or Conv

Low-Rank

High-Rank

Feature

：

agnostic

specific

GLYPH(cmap:d835)

GLYPH(cmap:d835)

f

GLYPH(cmap:d835)

× GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)ℎ

× GLYPH(cmap:d835)ℎ

Distribution shift

Reflect

Guide

GLYPH(cmap:d835)

GLYPH(cmap:d835)

&

GLYPH(cmap:d835)

GLYPH(cmap:d835)

, Eq.(4)

(a)

(b)

Eq.(3)

Eq.(2)

up

down

up

down

agnostic knowledge, addressing the challenges of error accumulation and catastrophic forgetting in 122 CTTA. To effectively adapt to the diverse distribution shifts, a Homeostatic Knowledge Allotment 123 (HKA) strategy is introduced to dynamically fuse the knowledge from different ViDA with different 124 domain representations. The overall framework is shown in Fig. 2. 125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

3.1 Motivation

The Continual Test-Time Adaptation (CTTA) faces significant challenges, primarily due to error accumulation and catastrophic forgetting [57, 16]. Meanwhile, adapters with different dimension prototypes demonstrate remarkable effectiveness in addressing these challenges. This encourages us to take a step further and investigate the principles underlying the use of domain adapters in CTTA.

Adapter with low rank prototype. Our hypothesis regarding the effectiveness of adapters in mitigating catastrophic forgetting is that their low-rank prototype representation plays a crucial role. To explore this further, we conduct a t-SNE study [53] on the third transformer block to analyze the feature distributions across four target domains (ACDC). The results are depicted in Fig. 1 (b). Our analysis reveals that the low-rank adapter exhibits a relatively consistent distribution across the different target domains, suggesting that its low-rank prototype can effectively disregard the impact of dynamic distribution shifts and prioritize the extraction of domain-invariant knowledge.

We adopt the domain distance definition proposed by Ben-David [4, 3] and build upon previous domain transfer research [18] by employing the H -divergence metric to further evaluate the domain representations of adapters across different target domains. H -divergence between D S and D T i can be calculated as d H ( D S , D T i ) = 2 sup D∼H | Pr x ∼ D S [ D ( x ) = 1] -Pr x ∼ D T i [ D ( x ) = 1] | , where H denotes hypothetical space and D denotes discriminator. Similar to [18], calculating the H -divergence directly is challenging. We adopt the Jensen -Shannon ( JS ) divergence between two adjacent domains as an approximation. To investigate the effectiveness of adapters in adapting to continual target domains, we compare the JS values obtained by using the source model alone, injecting low-rank adapter, and combining low-high adapters, as illustrated in Fig. 3 (a). Our results indicate that the feature representation generated by the low-rank adapter exhibits lower divergence compared to those of the original source model and closely resembles the values of low-high combination.

To provide clearer evidence for our assumption, we have developed an evaluation approach that 149 directly reflects the extent of domain catastrophic forgetting. Shown in Table 1, after one round of 150 CTTA on all target domains (ImageNet-C), we utilize the model and adapter from the last target 151 domain to directly test on previously seen target domains. As expected, the performance degradation 152 is observed in only 2 out of 15 corruption types, and there is an overall improvement of 1.0% in 153 the average classification error. These findings further support our assumptions and indicate that 154 low-rank adapters are more effective in preserving continual domain-shared knowledge. 155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

c6

c7

→

c12

c13

→

c3

c4

→

c9

c10

→

Domain Shifts

(a) Inter-Domain Divergence

c2

c4

c6

c8

c10

(b) Intra-class Divergence

0.6

1.0

-Source

Low-rank

←—High-rank

-Source

0.5

*-ViDA

*ViDA

0.4

0.3

0.2

0.1

+00

c12

c14

Adapter with high rank prototype. Regarding the domain representation of the adapter with a high-rank prototype, we propose that it is better suited to address error accumulation in the continual adaptation process. We verify this by visualizing the prototype distributions between different domains, as shown in Fig. 1 (b), and observe that there is a clear discrepancy between domains. And the distribution achieves a better aggregation in a single domain. This suggests that high-rank adapters primarily focus on extracting domain-specific knowledge in continual target domains. Inspired by intra-cluster dissimilarity proposed by k -means [41], we use normalized intra-class divergence to further verify the domain representations of high-rank adapters in CIFAR10C. As illustrated in Fig. 3 (b), the high-rank adapter is found to drive down divergence within almost all domains, indicating that it can better adapt to current domain distribution and extract domain-specific knowledge in continual target domains. To straightforwardly measure it, we quantitatively evaluate its performance. As shown in Table 6 Ex 2 , the classification error rate exhibits a sustained reduction (-4.6%) in the dynamic target domains with the use of a high-rank adapter. This finding supports our hypothesis that high-rank adapters can extract more reliable domain-specific knowledge.

3.2 Visual Domain Adapter

The above observation motivates us to introduce high-rank and low-rank Visual Domain Adapters (ViDAs) into the source pre-trained model, aiming to simultaneously adapt current domain distribution and maintain the continual domain-shared knowledge in CTTA.

The architecture. The design principle of injecting ViDAs into the pre-trained model is simple yet effective, which is illustrated in Figure .2 (b). As we can see there are three sub-branches, the linear (or Conv) layer in the middle branch is identical to the original network, while the right branch and left branch are bottleneck structures and separately indicate the high-rank ViDA and low-rank ViDA. Specifically, the right branch (high-rank) contains an up-projection layer with parameters W h up ∈ R d × d h , a down-projection layer with parameters W h down ∈ R d h × d , where d h (i.e., d h = 128 ) is the middle dimension of high-rank prototype and satisfies d h ≥ d . There is not any non-linear layer in the ViDA. And we utilize the linear layer as the projection layer when the original model is transformer architecture and adopt 1 × 1 Conv as the projection layer when the original model is a convolution network. In contrast, the left branch (low-rank) first injects a down-projection layer with parameters W l down ∈ R d × d l , then place an up-projection layer with parameters W l up ∈ R d l × d , where d l (i.e., d l = 1 ) stand for the middle dimension of the low-rank prototype ( d l ≪ d ). For a input feature f , the produced features of high-rank ViDA ( f h ) and low-rank ViDA ( f l ) are formulated as:

The two-branch bottleneck is connected to the output feature of the original network ( f o ) through the residual connection via scale factors ( λ h and λ l ). The fusion knowledge ( f f ) can be described as:

The domain knowledge scale factors ( λ h and λ l ) are adaptively obtained through the homeostatic knowledge allotment strategy, which is shown in Section 3.3.

Continual adapting. During the continual adaptation process, we freeze the parameters of the original model (middle branch) and update the high-rank ViDA and low-rank ViDA on the dynamic target domains with unsupervised loss. During inference, the different domain-represented ViDAs (linear relation) can be projected into the pre-trained model by re-parameterization [13], which ensures no extra parameter increase and maintain the plasticity of the original model.

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

3.3 Homeostatic Knowledge Allotment

Method motivation. In CTTA, the target domain data can only be accessed once and show different distribution shifts, which makes the efficiency of domain transfer crucial. Moreover, to tackle error accumulation and catastrophic forgetting effectively, it becomes necessary to extract different domain knowledge and handle them separately. This requires regularization of the knowledge fusion weight to ensure efficient capture of relevant domain-specific knowledge without sacrificing the retention of long-term domain-shared knowledge. HKAdesign. As depicted in Figure .2 (b), we draw inspiration from [44, 49, 17] and introduce an uncertainty value to quantify the degree of distribution shift for each sample. While the confidence score is a common measure to assess prediction reliability, it tends to fluctuate irregularly and becomes unreliable in scenarios characterized by distribution shifts. To address this limitation, we employ the MC Dropout technique [15] on linear layers, enabling multiple forward propagations to obtain m sets of probabilities for each sample. Subsequently, we calculate the uncertainty value U ( x ) for a given input x , which are formulated as:

Where p i ( y | x ) is the predicted probability of the input x in the i th forward propagation and µ is the average value of m times prediction. To dynamically adjust the scale factors ( λ h and λ l ) based on the uncertainty score, the formulation is as follows:

The threshold value of uncertainty is denoted as Θ , where Θ = 0 . 2 . To realize the homeostasis of different domain knowledge, when facing the sample with a large uncertainty value, we adaptively increase the fusion weight of domain-specific knowledge ( λ h ). Conversely, if the input has a low uncertainty value, the fusion weight of domain-agnostic knowledge ( λ l ) will be increased. By employing the HKA strategy, our approach ensures that the adaptation process effectively captures relevant domain-specific knowledge while retaining long-term domain-shared knowledge.

3.4 Optimization Objective

Following previous CTTA work [57, 16], we leverage the teacher model T to generate the pseudo labels ˜ y for updating ViDAs. And we adopt consistency loss L ce as the optimization objective.

Where ˆ y is the output of our student model S , C means the number of categories. Same as previous works[57, 16], we load the source pre-trained parameters to initialize the weight of both models and adopt the exponential moving average (EMA) to update the teacher model with ViDAs.

Where t is the time step. And we set α = 0 . 999 [52], which is the updating weight of EMA.

4 Experiment

In Section 4.2 and 4.3, we compare our method with other SOTA methods on classification and segmentation of CTTA. In Section 4.4, we employ the foundation model [32, 43] as the backbone and evaluate the efficacy of our method. In Section 4.5, we further evaluate the domain generalization ability of the proposed method. Comprehensive ablation studies are conducted in Section 4.6. More quantitative comparisons and qualitative analyses are shown in the supplementary materials.

4.1 Task settings and Datasets

Dataset. We evaluate our method on three classification CTTA benchmarks, including CIFAR10to-CIFAR10C(standard), CIFAR100-to-CIFAR100C [33] and ImageNet-to-ImageNet-C [24]. For

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

segmentation CTTA [57, 59], we evaluate our method on Cityscapes-to-ACDC, where the Cityscapes dataset [11] serves as the source domain, and the ACDC dataset [50] represents the target domains.

Baselines. We compare the proposed method against two types of CTTA approaches, including (1)Modal-based: source model [14, 58], Pseudo-label [36], Tent-continual [56], CoTTA [57], and, SATA [7]. (2) Prompt-based: visual domain prompt [16].

CTTA Task setting. Following [57, 16], in classification CTTA tasks, we sequentially adapt the pre-trained source model to the fifteen target domains with the largest corruption severity (level 5). The online prediction results were evaluated immediately after encountering the input data. Regarding segmentation CTTA [57, 59], the source model [58] is an off-the-shelf pre-trained on the Cityscapes dataset [11]. As for the continual target domains, we utilize the ACDC dataset [50], which consists of images collected in four unseen visual conditions: Fog, Night, Rain, and Snow. To simulate continual environmental changes in real-life scenarios, we cyclically repeat the same sequence of target domains (Fog → Night → Rain → Snow) multiple times.

Implementation Details. In our CTTA experiments, we follow the implementation details specified in previous works [57, 59] to ensure consistency and comparability. we adopt ViT-base [14] and ResNet [22] as the backbone in classification CTTA. In the case of ViT-base, we resize the input images to 224x224, while maintaining the original image resolution for other backbones. For segmentation CTTA, we adopt the pre-trained Segformer-B5 model [58] as the source model. We down-sample the input size from 1920x1080 to 960x540 for target domain data [57]. The optimizer is performed using Adam [31] with ( β 1 , β 2 ) = (0 . 9 , 0 . 999) . We set the learning rates to specific values for each backbone, such as 1e-5 for ViT and 3e-4 for Segformer. To initialize our visual domain adapters, we train the model with adapters for one epoch on the source domain. We apply a range of image resolution scale factors [0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0] for the augmentation method and construct the teacher model inputs [57]. All experiments are conducted on NVIDIA A100 GPUs.

4.2 The Effectiveness on Classification CTTA

ImageNet-to-ImageNet-C. Given the source model pre-trained on ImageNet, we conduct CTTA on ImageNet-C, which consists of fifteen corruption types that occur sequentially during the test time. Table .1 demonstrates that the majority of methods employing the ViT backbone achieve lower classification errors compared to those using the ResNet50 backbone. For ViT-base, the average classification error is up to 55.8% when we directly test the source model on target domains. In contrast, our method can outperform all previous methods, achieving a 12.4% and 6.6% improvement over the source model and previous SOTA method, respectively. Moreover, our method showcases remarkable performance across the majority of corruption types, highlighting its effective mitigation of error accumulation and its capability for continual adaptation. After completing the entire CTTA process, we evaluate the performance of our method on the seen target domains. As shown in Table 1, the performance degradation is observed in only 2 out of 15 corruption types. Additionally, we achieve an overall improvement of 1.0% in the average classification error. These findings demonstrate that our method successfully preserves continual domain-shared knowledge and avoids catastrophic forgetting during CTTA. In conclusion, our homeostatic ViDAs can extract the different domain knowledge and avoid CTTA main challenges simultaneously.

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

To further validate the effectiveness of our method, we conduct experiments on CIFAR10-toCIFAR10C and CIFAR100-to-CIFAR100C. As illustrated in Table .2, in CIFAR10C, our approach achieved a 3.4% improvement compared to the previous SOTA model. We extend our evaluation to CIFAR100C, which comprises a larger number of categories in each domain. Our approach surpasses all previous methods, which show the same trend as the above CTTA experiments. Therefore, the results prove that our method mitigates the challenges posed by continual distribution shifts, regardless of the number of categories present in each domain.

4.3 The Effectiveness on Segmentation CTTA

Time

t

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - →

Round

1

2

3

Mean

↑

Gain

Method

REF

Fog Night Rain Snow Mean

↑

Fog Night Rain Snow Mean

↑

Fog Night Rain Snow Mean

↑

Source [58]

NIPS2021

69.1

40.3

59.7

57.8

56.7

69.1

40.3

59.7

57.8

56.7

69.1

40.3

59.7

57.8

56.7

56.7

/

TENT [55]

ICLR2021

69.0

40.2

60.1

57.3

56.7

68.3

39.0

60.1

56.3

55.9

67.5

37.8

59.6

55.0

55.0

55.7

-1.0

CoTTA [57] CVPR2022 70.9

41.2

62.4

59.7

58.6

70.9

41.1

62.6

59.7

58.6

70.9

41.0

62.7

59.7

58.6

58.6

+1.9

DePT [20]

ICLR2023

71.0

40.8

58.2

56.8

56.5

68.2

40.0

55.4

53.7

54.3

66.4

38.0

47.3

47.2

49.7

53.4

-3.3

VDP [16]

AAAI2023 70.5

41.1

62.1

59.5

58.3

70.4

41.1

62.2

59.4

58.2

70.4

41.0

62.2

59.4

58.2

58.2

+1.5

Ours

Proposed

71.6

43.2

66.0

63.4

61.1

73.2

44.5

67.0

63.9

62.2

73.2

44.6

67.2

64.2

62.3

61.9

+

5.2

road

sidew

build

wall

fence

pole

tr.light

tr.sign

veget

terrain

sky

person

rider

car

truck

bus

train

m.bike

bike

n/a.

Snow

Night

Image

CoTTA

VDP

ViDA

Ground Truth

Cityscapes-to-ACDC. To demonstrate the effectiveness of our method in the semantic segmentation CTTA task, we conducted evaluations on four target domains from the ACDC dataset periodically during test time. As presented in Table 4, we observed a gradual decrease in the mIoUs of TENT and DePT over time, indicating the occurrence of catastrophic forgetting. In contrast, our method has a continual improvement of average mIoU (61.1 → 62.2 → 62.3) when the same sequence of target domains is repeated. Significantly, the proposed method surpasses the previous state-of-the-art CTTA method [57] by achieving a 3.3% increase in mIoU. This notable improvement showcases our method's ability to adapt continuously to different target domains in the pixel-level task. In Fig .4, our method correctly distinguish the sidewalk from the road, avoiding mis-classification.

4.4 Continual Adapting for Foundation Models

Foundation models [5] are trained on large-scale datasets, endowing them with powerful generalization capabilities and the ability to capture representations of common features. However, performing full fine-tuning on the foundation model is time-consuming and economically impractical. Hence, our adaptation method proves valuable by enhancing the continual transfer performance of foundation

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

models. As indicated in Table. 3, we introduce foundation models as the pre-trained model and adapt them to continual target domains (CIFAR10C). Our approach achieved a performance improvement of 4.8% on the representative image-level foundation model DINOv2 [43] and 5.2% on pixel-level foundation model SAM [32]. Our method consistently and reliably improves the performance of the foundation model on the unseen continual target domains. Note that, we only use the pre-trained encoder of SAM and add a classification head, which is fine-tuned on the source domain. During the inference phase, the ViDAs with a linear relationship can be projected onto the pre-trained foundation model through re-parameterization. This process empowers the foundation model with the learned different domain representations and maintains the model plasticity.

4.5 Domain Generalization on Unseen Continual Domains

To investigate the domain generalization (DG) ability of our method, we follow the leave-one-domainout rule [62, 38] to leverage 10/15 domains of ImageNet-C as source domains for model training while the rest (5/15 domains) are treated as target domains without any form of adaptation. Specifically, we first use our proposed method to continually adapt the pre-trained model to 10/15 domains of ImageNet-C without any supervision. Then we directly test on the 5/15 unseen domains. Surprisingly, our method reduces 7.6% on the average error on unseen domains (Table 5), which has a significant improvement over other methods. The promising results demonstrate that our method possesses DG ability by effectively extracting domain-agnostic knowledge. This finding provides a new perspective on enhancing DG performance. More DG experiments are provided in the supplementary materials.

4.6 Ablation study

Effectiveness of each component. We conduct the ablation study on ImageNet-to-ImageNet-C CTTA scenario and evaluate the contribution of each component in our method, including high-rank ViDA ( V iDA h ), low-rank ViDA ( V iDA l ), and Homeostatic Knowledge Allotment (HKA) strategy. As shown in Table .6, Ex 1 represents the performance of the source pre-trained model (only 55.8%). In Ex 2 , by introducing the high-rank ViDA, the average error decrease 4.6%, demonstrating that the high-rank prototype can extract more domain-specific knowledge to adapt in target domains. As illustrated in Ex 3 , low-rank ViDA gains 5.1% improvement compared to Ex 1 . The result proves that the domain-share knowledge extracted from low-rank prototypes can also improve the classification ability on continual target domains. Ex 4 has a remarkable improvement of 10.2% overall, demonstrating that the two types of ViDA can compensate for each other in the continual adaptation process. Ex 5 achieves 12.4% improvement in total, showcasing the effectiveness of the HKA strategy in maximizing the CTTA potential of both types of ViDA.

5 Conclusion and Limitations

In this paper, we propose a homeostatic Visual Domain Adapter (ViDA) to address error accumulation and catastrophic forgetting problems in Continual Test-Time Adaptation (CTTA) tasks. And we investigate that the low-rank ViDA can disregard the impact of dynamic distribution shifts and prioritize the extraction of domain-invariant knowledge, and the high-rank ViDA can extract more reliable domain-specific knowledge. Meanwhile, we further propose a Homeostatic Knowledge Allotment (HKA) strategy to dynamically fuse the knowledge from each ViDA with different rank prototypes. For limitations, the injected ViDAs and teacher-student scheme brings extra parameters and computational costs during the continual adaptation process.

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

References

Eduardo Arnold, Omar Y Al-Jarrah, Mehrdad Dianati, Saber Fallah, David Oxtoby, and Alex Mouzakitis. A survey on 3d object detection methods for autonomous driving applications. IEEE Transactions on Intelligent Transportation Systems , 20(10):3782-3795, 2019.

Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. 2022.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79(1):151-175, 2010.

Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems , 19, 2006.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Malik Boudiaf, Tom Denton, Bart van Merriënboer, Vincent Dumoulin, and Eleni Triantafillou. In search for a generalizable method for source free domain adaptation. 2023.

Goirik Chakrabarty, Manogna Sreenivas, and Soma Biswas. Sata: Source anchoring and target alignment network for continual test time adaptation. arXiv preprint arXiv:2304.10113 , 2023.

Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. ArXiv , abs/2204.10377, 2022.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

Shoufa Chen, GE Chongjian, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In Advances in Neural Information Processing Systems .

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence , pages 1-16, 2023.

Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13733-13742, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Yulu Gan, Xianzheng Ma, Yihang Lou, Yan Bai, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. arXiv preprint arXiv:2212.04145 , 2022.

Yulu Gan, Mingjie Pan, Rongyu Zhang, Zijian Ling, Lingran Zhao, Jiaming Liu, and Shanghang 384 Zhang. Cloud-device collaborative adaptation to continual changing environments in the 385 real-world. arXiv preprint arXiv:2212.00972 , 2022. 386

387

388

389

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research , 17(1):2096-2030, 2016.

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL-IJCNLP 2021 , pages 3816-3830. Association for Computational Linguistics (ACL), 2021.

Yunhe Gao, Xingjian Shi, Yi Zhu, Hao Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, and Dimitris N Metaxas. Visual prompt tuning for test-time domain adaptation. arXiv preprint arXiv:2210.04831 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. international conference on computer vision , 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, and Luo Si. On the effectiveness of adapter-based tuning for pretrained language model adaptation. arXiv preprint arXiv:2106.03164 , 2021.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2225-2240, 2022.

Yu Huang and Yue Chen. Autonomous driving with deep learning: A survey of state-of-art technologies. arXiv preprint arXiv:2006.06091 , 2020.

Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. 2022.

Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems , 34:1022-1035, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643 , 2023.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Jogendra Nath Kundu, Naveen Venkat, Rahul M, and R. Venkatesh Babu. Universal source-free 432 domain adaptation. 2020. 433

Qicheng Lao, Xiang Jiang, and Mohammad Havaei. Hypothesis disparity regularized mutual 434 information maximization, 2020. 435

436

437

Dong-Hyun Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. 2013.

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 , 2021.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision , pages 5542-5550, 2017.

Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303.15361 , 2023.

Jian Liang, D. Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML , 2020.

J MacQueen. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability , pages 281-297. University of California Los Angeles LA USA, 1967.

Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen. Test-time adaptation to distribution shift by confidence maximization and input transformation. arXiv preprint arXiv:2106.14999 , 2021.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems , 32, 2019.

Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, et al. Exploring low-dimensional intrinsic task subspace via prompt tuning. arXiv preprint arXiv:2110.07867 , 2021.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems , 30, 2017.

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multidomain deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8119-8127, 2018.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems , 28, 2015.

Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, and Arno Solin. Uncertainty-guided source-free domain adaptation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXV , pages 537-555. Springer, 2022.

Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acdc: The adverse conditions dataset with correspondences for semantic driving scene understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10765-10775, 2021.

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5227-5237, 2022.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged 480 consistency targets improve semi-supervised deep learning results. Learning , 2017. 481

482

483

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

484

485

486

487

488

489

490

491

492

493

494

495

Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 5039-5059, 2022.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726 , 2020.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR , 2021.

Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. ArXiv , abs/2203.13591, 2022.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems , 34:12077-12090, 2021.

Senqiao Yang, Jiarui Wu, Jiaming Liu, Xiaoqi Li, Qizhe Zhang, Mingjie Pan, and Shanghang 496 Zhang. Exploring sparse visual prompt for cross-domain semantic segmentation. arXiv preprint 497 arXiv:2303.09792 , 2023. 498

Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Generalized 499 source-free domain adaptation. international conference on computer vision , 2021. 500

501

502

503

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199 , 2021.

504

505

Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization in vision: A survey. arXiv preprint arXiv:2103.02503 , 2021.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: 506 Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 507 2020. 508

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of continual test time adaptation by proposing to add two branches of low-rank and high-rank adapters.\nThe paper claims that the low-rank adapter learns the domain-agnostic knowledge, whereas the high-rank adapter captures the domain-specific knowledge.\nThe paper also proposes a Homeostatic Knowledge Allotment (HKA) strategy to discern the contribution of the two branches.\nExperiments are conducted for the classification problem on the benchmarks ImagenetC, CIFAR10C, and CIFAR100C, and for the semantic segmentation on the Cityscapes-to-ACDC benchmark.\nThe experimental results demonstrate the effectiveness of the proposed approach as it achieves state-of-the-art results using some selected pre-trained deep neural network architecture.\n\nSTRENGTHS:\n1. The paper tries to capture the domain-specific and domain-agnostic knowledge using two separate adapters: low-rank and high-rank adapters.\n2. The HKA strategy to update the weight contribution of these adapters based on the uncertainty value of prediction is a nice idea.\n3. Ablation study demonstrates the contribution of different components.\n4. Experimental results for zero-shot generalization are new in the TTA setting.\n\nWEAKNESSES:\n1. No theoretical justification or even intuition is provided about why the low-rank adapter learns the domain-agnostic knowledge, whereas the high-rank adapter captures the domain-specific knowledge.\n2. The backbone architecture is not consistent with previous works such as CoTTA. Section 3.4 in the supplementary material has only for CIFAR10C. What about the performance of the proposed approach for Imagenet-to-ImagenetC on ResNet50? What about the performance of the proposed approach for CIFAR100-to-CIFAR100C for ResNeXt-29 architecture?\n3. Retraining the \"model added with low/high-rank adapters\" for some steps using the source data. This leads to the inability to use off-the-shelf pre-trained models without access to the source domain data. \n\nMinor Comments\n1. References should be conference/journal version, for example, 57 reference can cite the CVPR conference version\n2. Line 175: Figure .2 --> Figure 2, and many more \"Figure\"\n3. Line 237: Modal --> Model\n4. Line 248: comparability. --> comparability\n\nQUESTIONS:\n1. Equation 4 depicts the formulation to modify the scale factors $\\lambda_h$ and $\\lambda_l$ based on uncertainty scores. What if the conditions to update the scale factors are reversed? In other words, when facing a sample with a high uncertainty value, decrease $\\lambda_h$ (instead of increasing it), and vice versa for $\\lambda_l$. It would be interesting to see what happens in such a scenario.\n2. Why the low-rank adapter learns the domain-agnostic knowledge, whereas the high-rank adapter captures the domain-specific knowledge? Can the authors provide some intuition behind it, other than t-SNE plot of embeddings?\n3. The backbone architecture for most experiments is not consistent with previous works such as CoTTA. Section 3.4 in the supplementary material has only for CIFAR10C. What about the performance of the proposed approach for Imagenet-to-ImagenetC on ResNet50 (missing in Table 1)? What about the performance of the proposed approach for CIFAR100-to-CIFAR100C for ResNeXt-29 architecture?\n4. Why for the semantic segmentation task only the results for 3 rounds has been reported, as approaches such as CoTTA experiments for for 10 rounds?\n\nLIMITATIONS:\n1. The experimental results indicate the effectiveness of the proposed approach for transformer based architecture and not much for convolutional neural network based architecture.\n2. Addition of extra modules, such as adapters, requires further training of the model with adapters for one epoch on the source domain data. This is a limitation since the source domain data may not be available in many practical scenarios, and only the pre-trained model may be provided.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes to utilize domain-specific and domain-agnostic knowledge to tackle the error accumulation and catastrophic forgetting problem and boost the performance of continually test-time adaptation task. The proposed visual domain adaptor (ViDA) aims to adapt current domain distribution and maintain the continual domain-shared knowledge in CTTA, while a homeostatic knowledge allotment strategy is designed to fuse the knowledge. The experiments demonstrate that the method achieved SOTA.

STRENGTHS:
1. The logic of this paper is clear and the performance is godd.
2. The whole method is easy to understand and implement.
3. They conduct sufficient experiments to prove their proposal.

WEAKNESSES:
1. The authors mentioned that the proposed method ensures no extra parameter increase in Section 1 (L64) and Section 3 (L195). However, extra parameters and computational costs are considered as limitations in L335, which brings contradiction and unclarity. Adding clear explanations would be appreciated.
2. The authors argues that ViDA with a low-rank prototype focuses on domain-agnostic knowledge while ViDA with a high-rank prototype concentrates more on domain-specific knowledge. Explanation on the Fig.1 is not sound enough.
3. How are the different ViDAs projected into the pre-trained model by re-parametrizations? Please provide more details.
4. Other suggestions：
 L261 Table .1 -> Table 1
 L275 Table .2 -> Table 2
 L296 Table. 3 -> Table 3

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
Limitations on extra computational costs are discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address continual test-time adaptation (CTTA) with parameter-efficient fine-tuning techniques, i.e., adapter. The authors find that the low-rank adapter i.e., standard bottleneck structure, can extract domain-invariant knowledge. On the other hand, a high-rank adapter can extract more domain-specific knowledge. Experiments on three image classification benchmarks and one semantic segmentaiton benchmark demonstrate the effectiveness of the proposed method based on ViT backbone.\n\nSTRENGTHS:\n1. The paper is easy-to-follow. \n\n2. The topic is essential yet the idea is moderate. Both continual test-time adaptation and parameter-efficient fine-tuning are important topics for the community and the paper addresses two formulations simultaneously.\n\n3. A new method that seems well-motivated and performs well on classification and segmentation benchmarks.\n\nWEAKNESSES:\n1. The major concern is that are the learned features domain-invariant related to the structure of the adapter? That is, low-rank adapter can learn domain-invariant features while high-rank adapter can learn domain-specific features. How about all adapters adopt the same structures?\n\n2. What is the motivation of the teacher model? Just following CoTTA?\n\n3. More relevant works [a,b,c, d, e] should be discussed. \n\n4. What is the design of low-rank and high-rank adapter in CNN architecture? Why not the auhors report the result of ResNet50 in Table 1? \n\n5. It seems that the proposed ViDA module is memory-extensive since all experiments are conducted on A100 GPU. However, the backbone is just ResNet50 or ViT-base. Thus, does it mean that PEFT is memory-extensive or the proposed ViDA module is extensive?\n\n6. It can be seen from the results in Table 3 that the model performance with the pre-trained encoder parameters of SAM significantly decreases. But the reason is unclear. It might be that SAM is pixel-level foundation model but the performance gains with image-level foundation model DINOv2 are limited. Also, I am interested in the results for Cityscapes-to-ACDC with SAM pre-trained parameters.\n\nRefs:\n[a] Niu et al., Towards stable test-time adaptation in dynamic wild world.\n[b] Yuan et al., Robust test-time adaptation in dynamic scenarios.\n[c] Song et al., EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization.\n[d] Gong et al., NOTE: Robust continual test-time adaptation against temporal correlation.\n[e] Döbler et al., Robust mean teacher for continual and gradual test-time adaptation.\n\nQUESTIONS:\n1. What do t-SNE results of other blocks look like? Why the authors select the third transformer block to analye in Fig. 1 (b).\n\n2. What is the meaning of ``prototype``? \n\n3. Minor:\n\n-  ref [55] and ref [56] are duplicated.\n\nLIMITATIONS:\nThe authors claimed one limitation of this work, i.e., introducing extra parameters and computational cost.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a continual test-time adaptation method by designing a visual domain adapter (ViDA) to handle both domain-specific and domain-agnostic knowledge. To adapt to different distribution shifts, a homeostatic knowledge allotment strategy is proposed to adaptively merge knowledge from each ViDA with different rank prototypes. Experiments on four benchmark datasets demonstrate the effectiveness of the proposed method for both classification and segmentation CTTA tasks.\n\nSTRENGTHS:\n- The paper introduce homeostatic visual domain adapter for continual test time adaptation.\n- Good results are achieved on both classification and segmentation tasks.\n\nWEAKNESSES:\n- In general, the proposed components are not very well justified and verified. The details are listed in the following points.\n- Except for the empirical results, it lacks more in-depth analyses and justifications on why a low-rank prototype focuses on domain-agnostic features while a high-rank prototype focuses on domain-specific knowledge. In addition, Figure 3 (a) shows the reduction of inter-domain divergence due to the introduction of the low-rank adapters. The reviewer was wondering what would be the results of the inter-domain divergence by using the high-rank adapters.\n- In table 1, the experimental results are not clearly explained. For example, what are the difference between the last two rows?\n- In Table 6, both the ViDAh and ViDAl could improve the performance significantly. However, it is still unclear why the different adapters (high-rank and low-rank) necessary. How to effectively show that they are complement to each other? It would be interesting to know whether two high-rank adapters or two low-rank adapters could also achieve good results?\n\nQUESTIONS:\n- Why a low-rank prototype focuses on domain-agnostic features while a high-rank prototype focuses on domain-specific knowledge?\n- How to effectively show that ViDAh and ViDAl are complement to each other?\n\nLIMITATIONS:\nThe authors addressed some of the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}