Visual Prompt Tuning For Test-time Domain Adaptation
Authors: Yunhe Gao
Xingjian Shi
Yi Zhu
Hao Wang
Zhiqiang Tang
Xiong Zhou
Mu Li
Dimitris N. Metaxas
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep learning
test-time domain adaptation
unsupervised learning
visual prompt tuning
vision transformer
self-supervision

Abstract: Models should have the ability to adapt to unseen data during test-time to avoid performance drops caused by inevitable distribution shifts in real-world deployment scenarios. In this work, we tackle the practical yet challenging test-time adaptation (TTA) problem, where a model adapts to the target domain without accessing the source data. We propose a simple recipe called data-efficient prompt tuning (DePT) with two key ingredients. First, DePT plugs visual prompts into the vision Transformer and only tunes these source-initialized prompts during adaptation. We find such parameter-efficient finetuning can efficiently adapt the model representation to the target domain without overfitting to the noise in the learning objective. Second, DePT bootstraps the source representation to the target domain by memory bank-based online pseudo labeling. A hierarchical self-supervised regularization specially designed for prompts is jointly optimized to alleviate error accumulation during self-training. With much fewer tunable parameters, DePT demonstrates not only state-of-the-art performance on major adaptation benchmarks, but also superior data efficiency, i.e., adaptation with only 1\% or 10\% data without much performance degradation compared to 100\% data. In addition, DePT is also versatile to be extended to online or multi-source TTA settings.

Under review as a conference paper at ICLR 2023
VISUAL PROMPT TUNING FOR TEST-TIME DOMAIN ADAPTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Models should be able to adapt to unseen data during test-time to avoid perfor- mance drops caused by inevitable distribution shifts in real-world deployment scenarios. In this work, we tackle the practical yet challenging test-tima sit-timme adapta- tion (TTA) problem, where a model adapts to the target domain without accessing the source data. We propose simple recipe called Data-efficient Prompt Tuning (DePT) with two key ingredients. First, DePT plugs visual prompts into the vision Transformer and only tunes these source-initialized prompts during adaptation. We find such parameter-efficient finetuning can efficiently adapt the model repre- sentation the target domain without overfitting to the noise in the learning objec- tive. Second, DePT bootstraps the source representation to the target domain by memory bank-based online pseudo-labeling. A hierarchical self-supervised reg- ularization specially designed for prompts is jointly optimized to alleviate error accumulation during self-training. With much fewer tunable parameters, DePT demonstrates not only state-of-the-art performance on major adaptation bench- marks VisDA-C, ImageNet-C, and DomainNet- ciency, i.e., adaptation with only 1% or 10% data without much performance degradation compared 100% data. In addition, DePT is also versatile be extended to online or multi-source TTA settings.
but also superior data effi-
INTRODUCTION
Deep neural networks achieve excellent performance when the testing data (target domain) follow the same distribution as the training data (source domain). However, their generalization ability on the testing data is not guaranteed when a distribution shift occurs between the source and the target. Even simple domain shifts, like common corruptions (Hendrycks & Dietterich 2019) or appearance variations (Geirhos et al. 2018), can lead to significant performance drop. Sol Solving the problem of domain shift -ri-tiivial. On one hand, it is impossible to train a single model to cover an infinite number of domains. On the other hand, training individual models for each domain requires lots of annotated samples, which induces significant data collection and labeling costs.
In this paper, tackle the practical yet challenging test-time domain adaptation (TTA) problem. Compared with conventional unsupervised domain adaptation (UDA) (Long et al. 2015). TTA adapts the source domain initialized model with the unlabeled target domain data during testing without access to source data. TTA avoids the privacy issue of sharing the source data and is de- sirable for real-world applications. We focus on both offline and online TTA settings. For offline adaptation, also known as source- free adaptation (Kundu et al. 2020 Liang et al. 2020), the model first updated with unlabeled target data and then inference. For online adaptation, the model keeps updating and inferencing at the same time, given the testing data that comes in a stream.
The key challenges of TTA lie two folds. First, how to effectively modulate the source domain initialized model given noisy unsupervised learning objective? Tent (Wang et al. (020)) optimizes the parameters of batch normalization layers, which is parameter-efficient but lacks adaptation ca- pacity. On the other hand, SHOT (Liang et al.) 2020) tunes the feature encoder; AdaContrast (Chen) et al., 2022) trains the whole model. Given the current over peeepparamttriized models, these meth- ods are prone overfitting the unreliable unsupervised learning objective, especially when the amount of target domain data is limited. We present more evidences in Appendix A to illustrate motivation to use visual prompt tuning. Second, given only unlabeled target domain data,

---

Under review as a conference paper at ICLR 2023
what kind of learning objective for optimization? Existing works propose to use unsupervised objec- tives, including entropy minimization (Wang et al., 2020). self-supervised auxiliary task (Sun et al, 2019b), pseudo labeling (Lee et al., 2013),
combination of the above objectives (Liang et al, 2020 Chen et al. 2022 Liu et al. b) However, these unsupervised objectives are either not well aligned with the main task (Liu et al. 202Tb), or give noisy supervision (Liang et al.) 2020).
In this work, we propose Data-efficient test-time adap- tation via Prompt Tuning (DePT). Our recipe is sim-
ple yet effective, where two key ingredients are pro-
posed to address the two challenges mentioned above,
90
see in Fig. 2 First, a collection of learnable vi-
sual prompts (Jia al. 2022) are trained with labeled
source data alongside the vision Transformer (Doso-
vitskiy et al., 2020). In the adaptation phase, we only finetune the prompts and the classification head, while freezing the backbone parameters. Although the knowledge learned in the source domain is retained in the unchanged backbone, tuning solely the prompts can effectively modulate the model for target domain adaptation. Second, for the learning objective given only unlabeled target domain data, DePT bootstraps the source domain initialized model via self- trf-ntrannnng where the pseudo labels are first predicted and then re- fined by soft voting from nearest neighbor data points in memory bank. To further alleviate the accumula- tion of errors during self-training, we design a hierar- chical fine-grained self-supervised regularization term for the prompts encourage better target domain rep- resentation learning. These two objectives are comple- mentary and jointly optimized.
80
Adatona (85.8
DePT-G
76
10%
100%
Figure I: The test-time adaptation perfor- mance of different methods with respect to the data ratio on the VisDA dataset. The number in the legend denotes the num- ber of tunable parameters. DePT-G out- performs previous SOTA AdaContrast 100% target data with only O. 0.19% tunable parameters. The superiority of DePT is more significant on the low data settings.
The two ingredients in DePT offer multiple merits and lead to state-of -frthe--rt performance on major domain adaptation benchmarks: VisDA-C (Peng al. 2017]. ImageNet-C (Hendrycks & Dietterich] (209)) and DomainNet- (Peng et al. 2019). Tun- ing prompts during test-time is much more parameter-efficeen than full fine-tuning. As illustrated in [ig.[] with only 0.19% tunable parameters, DePT achieves 1.2% higher performance than pre- vious wate-of-the-are AdaContrast (Chen al.] 2022). Moreover, parameter efficiency brings data efficiency to DePT. Compared with AdaContrast, DePT has significant improvement in the low data regime. For example, with only 1% of unlabeled target domain data in VisDA-C. DePT achieves 88.0% average accuracy, which surpasses AdaContrast by 7.1%. Under the more challenging online TTA setting, DePT demonstrates superior performance with 85.9% average accuracy on VisDA-C. For robustness to corruptions, DePT achieves the least error rate 14 out of 15 types of corruptions on the severity level of 5 on ImageNet-C with 34.9% average error rate. Besides, we also show that DePT is flexible to extend more DA scenarios like multi-source (Peng et al. 2019 domain adaptation.
2 RELATED WORK
Unsupervised domain adaptation (UDA) setting is commonly used, where unlabeled data from the testing distribution (target domain) is available during training along with the labeled data from the training distribution (source domain). Many studies attempt to solve this problem using style transfer (Hoffman et al., 20188 Tang et al.) 2021]: Taigman et al., 2016), feature alignment (Long et al. 2015| Sun et al. [2017] Peng et al. 2019) or learning Nomain-invarian feature via adversarial training (Ganin et al. 2016 zeng et al. 2017). However, the UDA setting has a strong assumption that the source and target domain data are both accessible during training, which is not always true as it is difficult to access source data after the model is deployed. Moreover, these methods usually need retrain the whole framework if new domains come.
2

---

Under review as a conference paper at ICLR 2023
Test-time domain adaptation (TTA) is more challenging and practical setting where source domain data is available during adaptation. Such property makes TTA avoid the privacy issue and transmission burden of source data sharing, and enables the pipeline of training the model once and adapting the model to any unknown test distributions. The offline TTA is essentially source-free do- main adaptation (Kundu al. 2020 Liang al. (020), where the model can update several epochs on the unlabeled target domain data before inferencing. The online TTA setting requires the model to keep updating and inferencing at the same time batch-by-batch as long as there are testing data. Some recent works explore the TTA setting with different unsupervised objectives and model mod- ulation methods. Test-time training (TTT) (Sun et al.] (019b) introduces self-supervised rotation prediction task as the test-time optimization objective. Tent (Wang et al. (2020) modulates the batch normalization parameters by minimizing entropy. SHOT (Liang et al. (020) adapts the feature ex- tractor to the target domain with combination of information maximization and pseudo labeling. SHOT++ (Liang ct al., 202I) further applies a semi-supervised learning step to propagate the label of high-confidence samples to low-confidence samples. HDMI (Lao et al. 202T) also optimizes for information maximization, but introduces a disparity regularization on multiple hypotheses. Ada- Contrast (Chen et al.] 2022) tunes the whole model with memory bank-based pseudo labeling and contrastive loss. DePT has several critical differences from these approaches. Most of these meth- ods are proposed for ConvNet, while we switch vision transformer. For model modulation, we freeze the backbone parameters and learn the target-specific prompts. For the learning objective, we jointly optimize an online memory bank pseudo labeling loss and hierarchical supervisddd loss specially designed for the prompts. DePT is also flexible on a variety of DA scenarios.
Self-supervised learning methods have made great progress in unsupervised visual representation learning. Early attempts use pretext tasks, e.g. colorization (Zhang et al. 2016), rotation prediction (Gidaris et al. 2018). jigsaw (Noroozi & Favaro 2016). as the self-supervised objective. Another large body of works focus discriminative approaches (Chen et al.) 2020; He et al. 2020), named contrastive learning, which push the representation of positive pairs closer and spread negative pairs further. Recent works show that with carefully designed structure or learning objective. unsuper- vised features can be learned without discrimination between instances (Grill et al., 2020: Caron let al. 2021). Besides unsupervised visual representation learning, researchers also explore apply- ing self suprrvssee objectives for model robustness (Hendrycks et al. (019) and domain adaptation (Sun et al. 2019a b). Inspired by BYOL (Grill et al. 2020) and DINO (Caron et al. (021),
propose a hierarchical self-supervised regularization that are jointly optimized with pseudo labels.
Prompt tuning originates from NLP. Prompts are text instructions that are prepended the input as a hint to inform the pre-trained language model about the task (Brown et al., 2020). Besides text prompts that are mostly constructed via heuristics, soft prompts, i.c., continuous vectors that are optimized via gradient descent, achieve comparable performance with full fine-tuning but have much less tunable parameters (Li & Liang| 2021 Liu et al. 2021a). Prompt tuning is also explored in vision tasks. BRS (Jang & Kim] 2019) perturbs the input for prediction refinement. L2P (Wang) ett al.) 2022) is proposed for continual learning. [ahnn et al. (2022) proposed to pad prompts in the pixel space to adapt large-scale models in vision. VPT (Jia et al. 2022) is a highly related work proposed for transfer learning. VPT inserts randomly initialized prompts to the pre-trained model, then optimizes them with the downstream task's label. DAPL (Ge et al. 2022) introduced prompt learning to the UDA setting, yet it is highly dependent on the large-scale vision lision-language (Radford et al.) 202T). In contrast, DePT doesn't require text encoder, and the visual prompts learned with the source domain data are updated with unlabeled target domain data.
3 METHOD
We address the close-set test-time domain adaptation (TTA) setting where source data not acces- sible during adaptation. In source domain training, n, labeled samples (x), v² 1 from the source domain D. are given, where r E X₂ and yd V/ are data and the corresponding labels. A model has general architecture with parameters 0 trained to learn the function f. X, + V.. We assume the model is parameterized by a neural network and consists of two modules: the feature encoder gs: X, + Rd and a classifier ha: Rd + RC, where d is the dimension of the feature and C' is the number of classes. During target domain adaptation, unlabeled data { from target domain D. is available, where of Xt The goal of TTA is adapt the source domain trained model fa to
Sh
3

---

Under review as a conference paper at ICLR 2023
Head
Memory Bank
Transformer Encoder Layer
Aug,
00Q
Student
Transformer Layer
Lizm
Stage
Weight
00oDn
Transformer Layer
Lmal
Lp
Student
Unlabeled
EMA
Lys
Target Data
Transformer Encoder
Aug.
1000
Teacher
Stage
Image Token
Transformer Encoder Layer
4
CLS Token
Prompt
Aggregated Prompt
Stop Gradient
(A)
(B)
Figure 2: The overview DePT. (A) We split VIT into multiple stages and prepend prompts to the input of each stage. The prompts, along with the backbone, initialized with labeled source domain data. Only prompts and the classification head (in red) are finetuned during adaptation, while the backbone frozen (in blue). (B) The proposed adaptation framework. The pseudo labels of target data are first predicted and then refined by memory bank-based soft voting for self-training. hierarchical supe-supereiee objective is proposed improve target representation and alleviate error accumulation of self-training.
target domain f: : X - Vt. We assume the target domain share the same label space with source, ie., V. )/+ = =.
3.1 VISUAL PROMPT TUNING
How to modulate the model during TTA a non-trivial problem. The optimization objective of TTA is unsupervised and usually noisy, making full parameter tuning sensitive, easy to diverge, and prone to overfitting to the noise, especially when the number of target data is limited. On the other hand, tuning little parameters has limited adaptation capacity. Therefore, we propose to adopt visual prompt tuning to solve this dilemma. We provide more evidence and motivation for prompt tuning Appendix A
Visual prompts are a set of learnable continuous parameters that are prepended to the input of Trans- former layers, see in Fig. 2 (A). For Vision Transformer (ViT), the input image first divided into patches and then embedded into d-dimensional image tokens Ep. Similar to Jia et al. (2022). we concatenate the prompts with CLS token and image tokens to the input of Transformer layers after the image embedding layer. In order to adjust the adaptation capacity, DePT inserts new prompts at different levels of Transformer layers. To specific, we split the Transformer layers into M stages, where each stage has = N/M layers. A collection of prompts P are inserted into the first layer of each stage by concatenating with the image and CLS tokens. select M {1,4, 12) to form three variants of DePT: Dhallohallow (S), DePT-Group (G) and DePT-Deep (D). The prompt augmented ViT is formulated as:
[c;, Z,, E.] = L L,((c-1,, E,-1))
i = 1, m + 1, 2m 1m + 1, I,..., N - m
1mm+1.
[ci, Zi E.} Li ((c,11, Z- 1.)
else
= head(
where and Z; represent the CLS token and the aggregated prompt features computed by the i-th Transformer layer, respectively. By altering the number of stages M and the number of prompts p, can easily control the number of tunable parameters for different adaptation capacity.
4

---

Under review as a conference paper at ICLR 2023
The prompt augmented ViT model learns the function f. X, - Vg by labeled source data with cross-entropy loss, where all parameters, including the prompts and the backbone, are optimized. In the test-time adaptation phase, we initialize the target model with the source model's weights. We only fine-tune the parameters of the prompts and the classification heads, while all other pa- rameters are fixed. The knowledge learned in the source domain is retained in the frozen backbone. The prompts are finetuned to learn the target-specific knowledge to adjust the representations non- linearly through self-attention. The classification head is trained to learn a new decision boundary.
3.2 LEARNING ORJECTIVE
Our learning objective consists of two parts. First, an online pscudo labeling with a memory bank refinement mechanism is used bootstrap the source domain learned representation to the target domain. Second, we design a self-supervised objective on the CLS token and the aggregated prompt to form hierarchical regularization further reduce the noise in the pseudo labels and improve the target representation quality.
3.2.1 PSEUDO LABELING
Pseudo labeling has widely been used in semi/self- learning. Inspired by Tarvainen & Valpola (2017); SShn et al. (2020); Chen et al.) (2022), we use student-teacher model and an online memory bank refinement to generate pseudo labels. To be specific, see in Fig. (B), the student fa and the teacher f/ share the same architecture, and are both initialized with the source model's weight 0g at the beginning of adaptation. The teacher is not updated by gradients, but by exponential moving average (EMA) from the student's weights. The teacher maintains an memory bank that stores the feature embeddings and predicted probabilities of target samples. The memory bank is online updated as a first-in-frst-our queue from training mini-batches. Given an unlabeled target domain data It, weak and a strong random image transformation: Tw and To are applied. The embedding of the weakly augmented view is first obtained from the student. The pseudo label is then obtained by a soft voting, i.e. average of the probability, from the top-k nearest neighbors in the memory bank: Dt = arg max V (()))) where V denotes the soft voting with memory bank. The student is trained to enforcing the prediction for the with a cross-entropy loss:
view to match ýt
1 T4 Σ H (TT())))))
Cpt =
(2)
where H(a, b) a log b. The details can be found Appendix B.1 Intuitively, the student is fast updating during training, making the pseudo label produced by the student to be noisy. The EMA updated teacher can be seen as an ensemble of the student along training. Thus the memory bank maintained by the teacher stores more stable and high-quality predictions. The soft voting further leverages nearby data point cluster to produce more accurate pseudo labels.
3.2.2 HIERARCHICAL SELF-SUPERVISED REGULARIZATION
Due to the domain shift, the model trained the source domain cannot directly produce good rep- resentations of the target data, i.e., the representation of images from the same class may spread apart. Although using the student-teacher model and the memory bank refinement can significantly improve the quality of pseudo-labels, they still cannot completely avoid the accumulation of er- rors during self-training. Therefore, we design a hierarchical self-supervised regularization for the proposed prompt augmented ViT to improve the model's 's representation of target data.
For the prompt augmented ViT, the CLS token of the output of VIT encoder cN serves as the holistic image representation for classification. The prompts work similarly to the CLS token: prepending to the input of Transformer layers. With self-attention, the prompts can inject domain-specific in- formation into other tokens, and aggregate instance features from other tokens. Each prompt will attend to different attributes and features of the image. The prompts at the output of the Transformer layer, which we call aggregated prompts, serve as fine-grained representation of the image. More- over, the aggregated prompts from different stages may attend to distinctive levels of features and thus form hierarchical representation. Inspired by Caron et al. (2021) and Grill et al. (2020), given different views of the same image, we push both their holistic feature representations (CLS token) and their hierarchical fine-grained representations (aggregated prompts) closer.
5

---

Under review as a conference paper at ICLR 2023
2021) framework here with few key modifications. The student encoder 9r and EMA teacher encoder gf are reused with the ones in the pseudo labeling section. A series of projection heads q:: R - 12 are introduced to project the CLS token Ct or the aggregated prompts Zj of the target sample 22 into K dimensions, where each stage of the prompt augmented VIT has its own projection head. Two random arrong-augmente views of the target sample is generated. One is passed the student, and the other one is passed to the teacher. The projected CLS token and aggregated prompts are obtained from the student: ét (e(c)), {Z, qu (Zi)
We use the DINO (Caron et al.
and similarly for the teacher ट. (Z, where I = {m, 2m..... N is the index of the last Transformer layer in each stage. For the holistic regularization on the CLS token, we apply the DINO loss. The probability distribution of the student output is obtained by applying a softmax function: (x+) = softmax (C+), and so is the teacher's (.)e). The student aims matching the teacher's predicted distribution by minimizing the cross-entropy loss:
(3) The technique of centering and sharpening of DINO are also used to avoid collapse, where the details can be found in Appendix [..2 For the hierarchical fine-grained regularization, we minimize the mean squared error between the aggregated prompts of the student and teacher:
Leauts = (x)) P(zz)).
Ls prompt
1
ΣΣ 5=
2 2 2.
(4)
M
| 112 1 112
In addition, although the regularization is applied on the aggregated prompts one by one, i.e. the i-th aggregated prompt of the student is pushed closer to the i-th one of the teacher, we find the model tend to learn the trivial solution where all aggregated prompt become similar. To avoid this, we add a diversity term to encourage different prompts attend to different features, thereby enabling them to learn rich target-specific knowledge. Specifically, we maximize the cosine distance among the p aggregated prompts of the student:
dev
(5)
M NET
1| 2
where j and i are the index of the aggregated prompts at the output of the i-th Transformer layer. Therefore, DePT minimizes the following loss function for TTA:
L a + BiL.s dl + B2Lss prompt Xc div.
(6)
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate the effectiveness and versatility of DePT in a variety of TTA scenarios, covering three major domain adaptation benchmarks. VisDA-C (Peng al. (017)) is large-scale domain adapta- tion benchmark that focuses on 12-class synthesistto-real object recognition task. We use the train- ing set as the source and the validation set the target. The source domain contains 152k synthetis images generated by rendering 3D models while the target domain has 55k real images sampled large-scale benchmark
from Microsoft COCO. ImageNet-C (Hendrycks & Dietterich 2019) to evaluate models' robustness against 13 types of corruptions. DomainNet- -16 (Peng et al.) 2019) provides images from multiple domains for classification task. Since the original DomainNet dataset has noisy labels, we follow the previous work (Saito et al., 2019 Chen et al. (0022) to use a subset of NetainNet, which contains 126 classes from 4 domains (Real, Sketch, Clipart, Painting).
Models Different from previous works that use ConvNet as the backbone, e.g. ResNer26/50/101, we use VIT-B as a strong backbone network for adaptation. We use the model and ImageNet pre- trained weight from the timm library (Wightman 2019) in experiments.
Baselines We compare our method with both classical UDA methods and test-time adaptation methods. For UDA methods, compare to DANN (Ganin & Lempitsky 2015), CDAN (Long et al. 2018). CAN (Kang et al. 2019). SWD (Lee et al. 2019) and MCC (Jin et al. 2020). Note that all UDA methods need access both source and target domain data, while ours doesn's t. For TTA methods, we compare with Tent (Wang et al. 2020), SHOT (Liang et al. 2020), CFA (Kojmaa et al.
2022) and AdaContrast (Chen et al. 2022).

---

Under review as a conference paper at ICLR 2023
Table 1: Classification accuracy (%) on VisDA-C train val adaptation. Upper table: UDA meth- ods. Middle table: TTA methods. Bottom table: online setting. All models with ViT-B backbone are reproduced by us, while the results with R101 backbone are cited from the original paper. Bold and underline denote the first and second high results.
Method
Backbone
plane
beyel
bus
car
horse
knife
meyel
person
plant
skthrd
train
truck
Avg.
DANN
R101
81.9
82.8
44.3
81.2
29.5
65.1
28.6
54.6
7.8
57.4
CDAN
R101
85.2
66.9
83.0
50.8
84.2
74.9
88.1
74.5
83.4
76.0
81.9
38.0
73.9
82.5
97.8
96.2
90.8
80.7
96.3
59.9
87.2
SWD
RI01
90.8
82.5
81.7
70.5
91.7
69.5
86.3
77.5
87.4
63.6
85.6
29.2
76.4
MCC
R101
88.7
80.3
80.5
71.5
90.1
93.2
85.0
71.6
89.4
73.8
85.0
36.9
78.8
Source
VIT-B
99.0
69.1
78.2
71.9
87.7
58 4
96.4
35.0
52.7
92.5
96.0
17.6
71.2
Tent
VTT-B
99.0
76.9
79.2
80.8
93.9
84.2
95.9
54.5
74.6
92.9
6
22.9
79.2
SHOT
VII-B
99.5
91.8
88.7
65.1
98.6
98.0
96.0
66.1
95.1
98.9
96.8
52.4
87.3
AdaCom
VIT-B
99.5
94.2
91.2
83.7
98.9
97.7
96.8
71.5
96.0
98.7
97.9
45.0
89.2
DePT-S (Ouns
VTT-B
98.7
90.1
87.4
70.7
98.9
96.0
96.8
75.2
91.9
97.9
96.2
52.5
87.7
DePT-G (Ours)
VIT-B
99.2
93.0
93.4
85.6
99.4
98.6
96.7
76.2
98. 2
97.9
96.6
52.0
90.6
DePT-D (Ours)
VII-B
94.4
87.5
99.4
98.0
96.7
74.3
98.5
90.7
Online Setting
AdaCoon
95.4
83.4
64.2
95.8
93.2
92.4
65.3
82,9
81.3
DePT-G (Ours)
VIT-B
98.3
87.6
88,9
62.1
98.4
95.2
95.9
67.1
95.2
97.3
94.9
50.
85.9
4.2 VISDA-C DATASET RESULTS
DePT demonstrates state-of-the art performance even with much less tunable parameters. Tab. T compares DePT with UDA and TTA methods. Without access source data, DePT with more strong ViT-B backbone achieves better performance than the UDA methods. Among TTA methods, with much less tunable parameters, DePT-G (0.16M) and DePT-D (0.47M) outperforms the strong baseline AdaContrast (85.8M) by +1.4% and +1.5%. DePT-S, with only 0,048M tunable parame- ters, still has huge improvement (+16.5%) compared with adaptation. Although the backbone is frozen, tuning the prompts can effectively adjust the representation for the target domain.
DePT gains more advantages under limited data settings. Previous TTA methods usually assume the target domain data is sufficient for adaptation. However, even without label, obtaining a large amount of target domain data is still expensive in some scenarios. Therefore, we investigate the impact of the target domain data quantity on the performance of the TTA methods by evaluating with 1%, 10%, 50%, and 100% of target data, see Fig. (A). DePT shows strong data efficiency. DePT- G still has 87. 57.97% average accuracy under 1 % data, which is only 2.59% lower than with 100% data. In contrast, AdaContrast's average accuracy is only 80.8 80.87% at 1% data, which 8.48% lower than 100% data. For DePT-D, its adaptation capacity is the highest when target data is abundant, e.g., in 100% data. DePT-D has more decline than DePT-G and DePT-S under 1% data, but is still superior to AdaContrast. For DePT-S, the absolute performance is not high, but it shows the best robustness against data quantity changes, which only decreases 1.17% under 1% data compared to 100% data.
DePT has strong performance the more challenging online TTA setting. In the online setting, the model adapts to target data that comes in batch sequentially in a stream. Each data batch can only be used for training once. We do not apply learning rate decay and turn off the memory bank and directly use the model prediction as the pseudo label. The bottom of Tab. shows the performance of DePT online. DePT achieves 85.9% average accuracy, surpassing AdaContrast by +4.6%.
4.3 IMAGENET-C DATASET RESULTS
DePT is effective various corruption types. Tab. 2 shows the top-1 error rate the highest severity (level-5) corruptions on ImageNet-C C. In the offline adaptation, we compare with SHOT- IM (Liang et al. 2020). CFA (Kojima et al.) 2022 and AdaContrast (Chen et al. 2022). DePT-G achieves the lowest error for 14 out of 15 types of corruptions. In the online adaptation setting, DePT-G outperforms online AdaContrast and CFA by 3.7% and 1.2%, respectively.
7

---

Under review as a conference paper at ICLR 2023
DaPiiS DDt #Deft-3 RAteČaTA
Redoow
DaPT
Figure 3: (A) Performance comparison on VisDA-C dataset with different data ratio for TTA. DePT shows less performance degradation when the training data reduces. (B) t-SNE visualization of the DePT-G model before and after adaptation on VisDA-C dataset. Different color denotes classes.
Table 2: Top-1 error rate (%) 15 corruptions with highest severity (level-5) ImageNet-C. All models use the ViT-B backbone, where Source, AdaContrast and DePT-G are reproduced by us, while SHOT-IM and CFA cited from Kojima et al.) 20222).
Method
Gauss
Shot
Impul
Defoc
Glass
Motion
Zoom
Snow
Frost
Fog
Bright
Contr
Elast
Pixel
Jpeg
Avg
Source
52.2
51.5
51.
69.3
55.8
46.2
50.1
45.3
25.3
68.9
54.9
34.4
49.7
SHOT-IM
43.4
42.3
42.4
44.1
48.5
40.9
43.9
35.3
37.4
33.8
22.4
42.0
39.2
29.8
31.5
38.4
AdaConirast
43.6
42.2
43.5
45.7
52.0
40.2
44.1
33.2
38.1
31.7
23.3
42.9
37.1
28.5
30.8
DePT-G(Ours)
45.6
41.8
41.7
40.6
42.0
38.4
40.2
29.4
32.7
26.5
20.8
35.5
32.2
26.4
29.5
34.9
Online Setting
AdaContrast
45.6
44.2
35.4
39.9
28.1
CEA
43.1
42.0
41.9
45.6
51.1
40.1
43.4
33.6
35.9
32.3
21.0
41.2
35.7
28.3
29.8
37.6
DePT-G (Ours)
28.2
4.4 DOMAINNET DATASET RESULTS
DePT consistently improves seven domain shifts in the DomainNet-126 dataset. All three variants of DePT show consistent improvement on all seven domain shifts compared with no adaptation. DePT-G performs the best among them and achieves 81.7% average accuracy. Although the proposed DePT-G and DePT-D have much smaller tunable parameters, they outperform the previous SOTA AdaContrast. In the online setting, DePT achieves 1% higher performance than AdaContrast. We also find that models with ViT-B backbone demonstrate stronger robustness against domain shifts than the R50 backbone.
domain shifts. Tab. 3 shows the comparison the
Beyond one-on-one adaptation. We further evaluate DePT to multi-source TTA setting, details of the setting can be found Appendix [B.3 See Tab. DePT makes the best use of multi-source do- main knowledge by utilizing In contrast, SHOT (Liang et al., 2020) makes one-by-one domain adaptation separately and then makes predictions by ensemble, which does not jointly optimize for multi-source domains.
of domain-specific prompts and shared domain-agnosti prompt.
4.5 ANALYSIS
Visual prompt tuning helps to learn better target representation. Fig. (B) shows the t-sne visualization of the target feature embedding before and after adaptation. Before adaptation, the clusters are mixed and difficult to distinguish due to domain shift. After adaptation, the embeddings scattered, and the number of samples that fall into the wrong cluster is less. Although the backbone fixed, the prompts can effectively adapt the representation through attention layers.
Prompt tuning is highly effective for adaptation. Tab. 5 shows the impact of the number of prompts. Increasing the number stages or the number of prompts introduces more tunable param- eters. As the VisDA-C validation set has abundant unlabeled data, increasing tunable parameters usually results in better performance, e.g., DePT-G-100 and DePT-D-50 achieve SOTA adaptation performance. Surprisingly, even with prompt, DePT still achieves decent adaptation accuracy.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Classification accuracy (%) of seven domain shifts
Table 4: Classification accuracy
on DomainNet- 126, All models use ViT-B backbone.
(%) of multi-source DA setting on
Method
R-C
R-P
P-+C
C-+S
S->P
R-S
P--R
Avg.
DomainNet-122 dataset. R de- notes the remaining three domains
Source
68.7
75.9
69.8
67.7
74.1
60.4
86,4
71.8
Tent
69.1
77.7
70.1
67.0
75.3
61.7
88.0
72.7
except the single source. All mod-
SHOT
80.2
81.5
79.8
74.2
82.2
72.8
90.3
80.1
AdaCon
81.8
81.8
82.0
75.8
82.2
73.7
90.4
81.1
els use VTT-B backbone
DePT-S
81.2
81.4
80.9
75.1
81.7
73.1
88.9
80.3
Multi
ERM
SHOT
DeFT-G
DePT-G
83.3
81.6
82.6
77.8
81.3
75.1
89.9
81.7
R +R
88.7
90.7
91.0
76.9
81.2
74.6
81.5
R +C
76.7
83.1
83.7
Online Setting
R
69.2
75.9
78.3
AdaCon
72.1
77.9
71.8
75.7
65.5
74.2
R +P
78.7
83.4
84.0
DePT-G
74.8
78.3
74.2
72.5
77.2
68.5
87.2
76.1
Average
78.3
83.2
84.2
Table 5: Ablation study of DePT variants and prompt number on VisDA-C.
Prompt Num
50
100
Params/M
Parama/M
Params/M
ParmssMM
D
83.8
0.008
86.0
0.016
87.7
0.048
87.6
0.086
DePT.G
85.4
0.010
88.4
0.040
90.6
0.16
90.6
0.32
DePT-D
86.3
0.018
89.0
0.10
90.7
0.47
90.4
0.94
Table Ablation of the contribution of each component to the final performance on VisDA-C. The upper table uses DePT-G with 50 prompts and tunes the prompts. The lower table uses the same model but tunes all parameters including the backbone.
PL MB
CLS Reg.
Prompt Reg.
Prompt Div.
100%
85.6
78.4
3
89.3
85.6
88.2
84.6
90.6
88.0
Full
Contribution of each loss. Tab. showws the contribution of each loss under 100% and 1% data VisDA-C. The pseudo labeling and memory bank refinement can improve the accuracy in the target domain, but the improvement depends the amount of data. They improve by 12.4% under 100% data, but only improve by 7.2% under 1 % data. Adding the self-supervise objective on the CLS token allows the model learn better representation of the target domain. We tried to further add hicrarchical regularization on the prompts. However, directly adding regularization makes prompts collapse. By further introducing the diversity term, the final performance is boosted 90.6% under 100% data, and 88.0% under 1% data. The lower table shows the contribution of prompt tun- ing. With the same learning objective, tuning all parameters achieves good accuracy under abundant target data (100%), but is prone to overfit under limited data (1%).
5 CONCLUSION
We propose data-efficient test-time adaptation approach via visual prompt tuning. We pretrain collection of visual prompts alongside the model with labeled source domain data. During adap- tation, we only adapt the prompts to learn target-specific knowledge with the backbone fixed. A flexible scheme is proposed to insert a certain number of prompts to the input of each stage of the Transformer. By doing so, can easily control the adaptation capacity and tunable parameters by altering the number of stages and prompts. An online pseudo labeling with memory bank refine- ment is jointly learnt with a hierarchical self-supervised objective for test-time prompt tuning. We extensively evaluate the proposed method with VisDA-C, ImageNet-C and DomainNet- 126 datasets and show that our simple recipe can lead to SOTA performance in multiple settings and superior data efficiency. In addition, we further verify the effectiveness and flexibility of DePT on online and multi-source adaptation settings.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:220..17274, 1(3):4, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 9660, 2021.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 295-305, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pp. 1597-1607. PMLR, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by hackpropagation. In International conference machine learning, pp. 1180-1189, PMLR, 2015.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 17(1):2096-2333 2016.
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang, Domain adaptation via prompt learning. arXiv preprint A.K1:22222.06633 2022.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis. Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:/811.1223// 2018.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1e0..10030002 2018.
Sean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271 12222244, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 122611 2019.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learn- ing can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. Pmlr, 2018.
10

---

Under review as a conference paper at ICLR 2023
Won-Dong Jang and Chang-Su Su Kim. Interactive image segmentation via backpropagating refine- ment scheme. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5297-5306, 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning, arXiv preprint arXiv:2203./2119, 2022.
Ying Jin, Ximei Wang, Mingsheng Long, and Jianmin Wang. Minimum class confusion for versatile domain adaptation. In European Conference Computer Vision, pp- 464-480. Springer, 2020.
Guoliang Kang, Lu Jiang, Yi Yang. and Alexander Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4893-4902, 2019.
Takeshi Kojima, Yutaka Matsuo, and Yusuke Iwasawa. Robustifying vision transformer with- out retraining from scratch by test-time class-conditional feature alignment. arXiv preprint arXiv:2206. 13951, 2022.
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adap- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4544-4553, 2020.
Qicheng Lao, Xiang Jiang, and Mohammad Havaci. Hypothesis disparity regularized mutual in- formation maximization. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pp. 8243-8251, 2021.
Chen- Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10285-10295, 2019.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient supervupe learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00/190 2021.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028-6039. PMLR, 2020.
Jian Liang Dapeng Hu, Yunbo Wang, Ran He, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions Pat- tern Analysis and Machine Intelligence, 2021.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602, 2021a.
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexan- dre Alahi. Ttt+:: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 18-2188082110 2021b.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference computer vision, pp. 69-84. Springer, 2016.
11

---

Under review as a conference paper at ICLR 2023
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge, 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, PMLR, 2021.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised do- main adaptation via minimax entropy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8050-8058, 2019.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Baochen Sun, Jiashi Feng, and Kate Saenko. Correlation alignment for unsupervised domain adap- tation. In Domain Adaptation in Computer Vision Applications, pp. 153-171. Springer, 2017.
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A Efros. Unsupervised domain adaptation through self-supervision. arXiv preprint 19X.:::909.11822 2019a.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A Efros, and Moritz Hardt. Test-time training for out-of-distribution generalization. 2019b.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. arXiv preprint arXiv:1611.022000 2016.
Zhiqiang Tang, Yunhe Gao, Yi Zhu, Zhi Zhang, Mu Li, and Dimitris N Metaxas. Crossnorm and selfnorm for generalization under distribution shifts. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 52-61, 2021.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7167-7176, 2017.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
Zifeng Wang, Zizhao Zhang, Chen-Y Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Pro- ceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 139-149, 2022.
Ross Wightman. Pytorch image models. 2019.
https://gittab..commmm
pytorch- woacchh-imge--mooa
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 666. Springer, 2016.
12

---

Under review as a conference paper at ICLR 2023
A MOTIVATION OF USING PROMPT TUNING
How modulate the model is vital for the TTA setting. Since only unlabeled target data are available during adaptation, the model can only learn through the unsupervised learning objective. However, current unsupervised learning objectives are not always true and usually contain noises and errors. If there are too many parameters modulate, the current over- weer-paameteriieed models prone to overfit to the noise the unsupervised objective, especially when the amount of unlabeled target data is limited. On the other hand, if modulating too few parameters, although overfitting will be avoided, the adaptation capacity of the model will be limited.
We perform experiment to verify our hypothesis. We AdaContrast (Chen et al. 2022 and Tent [Wang et al. (2200) as examples to modulate the model. Among them, AdaContrast finetunes all parameters of the model. Tent only adjusts the transformation parameters of the Normalizioioo layers, which only contains very few parameters. For ViT, we adjust the transformation parameters of the LayerNorm (LN) layer instead. For these two model modulation approaches, we use the learning objective of AdaContrast for training, i.c., MoCo-based self-supervised combined with pseudo labeling.
100% Data
1% Data
85
AvalAvgAoo Aoc
Avg Aoc
80
AdaContrast
AdaContrast
LN
LN
DePT-G
DePT-G
70
1,000
2,000
3,000
4,000
5,000
6,000
1.000
2,000
3,000
4,000
5,000
6,000
Training Steps
Training Steps
Figure 4: The curve of average accuracy of evaluation v.s. training steps with 100% data and 1 data on VisDA-C dataset.
Fig. 4 shows the results of training the above modulation approaches with 100% and 1% VisDA- C unlabeled target domain data. Under 100% data, AdaContrast adapts well to the target domain. This is because the target domain of VisDA-C has 55K images, making AdaContrast not easy to overfit. For LN, its performance is limited by the adaptation capacity. Under 1% of the data, with the increasing of training steps, AdaContrast over-fits obviously and the performance of adaptation gradually declines. LN does not overfit but its adaptation performance still limited by the capacity. In the TTA setting, as no target domain label is available, we currently do not have a good metric to measure when to stop adaptation. Therefore, method that can avoid over-fitting and have enough adaptation capacity is necessary. Such observation motivate us to explore using visual prompt tuning to solve this dilemma.
B IMPLEMENTATION DETAILS
B.1 PSEUDO LABELING DETAILS
During adaptation, we use student-teacher model and an online memory bank refinement to gener- ate pseudo labels. The student and the teacher share the same prompt-augmented ViT architecture. They are both initialized with the source model's weight 8, at the beginning of adaptation. As the student is fast updating during training, the pseudo label directly produced by the student is usually unstable and noisy. We introduce slow momentum teacher model whose weights O'xxponential
13

---

Under review as a conference paper at ICLR 2023
moving average updated by the student's weights fa:
0 m0 (1 (1-mi))) m))0.
where O₂ is the student's weights, 0 is the teacher's weights, and is the momentum for updating. The teacher can be seen as an ensemble of student along training, thus is more stable and has higher pseudo label accuracy.
The teacher maintains an online memory bank with size to refine the pseudo labels from the student via soft voting, where the memory bank stores the feature embeddings and the predicted probabilities {c/, pl, p() of weakly augmented target data from the teacher. The memory bank B is initialized by randomly selected N samples, and then updated with data from training mini- batch as int-ii--fiss-out queue.
Given an unlabeled target domain data 2y, a weak and a strong random image transformations Tar and Tg are applied. The student produces the feature embedding c based on the weakly augmented view Tur (ze). To refine the prediction of the student, we use Ct to retrieve top-k nearest neighbors in the memory bank B by measuring the cosine distance between e2 and {c The refined hard
pseudo label is obtained by averaging the probability of the top-k neighbors followed by arg operation:
Ur arg max
P
Inspired by FixMatch (Sohn et al., 2020 the student is trained to enforce the cross-entropy loss against the model's output for the strongly- augmented view:
1 n n4
LPI.
H(g, (((T((x)))
B.2 SELF-SUPERVISED OBJECTIVE DETAILS
We use the DINO (Caron et al. 2021) framework for self-supervised learning with a few key modi- fications. We reuse the student and teacher model in the pseudo labeling with the DINO framework, which are both initialized by the source model. We apply the DINO loss on the CLS token. The sharpening and centering operation is also used avoid collapse. To be specific, the sharpening operation is applied for both the student and teacher:
Pr(x.) cxp(c(1/7)) /T)
(10)
Σ exp(c /T)
The teacher further applied a centering operation:
exp((E))) - (1) (()
(((x))
(1I)
ΣK exp((et - (8)
where 7 and r' are the temperature for the student and teacher, respectively. e is the center that is exponential moving average updated for more stable estimation. The multi-crop and local-to-global regularization are not applied in DePT. More details can be found in (Caron et al. 2021).
B.3 EXTENDING TO MULTI-SOURCE DOMAIN TTA SETTING
The multi-source domain test-time adaptation setting aims training model from multiple source domains and adapt to the unseen target domain. Many works aims at learning a domain-invariant feature across source domains. However, such methods are different to learn tures the source domain is diverse. They also can't I capture domain-specifee knowledge.
fea-
Our method is versatile extend multi-source domain test-time adaptation setting. See in Fig. 5) the basic idea is that domains share a domain-are aggnosicc backbone while each domain has its own prompts to learn the domain-specific knowledge. During source training, each domain prompts
14

---

Under review as a conference paper at ICLR 2023
along with shared general prompts randomly initialized. In each training iteration, at- tractively sample images from each domain. The backbone, shared prompts and the corresponding domain prompts optimized via supervised loss. During the TTA phase, prompts from all domain are averaged and concatenated with the shared prompts as the initialization. Then the optimization the same one-on-one TTA adaptation.
In our experiments, we use three domains in the DomainNet- 126 as the source domains, and use the left one the target.
Transformar Layer
O
O Shared
Clipart
Real
Painting
Skoetch
Aurrage
00
Intwlized Prompis
Figure 5: Our method is easy extend multi-source domain adaptation setting, where only the corresponding phase needs to be modified.
B.4 IMPLEMENTATION DETAILS
We use PyTorch for all implementation. For source training, we initialized the ViT-B-16 16 with Im- ageNet pretrained weights from timm library (Wightman) 2019). The model name used in timm library is "vit_base_ www.bbassppachhhh.e 1ttch16_224"" where the weight is pretrained in ImageNet21 K with AugReg and then finetumed on ImageNetIK. We follow Chen et al. (2022) to randomly split the source data with 90% for training and 10% for validation. For source training,
train around 1k iterations. For target training, we train around 6k iterations for all datasets unless noted. For all experiments, SGD optimizer with momentum 0.9 and weight decay le-4, and cosine annealing the learning rate. The training batch size is set as 128.
C MORE EXPERIMENT RESULTS
C. ABLATION ON THE HYPPER-PARAEETE CHOICE
Tab. shows the impact of the hyper-parameter choice, including learning rate, the teacher temper- ature in the DINO loss, and the weight to balance training loss.
For visual prompt tuning, we find that a larger learning rate would benefit the performance. For example, learning rates of 1e-3 and 5e-3 lead to decent performance. Continuing to amplify the learning rate may lead to instability in training.
The Teacher temperature is hyper- paremmerreetrr in the DINO loss that controls the sharpness of the teacher output distribution. A large temperature makes the teacher tend to output a uniform distribu- tion, which makes the initial optimization difficult. The performance of small temperature is slightly worse. Using a warmup mechanism on the teacher's temperature a good balance.
We finally investigate the effect of weights among the proposed losses in Eq. 6 We set to 1, and adjust the value of 3 = 82 = 3. We set = 3/20. We found that a small 3 value results in better performance, e.g., below 0.1.
15

---

Under review as a conference paper at ICLR 2023
Despite the performance varies slightly by different hyper-parameter choices, DePT consistently demonstrates state-of-the-ar thte-art performance compared to other methods, which shows the robustness of DePT against the choice of hyyper-prameert
Table 7: Ablation of the impact of werper-parameterr choices of DePT-G on VisDA-C.
LR
2c-4
5c-4
le-3
5c-3
Avg Acc
89.6
90.0
90.4
90.6
Teacher Temp
0.04
0.05
0.07
0.04-0.07
Avg Ace
90.2
90.6
90.5
B
0.01
0.05
0.1
0.5
Avg Acc
90.1
90.2
90.6
89.8
C.2 COMPUTATION EFFICIENCY ANALYSIS
As we insert visual prompts into multiple level the ViT, during test-time adaptation, the gradients need be propagated all the way the input for updating. Therefore, investigate the computa- tion cost in terms of training time and memory of different model modulation approaches, including tuning full model (AdaContrast), tuning feature extractor (SHOT), tuning layer normalization pa- rameters (Tent) and three variants of DePT with the number of prompts 50 and 100, see in 8 The results is measured with the same loss (entropy minimization) on the VisDA-C dataset. We run for 500 iterations and report the average results. The input images has batch size 16 (input size: 16 224 X 224). All results are measured with one NVIDIA A10G GPU. DePT consumed less time and memory than AdaContrast and SHOT with 50 prompts, while consumed slightly more time and memory with 100 prompts.
Table The comparison training time, memory consumption number tunable parapmeters of different model modulation approaches. Measured with ViT-B backhone.
Full
Feature
LN
DePT-s50
DePT-S100
DePT--G5
DePT-G100
DeFT-DD50
DePT-D100
Time's
0.106
0.106
0.077
0.102
0,123
0,102
0.124
0.105
0.126
Mem/M
5135
5133
4001
4647
5207
5323
#Params/M
85.81
85.80
0.038
0.048
0.086
0.16
0.32
0.47
0.94
C.3 DETAILED EXPERIMENT RESULTS
We present the detailed experiments results on VisDA-C and DomainNet- 126 dataset. The results of methods with ResNet backbone are presented.
16

---

Under review as a conference paper at ICLR 2023
Table 9: Detailed experiment results of T Classification accuracy (%) on VisDA-C train -> val adaptation. Upper table: UDA methods. Middle table: TTA methods. Bottom table: online setting- All models with VTT-B backbone are reproduced by us, while the results with R101 backbone are cited from the original paper. Bold and underline denote the first and second high results.
Method
Backbone
plane
hus
horse
knife
meyel
person
plant
skthrd
train
truck
Avg.
DANN
R101
81.9
77.7
82.8
44,3
81.2
29.5
65.1
28.6
51.9
54.6
82.8
7.8
57.4
CAN
R101
97.0
87.2
82.5
74.3
97.8
96.2
90.8
80.7
96.6
96.3
87.5
59.9
87.2
SWD
RIOI
90.8
82.5
81.7
70.5
91.7
69.5
86.3
77.5
87.4
63.6
85.6
29.2
76.4
MCC
R101
88.7
80.3
80.5
71.5
90.1
93.2
85.0
71.6
89.4
73.8
78.8
Source
R101
57.2
11.1
42.4
66.9
55.0
4.4
81.1
27.3
57.9
29.4
86.7
5.8
43.8
Tent
R101
86.5
23.0
79.8
51.9
78.3
17.6
87.6
64.22
79.9
23.6
63.9
1.1
57.3
SHOT
R101
95.3
87.5
78.7
55.6
94.1
94.2
B1.4
80.0
91.8
90.7
86.5
59.8
83.0
AdaCon
R101
97.0
84.0
77.3
96.7
93.8
91.9
84.8
93.1
94.1
49.7
87.2
Source
VIT-B
99.0
69.1
78.2
71.9
87.7
38.4
96.4
35.0
52.7
92.5
96.0
17.6
71.2
79.2
92.9
79.2
SHOT
VIT-B
99.5
91.8
88.7
65.1
98.6
98.0
96.0
66.1
95.1
98.9
96.8
52.4
87.3
AdaCon
VII-B
99.5
94.2
91.2
83.7
98.9
V7.7
96.8
71.5
96.0
98.7
97.9
45.0
89.2
S (Oux)
VTT-B
87.4
70.7
98.9
96.0
96.8
75.2
91.9
97.9
96.2
52.5
87.7
DePT-G (Ours)
VTTBB
99.2
93.0
93.4
85.6
99.4
98.6
96.7
76.2
98.2
97.9
96.6
52.0
90.6
DePT-D (Ours)
VIT-B
94.4
87.5
99.4
96.7
74.3
98.4
98.5
90.7
Online Setting
AdaCon
93.2
92.4
DePT-G (Ours)
VIT.B
98.3
87.6
88.9
62.
98.4
95.2
95.9
67.1
95.2
97.3
94.9
50.1
83.9
Table 10: Detailed experiment results of Table Classification accuracy (%) of seven domain shifts on DomainNet- 126. All models with ViT-B backbone are reproduced by us, while the performance of other baselines with R50 backbone cited from the original paper. Bold and underline denote the first and second highest results.
Method
Backbone
R-C
R-P
P-C
C-S
SP
R-S
P-
Avg.
Source
R50
62.7
53.0
46,9
50.1
46.3
75.0
55.6
TENT
R50
58.5
65.7
$7.9
48.5
52.4
54.0
67.0
57.7
SHOT
R50
67.7
68.4
66.9
60.1
66.1
59.9
80.8
67.1
AdaContrast
R50
70.2
69.8
68.6
58.0
65.9
61.5
80.5
67.8
Source
VIT-B
68.7
75.9
67.7
74.1
60.4
86.4
71.8
Tent
VIT-B
69.1
77.7
70.1
67.0
75.3
61.7
88.0
72.7
SHOT
VIT-B
80.2
81.5
79.8
74,2
82.2
72.8
90.3
80.1
AdaContrast
VIT-B
81.8
81.8
82.0
75.8
82.2
73.7
90.4
81.1
DePT-S (Ours)
VIT-B
81.2
81.4
80.9
75.1
81.7
73.1
88.9
80.3
DePT-G (Ours)
VIT-B
83.3
81.6
82.6
77.8
81.3
75.1
89.9
81.7
DePT-D (Ours)
VIT-B
82.8
82.5
82.8
76.9
81.2
74.6
89.8
81.5
Online Setting
AdaContrast
VIT-B
72.1
77.9
71.8
70.1
75.7
65.5
87.9
74.2
DePT-G (Ours)
VIT-B
74.8
78.3
74.2
72.5
77.2
68.5
87.2
76.1
17


Reviews:
[{"review_id": "x0ZPj9ku30", "reviewer": "Reviewer_zNVW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Test-time adaptation updates a model to reduce generalization on shifted data.\nSuch adaptation methods need to choose a loss for adaptation and parameters to update, and this work's main contribution is to introduce visual prompts as a parameterization.\nThe proposed Data-efficient Prompt Tuning (DePT) method includes additional learnable tokens in a ViT during training, then updates these tokens and the classification token during testing, by entropy minimization through pseudo-labeling and other losses.\nThe adaptation parameters are limited to the classification head and the proposed layer-wise prompts, which strikes a balance between expressivity and efficiency, where the number of prompts can be varied from just one layer to all layers.\nThis parameterization is justified by the accuracy achieved (Table 1) and analysis experiments that try alternatives like updating layer normalization parameters (Appendix B.1).\nThe other losses include a self-supervised regularization scheme derived from DINO, in which projections of the prompts (local representations) and the class token (the global representation) are updated to maximize their similarity.\nExperiments evaluate DePT against the prior state-of-the-art AdaContrast, reproduced with the same ViT backbone as the proposed method, and other test-time methods with ResNet backbones (which are therefore not comparable), along with unsupervised domain adaptation methods that require both source and target data.\nDePT achieves higher accuracies by offline adaptation on the whole test set and online adaptation on the streaming test set, with a larger margin of improvement for online adaptation.\nThese results are shown for domain adaptation and generalization datasets, VisDA-C and DomainNet-126, but not corruption or other robustness benchmarks like ImageNet-C or ImageNet-R.\nAblation studies show that each component of the method helps.\nAlthough pseudo-labeling makes the largest difference, and that is already a popular part of test-time adaptation, it is pseudo-labeling as a loss to update the proposed parameterization by prompts that is helping so much.\n\n", "summary_of_the_review": "In short, the method does work as shown by its evaluation on the standard datasets of VisDA-C and DomainNet.\nHowever, the evaluation could be broader in its datasets and more thorough in its comparable baselines by reproducing more than one method with the ViT backbone.\nThis is an expectation for work in this area, as many past test-time adaptation papers have evaluated on several datasets of shifts as a way to shown some measure of generality.\nAll in all, there is value in demonstrating that test-time adaptation applies to ViT models too, and in exploring the right choice of parameterization for this purpose.\nWhile this work could be improved, pushing test-time adaptation to use the most accurate architectures at present may be sufficiently informative to the community.\n\n*For Rebuttal*\n\n1. Please measure the time efficiency of DePT updates, and relate the computation required to update the prompts to the computation required to update normalization layers or other choices of parameters, as used by the compared methods like TENT and SHOT.\n2. Please discuss DePT as a source-free (SHOT) vs. test-time (TENT) vs. intermediate (TTT) method. The choice of setting should guide the choice of comparisons.\n3. Please report results for ImageNet-C, if possible, because it is a common benchmark for robustness and adaptation to corruption. (It is alright if this is not possible due to computational limitations, but doing it would be a plus.)\n4. Please motivate the low data regime reported in the results. Is there a practical deployment that this setting addresses?\n\n*Other Feedback*\n\n- Title\n  - Is the method a test-time adaptation method? Test-time methods update during testing, as DePT does, but in some usages \"test-time\" also indicates that the method can update online and that it does not require changes to training. DePT seems to be more like SHOT or TTT in that it alters training.\n- Abstract\n  - Name the datasets, benchmarks, and settings evaluated. A precise summary of experiments is more eye-catching to the potential reader.\n- Related Work\n  - The formatting with subsections takes up a lot of space. Consider revising it to remove the subsection headings and use more bolded heading instead. Then the paper would have more room for further explanation or experiments.\n- Analysis\n  - Table 3 is not an impressive comparison, because DePT has been augmented to include domain-specific and domain-shared parameters while SHOT has not been. Other methods like SHOT could also have domain-specific layers, as done by residual adapters (Rebuffi et al.) for example.\n  - Table 4 should include the accuracy without DePT in the caption to help ground the relative improvement of each number of prompts.\n- Proofreading\n  - \"Second, given only unlabeled target domain data, use what kind of learning objective for optimization.\" Should this be a question? Please revise.\n", "strength_and_weaknesses": "*Strengths*:\n\n- This work brings test-time adaptation up-to-date w.r.t. current model architectures and parameters: ViTs are current, strong models and prompting is a timely topic.\n  The choice of parameterization is novel for this purpose, low-dimensional for efficiency, and specifically-chosen for ViTs.\n- Online adaptation accuracy improves on the prior state-of-the-art AdaContrast by 4 points on VISDA-C and 2 points on DomainNet (which are both standard benchmarks).\n  Offline adaptation accuracy is no worse than the state-of-the-art and perhaps slightly better.\n- The choice of datasets is sound. VISDA-C is a standard benchmark, and DomainNet is a more recent benchmark that has also seen adoption. The evaluation protocol for these datasets follows prior work.\n- There is a self-supervised loss that does not depend on the pseudo-labels, for bottom-up adaptation to the target inputs, which may complement top-down adaptation to the model predictions.\n  This loss is closely-related to DINO, which is credited accordingly, but it is somewhat customized to the use of visual prompts in this work.\n\n*Weaknesses*:\n\n- While DePT efficient in its parameter dimension, that does not mean it is efficient in its update computation. For DePT, the prompts are distributed throughout the model, and so optimization has to compute many gradients across layers even if the parameter gradients are themselves much smaller.\n  The amount of computation required during testing is not described, nor is it compared against baselines.\n- There is only one comparable baseline with the same model architecture (AdaContrast). For thoroughness, it would be useful to adapt a prior method that reported results with ResNets to ViTs for comparison. Tent for instance simply minimizes the entropy of predictions for a given choice of parameters, and could be ported to ViTs by updating the layer norm parameters (for one example).\n- There are no results for other common choices of robustness or domain generalization benchmark, such as ImageNet-C (or its other variations ImageNet-V2, ImageNet-R, etc.), or PACS/VLCS/etc. These are included in many prior works such as TENT, SHOT, and TTT.\n- There is missing related work on learning and tuning visual prompts, although neither addresses the application of visual prompts to test-time adaptation against shifts, as done in this work.\n  - [Interactive Image Segmentation via Backpropagating Refinement Scheme](https://vcg.seas.harvard.edu/publications/interactive-image-segmentation-via-backpropagating-refinement-scheme/paper). Jang & Kim, CVPR'19. This paper prefigures visual prompts by perturbing the input according to a model's own predictions as a kind of self-training to improve accuracy. Citing it would help better trace the history of ideas.\n  - [Exploring Visual Prompts for Adapting Large-Scale Models](https://arxiv.org/abs/2203.17274). Bahng et al. arXiv'22. This is highly related to the cited VPT by Jia et al. 2022 and should be considered for reference alongside it.\n- The proposed method is not fully test-time because the prompts must be jointly trained with the source model parameters. Its setting should be better identified in the text, and the experiments should compare to the latest source-free methods accordingly, like SHOT++ (Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer. Liang et al. PAMI'21.)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "q5dgVlLhHg", "reviewer": "Reviewer_H4uU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper present a method for test time adaptation using learnable visual prompts integrated with visual transformers.  The paper uses various regularization methods to avoid error accomulation; knowledge distilation from ensemble or sourch checkpoints, self-supervised learning and learnable visual prompts. \nThe results are very promising. ", "summary_of_the_review": "The paper is interesting however, it misses some references (see above) and I cant seem to find an original point for novelty but the experimental results show significant improvement over the current state of the art. I hope the authors provide clarifications in the rebuttal phase. ", "strength_and_weaknesses": "strengths:\n- The paper is easy to read and the flow of the paper is generally good. \n- Figure 1 is highlights an important and intersting experiment with the number of available target examples in the semi-supervised learning setting. \n- Results are very promising\n\nweaknesses:\n- I think there needs to be discussion and ablation study on the importance of visual prompts. They describe how they are trained but whats missing is a discussion on why there are needed. \n- The paper misses some important citations. For example \"hypothesis disparity regularized mutual information maximization\" AAAi 2020 uses ensembles for regularization of TTA which is closely related to the proposed method.  \n- There needs to be more dicussion on what the self-supervised learning is set to achieve differently from that of the pseudo label training. Also what was the process of desing choice for selecting DINO over other SSL methods? \n-Error bars (repeated experiments with different parameter initialization seeds) are missing from expeirments which makes it hard to evaluate reproducibility and robustness. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GNc4Z0UlHPw", "reviewer": "Reviewer_n2AM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of test-time domain adaption. Specifically, the authors propose a data-efficient prompt tuning (DePT) approach, which only requires finetuning injected visual prompts and classifiers during training. In addition, a memory-based pseudo-labeling method is also introduced to improve the quality of pseudo-labels. Experiments on several datasets show the benefit of the proposed DePT.\n\n", "summary_of_the_review": "This paper is well-written and easy to follow.  However, the novelty and improvement of the proposed method are limited. The computation cost is not evaluated to show the real benefit of the proposed method compared to other methods. Also, this paper lacks a convincing explanation for the prompt tuning. Considering these factors, I tend to give borderline reject.", "strength_and_weaknesses": "\nStrength\n\n+ This paper is well-written and easy to follow. The motivation is clear and reasonable.\n\n+ Clear figures are provided to help us better understand the details of the proposed method.\n\n+ The proposed diversity term is novel and interesting.\n\n+ Extensive experiments are provided to evaluate the effectiveness of the proposed method.\n\nWeakness\n\n+ The novelty of the proposed method is somewhat limited. The proposed method includes two parts, memory-based pseudo-labeling and visual prompt learning [A]. However, both parts are already studied in the domain adaptation community, especially memory-based pseudo-labeling.\n\n+ Although the authors argue that the proposed method achieves significant improvement, the improvement is mainly obtained by pseudo-labeling (Table 5). The CLS regression can also improve the performance, but it is largely bought from DINO, i.e., distillation with teacher-student strategy. From Table 5, we can find that the improvements of the proposed memory-based PL and prompt tuning are somewhat limited.\n\n+ To verify the effectiveness of prompt tuning, it is better to show the ablation study of each component in more datasets.\n\n+ Although the proposed method requires few parameters to be tuned compared to other methods, I think the computation cost and training time are not reduced too much. This is because the prompts are injected into different layers, and the gradients should be calculated on all layers. In other words, tuning fewer parameters does not mean saving computation costs and training time. I thus think it is required to provide the computation cost (memory and training time) for different methods.\n\n+ How to ensure that the injected prompts could contain domain-specific information? In addition, it would help us better understand the learned prompts by visualization as shown in Figure 3.\n\n+ In the recent community, the test-time domain adaptation commonly refers to the online setting. While the experiments mostly conducted in this paper should be called source-free domain adaptation or test-time training adaptation. Please clarify them in the next version.\n\n[A] Domain Adaptation via Prompt Learning.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "28QWSjhheS", "reviewer": "Reviewer_Za8v", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a test-time adaptation method based visual prompt tuning. After training the deep model along with a set of prompts to convergence, the proposed DePT fixes the backbone parameters and only fine tunes the prompts on the target domain. Based on this learning scheme, DePT integrates multiple self-training techniques (with adequate modification) into the training objective, including a pseudo labeling under the student-teacher framework, modified DINO, consistency on both the class token output (through cross-entropy loss) and the prompt token output (through mean squared error). Experimental results validate the effectiveness and investigate the major components through ablation. ", "summary_of_the_review": "I am generally satisfied with the contribution of this paper. I would like to raise my rating if the authors could address my concerns above, especially the relation between their training objectives and the original self-supervision methods. ", "strength_and_weaknesses": "**Strength**\n\n- Using an efficient fine-tuning technique, i.e., prompt tuning for test-time adaptation is technically sound. \n\n- The achieved accuracy is competitive.\n\n\n**Weakness**\n\n- Introduction for some important components lacks intuition and is not easy to understand, e.g., the memory-based online pseudo labeling in the abstract and the introduction. It not easy to understand its name and how it performs until the detailed method. \n\n- While the training objectives are modified from recent self-supervised learning techniques, the authors have not provided necessary comparison or revisit. For example, the pseudo labeling part can be viewed as a modification from MoCo, but the authors introduce the momentum teacher as if it was invented for the first time. Moreover, the MSE losses in Eq. 4~5 (and their input features) have strong relation to BYOL.\n\n- Based on the above concern, it is important to explain why the proposed method makes those modifications (instead of directly applying the original MOCO or BYOL). \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject