SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models
Authors: Xiaosong Ma
Jie ZHANG
Song Guo
Wenchao Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Test-Time Adaptation
Prompt Learning
Unsupervised Representation Learning

Abstract: Test-time adaptation (TTA) is a special and practical setting in unsupervised domain adaptation, which allows a pre-trained model in a source domain to adapt to unlabeled test data in another target domain. To avoid the computation-intensive backbone fine-tuning process, the zero-shot generalization potentials of the emerging pre-trained vision-language models (e.g., CLIP, CoOp) are leveraged to only tune the run-time prompt for unseen test domains. However, existing solutions have yet to fully exploit the representation capabilities of pre-trained models as they only focus on the entropy-based optimization and the performance is far below the supervised prompt adaptation methods, e.g., CoOp. In this paper, we propose SwapPrompt, a novel framework that can effectively leverage the self-supervised contrastive learning to facilitate the test-time prompt adaptation. SwapPrompt employs a dual prompts paradigm, i.e., an online prompt and a target prompt that averaged from the online prompt to retain historical information. In addition, SwapPrompt applies a swapped prediction mechanism, which takes advantage of the representation capabilities of pre-trained models to enhance the online prompt via contrastive learning. Specifically, we use the online prompt together with an augmented view of the input image to predict the class assignment generated by the target prompt together with an alternative augmented view of the same image. The proposed SwapPrompt can be easily deployed on vision-language models without additional requirement, and experimental results show that it achieves state-of-the-art test-time adaptation performance on ImageNet and nine other datasets. It is also shown that SwapPrompt can even achieve comparable performance with supervised prompt adaptation methods.

SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models
Xiaosong Ma
Jie Zhang
Department of Computing
Department of Computing
The Hong Kong Polytechnic University Hong Kong, China
The Hong Kong Polytechnic University Hong Kong, China
xiaosong16 ma@connect. pectinnettpoll.uuuoo.
jie-comp. zhang@polyr eulyyu.ddee edu .kk
Song Guo
Wenchao Xu Department of Computing
Department of Computer Science and Engineering Hong Kong University of Science and Technology Hong Kong, China songguoOcse.uu ust.csseeu..the.
The Hong Kong Polytechnic University
Hong Kong. China wenchao xu@polyu.....oo edu. .kk
Abstract
Test-time adaptation (TTA) is special and practical setting in unsupervised do- main adaptation, which allows a pre-trained model in a source domain adapt unlabeled test data in another target domain. To avoid the computation-intent inteenive backbone fine-tuning process, the zero-shot generalization potentials of the emerg- ing pre-trained vision-language models (e.g.. CLIP, CoOp) are leveraged to only tune the run-time prompt for unseen test domains. However, existing solutions have yet to fully exploit the representation capabilities of pre-trained models as they only focus the entropy-based optimization and the performance far below the supervised prompt adaptation methods, e.g., CoOp. In this paper, propose SwapPrompt, a novel framework that can effectively leverage the self-supervised contrastive learning facilitate the test-time prompt adaptation. SwapPrompt employs a dual prompts paradigm, i.e., an online prompt and a target prompt that averaged from the online prompt to retain historical information. In addition, SwapPrompt applies a swapped prediction mechanism, which takes advantage the representation capabilities of pre-trained models enhance the online prompt via contrastive learning. Specifically, we use the online prompt together with an augmented view of the input image predict the class assignment generated by the target prompt together with an alternative augmented view of the same image. The proposed SwapPrompt can be easily deployed on vision-language models without additional requirement, and experimental results show that it achieves state-of- the-art test-time adaptation performance on ImageNet and nine other datasets. It is also shown that SwapPrompt can even achieve comparable performance with supervised prompt adaptation methods.
Introduction
When there discrepancy between the distribution of the training data and the testing data, the
generalization performance of deep neural networks can be compromised [ 2 3] The focus
of domain adaptation is to construct models that can adapt to variations in data distribution by
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Prompturge
FMA
P Promplatiie
Prompt
Prompt
Text Encoder
Text Encoder
Encoder
Image
Image
Image
Encoder
Encoder
Similarity
Encoder
Minimize
I Labeled Images
Unlaheled Images
Entropy
Unlabeled Images
Swapped Prediction
(a) CoOp
(b) TPT
(c) Ours
Figure 1: Comparison on V-L model architectures. (a) CoOp adapts prompt on labeled data.
(b) TPT optimizes prompt by minimizing marginal entropy. (c) Ours SwapPrompt leverages self- supervised contrastive learning to facilitate test-time prompt adaptation.
transferring knowledge from a source domain to new related target domain, which usually requires both the source and target domain data during the training phase [4] [5]. However, in practical scenarios, it is common to have only the model after it has been trained in the source domain, while without the access to the source data, or the authorization to alter the original training procedure 6 8. To address this problem. Test-time adaptation (TTA) 9 T0 has been proposed and shown its potential to adapt models to target/unseen domains by only leveraging the unlabeled test data streams. Existing works have developed techniques such as entropy minimization 77, [1]. class prototypes 22 13
image generation 14 15], and supervupe training [TO], which have already
demonstrated superior performance.
Although traditional model-based TTA methods shown effective, they typically rely computa- tionally intensive tuning to the parameters of the model backbone. The situation would be even worse with the advent of vision-language pre-trained models (e.g.. CLIP [16), CoOp [17] and CoCoOp (18), that have a massive number of parameters and are difficult to optimize. Therefore, it is promising to explore efficient techniques to fine-tune only small set of parameters for adapting models novel domains during testing while keeping the backbone fixed. Pre-trained vision-language models, which are trained on a significant amount of image-text pairs, have introduced a powerful paradigm that provides fresh insights for tackling this issuc. A straightforward way is to utilize the strong zero-shot capabilities of the pre-trained vision-language models discriminate various domains of test data via the fine-tuning over the labeled data of downstream tasks, however such way may not be feasible in TTA scenarios where the labeled downstream data unavailable. Shu al. [19] propose the test-time prompt tuning (TPT) to address the label scarcity problem in test-time. Nevertheless, it may lead to a risk of over-trust in the model (i.e., generating high confidence for a wrong result) from directly minimizing the entropy to tuning instance-specie prompts. The prediction confidence of TPT and proposed SwapPrompt can be found in appendix.
To this end, we propose SwapPrompt, novel test-time prompt adaptation method as illustrated in Figure [ Unlike previous approaches, SwapPrompt leverages self-supervised contrastive learning strategy in the test domain, which consists of two key components: exponential moving average (EMA) prompt and prompr swapped prediction mechanism. The EMA mechanism employs a dual prompts paradigm: the target prompt and the online prompt. We optimize the online prompt while the target prompt gradually updated through a slow-moving average process, which incorporates past information to increase stability and effectiveness. The prompt swapped prediction mechanism inspired by the unsupervised learning method SWAV (20). Based on an augmented view of image and the online prompt, SwapPrompt predicts the class assignment of an augmented view of the same image. This enables the online prompt to learn more representation knowledge. The rationale behind the swapped prediction strategy is that two different augmentations of the same image should have similar class assignments. The contrastive representation learning approach is leveraged to generate better decision boundaries (21].
In addition the loss function of supeviupe contrastive learning, we employ the conventional cross-entropy loss as in CLIP and CoOp, which tune the prompt with high-confidence pseudo labels generated by the zero-shot CLIP. Our approach can also be employed for online test-time scenarios, where test data arrive in flow of mini-batches. We break down the operation performed on all test data into multiple mini-batches, which is discussed in detail the experimental section. We evaluate our method on various test-time adaptation benchmarks, including ImageNet and four natural
2

---

distribution shift datasets based on it, as well as nine fine- fine-grained classification datasets. Experiment results show that our method achieves slate-of-the-- test-time adaptation performance. We present our main contributions as follows:
We propose SwapPrompt, a novel test-time prompt adaptation method that employs a self- supervised contrastive learning strategy, enabling prompts better adapt downstream image classification tasks.
To the best of our knowledge, we are the first to apply unsupervised representation learning in prompt adaptation for pre-trained vision-language models. We introduce EMA prompt and prompt swapped prediction strategies, which enable the prompt to learn more knowledge from the powerful representation capabilities of pre-trained models.
We conduct extensive experiments on ImageNet and its four variants, as well as nine other image classification datasets The empirical evaluation shows that approach significantly outperforms current TPT methods and can even compete with supervised prompt adaptation methods on most datasets.
2 Related Work
Test-Time Adaptation. Test-time adaptation refers to reducing the performance gap when source model deployed on a different target domain of test data. The challenge of this issue is that only the source model and unlabeled test data are available, the training process and source data should not be accessed. Many solutions have been proposed solve this problem, i.e., minimizing the entropy of the model's predictions [I, maintaining set of dynamically prototypes and measuring the similarity between test samples to each prototype [ [3] generating new data similar to the target domain to assist model adaptation [1, 15) and utilizing the idea of self-supervised training to improve the generalization capability [0). However, test-time adaptation in vision-language model is under-explored. Recently, Shu et al. [19] propose test-time prompt tuning (TPT) to extend the old entropy minimization method vision-language model, but it is limited in practice due to the potential obvious over-confidence risk on predictions.
Prompt Learning in Vision-Language Models. Prompt learning first emerged in the field of natural language processing (NLP), aiming to enhance the performance of pre- pre-trained models by utilizing different prompts. With the advent of vision-language models that integrate both visual and textual modalities. inspired by prompt learning in NLP, CoOp [66] is proposed for prompt learning, which transforms hand-crafted prompts into learnable continuous prompts and tunes them to adapt to downstream tasks. CoCoOp [17), an improvement upon CoOp, employs a meta-net and image features to generate individual prompts for each image. Additionally, there are also some other methods such as CLIPP adIPtad (22) and Tip-adapter (3] that do not modify the prompts but instead add additional classification layers after the backbone models. What they have in common that all of them heavily rely on set of labeled data, making them unsuitable for test-time settings. Another line work focuses on enabling prompt learning in an unsupervised manner during training process, i.e., unsupervised prompt learning (UPL) [24]. However, it simply extends pseudo-labeling methods to insion-languae models without fully leveraging the powerful representation capabilities of pre-trained models.
3 Methodology
In this section, we first introduce the preliminary and problem definition of test-time prompt adapta- tion, then elaborate the proposed SwapPrompt framework that leverages self-supervised contrastive learning to facilitate prompt adaptation, which is shown in Figure Finally, we present the workflow in Section 3.3
3.1 Preliminary and Problem Definition
We focus on test-time prompt adaptation for pre-trained vision-language models (e.g. CLIP), where the model is trained on the source domain, but the test data belongs to the target domain. In this scenario, zero-shot CLIP with a general prompt (e.g., "a photo of [CL.S]") has shown barely acceptable zero-shot generalization ability. However, these hand-crafted prompts cannot fully extract
3

---

Target Prompt
(CLASS)
Text Encoder
EMA
Online Prompt to
[CLASS]
Text Features
Image Features
1
p (trDE
Test image X
Image Encoder
A2
Di - -(PololAs(xD)l
4f (pa(cl426x)))
ppseado ((pt.))) +((((.)) (xwap(()) ((p..9p) <(p. q!)
Framework of the proposed SwapPrompt. We use the text features generated by the target prompt as prototypes and assign the image feature of an augmented view of an image to these prototypes to obtain a soft class assignment. The online prompt is trained to predict this class assignment with a different augmented view of the same image. The EMA of online prompt used to update the target prompt.
Figure 2:
the rich knowledge learned by CLIP from large-scale and diverse pre-training datasets. Optimized prompts can further improve CLIP's ability to retrieve knowledge about the target domain. There have been some related works on supervised target domain data, including the well-known and effective CoOp method, part of our approach also incorporates the ideas of CoOp. Then will briefly review CoOp as well as define the notation used in this paper.
Context Optimization (CoOp). CoOp is prompt adaptation method based CLIP. Similar to CLIP, CoOp includes an image encoder and a text encoder, which are denoted as f()) and g()), respectively. Let Dtarget input data sample, Y/ is the corresponding label and yi E {1, ......................... C} for a C'-lasss classification problem, N is the size of the dataset. Let t represent the learnable continuous prompt and (t; c) be the input e-th class for the text encoder. We then define z; = f(x;) and w. g((t: ) as the output features embedding of image encoder and text encoder, respectively. The probability of c-th class for v, is computed as:
{ y be the dataset the target domain, where x; is the i-th
...
exp(sim(z, wc)/T)
p(cla))) S exp(sim(z, w,)/T)*
where 7 denotes the temperature parameter, sim(-) denotes the cosine similarity. For all training data, CoOp calculates the probabilities of all classes by Eq. [ and minimizes the cross-entropy loss to tune the prompt.
Time-Time aae Prompt Adaptation. In the test-time scenario, labeled data from the target domain not available, thereby the prompt cannot be optimized as in CoOp. Consider a test dataset Drast [rr) without label information. The objective of test-time prompt adaptation can be formulated as:
arg min C(f.g,t. Dreat)),
t
where L is the cross-entrope loss function, our purpose is to design an unsupervised prompt adaptation method, facilitating prompt t compatibility with the target domain of Dreat, eliciting more target domain knowledge for CLIP.
3.2 Overview of SwapPrompt
In this section, we present our proposed test-time adaptation method SwapPrompt, which contains two key insights: Exponential Moving Average (EMA) Prompt and Prompr Swapped Prediction. As illustrated in Figure SwapPrompt employs dual prompts paradigm: the online and target prompts. These two types of prompts will be interacted and learnt from each other to adapt to the target domain by applying an EMA update strategy. Moreover, unlike most previous frameworks that only match one prompt to one image, SwapPrompt adopts prompt swapped prediction mechanism to establish self--uper representation learning in prompt adaptation with the goal of assigning similar classes for two different augmentations of the same image.

---

3.2.1 Exponential Moving Average Prompt
SwapPrompt's goal is to learn an online prompt t, which can be used on the test dataset. As described previously, SwapPrompt has a target prompt t, to guide the update of online prompt t.- The main motivation behind this design is: from a given target prompt G, we can train new and potentially improved prompt t, through predicting the representation knowledge generated by the t,- By repeatedly using subsequent online prompts as new target prompts for further training, we can create a sequence of prompts that improves in quality over time. Practically, SwapPrompt uses a slowly moving exponential average of the online prompt the target prompt, and we perform the following update after each training step:
4 ct2 + (1 c)
where e E [0, 1] is the decay rate of target prompt.
3.2.2 Prompt Swapped Prediction
In the field of self-supervised learning, cross-view prediction has been widely used in many existing works. These methods usually cast the prediction problem into a representation space, and then learn representations by predicting different augmented views generated from the same image. It is assumed that the different augmented views of an image should be relatively close in the representation space.
Among these methods, SWAV 120 proposes different approach. Instead of directly enforcing consistent mapping between the image features in the representation space, SwAV clusters a set of image features, uses the cluster centroids as prototypes, and matches different augmentations of an image to these prototypes to compute its cluster assignment. By comparing their cluster assignments rather than their features, SwAV performs contrastive learning across multiple image views.
Inspired by SwAV, we assign an image's augmented view with prototypes to obtain its soft class assignment and predict this class assignment with another augmented view of the same image. This approach well-suited for CLIP, as it has natural prototypes: text features outputted by the text encoder. Specifically, for all classes Y E (1,2. ........ C} in the test dataset Dieet, target prompt & will form C' inputs {1e =) for text encoder, which will generate C' text features w = g ((t;:)) of different classes. Due to the supervised contrastive pre-training on large-scale image-text pairs, the text features generated by the CLIP have high similarity with the image features of the same class and low similarity with the text features of different classes. Therefore, these C' text features (w") can serve set of nigh-quaiity prototypes.
For an image Ti, we use two different data augmentation methods A₁ and A2 to obtain two augmented views A (1x) and (zr)). The image features generated by the image encoder are zł ( and 2, (())))))) respectively. By applying Eq. [n prototypes {w']e can acquire the corresponding class assignments of two augmented image views:
q =
and q = {pr(c) (c(s))))
(4)
where the (c( (xre) and p. A2(r:) can be expressed as: exp(sim(z) w /T)
and P((c)2(()))) exp(sim(zi (2) w!)/T)
(e))) Σ exp(sim(z) w/)/T)
j== exp(simmmmm)m (2) w
w///T)
(5)
Similarly, the predictions generated by online prompt are defined as: p (po(c and p (p.(c(A2(::)))
(6)
We establish the prompt swapped prediction loss function for image o, as: Lavap(z;)) = (pl. q²) + (p, 9²),
where is the function that measures the difference between the predictions and the class assignments. In this paper, we utilize the cross-entrog cross-entropy loss function. i.e.,
[p q
Σ pr
(8)
The online prompt t, will be optimized by Eq. 7 while the target prompt t, will not be updated by this loss function.
5

---

3.2.3 Prompt Optimization by Pseudo Label
In addition to utilizing self-supervised representation learning methods to optimize prompts, we also employ a set of labeled data, similar to CoOp, to optimize online prompts. However, unlike CoOp where the labels are available for target domain, the test data is unlabeled in the test-time scenario. Therefore, we first perform inference on the test data with hand-crafted prompts (e.g., "a photo of [CLS)"), obtaining their pscudo-labels Vreet = {8.))= and then employ Eq. 6 and cross-entropy loss as:
Lesoudo (24) fce (pl (. + tee (P,,0))
(9)
should be noted that the pseudo-labels obtained through this approach may contain noise. Therefore, we cannot apply Eq. to all test data. The process of data selection is discussed in Section 3.3
3.3 Algorithm Workflow
In this subsection, we illustrate the overall prompt adaptation process of SwapPrompt and the training method when facing online test samples.
Before training with Eq. 7 Eq. and pscudo labels Peest, need to perform data selection to filter out potential noisy pseudo labels. Specifically, we first employ the zero-shot CLIP and a hand-crafted prompt obtain pseudo labels and classification confidences for the test data. Then, for each class, we only select the top K test data with the highest confidence. These selected test data form the adaptation set D Dadapt. which is subset of D Dreat- For all x; E Dadapt: given trade-off hyper-parameters a and 3, the following loss function will be used to do prompt adaptation:
Ladape (xt) = arl-wap (x:) + (llpaudd(()))
(10)
When target images arrive in flow of mini-batches, i.e., the test data is online, we cannot sort the confidences of the entire test dataset. However, we can still perform confidence- based sorting on mini-batches to select the top (k < K) test data with the highest confidence, while keeping the rest of the training process unchanged. When new mini-batch test data arrives, the available test data are sorted by confidence again to obtain new Dadapt for prompt adaptation.
Experiments
4.1
Experimental Setup
Dataset. We evaluate the proposed SwapPrompt over fourteen datasets, including ImageNet [25 and its four variants: ImageNet- V2 (26). ImageNet-A (27), ImageNet-R (28] and ImageNet-
Sketch 229. and nine other publicly available image classification datasets used in CLIP: Cal-
tech101 30 DTD B1. Flowers 102
(32, Oxford-Pets B3 UCF101 (34,, StanfordCars [35] Food101 36 EuroSAT 37 and SUN397 [8]. These datasets encompass a diverse range of visual
classification tasks, including general objects, fine-grained categories, and even texture classification, forming comprehensive benchmark. We only use the test data to do adaptation and also evaluate models with them.
Baselines. We compare the performance of SwapPrompt with the state-of-the-are methods. In addition to zero-shot CLIP [66,, we also include TPT [19], a test-time prompt tuning method that minimizes the marginal entropy of test data; UPL (24), an unsupervised prompt learning approach and we make some modifications on it to suit the test-time setting; CoOp [17). supervised few-shot
prompt tuning method. We use some labeled data from the same domain the test data during training this baseline, in order to use it as an upper bound performance of test-time prompt adaptation.
Implementation Details. all experiments, we use the publicly available CLIP model with the ResNet-50 (39] visual encoder as the backbone model. Unless otherwise specified, the prompt is initialized randomly with 4 learnable tokens in SwapPrompt, UPL and CoOp. As for TPT, the prompt is initialized as the default one "a photo of a". When comparing the performance with baselines, we select the top 16 test data with the highest confidence to train SwapPrompt and UPI. For SwapPrompt, the decay rate of target prompt is 0.99, both a and 3 are 1. We use the same image augmentation

---

Table Comparison of test test-time adaptation methods on 14 datasets. A denotes SwapPrompt's gain
over the better one of UPL and TPT. '+ Online' denotes SwapPrompt with online test data.
ImageNet- Sketch
Caltech101
Flowers 102
Oxford-Pets
StanfordCars
ImageNet- V2
ImageNet-A
ImageNet-R R
UCF101
Food101
EuroSAT
SUN397
ImageNet
DTD
Method
CoOp
17
88.76 54.62
83.98
87.44
66.71
61.83 73.79 61.68 64.33 61.23 55.29 23.41 56.96 35.64
CLIP
16
85.13 42.16
65.40
83.05
61.15
55.65
74.23
37.60
58.55
58.18
51.36
21.69
55.98 33.33
UPL.
24
86.37 45.04
67.11
88.53
63.63
58.46
74.38
41.40
61.07
61.19
52.07 23.59 57.09 36.40
TPT
87.22
42.17 65.42
84.60
61.18
58.49
74.88
43.82
61.46
60.74
54.35
26.24 58.72 35.02
SwapPrompt
89.90 47.34
70.22
89.14
65.66
59.60
75.08
46.64
63.93
61.80
53.94
24.46 60.88 38.21
A
+2.68 +2.30
+3.11
+0.61
+2.03
+1.11
+0.20
+2.82
+2.47
+0.61
-0.11 -1.78
+2.16 +1.81
+ Online
89.69 46.40
68.12 88.97
64.52 58.88 75.66
42.45
63.36 61.41 52.93
24.42 60.25 38.13
method as SimCLR 40) to generate two different augmented images for image. We optimize the prompts for 50 epochs with SGD optimizer and a cosine decay learning rate scheduler, the initial learning rate is 0.002. The batch size of images is 32 all datasets.
We do all experiments on a workstation with an RTX 3090 GPU, a 3.5-GHZ Intel Core 19-11900K CPU and 64GB of RAM.
4.2 Performance Comparison
First, we compare our SwapPrompt with the baseline methods over fourteen benchmark datasets. The classification accuracy is listed in Table [ should be noted that CoOp is trained with labeled target domain data, and we use 4-shot data per category. From the results, it can be observed that our proposed SwapPrompt provides superior test-time adaptation performance than baselines on most datasets. We not only outperform the better baseline in UPL and TPT, but also very close even outperform CoOp on many datasets (e.g., Caltech Oxford- Petss,-Pets, Food101, ImageNet, ImageNet-R and SetaggeettSSkethhe Figure 3 shows that the average accuracy over 14 datasets for all baselines. SwapPrompt outperforms TPT and UPL by 2.31% and 2.17% accuracy, respectively.
Strong performance in online test-time adaptation setting. We also add results of SwapPrompt with online test data. Under this setting, we received mini-batches which only have small part of data. For example, in DTD, mini-batch has only 64 test data samples, less than 4% of the entire dataset. On most datasets, there is slight decrease in accuracy because we cannot hold enough test data at the beginning of training to learn an appropriate prompt to classify the test data that come first. However, online SwapPrompt still outperforms UPL and TPT most of datasets, as well as the average accuracy on datasets in Figure It should be noted that on Food101, the accuracy of online SwapPrompt is better. It is because the prompt in the intermediate stage is better than the one in the final stage. This could be attributed to the presence of high-confidence noise in pseudo labels during the final stage, which is discussed in section 4.3.
4.3 Ablation Study
In this subsection, detailed analyses are shown to help understand the superiority of our SwapPrompt method, including the trade-off between accuracy and efficiency, analysis on objective functions Lexap and Lpeeudo the decay rate of target prompt, the effect of K value in data selection, and the effect of prompt's context length and initialization.
The Trade-Off between Accuracy and Efficiency. The main factor which affects the efficiency of
SwapPrompt is the prompt adaptation epoch. Figure 3b) shows the relationship between the epoch and the average acuuracy of SwapPrompt on datasets (Caltech101, DTD, Flowers102. Oxford-Pets and UCF101). It can be seen that SwapPrompt's 's accuracy increases quickly in the first 3 epoch, then
7

---

65
74
Acctraccy (%)
Accuracy
72
60-
70
68
SwapPromge
66
UPL
55
64
IPT
CoOp
CLIP
UPL
TPT
Swap +Online
5
10
15
35 40 45 50 Number of epochs
20
25
30
Prompt
(a) Average accuracy on 14 datasets
(b) Accuracy of different epochs on datasets
Figure 3: (a) The average accuracy on all 14 datasets, CoOp is compared as an upper bound. (b) The average accuracy of SwapPrompt on 5 datasets with different adpatation epochs, the accuracy of UPL and TPT the final epoch average accuracy.
reaches its highest value around epoch 20 and stabilizes there. Thus, when the test time is limited, SwapPrompt can do trade-off between the accuracy and adaptation epochs, e.g., only train 3 epochs for quick inference. Noteworthy, SwapPrompt outperforms the final epoch accuracy of UPL at only epoch 2, and the accuracy of TPT at epoch 1.
Analysis on Objective Functions. We evaluate the two objective functions Lavap and pesuddo of SwapPrompt on 5 datasets. Results in Table givees effectiveness of our proposed objective functions. First, We use the UPL. as the basic baseline, which has the confident test data selection and objective function Lpaoudo. Then, UPL+AUG means that only adds image augmentation to the baseline, so that the Lpseudo applies on augmented image views. can be observed that the accuracy has improved on all datasets, which demonstrates the benefits of data augmentation. Compare with SwapPrompt, UPL+Aug do not has the Levap function, and the performance is poor than SwapPrompt. Finally, the case of using all two objective functions. i.c., the complete SwapPrompt, has the best performance, which demonstrates that the loss function Levap for the swapped prediction mechanism can further improve prompt.
gives a clear ablation study to demonstrate the
Table 2: Analysis of objective functions.
ImageNet
Caltech101
DTD
Flowers102
Oxford-Pets
UCF101
Average
UPL (4
61.19
86.37
45.04
67.11
88.53
63.63
68.65
UPL+AUG
61.30
87.75
46.04
68.43
87.67
65.15
69.39
SwapPrompt
61.80
89.90
47.34
70.22
89.14
65.66
70.68
should be noted that we do not include the result of using only Lavap- Because considering the inherent generalization ability of CLIP, it is unfair to directly compare the performance of only Lexap with Lgseudo The standalone application of Lewap does not sufficiently leverage the rich pre- disregarding the pseudo-labels, which
trained knowledge embedded within CLIP. This can be encapsulate the most pre-trained knowledge. Our proposed method, SwapPrompt, combines Lavap and L Lipaaudo resulting in improved performance compared using Lpseudo alone.
Analysis on the Decay Rate of Target Prompt. SwapPrompt updates the target prompt by slow- moving average of online prompt, thus the target prompt represents delay and more stable version of the online prompt, a higher decay rate indicates the retention of a greater amount of historical information. As is shown Eq. 3 when the decay rate O, the target prompt is instantaneously updated to the online prompt at each step. It should be noted that using only one online prompt on Lewap loss (i.e., Levap (z:) Cpp. q.) + E(p? q )) cannot work because the gradient will collapse, the alternative is to maintain a target prompt that remains identical to the online prompt. (i.e., = 0). When the decay rate corresponding to its initialization. In this case we initialize the target prompt as "a photo of a",, thus it still has basic zero-shot generalization ability. There is a trade-off between updating the targets too often and updating them too slowly.
1, the target prompt never updated, and remains
constant value

---

Table shows the results of different decay rates on 6 datasets. When 0 and 1, the performance is poor than the other three eclectic values. All values of the decay rate between 0.9 and 0.999 yield have its best applicable datasets and 0.99 decay rate has highest average accuracy. Besides that, compared with maintaining fixed target prompt 1), the EMA strategy can increase the average accuracy by 1 1.35%.
Table 3: Results for different decay rates e.
value
ImageNet
Caltech101
DTD
Oxford-Pets Flowers102
UCF101
Average
60.87
88.49
45.21
68.57
87.01
65.85
69.33
0,999
61.88
89.71
47.58
68.70
88. 74
66,93
70.59
0.99
61.80
89,90
47.34
70.22
89.14
65.66
70.68
0.9
61.76
87.99
47.64
70.65
87.47
65,00
70.09
61.29
87.38
47.22
69.63
87.71
64.68
69.65
Analysis on the swapped prediction mechanism. Inspiring from the idea of self-supervised contrastive learning, SwapPrompt uses both image augmentation and EMA update strategies on target prompt. Both methods are effective and indispensable, and the combination of the two methods naturally leads "swap prediction" (i.e., Lavap).
There are two other loss functions which are similar to Lavap: L2(x;) = E(P P.) E(gi:og g and L2(x;) = {(pp.g)))((p) + ). However, compared with Lovap, both L, and L2 lost one of advantage. For L1, the two prompts will adapt independently without any interaction so the online prompt camnot be updated under the guidance of target prompt. The advantage of historical information will be lost. For L2, the contrastive phase is removed due to the lack of the approaching process between two augmented images. The advantage of image augmentation will diminish. performance using those three loss functions on datasets. The experimental setting is the same as Table I except for the part of Levap. The results indicate that Lavap outperforms L, and L2.
the
Table 4: Results for using different loss function.
Caltech101
DTD
Flowers102
Oxford-Pets
UCF101
Average
L
87.38
47.22
69.63
87.71
64.68
71.32
Lg
88.45
46.69
69.28
87.30
64.46
71.24
Lawap
89.90
47.34
70.22
89.14
65.66
72.45
Analysis on the Top-K Confident Data Selection. We have data selection before the prompt adaptation to filter out potential noise pseudo labels. Only top K confident test data will be used test-time adaptation. Too much data only will decrease model performance, but also slightly increased training time. 5 the performance of SwapPrompt with different K values on datasets, the result without data selection is also included. In general, larger values of K show better performance, the highest accuracy across all 5 datasets is achieved when K is set to 8 or 16, with K 16 having the best average accuracy. On the other hand, the performance using the entire test data does not surpass the performance when data selection is employed. Data selection can increase the average accuracy by 2.5%. This indicates that the negative impact of noisy pseudo labels outweighs the positive effects.
Analysis on the Context Length and Initialization of Prompt. To explore whether our Swap- Prompt works equally well prompts in different context lengths and initialization, repeat experiments on datasets by varying the context length from 4 to 8 to 16, and initializing with three different type.
The results of different context lengths are shown Table G(a), which indicates that having more context tokens sometimes leads to slightly decrease on accuracy. This is probably due to shorter prompt learned less overfitting from selected samples. SwapPrompt only uses part of test data to do adaptation, too much parameters in prompts may cause overfitting those data. Nevertheless,

---

Table 5: Results for different K in data selection. 'None' denotes no data selection.
K value
Caltech101
DTD
Flowers102
Oxford- Pets
UCF101
Average
87.51
41.13
65.04
87.83
62.20
68.7
2
88.72
43.50
64.66
$7.42
62.97
69.45
4
89.49
44.15
66.63
88.98
62.76
70.40
8
90.14
47.70
66.30
88.43
63.96
71.31
16 89.90
47.34
70.22
89.14
65.66
72.45
None
88.03
43.97
68.41
86.68
62.64
69.95
SwapPrompt still maintains advanced performance on different context lengths. In Table the three different initialization are hand-craft: "a photo of [CLS]", Pre-ImageNet: the prompt which is trained on 16-shot ImageNet with CoOp, and random initialization. only slightly affects the final accuracy. Because of the effective adaptation prompt, SwapPrompt demonstrates robust performance, which indicates that our method does not rely on any prior source domain prompt.
find that different initialization
Table 6: Analysis on the context length and initialization of prompt.
(a) Results for different context lengths.
(b) Results for different initialization.
Caltech101
Flowers102
Oxford-Ped PPts
Caltech111
Flowers10)
Oxford-Per Pees
UCF101
Average
UCF101
Average
DTD
DTD
Length
Initialization
89.90 47.34 70.22 89.14 65.66
72.45
Hand-craft
89.53 46.92 70.32 88.92 65.98
72.33
8
88.24 47.46 70.77 88.79
65.62
72.18
Pre-ImagNe
89.65 47.15 70.44
89.03
65.34
72.36
87.71 46.39 70.12
88.43 66.48
71.83
Random
89.90 47.34 70.22 89.14
65.66
72.45
Analysis on the sensitivity of hyper-parameters and B. To explore the sensitivity of Swap- Prompt about syper-parameters and 8, conduct experiments with different values of and on 5 datasets, other experimental settings are the same as Table The results are shown Table 7 It be seen that SwapPrompt is not sensitive the choice of hyperparameters and 3 in most cases. Results of SwapPrompt with different baselines in Table
settings in Table 7 still outperforms
Table 7: Analysis on the sensitivity of seper-parameters and B.
Caltech101
DTD
Flowers102
Oxford-Pees
UCF101
Average
0.6, 8 1.4
88.56
46.80
70.04
88.24
65.21
71.77
1.0, B 1.0
89.90
47.34
70.22
89.14
65.66
72.45
a 1.4, B 0.6
89.45
47.10
70.32
87.97
65.77
72.12
Conclusion
In this paper, we have investigated a novel test-time prompt adaptation method, SwapPrompt, to learn the prompt adapted to the test domain for pre-trained vision-language models. Specifically, we maintain an online prompt and an EMA updated target prompt which interact and learn from each other. A swapped prediction mechanism is designed to train the online prompt, enabling it to predict target prompt's class assignment the same image under different augmented view. Without any other requirement, SwapPrompt can be easily deployed on the test-time of vision-language models. Extensive empirical experiments have been conducted over various datasets verify the effectiveness and superior performance of SwapPrompt.
10

---

Acknowledgements
This research was supported by fundings from the Key-Area Research and Development Program of Guangdong Province (No. 202180101400033) Hong Kong RGC Research Impact Fund (No. R5060-19, No. R5034-18), Areas of Excellence Scheme (AoE/E-601/22-R) General Research Fund (No. 152203/20E, 152244/2IE, 152169/22E. 152228/23E), Shenzhen Science and Technology Innovation Commission JCCY120200109914200888
References
[1] J Quinonero Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. The MIT Press, 1:5, 2009.
[2] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[3] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of in-the-will distribution shifts. International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning visual features. Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[5] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. Pmlr, 2018.
[6] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pages 9229-9248. PMLR, 2020.
[7] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[8] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 295-305 2022.
[9] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-fres online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8344-8353, 2022.
[10] Fatemeh Azimi, Sebastian Palacio, Federico Raue, Jörn Hees, Luca Bertinetto, and Andreas Dengel. Self-supervised test-time adaptation video data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3439-34444 3448, 2022.
[11] Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen. Test-time adaptation to distribution shift by confidence maxi- mization and input transformation. arXiv preprint arXiv:2106 14999, 2021.
[12] Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039. PMLR, 2020.
[13] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances Neural Information Processing Systems, 34:2427-2440, 2021.

---

[14] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[15] Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsuper- vised domain adaptation without source data. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 9641-9650, 2020.
[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 1748-8766 PMLR, R, 2021.
[17] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009))2337-2348, 2022.
[18] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825 2022.
[19] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[20] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 9924,, 2020.
[21] Connor Shorten, Taghi M Khoshgoftaar, and Borko Furht. Text data augmentation for deep learning. Journal of big Data, 8:1-34, 2021.
[22] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:21 2021.
[23] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free free clip-adapter for better Dision-language modeling. arXiv preprint arXiv: 1.03930, 2021.
[24] Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision-language models. arXiv preprint arXiv:2204.03666 2022.
[25] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages
leee, 2009.
[26] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference machine learning, pages 5389-5400. PMLR, 2019.
[27] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[28] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8340 2021.
[29] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances Neural Information Processing Systems, 32, 2019.
12

---

[30] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pultern recognition workshop, pages 178-178. IEEF, 2004.
[31] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 3606-3613, 2014.
[32] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
[33] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498 3505. IEEE, 2012.
[34] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos the wild. arXiv preprint arXiv: 121..002 2012.
[35] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference computer vision workshops, pages 554-561, 2013.
[36] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
(ood-101-mining discriminative
[37] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):221722226, 2019.
[38] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages
IEEE, 2010.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[40] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
13


Reviews:
[{"review_id": "B26TfUSawa", "reviewer": "Reviewer_VTrP", "summary": "This paper proposes the test-time prompt adaption method \"SwapPrompt\" for the vision-language model. The author leverages contrastive learning strategy to learn the prompts on multiple augmented views of unseen data. To better adapt to out-of-distribution tasks, the learnable target and online prompts are applied to samples and interact with each other to retrieve the swapped prediction loss. Besides, a pseudo-labeling strategy and data selection process is applied to the test-time data. The results show that SwapPrompt has a better performance compared to other unsupervised prompt learning baselines.", "questions": "To my understanding, the proposed SwapPrompt can better leverage the prior knowledge of the pre-trained model during the adaption by using the swap prediction mechanism and pseudo labeling strategy. However, the experiment results might not convince the reviewer that SwapPrompt is better than the current method. Also, there exist some other kinds of OOD benchmarks, such as the ImageNet-C. Can SwapPrompt also be leveraged on multiple kinds of corruption types?", "limitations": "See abovementioned weakness and questions.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-organized and easy to follow. The reviewer thought the idea of using swapped prediction loss to optimize the prompt was good.", "weaknesses": "The related works might lack some existing works. For example, one of the similar works that the author didn't mention [1] also applied the multiple augmented views of samples for calculating the marginal cross-entropy loss to do the adaptation. \n\nThe benefit of applying the EMA update strategy on the prompt is not clear enough. The author mentions that the target prompts are slowly updated by the online prompt for the purpose of learning more knowledge from prior pre-trained models, but it seems like target prompts are updated after every training step (equation 3). Does one training step here mean a one-time adaption for just one batch or multiple-round adaption on multiple batches? \n\nFor the experiment part, why not compare with [1], which also uses multiple augmented views of samples and unsupervised objective function to improve test-time adaptation?  It seems like the idea is similar to this work which adapts the model by minimizing the marginal output distribution across the augmentations.\n\n[1] MEMO: Test Time Robustness via Adaptation and Augmentation. Marvin Zhang, Sergey Levine, Chelsea Finn", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and suggestions. We have tried to address all your comments and suggestions on Weaknesses and Questions as follows:\n\n**W1:** The related works might lack some existing works. For example, one of the similar works that the author didn't mention [1] also applied the multiple augmented views of samples for calculating the marginal cross-entropy loss to do the adaptation.\n\n**A1:** We agree with the reviewer that both MEMO in [1] and our SwapPrompt utilize multiple augmented views of samples. However, MEMO relies on the **marginal entropy minimization** rather than the **marginal cross-entropy loss** employed in our work. This makes MEMO more similar to TPT in the baseline, because both MEMO and TPT use multiple augmented views of samples and entropy minimization.\n\nIn fact, by comparing the published source code of MEMO and TPT, we found that TPT uses almost exactly the same loss function as MEMO, the difference between these two approaches is that MEMO adapts all parameters of a network model, while TPT only tunes the prompt. \n\nWe added experiments to combine CLIP with MEMO: we used MEMO's training method and the image encoder in CLIP, and fine tuned the image encoder at test time. The detailed results can be shown in the following table. From the experimental results, we can observe that SwapPrompt also outperforms MEMO.\n\n|  **Method**   | Caltech101  | DTD | Flowers102 | Oxford-Pets | UCF101 |\n|  :-  | :-:   | :-:  | :-:   | :-:   | :-:   | \n| CLIP  | 85.13 | 42.16 | 65.40 | 83.05 | 61.15 |\n| **MEMO**  | 85.44 | 42.47 | 62.63 | 83.62 | 61.28 |\n| UPL | 86.37 | 45.04 | 67.11 | 88.53 | 63.63 |\n| TPT | 87.22 | 42.17 | 65.42 | 84.60 | 61.18 |\n| SwapPrompt | 89.90 | 47.34 | 70.22 | 89.14 | 65.66 |\n\nBesides that, we improved the presentation of “Related Work” by conducting a more comprehensive literature review in the new version. \n\n**W2:** The benefit of applying the EMA update strategy on the prompt is not clear enough. The author mentions that the target prompts are slowly updated by the online prompt for the purpose of learning more knowledge from prior pre-trained models, but it seems like target prompts are updated after every training step (equation 3). Does one training step here mean a one-time adaption for just one batch or multiple-round adaption on multiple batches?\n\n**A2:** Applying the EMA update strategy on the prompt is to allow the target prompt to retain more historical information, and be more stable and able to guide the tuning of online prompt. Table 3 in the paper shows the experiment results without using the EMA update strategy, and it shows that EMA is an indispensable part to achieve better performance.\n\n“Target prompts are updated after every training step (equation 3).\" The training step here refers to a one-time adaptation for just one batch. The purpose is to make target prompts update more slowly (compared to online prompts) at an appropriate rate, so we set a fairly large delay rate of target prompts to ensure that the target prompt retains a large amount of history information.  At the same time, the decay rate can't be 1, because then the target prompt doesn't update, and it cannot learn from the test data.\n\nRegarding your comment about updating the target prompt after multiple-round adaptation on multiple batches, this is an interesting idea, and we believe that this approach also lowers the update rate of the target prompt, which we would like to discuss further in the future work.\n\n**W3:** For the experiment part, why not compare with [1], which also uses multiple augmented views of samples and unsupervised objective function to improve test-time adaptation? It seems like the idea is similar to this work which adapts the model by minimizing the marginal output distribution across the augmentations.\n\n**A3:** As answered in **A1**, MEMO is a baseline worth to compare, we thank the reviewer for his constructive comments and the experimental results can be found in **A1**.\n\n**Q4:** To my understanding, the proposed SwapPrompt can better leverage the prior knowledge of the pre-trained model during the adaption by using the swap prediction mechanism and pseudo labeling strategy. However, the experiment results might not convince the reviewer that SwapPrompt is better than the current method. Also, there exist some other kinds of OOD benchmarks, such as the ImageNet-C. Can SwapPrompt also be leveraged on multiple kinds of corruption types?\n\n**A4:** Thanks to the reviewer's suggestion, we added the experiments on **ImageNet-C**. Due to limited time in rebuttal period, we cannot do experiments on all the corruption types, so we experimented on four types of corruption, which are **snow** in weather, **contrast** in digital,  **defocus blur** in blur and **gaussian noise** in noise, all of them have a corruption severity level of 3. The results of the experiments are shown in the following table. We can see that our method still has a good performance on the corruption types.\n\n|  **Method**   | Snow  | Contrast | Defocus blur | Gaussian noise | \n|  :- | :-:   | :-:   | :-:   | :-:   | \n| CLIP  | 22.02 | 37.24 | 23.64 | 15.51 | \n| UPL | 25.85 | 41.11 | 25.09 | 17.01 | \n| TPT | 26.33 | 40.64 | 25.12 | 16.89 | \n| SwapPrompt | 26.54 | 41.32 | 25.47 | 17.24 | \n\n\n"}, {"review_id": "jcXMvlxzfC", "reviewer": "Reviewer_73P2", "summary": "The authors propose SwapPrompt, a novel test-time prompt adaptation method that leverages self-supervised contrastive learning to enhance prompt adaptation for pre-trained vision-language models. SwapPrompt employs an online prompt and an EMA updated target prompt, along with a swapped prediction mechanism, to facilitate prompt learning and improve performance. Experimental results demonstrate that SwapPrompt achieves state-of-the-art test-time adaptation performance on multiple datasets, and it can be easily deployed on vision-language models without additional requirements.", "questions": "While the proposed method predominantly focuses on text prompts, there have been recent studies that utilize visual prompts or multimodal prompts (combining visual and text). I am curious to know what trends or patterns would emerge if visual prompts or multimodal prompts were employed in the proposed framework.\n\n[1] Jia, M., Tang, L., Chen, B. C., Cardie, C., Belongie, S., Hariharan, B., & Lim, S. N. Visual prompt tuning. ECCV 2022\n[2] Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. Maple: Multi-modal prompt learning. CVPR 2023", "limitations": "The authors did not explicitly address the limitations of the proposed method.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and presents information in a clear and coherent manner, making it easy to understand.\n- The proposed method, SwapPrompt, can be easily implemented and used on test-time with vision-language models.\n- The authors conducted extensive experiments on multiple datasets, and the results obtained were promising and showed positive outcomes.", "weaknesses": "There is no comparison for robustness to natural distribution shifts which is done in TPT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and suggestions. We have tried to address all your concerns as follows:\n\n**W1:** There is no comparison for robustness to natural distribution shifts which is done in TPT.\n\n**A1:** If we understand your comment correctly, we have already shown the results of robustness to natural distribution shifts as you mentioned in TPT[1], i.e., Table 1. The experiment setting is essentially the same as Table 1 in TPT.\n\nSpecifically, natural distribution shifts in TPT refer to training with ImageNet and testing with four different ImageNet variants. While TPT does not require training data, its prompt is initialized as \"a photo of a\". Similar to TPT, our method does not require training data as well, thus we also did not use prompts trained on ImageNet. The comparisons in our Table 1 are fair. The following excerpt from the results in table 1 shows the results.\n\n|  **Method**   | ImageNet  | ImageNet-V2 | ImageNet-A | ImageNet-R | ImageNet-Sketch |\n|  :-  | :-: | :-:  | :-:  | :-: | :-: | \n| CLIP  | 58.18 | 51.36 | 21.69 | 55.98 | 33.33 |\n| UPL | 61.19 | 52.07 | 23.59 | 57.09 | 36.40 |\n| TPT | 60.74 | 54.35 | 26.24 | 58.72 | 35.02 |\n| SwapPrompt | 61.80 | 53.94 | 24.46 | 60.88 | 38.21 |\n\n[1] Shu M, Nie W, Huang D A, et al. Test-time prompt tuning for zero-shot generalization in vision-language models[J]. Advances in Neural Information Processing Systems, 2022, 35: 14274-14289.\n\n**Q2:** While the proposed method predominantly focuses on text prompts, there have been recent studies that utilize visual prompts or multimodal prompts (combining visual and text). I am curious to know what trends or patterns would emerge if visual prompts or multimodal prompts were employed in the proposed framework.\n\n**A2:** Thanks for your valuable comments. We believe that our proposed SwapPrompt will be also effective on visual prompts or multimodal prompts.  The principle behind our method is to utilize the powerful potentials of self-supervised learning in fine-tuning the prompt on test time, which can be considered as a general framework for prompt-based tuning methods. In future work, we would like to consider changing our method directly to do adaption on visual prompts.\n\n"}, {"review_id": "Qdfn5y5UtM", "reviewer": "Reviewer_N4YN", "summary": "This paper studies an interesting and practical task “test-time adaptation for large vision and language models”. The authors propose a novel SwapPrompt method by introducing self-supervised contrastive learning into prompt tuning. The proposed dual prompts paradigm and swapped prediction mechanism enhance performance, achieving state-of-the-art results on ImageNet and nine other datasets. However, I still have several questions that need to be further clarified, as outlined below.", "questions": "1. Could the authors summarize the advantages of Prompt Tuning over Image Encoder Tuning?\n\n2. How about the performance of SwapPrompt if SwapPrompt also updates the parameters (e.g., affine parameters in norm layers) in Image Encoder?\n\n3. Comparisons with “only one online prompt” + “consistency maximization (i.e., $l(p1, p2)$)” are also preferred.\n\n4. For Table 4, could the authors also include the results of CLIP? Additionally, when the batch size is very small during online adaptation, e.g., 1, how to select data? And in this case, will the performance gain of SwapPrompt degrade a lot?\n\n5. I am curious about the performance of SwapPrompt on corruption data (ImageNet-C)?\n\n6. Could the authors report more efficiency comparisons with TPT? Such as GPU runtime and Memory. I think SwapPrompt would be much more efficient and this is also an advantage.\n\n7. Will the authors release the source code after the acceptance of this work?", "limitations": "no limitations discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed SwapPrompt method is novel in the TTA community. It successfully introduces self-supervised contrastive learning into Prompt Tuning-based methods.\n2. Experiments are thorough and demonstrate the effectiveness of the proposed method.\n2. The paper is well-written and easy to follow. ", "weaknesses": "1. Two important ablation experiments are missing. $L_{swap}$ actually has two variants: $L_1 = l(p1, p2) + l(q1, q2)$ and $L_2 = l(p1, q1) + l(p2, q2)$. For these two variants, I think they are essentially the same as the consistency maximization used in TPT and MEMO. In this sense, comparison with $L_1$ and $L_2$ are necessary to verify the effectiveness of “Swap”.\n\n2. More in-depth analyses/motivation regarding why “Swap Predictions” is working are preferred. The current explanations are not thorough enough. So I would like to see more details in the authors’ response.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive and detailed comments, we have tried to address all your concerns one by one.\n\n**W1&W2:**  \"Two important ... in the authors’ response.\"\n\n**A1&A2:**  Thanks for your valuable comments. We will first explain why \"swap prediction\" works: inspiring from the idea of self-supervised contrastive learning, our SwapPrompt uses both image augmentation and target prompt with rich historical information to do adaption on the online prompt. Among them, image augmentation is a widely used design for constructing positive or negative pairs, in which the effectiveness has already been proved in many existing self-supervised contrastive learning frameworks. On the other hand, the same idea can be applied to textual modality by using EMA update strategies, which allows the target prompt to retain more historical information, and also be more stable and able to guide the tuning of the online prompt.\n\nBoth methods are effective and indispensable, and the combination of the two methods naturally leads to \"swap prediction\", which we will further illustrate by analyzing why we do not use $L_1$ and $L_2$ as you mentioned:\n\n* If $L_1$ is used: the two prompts will adapt independently without any interaction so the online prompt cannot be updated under the guidance of target prompt. The advantage of historical information will be lost. \n\n* If $L_2$ is used: the contrastive phase is removed due to the lack of the approaching process between two augmented images. In this case, the advantage of image augmentation will diminish. \n\nWe present the added experiment results of using $L_2$ or $L_2$ in the following table. \n\n|    | Caltech101  | DTD | Flowers102 | Oxford-Pets | UCF101 |\n|  :-  | :-:  | :-:  | :-:  | :-:  | :-:  | \n| $L_1$  | 87.38 | 47.22 | 69.63 | 87.71 | 64.68 |\n| $L_2$  | 88.45 | 46.69 | 69.28 | 87.30 | 64.46 |\n| SwapPrompt  | 89.90 | 47.34 | 70.22 | 89.14 | 65.66 |\n\nNote that our SwapPrompt uses swap prediction to take advantage of both image augmentation and historical information.\n\n**Q1:** \"Could the authors ... Encoder Tuning?\"\n\n**A1:** Prompt tuning generally works better than Image Encoder Tuning with small amounts of data. As the number of tuning parameters in prompt is far smaller than image encoder, prompt tuning requires less time compared to Image Encoder Tuning. On the other hand, directly altering a vision language model trained on extensive data could disturb its feature alignment between its image encoder and text encoder, possibly resulting in decreased performance. The results in Table 5 in [1] also show that fine-tuning the image encoder does not work well.\n\n[1] Zhou K, Yang J, Loy C C, et al. Learning to prompt for vision-language models[J]. International Journal of Computer Vision, 2022, 130(9): 2337-2348.\n\n**Q2:** \"How about ... Image Encoder?\"\n\n**A2:**  We added the experiment that SwapPrompt updates the parameters of all Batch Normalization (BN) layers in Image Enoder. The following table shows the results.  \n\n|    | Caltech101  | DTD | Flowers102 | Oxford-Pets | UCF101 |\n|  :-  | :-:  | :-:  | :-:  | :-:  | :-:  | \n| SwapPrompt  | 89.90 | 47.34 | 70.22 | 89.14 | 65.66 |\n| update BN layers  | 86.65 | 38.18 | 54.35 | 76.83 | 58.29 |\n\nWe also try to only update a part of norm layers but get worse results. Thus we think the adaptation on Image Enoder may lead to a decrease in performance. \n\n**Q3:** \"Comparisons with ... also preferred.\"\n\n**A3:** Thanks for your valuable comments. As we explained in **A1&A2**, “only one online prompt” + “consistency maximization” forfeits the benefits of historical information. The experiments in **A1&A2** also demonstrated that using $l(p_1, p_2)$ does not yield better performance than SwapPrompt.\n\n**Q4:** \"For Table 4... degrade a lot?\"\n\n**A4:** The result of CLIP is the same as Table 1. When the batch size is 1, we will not do the data selection. The results of this experiment are presented in the following table. It can be seen that SwapPrompt still outperforms baselines. \n\n|  Batch size = 1  | Caltech101  | DTD | Flowers102 | Oxford-Pets | UCF101 |\n|  :-  | :-:  | :-:  | :-:  | :-:  | :-:  | \n| TPT  | 87.22 | 42.17 | 65.42 | 84.60 | 61.18 |\n|  UPL | 88.18 | 42.07 | 65.24 | 85.44 | 62.71 |\n| SwapPrompt  | 88.64 | 42.49 | 65.98 | 86.24 | 63.05 |\n\n**Q5:** SwapPrompt on corruption data (ImageNet-C)?\n\n**A5:** Thanks to the reviewer's suggestion, we added the experiments on ImageNet-C. Due to the limited time in rebuttal period, we cannot do experiments on all the corruption types, so we experimented on four types of corruption, which are **snow** in weather, **contrast** in digital,  **defocus blur** in blur and **gaussian noise** in noise, all of them have a corruption severity level of 3. The results of the experiments are shown in the following table. We can see that our method still has a good performance on the corruption types.\n\n|  **Method**   | Snow  | Contrast | Defocus blur | Gaussian noise | \n|  :-  | :-:  | :-: | :-:  | :-: | \n| CLIP  | 22.02 | 37.24 | 23.64 | 15.51 | \n| UPL | 25.85 | 41.11 | 25.09 | 17.01 | \n| SwapPrompt | 26.54 | 41.32 | 25.47 | 17.24 | \n\n**Q6:** \"Could the authors ... an advantage.\"\n\n**A6:** Thanks for your valuable comments. We recorded the training time for SwapPrompt when its performance surpasses that of TPT (about 5 epochs' adaptation, refer to Figure 3 in the paper). The results show that SwapPrompt has lower GPU Memory utilization and shorter training time.\n\n| | Caltech101  | DTD | Flowers102 | Oxford-Pets | UCF101 |\n|  :-  | :-:  | :-: | :-:  | :-: | :-: | \n| TPT  | | | | | \n| Runtime (Min)  | 15.3 | 9.2 | 16.9 | 23.5 | 25.7 |\n| Memory (MiB) | 8985 | 9073 | 9023 | 9080 | 9074 |\n| SwapPrompt | | | | | \n| Runtime (Min)  | 4.3 | 4.1 | 4.2 | 3.9 | 4.2 |\n| Memory (MiB) | 4839 | 5042 | 4844 | 5092 | 4858 |\n\n**Q7:** Will the ... this work?\n\n**A7:** Yes, we will open the code once the paper is published.\n\n "}, {"review_id": "6ZMKIvUlck", "reviewer": "Reviewer_V8X4", "summary": "This work presents a framework to perform test-time prompt adaptation for pre-trained vision-language models. In test-time settings, one has no access to labeled test data. Therefore, the authors employ self-supervised contrastive learning techniques on the test domain. In particular, this works proposes an exponential moving average prompt mechanism, and a prompt swapped prediction mechanism. The proposed approach attains state-of-the-art results on image classification benchmarks against other unsupervised test-time adaptation methods. It shows competitive results when compared with approaches that actually use labeled data during test-time (CoOp).", "questions": "- Potential Ablation Study: one might argue that gathering a few groundtruth labels for the target domain is always a possibility. I understand this is not the scope of the paper. But I’d be curious to see how much this approach could improve if one has, say, 5-10 labeled images in the mini batches. Maybe one could downweight the auto-labeled samples in upweight the actual groundtruth samples? Just a potential ablation that could be interesting. Authors, please don’t tackle this ablation if you consider this not to be relevant and, especially don’t tackle this if it would take a long time to prepare. Some comments on this would be enough to satisfy my interest here. I think I’m just curious to see how much labeled data your approach would need to beat CoOp (ideally less, right?)\n- Could you talk about how your approach could be integrated into downstream tasks, such as object detection? For instance, how could one integrate SwaptPromt into OWL-ViT (Vision Transformer for Open-World Localization)?\n- Will the code become available? Could be interesting for reproducibility.", "limitations": "Limitations have been addressed implicitly. It might be a good idea to add a very small section that explicitly points out all the limitations that authors have identified. This is typically a helpful section for future researchers to get inspiration from for future work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality\n\n- This work presents a novel approach to tackle the following problem: vision-language models pre-trained on a source domain cannot reach their full potential when deployed on a new target domain where no labels are available. I find this a very relevant problem to tackle, with major repercussions on a wide variety of down-stream tasks.\n- Its two proposed mechanisms for test-time adaptation leverage smart unsupervised learning techniques to overcome the lack of target domain labels. Although these mechanisms are not novel per se, their combination tacke this specific problem is.\n\nQuality:\n\n- The paper is well-written in its most part. See “Weaknesses” below.\n\nClarity:\n\n- The proposed method is clear, and the authors do a good job in comparing their approach against previous works and problem statements.\n\nSignificance: \n\n- Tackling the problem of test-time adapting prompts to leverage pretrained vision-language backbones when no labels are available is quite a major and interesting challenge.", "weaknesses": "Please have a native english speaker review the grammar. \n\n- Sentences like “SwapPrompt’s goal is to learn an online prompt t_0 which can be used for test dataset.” (line 153) do not sound grammatically correct. You’re probably missing a “the” before dataset. And instead of using “for”, you should probably use “on”.\n- In line 229, the sentence is very long and hard to read. The comma in this line (“prompt tuning method, we use some labeled data”) should not be a comma, but rather a full stop.\n- Line 258 (”It should be noted that on Food101, the accuracy\n258 of online SwapPrompt is better, it is because the prompt …”). Again, the sentence is too long. I would say you need a full stop after “better”.\n- Line 315-317 (”This is probably due .. data”). This sentence is very poorly written. Please correct grammar. What do you mean by ”...shorter prompt learned less overfitting”? Also, it should be ”uses”, not “use”. In general, this sentence is again too long.\n- Lines 320-323. Another too long sentence. Please rewrite into 2 sentences. Will make what you’re trying to say so much more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for recognizing the value of our work and for the constructive comments, which have helped us a lot to improve the paper's quality and clarity. \n\n**W1-5:** Please have a native english speaker review the grammar.\n\n**A1-5:** We appreciate your highlighting of the grammatical errors and some unclear expression in our paper. We will rectify these issues in the new version and reduce the use of lengthy and complex sentences.\n\n**Q1:** Potential Ablation Study: one might argue that gathering a few groundtruth labels for the target domain is always a possibility. I understand this is not the scope of the paper. But I’d be curious to see how much this approach could improve if one has, say, 5-10 labeled images in the mini batches. Maybe one could downweight the auto-labeled samples in upweight the actual groundtruth samples? Just a potential ablation that could be interesting. Authors, please don’t tackle this ablation if you consider this not to be relevant and, especially don’t tackle this if it would take a long time to prepare. Some comments on this would be enough to satisfy my interest here. I think I’m just curious to see how much labeled data your approach would need to beat CoOp (ideally less, right?)\n\n**A1:** Thank you for presenting such a valuable experimental scenario.  It is true that labeled data may be scarce in many real-world applications due to the tremendous overhead and the requirement of corresponding expertise in data annotation. On this basis, some semi-supervised learning methods can be introduced to complement the current prompt-tuning framework. However, due to the time constraints for rebuttal, we cannot conduct detailed experiments on this matter. I believe our approach can achieve comparable performance to CoOp's requirements with less labeled data. We can incorporate some semi-supervised learning techniques to refine our solution. For example, we can fine-tune the prompt using labeled data and then use this refined prompt to generate pseudo-labels for unlabeled data. Alternatively, as you mentioned, giving more weight to ground truth samples is also a viable option. \n\nThis is an intriguing question, and we appreciate your suggestions. We will consider this question in future research.\n\n**Q2:** Could you talk about how your approach could be integrated into downstream tasks, such as object detection? For instance, how could one integrate SwaptPrompt into OWL-ViT (Vision Transformer for Open-World Localization)?\n\n**A2:** We believe our approach can be integrated into existing state-of-the-art methods, which utilize vision-language models for downstream tasks. For instance, as you mentioned, OWL-ViT. As far as we know, OWL-ViT, similar to CLIP, undergoes training using image-text pairs, but it doesn't adapt the prompt during inference. This is where our SwapPrompt can be embedded, allowing fine-tuning of the prompt, followed by obtaining the final detection results.\n\n**Q3:** Will the code become available? Could be interesting for reproducibility.\n\n**A3:** Yes, we will open the code once the paper is published.\n"}, {"review_id": "wvpHuFkxQM", "reviewer": "Reviewer_3xii", "summary": "Authors propose SwapPrompt for the CLIP-based test-time adaptation task. SwapPrompt consists of two loss functions, swapped prediction loss and pseudo label loss, with a prompt EMA strategy. Experimental results on 14 datasets validate that SwapPrompt outperforms several baselines.", "questions": "1. Could authors provide the results of FGVCAircraft dataset, which is used in CoOp and UPL experiments.\n2. Which augmentation is used in the prompt swapped prediction loss? Which augmentation is used in calculation of the pseduo label? Are choises of the augmentation affect the swapped prediction loss?", "limitations": "No potential negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The experimental results of the article look convincing.\n2. Authors conduct a detailed experimental analysis of the proposed method.\n3. The paper is well-organized.", "weaknesses": "1. The proposed method is too simple, authors add the prompt swapped prediction loss to the baseline UPL, which lack novelty in NeurIPS conference.\n2. \"A straightforward way is to .... tunning instance-specific prompts. \" in Introduction is confusing. Zero-shot inference by CLIP does not need fine-tuning and TPT is not designed to provide a correct confindence. Authors' comments of the previous work are strange.\n3. Eq.(3) is a line of code rather than a mathematical formula.\n4. Authors should provide sensitivity analysis about hyperparameters α and β.\n5. Both Eq.(8) and Eq.(9) use l to represent loss but they have different meaning, try to use different subscripts.\n6. Ablation study should provide the analysis about only Lswap in Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and suggestions. We have tried to address all your concerns as follows:\n\n**W1:** \"The proposed method is ... in NeurIPS conference.\"\n\n**A1:** Sorry for the insufficient explanation on the novelty of our proposed method. We would like to re-emphasize the novelty and contribution below: \n\nFirstly, we are the first to introduce the concept of unsupervised representation learning into test-time prompt adaptation in pre-trained vision-language models, which is achieved by designing a prompt swapped prediction loss.\n\nAdditionally, the prompt swapped prediction loss proposes two mechanisms that are employed for the first time in this problem: EMA prompt and prompt swapped prediction strategies. Their combined application to address this specific issue is innovative.\n\nFurthermore, we extensively conduct experiments on ImageNet and its four variants, along with nine other image classification datasets. The empirical evaluation demonstrates that our approach significantly outperforms current TPT methods and even competes with supervised prompt adaptation methods on the majority of datasets.\n\nThe simplicity of the SwapPrompt approach also facilitates its implementation and usage during test-time with vision-language models.\n\n**W2:** \"A straightforward way is ... are strange.\"\n\n**A2:** We apologize for the lack of clarity in our expression. The original meaning of this sentence is to clarify that techniques such as CoOp and CoCoOp are not suitable for deployment during test time. Regarding TPT, it is true that its primary design does not cater to the provision of precise confidence estimates. Nevertheless, this limitation may hold significance in specific scenarios. The prediction confidence of TPT and SwapPrompt can be found in Table 2 of **Supplementary Material**.\n\nTo improve the readability of the whole paper, we have carefully revised the presentation in the new version.\n\n**W3:** Eq.(3) is a line of code rather than a mathematical formula. \n\n**A3:** Thanks for the review but there seems to be a misunderstanding about the EMA strategy. We want to point out that Eq. (3) is exactly a mathematical formula rather than a line of code. The same equation can be found in many related literature, e.g., [1][2][3]. In the upcoming version, we will introduce subscripts to differentiate between the target prompts before and after the update, aiming to enhance the clarity of the expression of this formula.\n\n[1] Grill J B, Strub F, Altché F, et al. Bootstrap your own latent-a new approach to self-supervised learning[J]. NeurIPS, 2020, 33: 21271-21284.\n\n[2] Cai Z, Ravichandran A, Maji S, et al. Exponential moving average normalization for self-supervised and semi-supervised learning[C]//CVPR. 2021.\n\n[3] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C]//CVPR. 2020: 9729-9738.\n\n**W4:** The sensitivity analysis about hyperparameters α and β.\n\n**A4:** Thanks for your valuable comments. The following table presents the results.\nIt can be seen that SwapPrompt is not sensitive to the choice of hyperparameters α and β in most case. It still outperforms baselines.\n\n|    | Caltech101  | DTD | Flowers102 | Oxford-Pets | UCF101 |\n|  :-  | :-:  | :-:  | :-:  | :-:  | :-:  | \n| α=0.6, β=1.4  | 88.56 | 46.80 | 70.04 | 88.24 | 65.21 |\n| α=1.0, β=1.0  | 89.90 | 47.34 | 70.22 | 89.14 | 65.66 |\n| α=1.4, β=0.6 | 89.45 | 47.10 | 70.32 | 87.97 | 65.77 |\n\n**W5:** \"Both Eq.(8) and ... different subscripts.\"\n\n**A5:** Thanks for your valuable comments. We will change the $l$ in Eq.(9) to $l_{ce}$.\n\n**W6:** Ablation study should provide the analysis about only Lswap in Table 2.\n\n**A6:**  Thanks for your valuable comments. We have added ablation study to compare the performance of only using $L_{swap}$ with other versions. The experimental results can be seen in the following table.\n\n|  | Caltech101  | DTD | Flowers102 | Oxford-Pets | UCF101 |\n|  :-  | :-:  | :-:  | :-:  | :-:  | :-:  | \n| UPL  | 86.37 | 45.04 | 67.11 | 88.53 | 63.63 |\n| only $L_{swap}$  | 86.25 | 43.38 | 65.01 | 84.33 | 61.69 |\n| UPL+AUG | 87.75 | 46.04 | 68.43 | 87.67 | 65.15 |\n| SwapPrompt | 89.90 | 47.34 | 70.22 | 89.14 | 65.66 |\n\nAlthough the performance of only using Lswap is worse than UPL, it is in fact acceptable. Considering the inherent generalization ability of CLIP, it is unfair to directly compare the performance of only using $L_{swap}$ with $L_{pseudo}$.\n\nThe standalone application of $L_{swap}$ does not sufficiently leverage the rich pre-trained knowledge embedded within CLIP. This can be seen as disregarding the pseudo-labels, which encapsulate the most pre-trained knowledge. Our proposed method, SwapPrompt, combines $L_{swap}$ and $L_{pseudo}$, resulting in improved performance compared to using $L_{pseudo}$ alone.\n\n**Q1:** The results of FGVCAircraft dataset.\n\n**A1:**  The following table presents the results of FGVCAircraft dataset. It can be seen that SwapPrompt still outperforms baselines.\n|    | CLIP | UPL | TPT | SwapPrompt| \n|  :-  | :-:  | :-:  | :-:  | :-:  | \n| FGVCAircraft  | 17.07 | 17.45 | 17.69 | 18.03 | \n\n**Q2:**  Augmentations used in SwapPrompt.\n\n**A2:** We use the same augmentation method as SimCLR in swapped prediction loss and the regular image pre-processing methods in calculation of the pseduo label, which is the same as CoOp. We added experiments using the same augmentation method as FixMatch[4] in swapped prediction loss, and the results demonstrate that the choices of  augmentation will slightly affect the swapped prediction loss. \n\n|  Augmentation | Caltech101  | DTD | Flowers102 | Oxford-Pets | UCF101 |\n|  :-  | :-:  | :-:  | :-:  | :-:  | :-:  | \n| SimCLR  | 89.90 | 47.34 | 70.22 | 89.14 | 65.66 |\n| FixMatch | 89.61 | 46.51 | 70.24 | 88.89 | 66.26 |\n\n[4] Sohn K, Berthelot D, Carlini N, et al. Fixmatch: Simplifying semi-supervised learning with consistency and confidence[J]. Advances in neural information processing systems, 2020, 33: 596-608."}]

Author rebuttal:

Meta-review: This work argues that existing test-time adaptation (TTA) methods cannot fully exploit the representation capabilities of pre-trained models. To this end, the authors proposed a new method that leverages the self-supervised contrastive learning to facilitate the test-time prompt adaptation.

Reviewers recognize the importance of this problem, novelty of the proposed method and the effectiveness of the methods.

In the rebuttal the author has addressed the concerns regarding the requirements of more detailed comparisons and ablations, along with other technical details. Based on the rebuttal, two reviewers raised their scores. The final scores are unanimous accept, albeit with different degrees (5, 5, 5, 6, 8).

Overall, we feel that that the major concerns of the reviewers were addressed by the author. The proposed method is interesting and effective. We therefore recommend accept. In the final version, please add the additional experiments and discussions.
Decision: Accept (poster)