Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
Authors: Manli Shu
Weili Nie
De-An Huang
Zhiding Yu
Tom Goldstein
Anima Anandkumar
Chaowei Xiao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Foundation Model
Prompt Tuning
Generalization
Out-of-Distribution Robustness

Abstract: Pre-trained vision-language models (e.g., CLIP) have shown promising zero-shot generalization in many downstream tasks with properly designed text prompts. Instead of relying on hand-engineered prompts, recent works learn prompts using the training data from downstream tasks. While effective, training on domain-specific data reduces a model's generalization capability to unseen new domains. In this work, we propose test-time prompt tuning (TPT), a method that can learn adaptive prompts on the fly with a single test sample. TPT optimizes the prompt by minimizing the entropy with confidence selection so that the model has consistent predictions across different augmented views of each test sample. In evaluating generalization to natural distribution shifts, TPT improves the zero-shot top-1 accuracy of CLIP by 3.6\% on average, surpassing previous prompt tuning approaches that require additional task-specific training data. In evaluating cross-dataset generalization with unseen categories, TPTperforms on par with the state-of-the-art approaches that use additional training data.

Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
Manli Shu Weili Nie² De-An Huang?
Zhiding Yu 2
Tom Goldstein Anima Anandkumar 2.3.f
Chaowei Xiao
University of Maryland, NVIDIA, 3 Caltech,
Arizona State University
Abstract
Pre-trained vision-language models (e.g.. CLIP) have shown promising zero-shot generalization in many downstream tasks with properly designed text prompts. Instead relying on hand-engineered prompts, recent works learn prompts using the training data from downstream tasks. While effective, training on domain- specific data reduces model's generalization capability to unseen new domains. In this work, propose test-time prompt tuning (TPT), method that can learn adaptive prompts on the fly with a single test sample. For image classification, TPT optimizes the prompt by minimizing the entropy with confidence selection so that the model has consistent predictions across different augmented views of each test sample. In evaluating generalization natural distribution shifts, TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average, surpassing previous prompt tuning approaches that require additional task-specific training data. In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data. Project page: https://azshue..gnn http:///ih...iiiuuiibo///...
Introduction
Recent advances in vision-language pre-training, such CLIP [1] and ALIGN [2]. present promising direction for developing foundation models for vision tasks [3, 4]. These foundation models encode wide range of visual concepts after training millions of noisy image-text pairs and can be applied to downstream tasks in zero-shot manner without task-specific training data [5-11] 1]. This is made possible by appropriately designed instruction prompts. Take image classification in Figure as an example: We can prepend category name with a prompt "a photo of a" (e.g., "a photo of a dog"). Images can then be classified by using CLIP measure their alignment with the various class descriptions. Designing such prompts thus plays a crucial role in applying foundation models to downstream tasks in zero-shot manner. However, such hand-crafted prompts require domain-specifer heuristics and may not be optimal.
Recent works address this by proposing prompt tuning to directly learn prompts using training data from downstream tasks [12]. We can fine-tune prompts with training data in the same way we fine- tune model parameters since prompt embeddings are part of the model input and are differentiable with respect to the loss function. Such an approach can find better prompts compared to hand-crafted ones, but the learned prompts are limited to the distribution and tasks corresponding to training data and may have limited generalization beyond that. In addition, this approach requires training data with annotations, which can be expensive and is not available for zero-shot tasks.
Work done during an internship at NVIDIA, manlis@cs.umd.ccom cs. udd. Equal advising
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Dog
Hind
promptp p
class
A
P,0%A,000
A
average
H(p
encoder
Av
accepp
X
reject
eernable Parameters
Back Propagation
single image
augmented views
confidence selection
Figure I: Test-time Prompt Tuning (TPT) for image classification. We tune adaptive prompts the fly with a single test sample, without the need for additional training data or annotations. TPT optimizes the prompt encourage consistent predictions across augmented views by minimizing the marginal entropy. We introduce confidence selection to filter out noisy augmentations.
Our Approach. To address the aforementioned challenges, propose test-time prompt tuning (TPT) that tunes the prompt on the fly using only the given test sample. The tuned prompt is adapted to each task, making it suitable for zero-shot generalization without requiring any task-specific training data or annotations. TPT retains the zero-shot generalization setting since no additional training data or annotations used.
We explore TPT on two different downstream tasks: image classification and context-dependent visual reasoning. For each downstream task, we design customized test-time tuning strategy that fits the nature of the task. Without loss of generality, we consider CLIP [1] as our vision- Dision-language foundation model, for its simplicity in design and its wide applicability [13].
For image classification, test sample is an input image. Given a single sample at test time, we perform prompt tuning by generating multiple randomly augmented views, and optimizing the text prompt so that the model has consistent predictions across different augmented views. This done by minimizing the marginal entropy among the outputs of the augmented views. In addition, since some augmentations may lead to misleading model predictions, we propose confidence selection to filter out those "noisy" augmented views. We discard augmented views with high prediction entropy (i.e., low confidence), and only include high confidence views the consistency optimization.
We evaluate the zero-shot generalization of TPT in two image classification settings: natural distribu- tion shifts [14] and cross-dataset generalization [15]. For the setting evaluating natural distribution shifts, TPT boosts the T accuracy of CLIP in the zero-shot setting by 3.6% on average compared using a hand-crafted prompt, achieving on-par accuracy with previous prompt tuning methods that require additional training data (i.e., ImageNet). TPT achieves maximum improvement of 6.9% on ImageNet-A compared to using a cand-cratted prompt, surpassing the existing few-shot prompt tuning method by 5.1 %. For the setting of evaluating cross-dataset generalization with possibly unseen categories, TPT achieves on-par performance with state-of-the-art few-shot prompt tuning method [15] without the need for additional training data or annotations.
For the second task of context- -deeedt-d visual reasoning, such as Bongard-Hon HOI [16], sample contains two sets of support images and a query image for evaluation. The two sets of support images exemplify the presence and the absence bike"). The model is then asked to infer whether the query image contains the underlying concept. Given such test sample, we apply TPT by tuning prompts to better differentiate between the two support sets, so that the query image can be better classified (Figure 4). Despite the use of support sets, our approach still considered zero-shot training tasks from other concepts or the annotation of the query image at test time update the prompt of the test task. By adapting TPT to context-dependent visual reasoning, we outperform the Mlateoofthe--a method [17] by 4.1% Bongard-HOO benchamrk [16].
human-object interaction (HOI) concept (e.g. "ride
visual reasoning, because we do not use either

---

We summarize our main contributions as follows:
We propose test-time prompt tuning (TPT) that does not need any training data or annotations to optimize the prompt. To the best of our knowledge, are the first to perform prompt tuning single test sample in a zero-shot manner.
We introduce confidence selection as simple plug-and-play module of for image classification. It improves entropy minimization among augmented views by filtering out "noisy" augmentations that lead to low-confidence predictions.
We conduct extensive experiments on image classification under natural distribution shift, cross- dataset generalization, and context- context-dependent visual reasoning. TPT improves CLIP in a zero-shot manner to be on par with prompt tuning methods that require additional training data.
2 Related Work
Prompting for foundation models. Foundation models are those trained on large-scale hetero- geneous data, of which the knowledge can be transferred various downstream tasks in natural language processing [18, 19J. computer vision [1. 2, 20]. etc. Recent work has proposed different ways to efficiently and effectively transfer such knowledge to downstream task [21-24]. Prompting is a heuristic way to directly apply foundation models to downstream tasks in a zero-shot manner. Prompt works as part of the text imput that instructs the model perform accordingly on a specific task. However, such zero-shot generalization is highly dependent on a well-designed prompt. Prompt tuning [12, 25-27] proposes to learn the prompt from downstream data in the continual input embed- ding space, which presents parameter-efficient way of fine-tuning foundation models. Although initially developed for language models, prompt tuning has later been applied to other domains, including vision-language models [28, 15, 29] and continual learning [30]. CoOp[28] applies prompt tuning to CLIP. By tuning the prompt on a collection of training data, CoOp effectively improves CLIP's performance on the corresponding downstream tasks. CoCoOp [15] points out that CoOp lacks in generalization to the prompt conditioned on model inputs. Despite being effective the given task, this line of work requires access to downstream training data with annotations, restricting the zero-shot knowledge transfer of foundation models. Another line of work proposes to tune the prompt in an unsupervised manner [31, 32]. However, it requires access to multiple samples from either the training or testing split. In this work, we propose test-time prompt tuning that works on single test sample. Our method can directly work with the zero-shot applications of foundation models.
data, and proposes to alleviate the problem by making
Generalization under data distribution shifts. A reliable machine learning model is supposed to perform well under data distribution shifts for real-world applications. For model trained on a given set of data, distribution shift refers to the discrepancy between the underlying distributions of the test and the training data. Distribution shifts occur naturally in the real world due to variations in the environment [33] or the encounter of unseen concepts [34]. For example, in the meta-learning literature [35], each test sample consists of novel task (i.e., distribution), and the models should be able to quickly adapt to the novel distributions. Even in the standard evaluation protocol for machine learning models, there exists a subtle difference in the data distribution between the training and testing splits [36, 37J, which is also one type of distribution shift. Pre-trained vision-language models like CLIP can generalize downstream tasks with various distribution shifts zero-shot manner. Such zero-shot generalization ability presents a promising direction for realizing reliable and generic machine learning models. Our method aims to improve CLIP towards a better generic model in this work, instead of adapting it to specific downstream tasks or target datasets. We leverage the assumption that robust model should have decision boundaries lying low-density data regions [38]. ciiisteecy-rguulriiattii based methods [39, 20] achieve this goal by making the network outputs invariant to small input noises. For classification tasks, we use consistency regularization as our test-time prompt tuning objective with the confidence selection module.
Test-time optimization. The idea of adapting machine learning models to test samples on the fly has been applied to different tasks [40-43]. This work mainly focuses on applying the technique to improve model generalization. One challenge in this area is to design a practical test-time objective. Test-time training and its variants [44, 45] modify the training objective and the network architecture by adding self-supervised multi-task branch, which will then be used at test time for computing
3

---

optimization objectives and adapts the network to the test sample. Entropy minimization is another common technique for developing self-supervised objectives [46, 47]. TENT [48] performs test-time optimization by minimizing the entropy of the batch-wise prediction probability distributions, but it needs more than one test sample to get a non-trivial solution. Zhang et al. [49] propose marginal entropy minimization that works on a single test sample with data augmentations. Another major challenge is to choose the right parameter group for optimization. Batch normalization (BN) layers have been shown to capture the domain discrepancies in image data [50, 51J. It is a straightforward way to directly adapt the BN statistics at test time to enhance model robustness against distribution shifts [52]. However, adapting BN layers puts restrictions model architectures. Another choice is to update the feature extractor while freezing the prediction module [44, 53]. Zhang et al. [49] show work as well. Our method addresses both of the
that optimizing the entire model at test time challenges above. For the choice of parameter group, we optimize the text prompt while keeping the model intact. Our motivation is to avoid distorting the pre-trained features and preserve the zero-shot generalization ability of pre-trained models. In section 5, we empirically show that the prompt works as the most effective parameter group for CLIP. Different from the previous single-point method [49], we refine the entropy minimization by proposing confidence selection, which helps filter out noisy augmentations that may lead to misleading predictions.
3 TPT: Test-Time Prompt Tuning
In this section, we first discuss how to apply CLIP to downstream tasks in a zero-shot manner with hand-crafted prompt. Next, we briefly review recent progress in prompt tuning approaches for CLIP using downstream training data. Finally, we give detailed introductions of how to apply our method to the image classification task and context-dependent visual reasoning, respectively.
3.1 Background
Contrastive Language-Iman Pre-training (CLIP). CLIP consists of two parallel encoders, one that maps the text input into feature vector, and the other does the same for the image input. The model is trained with a contrastive loss that promotes similarity between the two vectors so that the text and image align in the feature space. We denote CLIP model as F = (Evieusl, Erext }. with Evisual and Evext being the image and text encoders.
We first review how apply CLIP to downstream tasks in a zero-shot manner with hand-crafted prompt. We take image classification as an example. Consider a single test image Xtest of class y, where X E RCXH*W and y RKK for K-class classification problem. In the baseline zero- shot setting, we prepend a hand-crafted prompt prefix, such as ="a photo of a", to every y: in y == (11, ..,,... У to form the calegory-specific text inputs {p: yi We then feed these class descriptions to the text encoder to get the text features {tr, .2,,,,,,, Each text feature 6; paired with the image feature Eviiuul (X) ) to compute a similarity score 8 sim(t, v), where sim(.) denotes the cosine similarity. The prediction probability on X can exp(sim(tt,e)) be denoted by p(y: X)- E exp(ein(t:-o)r))
where & = Eraxe ((P; 3: F).
where is the temperature of the softmax function.
Prompt tuning using downstream training data. Instead of using a hand-crafted prompt, prompt tuning methods train a prompt maximize performance on a downstream task for which labeled data is available. Prompt tuning optimizes the prompt p E RTx D in the text embedding space, with the number of tokens L and embedding size D, using training data with annotations D Purin = {X,, yi) from the downstream task. The goal is to obtain text inputs {p: }} ((p.yd } for ys }}} that can provide the model with the most helpful context information about the task. For image classification with cross-entrops loss L. the problem can be formulated as:
p' = arg min ((X.x) ((Fp(X,,
where F, (X) sim Etext ((p) Evieuar (X)).
(2)
Context-dependent visual reasoning. For the task context-dependent visual reasoning, such as Bongard-HOO [16], a test sample contains two sets of support images and query image for evaluation. The two sets of support images exemplify the presence and the absence of human-object interaction (HOI) concept (e.g., "ride bike"). The model is then asked to infer whether the query image contains

---

the underlying concept. Specifically, each concept in this task is a visual relationship c = (s, a. o). with s being the subject (s ="human" for HOI tasks), a denoting the action and for the object. Fach test sample Xieu captures a concept by presenting c = (s, a, o) in one set of support images (positive examples), while having the other set (negative examples) to demonstrate d' = (s, a' o). where a' o a. Note that neither nor a is given explicitly, and it relies on the model's s reasoning ability to predict whether the query image contains the featured concept of the test sample.
Existing methods [54, 55] approach the Bongard-HOO problem by training the model on a collection of similar tasks (using the Bongard-Ho HOI training split) so that can make similar inferences on test samples at test time. When applying CLIP to this task, we do not use additional training data because CLIP has learned abundant visual concepts and thus a natural fit for such visual reasoning tasks.
3.2 TPT: Test-Time Prompt Tuning
Why optimize prompts? CLIP contains rich knowledge obtained from pre-training on a massive and diverse dataset. However, how to more effectively extract such knowledge remains open question. A simple strategy is to directly fine-tune the model, either end-to-end or for a subset of layers, category of inputs. However, previous work has shown that such fine-tuming strategies result in bomain-speciffe behaviors that lose the out-of-distributie of foundation models [13, 56]. Prompts, the other hand, work outside the pre pre-trained model by modifying the context of the model input, thus do not distort pre-trained features.
generalization and robustness
In this work, our goal is to leverage the existing knowledge of CLIP to boost its generalization in a zero-shot manner. Therefore, prompt tuning serves as an ideal handle to approach the goal. Furthermore, we regard test-time prompt tuning as a way to provide the model with the context tailored to the single test sample, which helps precisely retrieve the knowledge of CLIP.
At the inference stage, the only information available is the single test sample Xust without label information. TPT, therefore, manages optimize the prompt p test time based on the single test sample. In general, our objective can be formulated in the form of
= arg min L(F, ,, Xiett)
(3)
for some carefully constructed loss. Note that, unlike equation (I), our method does not require any labels or any data beyond the zero-shot test sample.
TPT for image classification. Because labels are not available for test time tuning, we must select an unsupervised loss for prompt tuning. We design our TPT objective promote the consistency of the model's predictions across different augmented views of given test image. Specifically, we
generate N randomly augmented views of the test image using family of random augmentations A, and minimize the entropy of the averaged prediction probability distribution:
p' arg min Σ pp (ress) log (vil Xres);
(4)
where Pp Xwaa)
1 N
Σ Pp(2))AA
(5)
Here, (p is the vector of class probabilitics produced by the model when provided with prompt p and the i-th augmented view of the test image.
In addition, to reduce the noise from random augmentations, we propose confidence selection filter out views that generate high-entropy (i.e., low-confidence) predictions. Such views of an image may lack important information needed to classify it correctly, e.g., a random crop may have removed important image content. We select confident samples with prediction entropy below a threshold r. We adapt for each test sample, by taking the entropy value at the p-percentile among the self-entropy of N augmented views ranked from low to high (i.e., confidence from high to low). With т, the confidence selection can be written as a mask over the augmented samples 1l (p.)) S r). with H measuring the self-entropy of the prediction
augmented view. Using confidence selection with
5

---

cutoff percentile on N augmented views, the averaged probability in Eq. (4) now becomes:
1 1H(pp) S ( (,(ros))
Po(y)Xum)
(6)
PN
TPT for context-dependent visual reasoning. Different from image classification, where every image has one and only ground-trume truth label, the correctness of the prediction Bongard-Ho Hor depends on the context (i.e., example images), which is binary (containing the concept or not), In the case of binary labels, a straightforward prompting strategy is hand-craft "labels" for positive and negative examples, such as "True/False" or "Yes/No". With TPT. on the other hand, we can directly learn an optimal label token cl.s on the example images in the test sample. More importantly, for visual reasoning. TPT can explicitly learn the context (i.e., visual concept) in the form of text prompts, and assists visual reasoning of vision-langee language models with language context. Formally, given M support images in each test sample, the TPT objective for context-dependent reasoning can be written as:
p = arg min p.dls M
Σ XE(Xp.Xx)
[(Fc.de(X).))
where we assign y E {0,
negative and positive example images respectively for computing the cross-entropy loss L. Unlike in image classification, we tune the binary label tokens cls (cls', ), cl.s E RI.D and prompt p E RL.D simultaneously. For each image, we assemble the text input to CLIP as T (Ti, T2 T; {p, cls')).
Note that the support is an essential part of Bongard-HOl sample, which provides the context for this context-dependen task. Therefore, our approach is still considered to work purely at test time, without training data or annotations (i.e., TPT has been trained on a collection of similar tasks from the Bongard-HOO training split).
4 Experiments
In this section, we describe the tasks and benchmarks used for evaluating our method, along with the implementation details. Our main results cover three aspects of the model's generalization: robustness to natural distribution shifts, cross-dataset generalization, and context-dependent visual reasoning. We also provide ablation experiments in section 5, analyzing different network components for test-time tniigg, and other design choices of our method.
4.1 Robustness to Natural Distribution Shifts
Datasets. CLIP has been shown be robust to distribution shifts that can occur naturally in real- world scenarios. We follow the setting in Radford et al. [1] and evaluate the model's ss robustness to natural distribution shifts on 4 ImageNet Variants follows, which have been considered as out-of-distributtio (OOD) data for ImageNet [57] previous work.
ImageNet-V2 [58] is independent test containing natural images, collected from different source, including 10,000 images of 1,000 ImageNet categories.
ImageNet-A [59] is challenging test set of "natural adversarial examples" misclassified by standard ResNet-50 [60], consisting of 7,500 images of 200 of ImageNet categories.
ImageNet-R [14] collects images of ImageNet categories but with artistic renditions. There are 30,000 images in total, including 200 ImageNet categories.
ImageNet-Sketchh [61] is a dataset of black and white sketches, collected independently from the original ImageNet validation set. The dataset includes 50,000 images in total, covering 1,000 ImageNet categories.
Baselines. We compare TPT with existing few-shot prompt tuning methods that are designed for CLIP. CoOp [28] a few-shot prompt tuning baseline that tunes fixed dataset-specific prompt on each downstream dataset. CoCoOp [15] is the state-of-the-an prompt tuning method for CLIP. produces input-dependent prompts with a network module, of which the input is the image feature. The network module of CoCoOp is also trained on downstream data in datasct- way.

---

Following their original configuration, we train both methods on ImageNet using 16-shot training data per category with 4 learnable prompt tokens and directly test the tuned prompt on OOD benchmarks. We also include two versions of the baseline zero-shot performance of CLIP, using a default prompt "a photo of a", and the ensemble of 80 hand-crafted prompts from Radford al. [1].
Implementation details. For TPT, we initialize the prompt the default hand-crafted one "a photo of a", and optimize the corresponding 4 tokens in the text input embedding space based on a single test image. We augment single test image 63 times using random resized crops and construct batch of 64 images, including the original one. Among the 64 predictions, we select the top 10% (p=0.1) confident samples (lowest 10% in self-entropy) and compute the entropy of the averaged probability of the selected predictions (i.c., marginal entropy). We optimize the prompt to minimize the marginal entropy for step, using the AdamW optimizer with learning rate of 0.005.
Results. In Table 1, the standalone TPT achieves higher accuracy than both prompt ensemble and existing few-shot prompt tuning methods, including CoCoOp. Furthermore, since TPT works at test time, it is complementary to existing baseline methods. We show that by applying TPT to prompts learned by CoOp CoCoOp, can further improve the accuracy of their in-domain ImageNet data, as well as generalization ability to OOD data. We also compare TPT to the ensembles of baseline models in Appendix A.3, where we find that applying TPT baseline methods can bring more substantial improvement than model ensembles. In addition, among the five datasets, few-shot prompt tuning methods bring the most accuracy gain on the ImageNet validation set and ImageNet- V2. However, on datasets with more significant distribution shifts, few-shot prompt tuning methods trained on ImageNet show no advantage over the ensemble of hand-crafted prompts.
Table I: Robustness to Natural Distribution Shifts, CoOp and CoCoOp are tuned on ImageNet using 16-shot training data per category. Baseline CLIP, prompt ensemble, and TPT do not require training data.
ImageNet Topl acc.
ImageNet-A Topi acc.
ImageNet-VV V2. Top1 acc.
ImageNet-R. Top1 acc.
ImageNat-SKetch Topl acc. 1
Average
0OD Average
CLIP-RN50
58. 16
21.83
51.41
56.15
33.37
44.18
40.69
Ensemble
59.81
23.24
52.91
60.72
35.48
46.43
43.09
CoOp
63.33
23.06
55.40
56.60
34.67
46.61
42.43
CoCoOp
62.81
23.32
55.72
57.74
34.48
46.81
42.82
IPT
60.74
26.67
54.70
59.11
35.09
47.26
43.89
TPT + CoOp
64.73
30,32
57.83
$8,99
35.86
49,55
45.75
TPT + CnCoOp
62.93
27.40
56.60
59.88
35.43
48.45
44,83
CLIP-VIT-B/16
66.73
46.09
57.20
Ensemble
68.34
CoOp
71.51
49.71
64,20
75.21
47.99
61.72
59,28
CoCoOp
71.02
50.63
64.07
76.18
48.75
62.13
59.91
TPT
68.98
54.77
63.45
77.06
47.94
62.44
60.81
TPT CoOp
73.61
$7.95
66,83
77.27
49.29
64,99
62,83
IPT CoCoOp
71.07
58.47
64.85
78.65
48.47
64.30
62.61
4.2 Cross-Datasets Generalization
Pre-trained vision-language models like CLIP are ideal for "open-world" problems. For example, we can apply CLIP classify arbitrary categories in a zero-shot manner in image classification.. However, prompt tuned on a specific downstream dataset can be less generalizable to categories outside its training set. In this section, evaluate the cross-dataset generalization of existing few-shot prompt tuning methods (same as in section 4.1), and compare them with TPT, which is not dataset-specific.
Setup. We conduct cross coos--ataset evaluation on the task of image classification. We consider 10 datasets, covering fine-gre -raiind classifications including species of plants or animals (Flower102 [62], OxfordPets [63]). scenes (SUN397 [64]), textures (DTD [65]], food (Food101 [66]). transportation (StanfordCars [67], Aireraft [68]), human actions (UCF101 [69], satellite images (EuroSAT [70]), and general objects (Caltech10) 101 [71]). We consider two different settings of cross-dataset general- ization. the first setting, we consider ImageNet with 1000 categories as a comprehensive source
datasct, and use other fine-grained datasets as target datasets for evaluation. We implement CoOp
7

---

Flower102
0.31
0.31
Flower
0.56
10.11
4011 10.2 0024
0.4
DTD
0.29
0.21
DTD
4.04 0.29
Pcts
465-4m
0.23
0.53
0.65
0.52
1024-0.. 0.38
Pots 0.11
-.16
oup 4.02 0.11 wot 4.07 10.0 0.26
0.00
Cars
4.77
025
035
0.60
4040
0.54
15-40455
Cars 0.66 -3
0.54 0.10 4.54 4025 4 0.45 0.35
0.35
0.2
VCF100
435
0.57
U/C110
-225
0.10 4.21
0.18
10
233
Caltech101
Food101
434
0.49
0.00
0.29 RM6
9.41
0005 0,30
Food101
0.54 9.32 9:25
0.15 0.15 4.24 1un 0.5, 0.63
0.14
SUN397
235
022
0.05 0.20 0.05 0.17 0.1 M08 0.16
SUN397
n02 0.10 0.02 4:03 0.t 4010 0:14 025
0,10
0.2
Aircraft
Aireraft
260 0.34
055
eurosat 4.76 0.73 4:53
0.77
-0.4
Nome (TPT)
n02 oot
out
0.5 0.0 oot not
6.05
6a2 0.20 0.33
Nove (TPT)
0.0z not our 0.55 0.8. wot oot e.s
6.12 0.00 0:33
Flowor102
DTD
Pots
Care
UCRIOI
Catechion Food
SUN397
Aircrunt
eurosat
Avrage
Flower 102
DTD
Pats
Cars
UCF I01
Food Calech101
SUN399
Aitoraft
curosat
Average
(a) CoOp with CLIP-RNSO.
(b) CoCoOp with CLIP-RNSO.
Figure 2: Cross- improvement normalized by the zero-shot baseline performance. In each matrix A, A is the normalized relative improvement on the ju dataset using the prompt tuned on the i-th dataset. The value Ai, stands for how well a method trained on a target dataset j. in comparison with a zero-shot CLIP baseline (using a hand-crafted prompt). Thus, the higher, the better. The last row is the performance of TPT, which is not tuned any source dataset. The last column summarizes the average improvement over 10 datasets, measuring the overall generalization ability across the 10 datasets.
source dataset performs
and CoCoOp using the same setting as in section 4.1, and evaluate their generalization performance the 10 datasets. In the second setting, consider more challenging scenario, where the source data for few-shot prompt tuning also comes from the specialized fineggriied datasets, and there no overlapping in categories between source-target pair.
Implementation details. We implement CoOp and CoCoOp on each source dataset following their original configurations. For TPT, initialize the prompt as "a photo of a" for every datasets. We adopt the same hyper-parameter setting as in section 4.1. We use AugMix [72] as a stronger data augmentation for this task.
Table 2: Cross-dataset generalization from ImageNet fine-grained classification datasets. CoOp and CoCoOp are tuned on ImageNet using 16-shot training data per category. Baseline CLIP, prompt ensemble, and TPT do not require training data or annotations. We report the top-1 classification accuracy on each dataset.
Flower102
DTD
Pcts
Cars
UCF101
Caltsch101
Food101
SUN397
Aircraft
EuroSAT
Average
CLIP-RNS0
61.75
83.57
15.66
23.69
Ensemble
62.77
40.37
82.97
55.89
59.48
87.26
74,82
60.85
16.1 I
25.79
56.63
CaOp
75.59
CaCoOp
65.57
38.53
88.39
56.22
57.10
87.38
76.20
59.61
14.61
28.73
57.23
TPT
62.69
40.84
84.49
58.46
87.02
74.88
61.46
17.58
28.33
57.66
CLIP-VIT-/IIA
67.44
44.27
$8.25
65.48
83.65
23.67
42.01
Ensemble
66.99
45.04
86.92
66.11
65.16
93.55
82,86
65.63
23.22
50.42
64.59
CoOp
68.71
41.92
89.
64.51
66.55
93.70
85.30
64.15
18.47
46.39
63.88
CoCoOp
70.85
45.45
90.46
64.90
68.44
93.79
83.97
66.89
22.29
39.23
64.63
TPT
68.98
87.79
84.67
24.78
Results. In Table compare TPT with few-shot prompt tuning methods on generalization from ImageNet to fine-grat fine-grained datasets. Note that TPT works in a zero-shot manner, thus it is not trained on ImageNet. Nonetheless, we find TPT to achieve on-par generalization as ImageNet trained CoCoOp. In Figure 2, we present the results of the more challenging setting of cross cross-dataset generalization, where there is no overlap between the source and target dataset. For better visualization, we plot the relative accuracy improvement acc' (acc- acCtase) /acchases normalized by the zero-shot baseline accuracy acCbose of a CLIP-RN50. For example, baseline CL.P with hand-crafted prompt achieves 61.75% accuracy Flower102, while CoOp trained DTD only has 33.41% % on Flower102. In

---

this case, we calculate acc' as (33.41 - 61.75) /61.5 = -0.46. From Figure 2, we can see that the averaged accuracy improvement (in the last column of each matrix) of few-shot prompt tuning methods is always negative, meaning that they do worse than the zero-shot baseline. TPT, on the other hand, shows consistent improvement in each of the 10 datasets.
4.3 Context-dependent Visual Reasoning on Bongard-HOI
Baselines. We include three previous methods for comparison: (1) The CNN-baseline [54] is
simple classifier trained on Bongard-HOO training data, for which the model trained to map a training sample as whole (including the support and query images) to binary output, indicating whether the query image contains the corresponding concept; (2) The Meta-baseline [55] regards each sample in Nongard-HON a few-shot task and the model is trained the Bongard- Mongard-HOI training data with a meta-objective that aims to quickly adapt the model to new tasks; (3) HOITrans [17] is the previous best method on Bongard-HOr It is transformer-based HOI detection model that
achieves state-of-ther off-the-art accuracy on various HOI detection benchmarks. It solves Bongard- HOI by comparing the detected HOIs of the query images those of the support images.
Table Evaluation on the Bongard-HOO benehmark. CNN and Meta baselines are implemented based on a ResNet-50 (RN50). (+ denotes that the method uses ground truth bounding boxes to assist the inference.)
Test Splits
Method
act,, unseen act.,
seen act,, unseen obj..
unseen act., unseen obj..
Average
obj.,
obj..
50.03
49.89
49.77
50.01
49.92
Meta-baselimee
58.82
58.75
58.56
57.04
58.30
HOlTrans
59.50
64.38
63.10
62.87
62.46
TPT (w/ (NP-)
66.39
68.50
65.98
65.48
66.59
Implementation details. For each Bongard- test sample, TPT tunes the prompt prefix and class tokens simultaneously from scratch. All learnable tokens are initialized in the text embedding space from Gaussian distribution with 0.02. We optimize the prompt on all support images of a test sample for 64 steps, using the Adam W optimizer with learning rate of 0.005, and then infer the query image with the updated prompt. For other baselines, we directly report the results from Jiang et al. [16], and we refer interested readers to this paper for more details. Note that the HOlTrans model is trained on all possible HOI concepts, including the ones in the testing splits.
Results. In Table 3, we follow the setup Jiang et al. [16], and compare TPT with previous methods on 4 test splits of Bongard-HOI respectively. In Bongard-HOI, test images are split into four sets by their overlap the HOI concept with the training data: whether the action a or the object o has appeared in the training data. Note that our CLIP-based TPT is not trained on the training split of Bongard-HOO and thus the definition of the four splits is not strictly applicable to TPT.
5 Ablation Study
In this section, we analyze our design choices and provide ablation study on the effects of different components of TPT. For simplicity, if not otherwise specified, analyses in this section are conducted on the natural distribution shifts benchmarks. We first compare test-time optimization on different parameter groups of CLIP, showing that prompt tuning achieves the most accuracy gain. Next, we show the improvement brought by confidence selection and analyze how the confidence threshold affects performance. Additional ablation studies can be found in Appendix A.3. We also provide qualitative study on the effect of TPT on model prediction probability distributions in Appendix A.8.
Test-time optimization on different parameter groups CLIP. Existing test-time optimization methods have worked on different parameter groups of a model. Although there is a strong intuition for tuning prompts on CLIP, it is unclear whether it is the most effective choice. In Figure (a), we evaluate the performance of test-time optimization on different parameter groups of CLIP. We consider four different parameter groups: 1) the entire model, 2) the text encoder, 3) the visual encoder, and 4) the text prompt. For fair comparison, we adopt the same setup as MEMO [49] by

---

using AugMix [72] as the data augmentation. Confidence selection is not used in this ablation study. For each design choice, we run grid search for hyper-parameter tuning (on the learning rate and the number of optimization steps) and report the best result.
The result suggests that text prompt is the most effective parameter group. In addition, find optimizing the visual encoder to have the worst result. This observation is in alignment with previous work that suggests fine-tuning the image encoder can distort pre-trained features [73, 56].
Average Top-1 Accuracy (%)
Average Top- Aecuracy (%)
48
Zero-shot CLIP
35
46
All
Visual
Text
Prompt
0.2
0.3
0.4
0.5
0.7
Encoder
Encoder
Cutoff Percentile
Test-time optimization on different modules.
(b) Different cutoff percentile in confidence selection.
Figure 3: Ablating the effects of different components of TPT. We evaluate the top-1 accuracy on the distribution shifts benchmarks in section 4.1. Methods are implemented based on CLIP-RN5O.
The effect confidence selection. We present confidence selection as a major component of our method, which filters out "noisy" augmented views that provide little information. In Table 4, we provide the performance of TPT without confidence selection, comparison with the full method. Confidence selection brings non-trivial performance improvement our bascline TPT. We further show the effect of confidence threshold p in Figure 3 (b). The result suggests that using the top- 10% confident sample leads to the highest average accuracy. In addition, we find that the effect of confidence selection is generalizable to other entropy-basee test-time optimization methods. More details about this analysis are included in appendix A.4
Table 4: The effect confidence selection. The last row is the performance our full method.
ImageNet Topl
ImageNet-A Topl
ImageNer-Ve V2 Topl acc.
ImageNet-R. Topl
Image Ver Ker-Shh Topl
Method
Average
OOD Average
CLIP-RN50
58.16
21.83
51.41
56.15
33.37
44.18
40.69
baseline TPT
60.31
23.65
53,66
57.48
34.31
45.88
42.28
confidence selection 60.74 (+0.43)
26.67 (+3.02)
(4.70
59.11 (+1.63)
35.09 (+0.78)
47,26 (+1.38)
43.89 (+1.61)
6 Conclusion
In this work, we investigated how to fully exploit the potential of pre-trained vision-language foundation models better zero-shot learners. We developed Test-time Prompt Tuning (TPT), new prompt tuning method that can learn adaptive prompts on the fly with a single test sample. We demonstrated the effectiveness of our method the robustness to natural distribution shifts and cross-dataset generalization, by using CLIP as the foundation model. Without the need for any training data or annotations, TPT improves the zero-shot generalization ability of CLIP.
Limitations. While TPT does not require training data or annotations, our method requires a one-step backpropagation when optimizing the prompt at test time. Since TPT generates multiple augmented views of a single test sample, it increases the memory cost during inference.
Future directions. The idea of TPT can be applied to other foundation models for various downstream tasks, including other vision-language models [6, 74] and foundation models of other modalities (e.g.. pre-trained argee-ccaee/a language models [19, 18]) to further boost their zero-shot generalization. The most interesting part this direction is to design test-time objective that fits the nature of the model and the downstream task. Besides, it is also interesting explore how to reduce the memory cost of TPT and make it more computationally efficient.
7 Acknowledgements
Shu and Goldstein were supported by the ONR MURI program and DARPA GARD.
10

---

References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML,
2021. 1, 2, 6, 7, 19, 20, 21
[2] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision vision-language representation learning with noisy text supervision. In JCML, 2021. 3,3
[3] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint S.kiv:2008.0725555 2021.
[4] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. arXiv preprint arXivv2209.07526, 2022.
[5] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR- modulated detection for end-to-end multi-modal understanding. In ICCV, 2021.
[6] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022. 10
[7] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, 2021.
[8] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.06 06125, 2022.
[9] Zhaowei Cai, Gukyeong K Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, and Stefano Soatto. X-DETR: A versatile architecture for instance-wise liiounlanguage tasks. CoRR, abs/2204.05626, 2022.
[10] Tianyi Liu, Zuxuan Wu, Wenhan Xiong, Jingjing Chen, and Yu- Yu-Gang Jiang. Unified multimodal pre-training and prompt-base tuning for vision- lasoua-lagguag understanding and generation. CoRR, abs/2112.05587,, 2021.
[11] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model. CoRR, abs/2112.14455, 2021. 1
[12] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcieet prompt tuning. In EMNLP, 2021. 1,3
[13] Mitchell Wortsman, Gabriel Ilharco, Mike Li. Jong Wook Kim, Hannanch Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt, Robust fine-tuning of zero-shot models. CoRR, 0bs/2109.01903, 2021. 2,
[14] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis out-of-distribution generalization.
In ICCV, 2021. 2, 6, 19
[15] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for lasion-e models. In CVPR, 2022. 2, 3, 6, 19, 20, 21
[16] Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard-hoi: Benchmarking few-shot visual reasoning for human-object interactions. In CVPR, 2022. 2, 4,
11

---

[17] Cheng Zou, Bohan Wang, Yue Hu, Jungi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, and Jian Sun. End-to-end human object interaction detection with HOI transformer. In CVPR, 2021. 2,
[18] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-raining of deep bidirectional transformers for language understanding. In NAACL- Association for Computational Linguistics, 2019. 3, 10
[19]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020. 3, 10
[20] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In JCML, 2020.
[21] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-base masked language-models. In ACL, 2022. 3
[22] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. CoRR, abs/211 1t//22111.0330 1.03930, 2021.
[23] Renrui Zhang, Longtian Qiu, Wei Zhang, and Ziyao Zeng. VT-CLIP: enhancing vision- vision-language models with visual-guided texts. CoRR, abs/2112.02399 2021.
[24] Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Jianwei Yang, Xiyang Dai, Bin Xiao, Haoxuan You, Shih-Fu Chang, and Lu Yuan. CLIP-TD: CLIP targeted distillation for vision-language tasks. CoRR, hass/2001.05229 2022.
[25] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation.
In ACL/JCNLP. Association for Computational Linguistics, 2021. 3
[26] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V. G. Vinod Vydiswaran, and Hao Ma. IDPG: instance-dependent prompt generation method. In NAACL, 2022.
[27] Zonghan Yang and Yang Liu. On robust prefix-tuning for text classification. In ICLR, 2022. 3
[28] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. CoRR, abs/2109.01 134, 2021. 3, 6, 19, 20, 21
[29] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR, 2022. 3
[30] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. Learning to prompt for continual learning. CoRR, abs/21 12.08654, 2021. 3
[31] Tony Huang, Jack Chu, and Fangyun Wei. Unsupervised prompt learning for vision- langun-aa models. CoRR, abs/2204.03644, 2022. 3
[32] Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Kerg-Kirkpatrick. and Graham Neubig. Prompt consistency for zero-shot task generalization. CoRR, abs/2205.0000 2022. 3
[33] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga. Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang, WILDS: A benchmark of in-the-wild distribution shifts. In ICML, 2021.
[34] Akshay Raj Dhamija, Manuel Günther, and Terrance E. Boult, Reducing network agnostophobia.
In NewrIPS, 2018.
12

---

[35] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In Doina Precup and Yee Whye Teh, editors. ICML, 2017. 3
[36] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be at odds with accuracy. In ICLR, 2019. 3
[37] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent EI Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
[38] Olivier Chapelle and Alexander Zien. Semi-supervised classification by low density separation. In International Workshop on Artificial Intelligence and Statistics, 2005.
[39] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic trans- formations and perturbations for deep supervise learning. In NIPS, 2016. 3
[40] Assaf Shocher. Nadav Cohen, and Michal Irani. "zero-shot" super-resolution using deep internal learning. In CVPR, 2018. 3
[41] David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun- Yan Zhu, and Antonio Torralba. Semantic photo manipulation with generative image prior. ACM Trans. Graph., 2019.
[42] Jogendra Nath Kundu, Naveen Venkat, Rahul M. V., and R. Venkatesh Babu. Universal source-free domain adaptation. In CVPR, 2020.
[43] Yujia Huang, James Gornet, Sihui Dai, Zhiding Yu, Tan M. Nguyen, Doris Y. Tsao, and Anima Anandkumar. Neural networks with recurrent generative feedback. In NeurIPS, 2020. 3
[44] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020. 3,
[45] Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tit++: When does self-supervised test-time training fail or thrive? In NeurlPS, 2021.
[46] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In NIPS, 2004. 4
[47] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In ICCV. IEEE, 2019. 4
[48] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. ICLR, 2021. 4,
[49] Marvin Zhang, Sergey Levine, and Chelsea Finn. MEMO; test time robustness via adaptation and augmentation. CoRR, abs/21 2021. 4,9, 19
[50] Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. In ICLR Workshop, 2017. 4
[51] Manli Shu, Zuxuan Wu, Micah Goldblum, and Tom Goldstein. Encoding robustness to image style via adversarial feature perturbations. In NeurlPS, 2021. 4
[52] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In NeurlPS, 2020. 4
[53] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In JCML, 2020.
[54] Weili Nie, Zhiding Yu, Lei Mao, Ankit B. Patel, Yuke Zhu, and Anima Anandkumar. Bongard- logo: A new benchmark for human- level concept learning and reasoning. NeurlPS, 2020. 9
13

---

[55] Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta- meta-baseline for few-shot learning. CoRR, (bss/203.004390, 2020.
[56] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning distort pretrained features and underperform uut-of-distributien CoRR, abs/2202.10054, 2022. 5, 10
[57] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPRO9, 2009. 6, 19
[58] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICML, 2019. 6, 19
[59] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pages 15262-15271) 2021. 19
[60] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[61] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In NeurIPS, 2019. 6, 19
[62] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. Indian Conference on Computer Vision, Graphics and Image Processing, 2008. 7
[63] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs, In CVPR, 2012. 19
[64] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2010. 7, 19
[65] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, 2014. 7,
[66] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 101 - mining discriminative components with random forests. In ECCV, 2014. 7
[67] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops, 2013. 7, 19
[68] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013. 7, 19
[69] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012. 7
[70] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote, Sens., 2019. 7
[71] Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshops, 2004. 7
[72] Dan Hendrycks, Norman MU, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan, Augmix: A simple data processing method to improve robustness and uncertainty. In ICLR, 2020. 8, 10
[73] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022. 10
14

---

[74] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. CoRR. hts/22011120866 2022. 10
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] . You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit, In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See our main results and discussion in Section 4
(b) Did you describe the limitations of your work? [Yes] See section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss the social impacts in Appendix A.1
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] See implemen- tation details in section 4 and the supplemental materials.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? (Yes) See implementation details in section
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We include analysis in Appendix A.3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the supplemental materials.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] We provide the license information in Appendix A.5
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
15

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "tXTHngj6q6H", "reviewer": "Reviewer_wmVL", "summary": "The authors introduce test-time prompt tuning to improve CLIP's zero-shot classification accuracy by optimizing the consistency of probability distributions across multiple augmented views. To reduce noise samples from random augmentation, the authors proposed confidence selection to remove samples with low confidence. Numerous experiments show the robustness and generalisation of the proposed method.", "questions": "1. Compared to the baseline (CLIP, COOP, COCOOP), how much time complexity does the proposed approach increase while improving performance?\n\n2. I think that the optimization objective function (4) proposed by the authors is not reasonable, as I discussed before. Please correct me if I have misunderstood.\n", "limitations": "The authors have discussed the limitations of the proposed approach. I do not see the potential social impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method improves the performance of the baselines.\n2. This paper is well-organized and easy to follow.\n\nWeaknesses:\n1. Although the authors have presented the inference time as a limitation, the paper still lacks a comparison with the time complexity of CLIP, COOP, etc.\n2. The authors' motivation is reasonable, but the proposed optimization objective is not. I agree that the predictions should be consistent across perspectives, but it doesn't make sense to keep the predicted class probability vectors consistent. Predictions of non-correct label positions in the probability vector are irrelevant, and it is reasonable to be inconsistent across perspectives.\n3. This paper lacks significant contribution and the improvement results from data augmentation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_eo62iYHb9H", "reviewer": "Reviewer_Tt1G", "summary": "This paper proposes test-time prompt tuning (TPT), a method that can learn adaptive prompts on the fly with a single test sample. It optimizes the prompt by minimizing entropy. The authors show TPT can improve 0-shot performance for classification.", "questions": "I'm not sure if the four ImageNet variants are really OOD data? What about other tasks? ", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Pros\n- The method achieves good performance compared to other language-vision prompting methods\n- The method shows a new way to exploit a text-image model like CLIP for mono-modal tasks.\n- This method doesn't require additional data or supervision.\n\n### Cons\n- The 0-shot performance is still behind the fine-tuning by a large margin (although understandable)\n- The idea is not completely novel since consistency regularization has been explored before.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Y4V1VI8FkG", "reviewer": "Reviewer_SAYF", "summary": "The method proposes entropy-based optimization of prompt tokens during test time. To this end, the image is augmented with different views to make a batch of 64 samples. Noisy samples are filtered based on the entropy score. The Adam-based optimizer steers the tokens' representation with one step. \nAn ensemble with a previous baseline (e.g., CoCoOp) significantly improves various classification tasks. \n\n\n", "questions": "\nCan the authors provide results of ensemble between different methods? \nWhat classes are improved by applying TPT?\n", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The work proposes a new approach (TPT) for zero-shot image classification that does not require pre-training and work during inference. The method is plausible and intuitive and suggests some novel elements, such as the confidence selection term. One benefit of this approach compared to CoCoOp is that it doesn't require additional training. In most experiments that check classification capabilities, an ensemble of TPT with  CoCoOp shows a significant improvement in several presents compared to the CoCoOp baseline. \n\nWeaknesses: [W1] The method alone is on par with previous works (i.e., approximately 1% difference between a naive \"hand-crafted\" ensemble). The most significant gap comes from the ensemble of CoCoOp with TPT. However, as an ensemble, in general, is known to improve performance, the method should check other ensembles, e.g., CoCoOp (with different seeds), CoCoOp + CoOp, CoCoOp + Ensemble,  etc. \n[W2] The work lacks discussions and qualitative study. The study does not contribute much to the reader in its current form. Yes, metrics are improved. But why? Which classes improved? I suggest the authors follow this work in the analysis [1]. \n[W3] I'm not sure the method is significantly new. I would appreciate a better discussion on the differences from MEMO. \n\nTypos:  Tab. 1 caption requires -> require; L293 is -> are.\n\n\nRefs: \n[1] - The effects of regularization and data augmentation are class dependent; Balestriero et al.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4K7z8ZpUeGn", "reviewer": "Reviewer_D5Y3", "summary": "This paper proposes a test-time tuning method that adapts the pretrained CLIP for each test example using the example itself. \nSuch tuning is conducted with the objective of encouraging consistent predictions for different augmented views of the test image, and only soft prompts are updated during this process. The authors demonstrate the effectiveness of this method for image classification task under natural distribution shifts setting and cross-dataset generalization setting, outperforming existing methods that even use additional image classification data for training.\n\n  ", "questions": "- The method uses multiple augmented images for one classification, and thus can be regarded as an ensemble method. Did the author compare with a baseline that runs inference on these images and then average the probability? Basically, if we have those augmented images, are there other methods to use them without the optimization step? \n- The 1-step optimization at test time works surprisingly well. I am wondering if the author observed any unstableness of this optimization?", "limitations": "The author discussed the limitations pretty well.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The general idea of test-time tuning by encouraging consistency of different augmented views is novel.\n- The performance is surprisingly good, outperforming strong baselines in the two generalization settings.\n- The analysis on what parameter groups to optimize at test time is inspiring in that it finds prompt tuning is more effective than tuning other set of parameters or the entire model.\n- It's also surprising that only one step of optimization can increase the performance stably.\n\nWeaknesses:\n- Although the paper demonstrate the empirical improvement of the proposed method, it's not very clear why encouraging the consistency can lead to such much improvement. I would suggest the author discuss more about the underlying assumption.\n- Despite that only one step of backpropagation is required for this test-time training, the method still requires computation on multiple augmented images (8-64), which significantly increases the inference computation (roughly 8-64 x 2 slower).\n- The proposed test-time tuning involves several hyper-paramters to be tuned, such as the optimization step or the confidence selection interval. However, it's not clear how they are tuned, which raises the concern of using test data for selecting models.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept