Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization
Authors: Jameel Hassan Abdul Samadh
Hanan Gani
Noor Hazim Hussein
Muhammad Uzair Khattak
Muzammal Naseer
Fahad Khan
Salman Khan
Venue: NeurIPS 2023 poster (2023)
Keywords: Vision-Language models
Prompt Learning
Test-Time Adaptation

Abstract: The promising zero-shot generalization of vision-language models such as CLIP has led to their adoption using prompt learning for numerous downstream tasks. Previous works have shown test-time prompt tuning using entropy minimization to adapt text prompts for unseen domains. While effective, this overlooks the key cause for performance degradation to unseen domains -- distribution shift. In this work, we explicitly handle this problem by aligning the out-of-distribution (OOD) test sample statistics to those of the source data using prompt tuning. We use a single test sample to adapt multi-modal prompts at test time by minimizing the feature distribution shift to bridge the gap in the test domain. Evaluating against the domain generalization benchmark, our method improves zero-shot top-1 accuracy beyond existing prompt-learning techniques, with a 3.08% improvement over the baseline MaPLe. In cross-dataset generalization with unseen categories across 10 datasets, our method improves consistently across all datasets compared to the existing state-of-the-art. Our source code and models are available at [https://jameelhassan.github.io/promptalign](https://jameelhassan.github.io/promptalign)

Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization
Jameel Hassan
Hanan Gani
Noor Hussein' I Muhammad Uzair Khattak
Muzammal Naseer
Fahad Shahbaz Khan
Salman Khan
Mohamed Bin Zayed University of Al Linköping University
3 Austtralian National University
hameel.hassan hanan.ghani, noor. hussein,
uzair khattak
muzammal naserr khan, salman. khan)
@mbzuai.ac.ae
Abstract
The promising zero-shot generalization vision-language models such CLIP has led to their adoption using prompt learning for numerous downstream tasks. Previous works have shown test-time prompt tuning using entropy minimization to adapt text prompts for unseen domains. While effective, this overlooks the key cause for performance degradation unseen domains distribution shift. In this work, we explicitly handle this problem by aligning the out-of-distribution (OOD) test sample statistics to those of the source data using prompt tuning. We use a single test sample to adapt multi-modal prompts at test time by minimizing the feature distribution shift to bridge the gap in the test domain. Evaluating against the domain generalization benchmark, method improves zero-shot top- accuracy beyond existing prompt-learnin techniques, with 3.08% improvement over the baseline MaPLe. In cross-dataset generalization with unseen categories across 10 datasets, our method improves consistently across all datasets compared to the existing state-of- state-of-the-arts Our source code and models are available at hatpss:/jameeellassan.. htw.://wwelllasaaaaiuu......oo.. wttthubii/prompptllg..
Introduction
Deep neural networks (DNNss have outperformed humans in numerous visual recognition tasks 112 8). However, their impressive performance generally holds in the case when test data originate from the same distribution as the training data. most real-world applications, the train and test data distributions can significantly differ due to factors such as natural variations or changes in sensing equipment. The sensitivity of models such unseen distributional shifts during inference results in their performance degradation [14] 31 22] A plethora of previous works [36, (3] 37 have explored test-time adaptation mechanism overcome the crucial problem of distribution shift in test data, However, test-time adaptation has been minimally explored for the increasingly popular foundation models, which are large DNNs trained on massive Language datasets
41
Foundation models emerging at the intersection of various modalities such as vision, language, and audio, are proving be effective numerous downstream applications. Among these models, Vision- Language (V-L) model CLIP (Contrastive Language-Imag Image Pretraining) (30] has been pre-trained on large-scale image-text pairs from the web, and can generalize well zero-shot recognition tasks. The CLIP model has parallel vision and language encoding branches and the similarities between the embeddings obtained from both the branches are used to classify an input image. At inference, handcrafted prompt such as 'a photo of <CLS>' is used as query for the text encoder. However, adapting CLIP efficiently specific downstream tasks still a challenging problem. Naively fine- tuning such models poses the risk of losing their inherent generalization abilities (21.. Instead, recent
Equaa contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

PromptAlign
Stanford Cars
15F
Euroset
of
582
(elT:p) pT():p)
Oxford Pets
SUN397
Soure Stabiks
Sumple Staiatio
sample Ulamrtatiors
CJP
CoOp C50p-TPP
Promablig
cours!
CUPT
(a) Proposed PromptAlign method
(b) Performance comparison in cross datasets evaluation
Figure 1: (a) PromptAlign matches the distribution statistics μ4: p)) (T: obtained from multiple augmented views of single test sample, with the source data distribution statistics fa, o7 This effectively brings the test sample closer to the distribution of the souree data, where the domain shift is denoted by è A.. denotes the distribution of the test sample, p represents the prompts that are updated and refers to the ssion-backbone layers. (b) Owing to the distribution matching via prompts, PromptAlign surpasses the existing state-of-the-art prompt learning approaches on 8 out of 10 datasets in cross-dataset generalization benchmarks.
methods have shown the promise of prompt learning on training data 48 47 to handcrafted prompts, allowing the model to better adapt to the training data distribution.
19
42
as opposed
Existing prompt learning approaches are deployed the training phase to learn representative prompts based on the training data for the downstream task. This conventional approach does not explicitly handle the distribution shift in the test set. Leveraging the capability of prompt learning, recent approach TPT 34 performs test-time adaptation by tuning the text prompts on the fly to adapt the model the test sample. For image classification, the model updates the prompts by minimizing the entropy of the top confident samples which are obtained using different augmented views. However, TPT does not explicitly align the pre-trained CLIP become aware of the test sample distribution.
For the effective test-time adaptation of V-L foundation models, it is crucial to bridge the distribution gap between the pre-training dataset and the downstream evaluation set for high zero-shot generaliza- tion. To this end, we propose PromptAlign, a test-time token distribution alignment strategy using prompt learning. The TPT setting, which tunes the prompts on the text encoder branch alone poses an architectural limitation in performing distribution alignment of tokens. Further, at test time, there no knowledge transfer between the text and vision branches in the intermediate stages. Since the text encoder features will be static given dataset (as the input is the same class labels), distribution alignment of tokens can only be performed on the vision branch. Given these constraints and to further extend the strength of prompts for test time adaptation, we propose to employ a multi-mode prompt learning model (MaPLe) [8] for distribution alignment. PromptAlign explicitly aligns the mean and variances of the image token embeddings of proxy source dataset, computed offline, with the image token embeddings of the test sample. We extend TPT with the token alignment strategy, which enforces to bridge the distribution shift in the test data (Fig. a For each input test sample, we obtain randomly augmented views that are fed into the model to obtain the token embedding statistics. Without any noticeable compute overhead, we update the prompts both the text and vision branches of CLIP to minimize jointly the feature distribution shift and entropy of predictions.
We extensively demonstrate the effectiveness of PromptAlign by evaluating the zero-shot generaliza- tion on two representative benchmarks: domain generalization and cross-dataset generalization. the domain generalization setting, our method improves the baseline model by 3.08% on average across four datasets and has the highest average Top-1 accuracy compared to existing state-of-the-art methods. In the cross-dataset setting, our method achieves absolute average improvement of 1.82% over the existing state-of-the-art method which uses test-time prompt tuning, while attaining the best Top-1 I accuracy in 8 out of the 10 datasets. Our contributions can be summarized as follows:

---

Given only single test sample, we introduce a distribution alignment strategy for V-L models to improve test-time adaptation. The distribution-awed aware pre-trat pre-trained CLIP effectively narrows the distribution gap on the test domains. To the best of our knowledge, this is the first study to explore the potential of distribution alignment in V-L, models at test time.
The proposed strategy formulates a distribution alignment loss that utilizes offline computed source data statistics to encourage the test sample token distributions to be aligned with the source data token distributions. We harmonically combine the benefits of token distribution alignment and entropy minimization using multi-moda prompt learning approach.
Since ELIP-pre-trainiige data is not publicly released, we study the statistics of ImageNet as a possible candidate for the source distribution, and our empirical results show that ImageNet is an effective proxy source dataset for large-scale V-L models such as CLIP.
We validate our method PromptAlign through extensive experiments in domain generaliza- tion and cross-datase benchmarks. PromptAlign improves the generalization of CLIP at test time beyond existing prompt-tuning methods, achieving state-of-the results.
2 Related Work
Prompting for
models. Vision Language (V-L) foundation models 30 17
40 41] have emerged with the convergence of image and text modalities. Having been pre-trained on massive image-text pairs from the web in a contrastive self-supervised manner, these models like CLIP 30 and ALIGN [17] have shown strong generalizability towards downstream zero-shot recognition tasks. However, adapting them efficiently to specific downstream tasks with limited data still a challenging problem. Prompting in CLIP-like models has been explored in the form of a text query that is usually given to the text encoder to instruct it to perform specific task. Recent methods propose to learn these prompts by treating them as continuous learnable vectors and training them in end-to-end manner while keeping the model parameters frozen. CoOp [48] proposes to fine-tune CLIP by learning a set of prompts in the text encoder. CoCoOp 47 highlights the inferior generalization capability of CoOp and conditions the text prompt tokens on image embeddings on the fly. MaPLe [18] proposes to jointly learn deep prompts at both vision and text encoders CLIP. text prompts via V-L coupling function. While
The vision prompts are further conditioned all these approaches promote the effective transfer of CLIP, they require training data to learn the prompts which restricts such adaptation on novel datasets at test time. A recent method TPT (34], attempts to learn prompts solely at test time with the objective of enforcing consistency regularization between multiple views of a test sample by minimizing their averaged entropy. However, this approach struggles to explicitly address the distribution misalignment between the pre-training data of CLIP and the downstream test data. Our method builds on multi-modal prompting variant [18] and explicitly enforces to match the distribution via a distribution alignment technique using a proxy dataset that serves as a replacement for unavailable pre-training data of CLIP. To the best of our knowledge, ours is the first work to explicitly align the distributions learned by V-L foundational models with test time data distribution.
37, 23 aims to bridge the distribution gap between the train and test data distributions at test time. These methods can broadly be categorized into two
Test-Time/A Adaptation (TTA). TTA 36
streams. The first stream of approaches [1 36 typically utilizes a self-supervised proxy task such as predicting image rotations and thus alters the training phase. For example, Test-Time Training (TTT) [36] jointly trains the model for rotation prediction and image classification during training. auxiliary task during training and test-time feature alignment module to adapt the model. By incorporating additional auxiliary tasks, these techniques aim to mitigate over-fitting and improve the generalization performance of the model. The second stream of approaches solely adopts the pre-trained model without altering the training phase. These methods improve model generalization consistency regularization constraints or aligning statistics of training and test samples in model. Approaches with self- self-consistency regularization constraints typically utilize entropy minimization within samples of batch or multiple views of single test sample. TENT [77] proposes entropy
TTT+ 23 adopts a self-supervised contrastive learning approach
test time by imposing self-
minimization of batch-wise prediction probability distributions. The need to have batch of samples by generating multiple views via augmentations at test time is eliminated in (5) Other methods,
such as NORM 32 and DUA (55 adapt the model's 's normalization layers by matching the batch
normalization statisties computed during training with the test set distribution. Both (22 and ActMAD
3

---

Word Embed
Ft
Distribution Alignment
Labels
pi (itat)
Encoder g
Encoder Layer
Distributton Alignment
Luntropy
Patch Embed
F
aalign
Test Sample
P1.
Lilign
Entooder Layer
Encoder
Trainable
F
Frozen
Text Prompts
Offinn Computed Source Data Statishics
Visual Prompts
Source Data
Overview of proposed PromptAlign method for zero-shot image classification. At test time, single test sample along with augmented views is passed through CLIP image encoder, and text labels are passed the CLIPP text encoder. The token distribution statistics mean and variance of the test sample are aligned with the offline computed source data statistics using a distribution alignment loss. The resulting alignment loss from the distribution shift is combined with the entropy loss to update the multi-modal prompts.
Figure 2:
26 updates all model parameters using a batch of samples in a continuous manner, with distribution matching loss between the source statistics and the batch statistics of activations across the network. However, it is unclear how these statistics aligmment techniques can be adopted for foundational V-L models in a lightweight manner for single sample. In this paper, we propose a multi-modae test time prompting approach without altering the pre-training phase of V-L models along with a strategy to obtain training distribution statistics. This effectively paves the way for distribution alignment together with entropy minimization for improved test-time optimization.
Test-time prompt tuning. As mentioned earlier, plethora of works has been introduced on learning prompts adapt V-L models using the downstream task training data. However, the learning of prompts at test time remains largely unexplored in the literature. Very recently, TPT (34) proposed test time prompting approach by learning prompts the text side with an entropy minimization objective. While effective, TPT struggles with addressing the explicit alignment of the pre-training data distribution of CLIP with the data distribution. Further, such an alignment is trivial as access to the pre-training data of CLIP is not available. Our work addresses these limitations by extending TPT and introducing a distribution alignment objective for CLIP. This objective leverages token distribution statistics from proxy source data and a single test sample using multi modal prompt learning, and explicitly aligns train and test sample distributions to mitigate the domain shift.
3 Methodology
provide an overview of CLIP [30 and prompt learning their application for zero-shot
generalization to downstream tasks in Section 3.1 We then discuss test-time prompt tuning and briefly review previous approaches. Finally, we introduce PromptAlign Section 3.22 and provide detailed explanation of how apply it achieve improved zero-shot image classification.

---

3.1 Preliminaries
Contrastive Language-Image Pre-training (CLIP). CLIP consists of two parallel encoders, one for mapping the text imput into feature vector and the other for mapping the visual input. We denote the CLIP image and text encoders by Fu and Fr respectively, and their pre-trained parameters are represented by fLLI = (0.., 0t) respectively. The input image X is divided into M patches, which are projected to produce patch tokens, and class token CLS is prepended to it, resulting in X. = =CCLS, E1, .2,..., e where e; is the embedding of the 7th patch. The image encoder Fu encodes the input patches via transformer blocks to produce latent visual feature representation fo (Xoo 0g). The corresponding class label y is embedded within text template, such 'a photo of <CLS>', which is formulated Yo {sos, t,, 22...., tg. Ck2 EOS}. Here, t/ and Ct are the word embeddings corresponding to the text template and the class label, respectively, while SOS and FOS are the start and end token embeddings. The text encoder F, encodes Y via transformer blocks to produce the latent textual feature as ft Fe F 0z). For zero-shot inference, each text feature with class labels y 21, C} is paired with the image feature to compute similarity score 8; = sim(f ft. fo), where sim(.) denotes the cosine similarity. The prediction probability on X can be denoted by p(y [X) E exp(eim(f=-f.)r>)
exp(sim( f.)r) where T the temperature of the softmax.
Prompt tuning on downstream tasks. CLIP contains a plethora of knowledge leveraged from training on millions of noisy image-text pairs. To effectively extract the rich features learned by the CLIP model, recent approaches [48] 47
append extra learnable prompts while keeping image and text encoders frozen. These prompts modify the context of the model input without distorting the pre-trained CLIP features. Prompts are appended either at the image or text encoder side and learn contextual information tailored towards a specific task. In our work, we use a recently introduced multi-modal prompting baseline [18] that learns prompt tokens both the text and image encoders.
18
2
42
Specifically, we append learnable V visual and T text prompts given as P. = (p, p!..... P. and Pt [pl pi.....P? P with the visual and textual input tokens respectively. The image
encoder processes the input X = {CLS, Pu, E1, e,, EM} to generate prompted visual feature representation denoted as fg. Similarly, the text encoder processes the input Y {sos, Pt:
Pt: t1, t,.,,.., .t, CR. EOS} producing the textual feature representation ff Our approach uses deep prompting as utilized in (I8). along with text prompts and conditional image prompts at subsequent transformer blocks. We jointly represent the visual and textual prompts by p. We refer the readers to for more details on the baseline architecture.
Test-time prompt tuning, Test-time prompt tuning introduced by 34 aims to benefit from the rich knowledge of CLIP boost its generalization in a zero-shot manner. TPT can be viewed as a means to provide the model with a context that is customized for each individual test sample in order to more accurately recall the knowledge contained within CLIP. During inference, several randomly augmented views are generated from the given test sample Xuen- The predictions having entropy below a certain threshold are kept, while other views are discarded using a confidence selection filter. The averaged entropy of the filtered predictions is then used to update the prompts p in an unsupervised fashion using the following objective function.
Centropy arg min
Σ Pp(ys (Xuea) log Pp(1 ( (eest)
(1)
where (v: (rss) ( represents the mean of vector class probabilities produced by the model across the different augmented views preserved after the confidence selection filter.
3.2 PromptAlign
Unimodal Test-time Prompt Tuning (TPT) 34
updates the text prompts on the fly at inference by minimizing the entropy loss. This approach does not explicitly handle distribution shift that arises in the test set which is sub-optimal. One solution to this problem is to align the source and target distributions by bringing the test sample into the domain of the source data distribution. However, TPT updates the prompts only on the text encoder branch with static labels which poses an architectural limitation in performing the distribution alignment of tokens. Hence, distribution alignment of tokens
5

---

can only be performed on the vision branch. We. therefore, utilize multi-modes prompt learning VL models [8] to explicitly handle the distribution shift of each test sample from the source distribution.
Given a test sample Xiea, we take multiple augmented views and pass them through the visual encoder with deep learnable prompts as shown in Fig. [2] At each layer of the visual encoder, compute the token alignment loss between the means and variances of the test sample with that of the means and variances of proxy dataset which mimics the behavior of the source data distribution. Our final objective combines the entropy and alignment losses to update the prompts for a given test sample.
Proxy source dataset. In order to compute the token embedding statistics on the source dataset, we require the pre- pre-training dataset of the CLIP model. The CLIP model was trained on over 400 million image-text text pairs, which is not publicly available. However, previous works have shown that LAION400M 33 can be used as a training dataset to achieve the performance of CLIP [5] leading proxy dataset for the CLIP training dataset. In addition to this, CLIP has also been heavily tuned to achieve excellent zero-shot performance on ImageNet []. Therefore, we use ImageNet as the proxy source dataset for computing the mean and variance of token distributions. Source dataset statistics are computed offline and utilized directly at test time.
to subsets of LAAON400M. being used
Token Distribution Alignment via multi-modal prompting. We generate N₁ random views of the test sample using a set of augmentations H. The mean and variance statistics of token embeddings of the test sample are computed at the output of each transformer layer of the CLIP model's 's visual encoder, across the N₂ views. Similarly, the source data statistics are pre-computed in an offline manner. We represent test sample distribution by (7) and source distribution by (D). Specifically, compute the token means and variances for the alignment as follows.
(i(T:p)
Σ X
(2)
N&
xE
(T:T:))
1
Σ
XL 15--Pi -M(T:T)))
2
NO
xER(X)
where (T(T;)) p) and PT: are the vector means and variances of the test sample tokens at the layer in the visual encoder and X represents the prompted token embeddings layer for the augmented view input Similarly for each layer I in the visual encoder, we pre-compute the source data statistics as,
A (D,0,) and of
where fg denotes the parameters of the visual encoder from the pre-trained CLIP model. We compute the token distribution alignment loss between the mean and variances of the test sample and the source dataset statistics as follows.
1 Σ T. 1=1
(IIU(T:D)) - Arll + (TTp(T:)) of -070)) o?ll1)
Lalign
(5)
As shown above, we use L1 loss to enforce the distribution alignment of the test sample with the source distribution. The alignment loss Lalign is added to the entropy loss (Eq. (T) to obtain the final objective Cinat- Finally, the combined objective is optimized to update the prompts p.
Cinal Lentrpy + 3 C
(6)
where the hyperparameter / controls the contribution of alignment loss to the overall objective function. Our alignment loss in Eq. 6 bridges the gap between source and token distributions by updating the prompts to align them with the source distribution.
Discussion on Limat- As derived above, our test-time loss combines the entropy minimization (Cennopy) and distribution alignment objectives (Calign) together. The Lentropy objective enforces prediction consistency among different views of sample, leading to robustness against various variations that could possibly occur at test time. On the other hand, Latign effectively narrows the domain shift and brings the test sample distribution closer to the pre-trained CLIP distribution space, which enforces CLIP to better understand the test sample. The combination of these loss objectives harmonically adapts CLIP to be robust to different sample variations and at the same time, enhances CLIP's s understanding of the underlying test sample domain for better generalization.

---

Table 1: Effect of token distribution alignment strategy for domain generalization. The base model MaPLe is trained on ImageNet and evaluated on datasets with domain shifts.
Imagenet V2
Imagenet Sketch
Imagenet
Imagenet
OOD Avg.
MaPLe
64.07
49.15
50.90
76.98
60.28
MaPLe+TPT 64.87
46.16
58.08
78.12
62,31
PromptAlign 65.29
50.23
59.37
79.33
63.55
4
Experiments
Datasets. For the domain generalization setting, we evaluate the four out-of-distribution (OOD)
variants ImageNet 77; ImageNetV2 [3T], ImageNet-Sketch [38], ImageNet-A [16] and ImageNet-
R [15. We also evaluate the domain generalization setting the recent challenging Photorealistic
Unreal Graphics (PUG) dataset [3], comprising different textures, backgrounds, sizes and orientations. In the cross-dataset evaluation, we follow TPT [4 and evaluate the performance of methods on 10 image classification datasets covering a wide range of visual recognition tasks. This includes one generic-objects dataset Caltech101 [10]: five finegrained datasets OxfordPets (28), StanfordCars [20]. Flowers 102 [27], Food101 4 and FGVC-Airerafi [24]. which contain images of animals, flowers and transportation; and four datasets of scenes, textures, satellite imagery and human actions
- -SUN397 36
DTD 6), EUROSAT
13 and UCF101 35 respectively.
Baselines. We evaluate PromptAlign with existing few-shot prompt learning methods for adapting CLIP including CoOp [48] and CoCoOp [47]. and TPT 34 method. MaPLe [88] is a multi-modal prompt learning baseline, which adapts CLIP by learning deep prompts both the text and vision branches. TPT a test-time prompt tuning method, that tunes the prompt at test time per input sample achieving state-of-the-art performance in prompt learning when combined with CoOp.
Implementation details. Following MaPLe [8], we train on ImageNet using 16-shot training data with 2 prompt tokens for depth of 3 layers. We optimize the prompts on both the text and vision branches using a single test image. We obtain 63 augmented views using random resized crops and horizontal flip augmentations to construct a batch of 64 images including the original image to mimic the setting of TPT. From the 64 predictions, we obtain top 10% confident predictions based on the lowest entropy and compute the average prediction probability. We compute the token distribution aligmment loss between the tokens of all 64 images. We optimize the prompts to minimize the combined loss of average prediction entropy and the token distribution alignment loss using AdamW optimizer. We use a learning rate 54 for the fine-grained datasets Flowers 102, OxfordPets, Food101, SUN397. FGVCAireraft, and EuroSAT and learning rate of 0.04 for the rest of the datasets, and set the loss scale factor / equal to 100. Further details are listed in Appendix A
4.1 Token distribution alignment in V-L models
We first evaluate the performance of the token distribution alignment strategy over the baseline for test-time prompt tuning. Since the alignment can only be performed on the vision branch, we explicitly compare PromptAlign to vanilla MaPLe, and MaPLe with TPT using entropy minimization. Table presents the results in domain generalization across ImageNet variants. TPT improves the performance of MaPLe from 60.28% to 62.31% on average, while the incorporation of token distribution alignment further improves the performance by 1.24%. This indicates that explicitly aligning the distribution of training and testing data enhances CLIP's generalization. Moreover, TPT is intended to improve test-time adaptation a plug-in module, but its effectiveness is not uniform, as observed from the drop in performance on the ImageNet-Sketee Sketch dataset. On the other hand, PromptAlign does not show any performance degradation and improves beyond vanilla MaPLe consistently on all datasets. We further present qualitative results of PromptAlign in Appendix C
4.2 Domain Generalization
Compared to CLIP, all test-time adaptation methods demonstrate better performance, indicating that V-L models benefit from test-time adaptation approaches (Table PromptAlign achieves the highest Top-1 1 accuracy averaged across all ImageNet variants. In comparison to the previous state-of-the-an CoOp+TPT, PromptAlign achieves average improvement of 0.71%. Out of the four different
7

---

Table Comparison PromptAlign in domain generalization setting. Prompt learning methods are trained on ImageNet and evaluated on datasets with domain shifts.
Imagenet
Imagenet Sketch
Imagenet A
Imagenet
OOD Avg.
CLIP
E0
60.86
46.09
47.87
73.98
57.20
CLIP+TP
64.35
47.94
54.77
77.06
CoOp 48
64.20
47.99
49.71
75.21
59.28
CoOp+TPT 34
66,83
49.29
57.95
77.27
62,84
Co-CoOp 47
64.07
48.75
50.63
76.18
59.91
Co-CoOp+TPI
64.85
48.27
58.47
78.65
62.61
PromptAlign
65.29
50.23
59.37
79.33
63.55
Table Effect of token distribution alignment strategy for domain generalization, The base model MaPL.e is trained on ImageNet and evaluated on PUG-ImageNet.
Camera
Puse
Scale
Texture
Lighting
Worlds
(Yaw/ Pitch/ Roll)
(Yaw/ Pitch/ Roll)
MaPLe 18
48.73/ 39, 32.13
48.10/ 28 40/ 27.80
46.90
37.90
15.50
32.13
MaPLe+TPT
57.04/ 45.99/ 39.23
56.26/ 35.64/ 33.26
54.87
43.73
22.52
42.00
PromptAlign
58.14/ 46.93/ 40.45
57.43/ 36.31/ 34.32
56.18
44.97
23.06
43.24
Table 4: Comparison of liomptAlign cross-dataset evaluation. Prompt learning methods are trained on ImageNet and evaluated on cross-datasets.
Caltech
Flowers
Foud101
Aircraft
SUN397
UCF101
Avenage
Pets
Cars
DID
CLIP 30
93.35
88.25
65.48
67.44
83.65
23.67
62.59
44.27
42.01
65.
13
63.58
CLIP+TPT
34
94.16
87.79
66.87
68.98
84.67
24.78
65.50
47.75
42.44
68.04
65.10
CoOp
48
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
46.39
66.55
63.88
CoCoOp
47
93.79
90.46
64.90
70.85
83.97
22.29
66.89
45.45
39.23
68.44
64.63
ProDA
46
86.70
88.20
60.10
77.50
80.80
22.20
50.90
58.50
65.62
MaPLe
93.53
90.49
65.57
72.23
86.20
24.74
67.01
46.49
48.06
68.69 68.69
66.30
MaPLe+TPT
93.59
90.72
66.50
72.37
86.64
24.70
67.54
45.87
47.
66.50
PromptAlign
94.01
90.76
68.50
72.39
86.65
24.80
67.54
47.24
47.86
69.47
66.92
domains, method improves beyond the previous best, except Imagenet- V2. We hypothesize this is due to the extensive training of CoOp on ImageNet which has a very similar distribution to the
ImageNet-V. V2. We further evaluate the effectiveness of our method in domain generalization challenging PUG dataset as reported in Table PromptAlign consistently improves upon TPT here as well. These results demonstrate the effectiveness of PromptAlign, which adapts CLIP simultaneously for both robustness against view changes and alignment of train-test set distribution.
4.3 Cross-Dataset Transfer
In T evaluate the performance of method in generalizing across diverse cross-datasets in comparison with existing state-of-the-art methods using prompt learning. PromptAlign provides consistent improvements and outperforms the previous best method MaPLe combined with TPT in all cross-datasets with an average improvement of 0.42%. In comparison to prompt learning methods, can be observed that both CoOp and CoCoOp on average are inferior to zero-shot CPI while PromptAlign consistently demonstrates improvements, which validates the necessity of token distribution alignment for preserving CLIP's generalizability. This further affirms that PromptAlign harmonically combines the effect of entropy minimization and token distribution alignment in V-L models. We also analyze the effect in Appendix 1 proving the effectiveness of distribution alignment with source dataset statistics. Furthermore, we note that as opposed to our method, existing prompt learning methods with TPT are not consistent across cross-datasets, which we discuss in detail in Appendix
better proxy source dataset, using a subset LAION400M

---

Method
Entropy loss Distribution alignment
Top-1 Acc.
Table Analysis on the align- ment and entropy minimization loss. The average of Top-1 accuracy (%) across three seeds is reported. PromptAlign' denotes our method excluding entropy minimization loss.
MaPLe
50,90
MaPLe+TPT
X
58.08
PromptAlign'
50.85
PromptAlign
59.37
5
Ablation
We perform ablative analysis on various components of PromptAlign. Unless stated otherwise, we show ablations ImageNet-A dataset, the smallest domain generalization variant for simplicity. Detailed ablation trends across other ImageNet variants and datasets are presented in Appendix D
Token Distribution Alignment. Table summarizes the effect of token distribution alignment in PromptAlign. It can be observed that removing the entropy loss results in almost the same performance vanilla MaPLe. Whereas combining alignment and entropy losses improves the performance remarkably. This because the distribution alignment loss is regularizer that functions together with the self-entropy loss. Since distribution alignment does not promote any discriminative learning as opposed to entropy loss, is not expected improve test-time adaptation on own.
Loss variants for distribution alignment. We show the variation in performance with different loss choices for distribution alignment objective. Figure compares the three loss choices Li, L2, and KL divergence. We observe that L, constraint performs best for the distribution aligmment objective. Further analysis is shown in Table of supplementary material.
Loss balancing factor B. We ablate the loss scale factor 3 to analyze the significance of the the baseline model with TPT depicted in Figure 3 even with smaller weight scale and gradually plateaus close to a scale of 100. We ImageNet validation set for this ablation and choose 3 100 across all experiments. We find
distribution alignment loss. The performance improves
a similar trend of the scaling factor on other datasets, which is discussed in Appendix D For the MaPLe+TPT model, the value of 3 is there is alignment objective in this case.
Higher Order Statistics. Since we align distributions computed using a small set of samples, we use relatively simpler (mean and variance) measures for alignment strategy over higher order statistical measures. We evaluate this choice for the distribution aligmment in PromptAlign using the 5th order statistics of Central Moment Discrepancy measure 45 in Table 6 Our observation reveals that the utilization of higher order statistics results in only a slight improvement implies that there is minimal distinction between opting for higher order statistics validating mean and variance based approach as a suitable measure for alignment in our case.
Prompt Regularization. We compare PromptAlign a continuous test-time adaptation setting, in which the model gradually adapts to the incoming distribution. We regularize the prompts in relation to the previous prompts, denoted by PromptReg in We also lower the learning rate to le -5 to prevent diverging in the continuous setting. We Table
naive prompt regularization method in
&8 71.60
660.
Top-1 Iverage accura
71.55
59.5
71.40
58.5
71.35
58.00
71.30
PromptAlign
TPP
71.25
10
50
100
150
200
257.0
KL div.
42
41
Weight scale factor
Loss function
Figure 3: Effeet of the loss scaling factor 3 on ImageNet. Scale factor plateaus after / 100.
Figure 4: Effect of performance on the choice of loss function for distribution alignment.

---

show that the naive regularization setting, even in continuous setting is not robust as the proposed alignment strategy, which solely aligns based on a single test sample, individually.
Table 6: Comparison of the effect of higher order statistics for distribution alignment.
PromptAlign' utilizes Central Moment Discrepancy measure for the distribution alignment loss.
Caltech
Flowers
Food101
Aircralt
SUN397
EuroSAT
UCF101
Average
Pets
Cars
DTD
PromptAlign
72.39
47.24
90.76
68.5
69.47
94.01
86.65
67.54
24.8
47.86
66.92
PromptAlign
72.36
47.24
90.74
68.3
69.48
94.04
86.65
67.85
24.75
48.5
66.99
Table 7: Comparison of PromptAlign with naive prompt regularization.
Target
Caltech
Flowers
Food101
Aireraft
SUN397
EuroSAT
UCF101
Average
Pets
Cars
DTD
MaPLe
93.53
90.49
65.57
72.23
86.20
24.74
67.01
46.49
48.06
68.69
66.30
PromptReg
93.71
90.60
65.02
72.05
86.05
22.32
65.46
46.24
19.03
68.23
62.87
PromptAlign 94.01
90.76
68.50
72.39
86.65
24.80
67.54
47.24
47.86
69.47
66.92
Trade-off between compute resources and performance. Figure 5)a) demonstrates the improve- ment in performance as the number of augmented views at test time increases. PromptAlign shows better improvement with more augmented views, owing to a better approximation of the test sample distribution. While MaPLe+TPT performance plateaus around 64 augmented views. our method with aligmment loss shows further scope for improvement given more augmented views. FigureBJbb shows the variation in performance with the number of prompt update steps. We note that, with more update steps, PromptAlign can consistently better adapt to the test sample comparison to MaPLe+TPT alone. In Figure 5jc) we analyze the computational overhead of PromptAlign PromptAlign shows no additional compute cost terms of latency. The average inference time per sample across three seeds with and without the distribution alignment strategy are 0.216s and 097s. Considering the memory usage and increase latency, we use views and a single-step update following TPT
Top-1 average accurrac aarraay %
3.5
Top overage accuracy
58
3.0
2.5
2.0
14
1.5
Promptalign MPPLe-TPT
18
Time
1.0
Prom Promptülgg
0.5
16
Number of sugmented views
37
64
128
promot update steps
Number promot update
Figure 5: Analysis of compute resource constraints on performance. (a) The Top-1 accuracy increases with the number of augmented views. (b) The Top-1 accuracy improves consistently with the number of prompt update steps. (c) Impact latency with the number of prompt update steps is similar for both methods.
Conclusion
In this paper, we introduce PromptAlign, novel approach for enhancing test-time adaptation of VisguuLLanguage (V-L) models for zero-shot generalization. Our proposed approach bridges the gap between the test sample and source distributions by explicitly aligning the test sample statistics with that of the source data distribution through token distribution alignment. To achieve this, we incorporate multi-modal prompting to facilitate the alignment of token distributions across transformer layers during test time. Through extensive experiments, PromptAlige demonstrates superior performance over existing state-of-the-an art CLIP zero-shot generalization methods in domain generalization and cross-dataset evaluation settings.
10

---

Acknowledgements
The computational resources were provided by the National Academic Infrastructure for Super- computing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement No. 2022-06725, and by the Berzelius resource, provided by Knut and Alice Wallenberg Foundation at the National Supercomputer Center.
References
[1] Hyojin Bahng, Ali Jahamian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint 22t.::220..224444 1(3)4, 2022.
[2] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modifying pixel space adapt pre-trained models. arXiv preprint arXiv:2203. 22222223..7774 17274, 2022.
[3] Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari S Morcos. Pug: Photorcalistic and semantically controllable synthetic data for representation learning. arXiv preprint arXiv:2308.00 03977, 2023.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mimmminn discriminative components with random forests. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings, Part 13, pages 446-461. Springer, 2014.
[5] Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabricl Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. arXiv preprint 2rX1v:2212.07143, 2022.
[6] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 6606-361,, 2014. 2014.
[7] Jia Deng, Wci Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages IEEE, 2009.
[8] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan Swetter, Helen Blau, and Sehastian Thrun. Dermatologist-ler level classification of skin cancer with deep neural networks. Nature, 542(7639)) 115-118 2017.
[9] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19358-19369, 2023.
[10] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pottern Recognition Workshop, pages 178 178. IEEE, 2004.
[11] Yossi Gandelsman, Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 35:29374-2993 29385,, 2022.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedingy of the IEEE International Conference Computer Vision, pages 1026-1034, 2015.
[13] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 12(7):2217-222 2019.
[14] Dan Hendrycks and Thomas Dietterich. Benehmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903. /2261, 2019.
[15] Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of distribbtiio generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8349, 2021.

---

[16] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15271, 2021.
[17] Chao Jia, Yinfei Yang. Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, fun-Hsuan Sung. Zhen Li. and Tom Duerig. Scaling up visual and visionlanguage representation learning with noisy text supervision. In ICML, pages 4904 4916. PMLR, 2021.
[18] Muhammad Uzair Khaitak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi- modal prompt learning. In The IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
[19] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer. Salman Khan, Ming- Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 15190-15200, 2023.
[20] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- Ene-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 561, 2013.
[21] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. International Conference on Learning Representations, 2022.
[22] Wei Lin, Muhammad Jehanzeb Mirza, Matcusz Kozinski, Horst Possegger, Hilde Kuehne, and Horst Bischof. Video test-time adaptation for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 22952-22961, 2023.
[23] Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet Taylor Mordan, and Alexandre
Alahi. Ttt+: When does self-supervised test-time training fail or thrive? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing
Syxiems, volume 34, pages 21 808 2 1820. Curran Associates, Inc., 2021. URL https: lttpps:/proceeeding proceedings neurips.co/ pourips.cc/pape... files/ wwwps/papeeeeeeeeepppppaaaapeeea 2021/ httpappe//////ll////eeaa2/2 /ttph////66.22220000000002220111
Papeee2-PPpppppppe .pdff
[24] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
[25] M Jehanzeh Mirza, Jakub Micorek, Horst Possegger, and Horst Bischof. The norm must go on: Dynamic unsupervised domain adaptation by normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14765-14775, 2022.
[26] Muhammad Jehanzeb Mirza, Pol Jané Soneira, Wei Lin, Mateusz Kozinski, Horst Possegger, and Horst Bischof. Actmad: Activation matching to align distributions for test-time-training. arXiv preprint arXiv:2211.12870.. 2022.
[27] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
[28] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498-3505. IEEE, 2012.
[29] Joaquin Quinonero-Candell Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset Shift in Machine Learning. MIT Press, 2008
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-88 8763. PMLR, 2021.
[31] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Muchine Learning, pages 5389-5400. PMLR. 2019.
[32] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33:11539 11551, 2020.
12

---

[33] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-texi pairs. arXiv preprint 1rx0v:2111.021144 2021.
[34] Manli Shu, Weili Nie. De-An Huang, Zhiding Yu. Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization vision-language models. Advances in Neural Information Processing Systems, 35:14274-14288 2022.
[35] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. dataset 101 human action classes from videos in the wild. Center for Research Computer Vision, 2(11), 2012.
[36] Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, pages 9229-9244, PMLR, 2020.
[37] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10772 2020.
[38] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
[39] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7201-7211, 2022.
[40] Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive anguage-image pre training, arXiv preprint arXiv:2111.07783 2021.
[41] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: new foundation model for computer vision. arXiv preprint ar Xiv:21111.1142.222
2021. 2021.
[42] Yuhang Zang. Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225,, 2022.
[43] Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger- Platz. Central moment discrepancy (cmd) for domain-invariat invariant representation learning. arXiv preprint orXiv:/702.088 2017.
[44] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, pages 111133,,,, 2022.
[45] Marvin Zhang, Sergey Levine. and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629-38644 2022.
[46] Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12414 12414-12424, 2021.
[47] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[48] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. International Journal of Computer Vision (IJCV), 2022.
13

---

Appendix
In the following, provide further details implementation under Section A Section B provide additional results our proposed PromptAlign method on cross-dataset generalization, base-to-novel generaliza- tion and for different CLIP backbone architectures; and Section shows additional qualitative results. Finally, Section D presents ablation trends across diverse datasets for the effect of the proxy dataset, the effect of test sample statisties and different parameter choices.
A
Additional Implementation details
A.1
Hardware and Software details
We implement method with the MaPLe 18 multi modal prompting model with the CLIP ViT-B/16 backbone architecture. Our models were implemented on a single NVIDIA A100 GPU using the Py Torch framework.
A.2 Effect of layer depth for token distribution alignment loss
The token distribution alignment loss is computed from all the corresponding tokens of 64 crops spanning the first layers of the visual encoder. This chosen in line with the baseline multimodal prompt learning model [8] where the deep prompts are present in layers 1-3. In order to analyze impact different layer depths on the alignment loss, we conduct an ablation study. Figure displays the performance comparison of alignment losses obtained from various layer combinations using the ImageNet-A A dataset. As observed in the Figure 6 enforcing the token alignment loss across the first three layers of the vision encoder of CLIPP provides the best. performance.
Figure 6: Effect of alignment depth on performance: We ex- amine how the performance of PromptAlign is affected by alter- ing the number of alignment lay- that contribute ment loss. Our findings indicate that the token distribution align- ment loss is most impactful when applied to the lower layers (specif- ically, layer + 3). We hypoth- esize the effectiveness token distribution alignment loss in these layers can be attributed to the presence of prompts, which are located within the first three layers of the visual encoder.
12
6 - 12
3 - 12
-
12
Top-1 average accuracy de
60.0
59.0
the align-
58.0
57.0
56.0
bottom top top bottom
55.0
1-3
1
1
1 - 9
1 - 12
Alignment Depth
B
Additional results
B.1 Cross dataset evaluations
To the main paper, we compared the performance PromptAlign with the current state-of-the-art prompt learning methods, CoOp and CoCoOp, for cross-dataset generalization. Additionally, in Table we evaluate existing prompt learning methods that have been integrated with TPT and present further comparisons of method with them. As shown in Table PromptAlign outperforms both CoOp+TPT and CoCoOp+TPT out of datasets. It worth noting that when integrated with TPT, performance of CoOp CoCoOp decreases for the majority of cases. contrast, PromptAlign consistently improves performance across diffcrent datascts.
B.2 Base to Novel Generalization
We also evaluate the performance our method on base-to-novel generalization benchmark setting. average, our distribution alignment strategy improves the model performance by nearly 1% in the base classes and by 0.79% on novel classes. We observe that TPT alone drops the performance of the model in some instances such for OxfordPets, Eurosat and UCF101, On such instances, PromptAlign regulates the drop in accuracy cause by TPT, while in most cases achieves better performance novel classes of these datasets.
14

---

Table Comparison of PromptAlign to vanilla TPT in cross-dataset evaluation. Prompt learning
methods are trained on ImageNet and evaluated on cross-datasets at test-time.
Target
Caltech101
OxfordPets
StanfordCars
Flowers 102
Food101
Aircraft
SUN397
EurOSAT
UCF101
Average
DTD
CoOp 48
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
46.39
66.55
63.88
Co
R4
93.15
89.48
66.77
68.48
86.48
20.51
66.06
43.32
37.73
68.91
64.08
CoCoOp
4
93.79
90.46
64.90
70.85
83.97
22.29
66.89
45.45
39.23
68.44
64.63
CoCoOp+TPP
88.57
85.33
59.68
55.31
80.64
16.89
60.24
38.93
48.55
63.35
59.75
PromptAlign
94.01
90.76
68.50
72.39
86.65
24.80
67.54
47.24
47.86
69.47
66.92
Table 9: Comparison on Base-to-novel generalization of PromptAlign with previous methods. PromptAlign shows consistent improvement over TPT.
Dataset
ProDA
MaPLe MaP
PromptAlign (Ours)
46
[8
Base
81.56
82.24
82.16
83.19 (+0.95)
Avg. on 11 datasets
Novel
72.30
75.09
74.95
75.88 (+0.79)
Base 75.40
76.67
77.73
78.26
ImageNet
Novel
70.23
70.54
72.24
72.59
Base 98.27
98.00
98.54
98.60
Caltech101
Novel
93.23
94.27
94.29
94.50
Base 95.43
95.43
95.23
95.38
OxfordPets
Novel
97.83
97.80
97.37
97.56
Base 74.70
72.90
74,00
75.02
Stanford Cars
Novel
71.20
73.97
75.20
75.71
Base 97.70
95.93
96.24
96.61
Flowers
Novel
68.68
72.40
72.10
72.34
Base 90.30
90.70
91.13
91.63
Food101
Novel
88.57
92.07
92.03
92.68
Base 36.90
37.27
34.31
37.21
FGVC Aireraft
Novel 34.13
35.53
35.81
37.27
Base 78.67
80.80
81.15
81.57
SUN397
Novel 76.93
78.70
79.18
79.48
Base 80.67
80.30
82.20
82.60
DTD
Novel 56.48
59.23
59.91
60.55
Base 83.90
93.63
91.02
94.10
Eurosat
Novel
66.00
72.87
68.96
72.71
Base 85.23
82.97
82.23
84.11
UCF101
Novel 71.97
78.57
77.34
79.30
B.3 Different CLIP backhone architectures
We evaluate our method the VIT-B/32 backbone architecture variant of CLIP. Here, we train CLIP VIT-B/32 in the original MaPLe setting without any hyperparameter tuning specific to MaPLe. Our distribution alignment strategy improves upon TPT on average in both the domain generalization setting and cross dataset settings reported in Table 10 and Table [I]
15

---

Table 10: Comparison of PromptAlign in domain generalization setting for ViT-B/32. Prompt learning methods are trained on ImageNet and evaluated on datasets with domain shifts.
Imagenet V2
Imagenet Sketch
Imagenet A
Imagenet R
OOD Avg.
MaPLe
57.63
42.15
32.12
67.64
49.89
MaPLe+TPT 60.01
43.77
37.52
71.11
53.10
PromptAlign 60.43
44.24
38.02
71.44
53.53
Table 11: Comparison of PromptAlign in cross-dataset evaluation on ViT-B/32. Prompt learning methods are trained on ImageNet and evaluated on cross-datasets.
Caltech
Flowers
Food101
Aireraft
SUN397
EuroSAt
UCF101
Average
Pcts
Cars
DID
MaPLe [8]
92.50
88.13
59.93
65.33
81.00
17.53
65.00
41.70
40.80
63.63
61.56
MaPLe+TPI
34
91.44
88.47
59.35
66.08
82.08
18.71
66.07
40.01
39.67
64.40
61.63
PromptAlign
92.10
88.44
63.48
66.14
82.07
18.76
66.08
42.54
39.68
65.57
62.49
C Qualitative Results
We generate attention map visualizations from the heads of the last block of the CLIP visual encoder on the test samples from the Imagenet-A dataset. Figure 7 we compare attention maps produced by PromptAlign and TPT. Our findings reveal that PromptAlign exhibits stronger focus on discriminative regions while reducing attention on background regions compared to TPT. For the first test sample (on the left), PromptAlign intensifies attention the object of interest and diminishes attention on the background regions. Similarly, in the second image (on the right), PromptAlign increases attention toward the salient object.
D Extended Ablations
D.1 Proxy source dataset
We assess the effectiveness of aligmment strategy, in Table by Tayeeoyniyyemmmmoyiing source data statistics. utilize subset 2 Million images from the LAION400M dataset (denoted as PromptAlign ). We show that using LAION400M to compute the source dataset statisties enhances the distribution alignment objective, and thus the overall performance in Table T2 This emphasises the effectiveness of distribution alignment, since LAION400M has been shown valid training dataset for CLIP (3]
employing LA1ON400M 33 compute
In Table present the performance of our method when using the training of the respective datasct as the proxy source to compute the training statisties used the token distribution alignment loss. We choose
analysis. We observe that using ImageNet as the proxy source dataset gives hetter performance across all datasets, except in UCF101, where the performance is the same. Experiments in different Vision-Lane Language models such as
datasets: DTD [). StanfordCars 20 UCF10 (35) Caltech101 10), and FGVC Airerafts (24)
EVA-CLIP [. with different training recipes has similar trend validating the choice of ImageNet proxy dataset.
'Due to compute and time limitations we use a subset of LAION400M dataset
Test sample
TPT
PromptAlion (Ours)
Test sample
TPT
PromptAlign (Ours)
Figure 7: Comparison of attention map visualizations. PromptAlign focuses highlighting the discrimina- tive arcas while reducing attention background regions opposed to TPT.
16

---

Table 12: Experiment with different source dataset statisties. PromptAlign' refers to source
dataset statistics computed using subset of LAION400M for distribution alignment.
StanfordCars
Caltech101
OxfordPets
Flowers102
Food101
Aircraft
SLN397
EuroSAT
UCFI01
Averuge
DID
PromptAlign 94.01
90.76
68.50
72.39
86.65
24.80
67.54
47.24
47.86
69.47
66.92
PromptAlign' 94.32
90.76
68.11
72.38
87.26
24.78
68.02
47.97
47.88
70.20
67.17
Table 13: Comparison of the effect of proxy source dataset in cross-dataset evaluation for different V-L models. PromptAlign' refers to PromptAlign with training set of the respective test dataset used the proxy source dataset to compute statistics for the alignment loss.
Target
Model
Method
Caltech101
StanfordCars
UCFI01
Aircraft
DTD
PromptAlign
93.98
68.44
69.47
23.85
47.10
CLIP
PromptAlign
94.01
68.50
69.47
24.80
47.24
PromptAlign' 95.66
79.44
65.39
23.57
50.06
EVA-CLIP
PromptAlign 95.69
79.50
65.41
23.48
50.06
Our hypothesis the effectiveness of ImageNet proxy source dataset is two-fold. Firstly, the token distribution alignment intends align tokens to that the CLIP pre-training dataset token distributions. The larger scale of ImageNet in comparison to the training set of the above datasets, and the CLIP pre-training strategy, finetuming it to improve performance on the ImageNct benchmark [ renders ImageNet a hetter proxy source dataset for CL.PP Furthermore, using ImageNet acts regularizer, maintaining the generalization capability of CL.IP while performing test-time prompt learning using entropy minimization. Similarly, since LAION400M a better representation of CLIP training dataset, the subset LAION400M used as proxy shows an improvement over ImageNet in being used as a proxy source dataset.
D.2 Effect of test sample statisties
To analyze effect the test sample statistics quality on distribution alignment strategy, we experiment with the use of more samples for alignment. For each test example, we use a bag of samples (N=5) obtained from the same class (while the class label is kept unknown) to estimate the test sample statistics. We present the results in Table [14] which shows that test sample statisties can be better approximated with more samples, improving the performance through the alignment objective. However, the actual setting of test-time adaptation, only single sample is available.
D.3 Trend across parameter choices
We present ablations the distribution loss scaling factor 3, the number of augmented views, and the number prompt update steps across diverse datasets. We choose 5 datascts: DTD [6. StanfordCars 20),, UCF101 35 and Caltech101 (10), and ImageNet-A [16], and compare the average accuracy across them as shown in Figure
Table 14: Evaluation the effect of more samples for distribution alignment. PromptAlign'
utilizes bag-of-samples from the same class for the distibution alignment.
Caltech
Flowers
Food101
Aircraft
SUN397
EuroSAT
UCF101
Average
Pets
Cars
DTD
PromptAlign
94.01
90.76
68.50
72.39
86.65
24.80
67.54
47.24
47.86
69.47
66.92
PromptAlign'
96.12
91.60
74.00
74.49
87.67
25.68
68.83
51.52
50.86
74.81
69.59
17

---

The loss scaling factor plateaus close to 100, whereas the performance increases with the increasing number of augmented views and prompt update steps. As discussed in Section the token distribution alignment scaling factor /3 set to 100 using the ImageNet validation set, which used experiments.
87.2
20- average accuracy
67.0
67.0
67.7
67.6
66.8
67.5
2006
06.4
66.2
Mal T
Promptalign
67.2
66.0
50
300
150
200
65.0
16
32
64
120
67.1
Weight sca factor
Number of sugmented views
Namber prompt upcate steps
Figure 8: Effect of loss scaling factor 3 (left), number of augmented views (center), and number of prompt update steps (right) averaged across the 5 datasets. The loss scaling factor trend is similar to that on ImageNet validation set, and the average accuracy increases consistently with the number of augmented views and the number prompt update steps.
E Analysis of losses for distribution alignment
The comparison of different alignment loss functions is presented in of the main paper. All three losses improve top of the test-time prompting approach. In terms of a test-time setting. the LI loss appears better suited equally valuing all errors opposed to L2 which primarily penalizes outliers. Moreover, the KL divergence loss docs enforce strict distribution alignment hence leading to sub-par performance. We further show a comparative analysis of alignment loss gradients Table Each row indicates the ratio the average gradient in comparison to the gradient when using cach of the other losses. In our experiments we notice that the shift in the statistical measures are small, which would lead to very small gradients in the case of L.2 and KL divergence losses. This be observed from Table 13 Specifically, the gradient of the LI loss is almost 80 times greater than of evident that the LI Loss results in improved alignment, primarily due to its higher gradient values, indicating that prompt updates with an Ll penalty can yield better alignment.
loss and approximately times greater than that the divergence loss. It
Table 15: Comparison of different alignment losses. LI Loss yields better alignment due smaller deviation statistical measures, thus higher gradient values.
Li-Loss
L2-Loss
KL-Divergence
LI- Loss
81.40
3.95
L2-Loss
0.0123
1
0.048
KL-Divergence
0.25
20.5
F Limitations and Future Directions
As test-time adaptation strategy. PromptAlign improves CLIP performance but relies the proxy dataset statistics for aligning the train data distribution with the test sample statistics. Further, involves single update step during runtime to optimize the prompts. A potential future direction would be to explore the adaptation of PromptAlign to other downstream tasks such as object detection and segmentation.
18


Reviews:
[{"review_id": "GvruYi7wsP", "reviewer": "Reviewer_wdgw", "summary": "In this paper, the authors proposed a novel approach for enhancing test-time prompting with distribution alignment of Vision-Language (V-L) models for zero-shot generalization. A loss between the means and variances of the test sample and the proxy source dataset statistics was proposed to align the distributions during testing. The authors conducted several experiments on multiple datasets and showed improved performance over several baseline methods.", "questions": "Please refer to the weaknesses section for details.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Combining Test-Time Prompting and Distribution Alignment for Zero-Shot Generalization makes sense to me and the proposed alignment loss is also technically sound. \n2. The authors conducted extensive experiments and ablation studies to sufficiently validate the proposed approach over multiple baseline methods on several benchmarking datasets.\n3. Writing is good and easy to follow. The tables and figures are also easy to understand.", "weaknesses": "1. I am not sure the mean and variance of a single test example would be sufficient for distribution alignment even though multiple augmented views are used. Could this lead to over-fitting? Would there be further gain with more test examples?\n2. I am not sure how the proxy source dataset would affect the performance. Since LAION400M is more closer to the training data of CLIP, it could also be used as a proxy source dataset. It would be interesting to ablate the effect of ImageNet versus LAION400M as proxy source dataset.\n3. The authors use means and variances of tokens to align the distribution. I am wondering would these be sufficient or more sophisticated statistics or losses are needed for better alignment. I would expect more insights on why these two was used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Using higher-order statistics for alignment**\n\nHigher-order statistical alignment is a classical approach in test-time domain adaptation. However, since we align distributions computed using a small set of samples, we use the relatively simpler (mean and variance) measures for our alignment strategy. We evaluate the choice of higher-order statistical measures for the distribution alignment using the Central Moment Discrepancy measure [1]. However, we note that the increment is minimal thus using the mean and standard deviation for alignment.\n\n### Comparison of PromptAlign vs Higher order statistics for Alignment\n\n| Method|Flowers|DTD|Pets|Cars|UCF|Caltech|Food|SUN   |Aircraft|Eurosat| Avg |\n|-----|-----|---|---|---|---|-----|----|---|-----|-----|----|\n| PromptAlign | 72.39  | 47.24  | 90.76  | 68.5  | 69.47  | 94.01  | 86.65  | 67.54  | 24.80  | 47.88  |66.92 |\n| w/ CMD  | 72.36  | 47.24  | 90.74  | 68.3  | 69.48  | 94.04 |  86.65 | 67.85  |  24.75 | 48.50 |66.99 |\n\n\n## **Utilizing more samples for distribution alignment**\n\nWe experiment with the use of more samples for distribution alignment. For each test example, we use a bag of samples (N=5) obtained from the same class (while the class label is kept unknown) to estimate the test sample statistics. We present the results in the table below which shows that test sample statistics with more samples can be better approximated, improving the performance through the alignment objective. However, in the actual setting of test-time adaptation, only a single sample is available.\n\n### Bag of Samples Comparison\n\n| Method|Flowers|DTD|Pets|Cars|UCF|Caltech|Food|SUN   |Aircraft|Eurosat| Avg |\n|-----|-----|---|---|---|---|-----|----|---|-----|-----|----|\n| PromptAlign | 72.39  | 47.24  | 90.76  | 68.5  | 69.47  | 94.01  | 86.65  | 67.54  | 24.8  | 47.88  |66.92 |\n| PromptAlign*  | 74.49  | 51.52  | 91.60  | 74.00  | 74.81  | 96.12 |  87.67 | 68.83  |  25.68 | 50.86 |69.56 |\n\nPromptAlign* is fed with a bag of samples (N=5) along with the original test sample at the test time.\n\n## **Experiments with LAION400M dataset statistics**\n\nAs suggested, we ablate on the performance of the alignment strategy using LAION400M to compute the source data statistics. Given the time constraint, we use a subset of LAION400M with 2.5 million images (2x of ImageNet). The similarity of LAION400M to the CLIP training dataset yields a better statistical measure of the distributions, thus improving the distribution alignment objective and thus the performance. Given the large scale of the LAION400M dataset and since CLIP achieves excellent zero-shot performance for ImageNet, we found ImageNet as a feasible choice as the proxy source dataset which is consistent with previous work [2]. We will include these results in the paper. \n\n### LAION vs. Imagenet statistics for distribution alignment loss.\n\n| Method|Flowers|DTD|Pets|Cars|UCF|Caltech|Food|SUN   |Aircraft|Eurosat| Avg |\n|-----|-----|---|---|---|---|-----|----|---|-----|-----|----|\n| ImageNet | 72.39  | 47.24  | 90.76  | 68.5  | 69.47  | 94.01  | 86.65  | 67.54  | 24.8  | 47.88  |66.92 |\n| LAION  | 72.38  | 47.97  | 90.76  | 68.11  | 70.2  | 94.32 |  87.26 | 68.02  |  24.78 | 47.88 |67.17 |\n\n[1]: Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning\n\n[2]: Bahng, Hyojin, et al. \"Exploring visual prompts for adapting large-scale models.\" arXiv preprint arXiv:2203.17274 (2022).\n"}, {"review_id": "HJ0a6zZZ5X", "reviewer": "Reviewer_H4M3", "summary": "This paper introduces a distribution alignment strategy, termed PromptAlign, for V-L models to improve test-time adaptation. To be specific, they propose a distribution alignment loss that utilizes offline computed source data statistics to encourage the test sample token distributions to be aligned with the source data token distributions. Since CLIP-pre-training data is not publicly released, they directly use ImageNet as a possible candidate for the source distribution. Experiments on multiple benchmark datasets reveal the zero-shot generalization ability of the proposed PromptAlign during test-time adaptation.", "questions": "Please refer to the weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written with a clear structure. The research on prompting for large Vision-Language models holds practical value.\n\n- The proposed PromptAlign, with its straightforward implementation, has achieved good results on a variety of test benchmarks.", "weaknesses": "- *Proxy source dataset*. Although using an off-the-shelf large-scale dataset for distribution alignment is intuitive. It lacks theoretical justifications or insights regarding the choice, making the whole process somewhat heuristic. Moreover, as the statistics of the source dataset are computed offline, how can the method be applied to some dynamically changing environments that may significantly differ from the source data distribution? \n\n- *Distribution alignment*. Using mean and variance for distribution alignment still lacks theoretical or more in-depth analyses. This appears to be the foundation of the paper, yet I haven't observed a particularly robust theoretical explanation. Could the authors provide more insights regarding this point?\n\n- *Experimental results*. Based on the reported experimental results, the proposed method does not demonstrate a significant performance improvement compared to TPT-based methods. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **ImageNet as proxy source dataset and precomputed source data statistics**\n\nGiven the excellent zero-shot performance of CLIP on ImageNet we choose ImageNet as a feasible proxy source dataset, which is consistent with previous work [1]. Additionally, we conduct an ablation study on the alignment strategy, employing LAION400M as the source dataset, which has been shown to reflect the training dataset of CLIP [2]. Due to time constraints, we utilize a subset of LAION400M, containing 2.5 million images (2x of ImageNet size). Notably, the performance impact of using ImageNet does not considerably change when employing this subset of LAION400M. We will include these results in the main paper.\n\n### LAION vs. ImageNet statistics for alignment loss.\n\n| Method|Flowers|DTD|Pets|Cars|UCF|Caltech|Food|SUN   |Aircraft|Eurosat| Avg |\n|-----|-----|---|---|---|---|-----|----|---|-----|-----|----|\n| ImageNet | 72.39  | 47.24  | 90.76  | 68.5  | 69.47  | 94.01  | 86.65  | 67.54  | 24.8  | 47.88  |66.92 |\n| LAION  | 72.38  | 47.97  | 90.76  | 68.11  | 70.2  | 94.32 |  87.26 | 68.02  |  24.78 | 47.88 |67.17 |\n\nThe source data statistics are computed on the proxy source data to obtain the distribution of tokens that are used for the alignment during test time. Since this proxy dataset is based on the similarity to the training set of the model, this offline computation will not change regardless of the changing environment and only needs to be computed once.\n\n## **Distribution alignment strategy**\nHigher-order statistical alignment is a classical approach in test-time domain adaptation. However, since we align distributions computed using a small set of samples, we use the relatively simpler (mean and variance) measures for our alignment strategy. We evaluate the choice of higher-order statistical measures for the distribution alignment using the Central Moment Discrepancy measure [3]. However, as shown in the Table below, we note that the increment is minimal which validates that the mean and variance based approach is a suitable measure for alignment in our case.\n\n### Comparison of PromptAlign vs Higher order statistics for Alignment\n\n| Method|Flowers|DTD|Pets|Cars|UCF|Caltech|Food|SUN   |Aircraft|Eurosat| Avg |\n|-----|-----|---|---|---|---|-----|----|---|-----|-----|----|\n| PromptAlign | 72.39  | 47.24  | 90.76  | 68.5  | 69.47  | 94.01  | 86.65  | 67.54  | 24.80  | 47.88  |66.92 |\n| w/ CMD  | 72.36  | 47.24  | 90.74  | 68.3  | 69.48  | 94.04 |  86.65 | 67.85  |  24.75 | 48.50 |66.99 |\n\n## **Experimental results**\nThe proposed method is a lightweight adaptation strategy for V-L models of which the improvements are also appreciated by reviewers wdgw, JL6L. Being a test-time adaptation method using a single sample, the improvements are consistent across multiple datasets and benchmarks (domain generalization and cross-datasets transfer) and improve over existing works such as [4] which performs test-time prompt tuning on a single sample. \n\nIn addition, we also evaluate our method in the Base to Novel benchmark, where the model is trained on the base class training set and tested on both the test sets of base classes and novel classes. As reported in Table 3 in the global rebuttal response, averaged across 11 datasets, our method shows consistent improvements with +0.95 on the base classes and +0.79 on novel classes whereas, without the alignment strategy, the performance drops below the baseline.\n\n[1]: Bahng, Hyojin, et al. \"Exploring visual prompts for adapting large-scale models.\" arXiv preprint arXiv:2203.17274 (2022).\n\n[2]: Cherti, Mehdi, et al. \"Reproducible scaling laws for contrastive language-image learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3]: Zellinger, Werner, et al. \"Central moment discrepancy (cmd) for domain-invariant representation learning.\" arXiv preprint arXiv:1702.08811 (2017).\n\n[4]: Shu, Manli, et al. \"Test-time prompt tuning for zero-shot generalization in vision-language models.\" Advances in Neural Information Processing Systems 35 (2022): 14274-14289."}, {"review_id": "mpdYYqp62Y", "reviewer": "Reviewer_ftF2", "summary": "Building on top of the prompting method and architecture from MaPLE and test-time adaptation strategy of TPT, the work ads an additional alignment penalty, based on pre-computed statistics. The statistics are computed offline on a target source dataset. The method is evaluated then for domain generalisation on a set of datasets.", "questions": "See the above section", "limitations": "Adequate limitation statement is present.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper reads well and its easy to follow.\n- Table 1 reports gains compared with the TPT method baseline.\n- The conept of test-time adaptation is interesting.", "weaknesses": "- Somewhat low novelty: The method combines 2 existing works: takes the architecture and prompting strategy from MaPLE and the adaptation mechanism from TPT. The only addition is an alignment loss based on pre-computed statistics.\n- Can this approach be combined with other methods (ie. CoOp, CoCoOp etc), similarly as its the case for TPT. If so, what is its performance? Further evaluations are needed in this direction.\n- Similarly, the comparison from Table 2 and 3 are unfair. The proposed method should be compared with a similar baseline, i.e. TPT+MaPLE, as done in Table 1. Currently, especially for Table 3 its unclear what are the gains, if any.\n- Missing comparisons with current state-of-the-art methods (e.g. Neural Prompt Search, LASP, ProDA etc). \n- No evaluation on the standard settings from CoOp (i.e. without ImageNet pretraining) and CoCoOp( base-to-new setting).\n- Missing analysis of certain ablation. For example, why is L1 offering the best results in this case? What makes it more suitable?\n- What is the impact of the quality of the statistics on the overall performance?\n- No FLOPs/compute analysis: What is the inference time and FLOPs (i.e. including adaptation time) per sample? How does this compare with TPT? How much extra time is spent on pre-computing the statistics? It looks than in some case ity may be simply faster to train and eval CoOp or a similar method, especially as the size of the dataset and/or the number of classes increases.\n- L154, sum from 1 to K, where is the index i used on the denominator? It's a bit ambigous currently which f_t abd f_v are used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Contributions and Novelty**\nKindly note that the multimodal prompting architecture of MaPLe and the adaptation strategy are independently pre-existing methods. However, the holistic combination of them using the proposed distribution alignment strategy to effectively use the synergy in V-L models for test-time adaptation is novel to the best of our knowledge. Further, the proposed method consistently improves on top of the individual methods across extensive benchmarks including a newly reported base-to-novel generalization benchmark. The test-time adaptation strategy using a single sample, consistently improving across numerous benchmarks showcases the effectiveness of the proposed method.\n\n## **Combining PromptAlign with other prompting methods**\nThe existing prompt learning methods such as CoOp and CoCoOp learn prompts only on the language side.\nIn the test-time adaptation setting, the prompt vectors are reset to the initial states for each incoming sample. For a given dataset, since the class names and the prompt vectors are the same, the tokens throughout the text encoder will be the same. Therefore, the mean would be the token itself and the variance zero, thus leading to no contribution from a distribution alignment loss. This architectural limitation of text-side-only prompting prevents naive incorporation of a distribution alignment. However, through multimodal prompting, the distribution alignment on the vision branch allows prompts on both branches to be updated together.\n\n## **Main paper table comparisons**\nWe update the comparisons in Table 2  (in the global rebuttal response) on the cross-dataset evaluation with MaPLe, MaPLe+TPT, and our method PromptAlign. It can be noted that the gain from our alignment strategy on top of MaPLe+TPT is relatively higher than the improvement of TPT on MaPLe. We further incorporate the performance of our method with the source dataset statistics computed using a subset of LAION400M with 2.5 Million images (2x of ImageNet), since LAION400M is closer to the training dataset of CLIP [1]. These results further validate our method where the performance improves with a better proxy source dataset. \n\n## **Comparison to the state-of-the-art methods**\nWe report comparisons to ProDA from the original paper [2] since the official code is not publicly available. The cross-dataset evaluation is reported in Table 2 and the base to novel benchmark settings, in Table 3. Both Neural Prompt Search and LASP works have not been published at the time of this submission.\n\n## **Comparisons with the standard setting of CoOp and CoCoOp base to novel benchmark**\nWe compare the standard CoOp setting without ImageNet pretraining in the cross-dataset setting (mentioned as CLIP+TPT) in Table 2. We also evaluate our method in the base to novel benchmark setting in Table 3 in comparison to the state-of-the-art method MaPLe. We show that our alignment strategy improves upon MaPLe in both base and novel class settings by +0.95 and +0.79 respectively, whereas TPT alone drops in performance slightly. \n\n## **Analysis of matching losses for Distribution Alignment**\nThe comparison of different alignment loss functions is presented in Figure 4 of the main paper. All three losses improve on top of the test-time prompting approach. In terms of a test-time setting, the L1 loss appears better suited to equally valuing all errors as opposed to L2 which primarily penalizes outliers. Moreover, the KL divergence loss does not enforce strict distribution alignment hence leading to sub-par performance. Further analysis of this will be included in the final draft of the paper.\n\n## **Impact of statistics quality on performance**\nWe evaluate the quality of statistics in terms of the pre-computed source data statistics and the test sample statistics. We use a subset of LAION400M with 2.5 Million images as the proxy source dataset to compute the offline statistics. This further improves the alignment objective (Table 2, last row), suggesting that better quality in the source data statistics improves the alignment objective further. \n\nTo evaluate the effect of the quality to test sample statistics, we conduct the following experiment. We randomly sample 5 images from the same class of the test sample (while the class label is kept hidden). We use these five samples to calculate the statistics of the test sample and perform test-time adaptation. The results in Table 1 reflect that the performance increases with better quality in test-sample statistics as well.\n\n## **Compute Analysis**\nAs recommended, we report the compute cost in terms of time and FLOPS including the adaptation step, in comparison to TPT in Table 4. Figure 5(c) in the submission paper denotes the compute time averaged across three seeds for the ImageNet-A dataset, for different number of prompt update steps. In comparison to the average inference time and FLOPS per sample, our method with the alignment strategy has minimal overhead. \n\nThe offline pre-computation of source data statistics requires 2 forward passes of the ImageNet training set which takes nearly 3h. First, the mean is calculated and in the second pass the variance is computed using the mean that has been computed. However, this is a single-time offline computation that need not be repeated again which does not depend on the test dataset size nor the number of classes. Furthermore, since we consider the test-time adaptation setting using a single sample, we presume no training data to be available for our adaptation.\n\n## **Minor correction in L154**\nThank you for pointing out this error. This has been rectified. Here, 'i' refers to the index of the class, which has been now added.\n\n[1]: \"Reproducible scaling laws for contrastive language-image learning.\" CVPR 2023.\n[2]: \"Exploring visual prompts for adapting large-scale models.\" arXiv preprint arXiv:2203.17274 (2022).\n[3]: \"Prompt distribution learning.\" Proceedings of the IEEE/CVF CVPR. 2022."}, {"review_id": "3KhBqVug1Y", "reviewer": "Reviewer_JL6L", "summary": "This paper tackles the task of test-time prompt tuning. It proposes to align the out-of-distribution test sample statistics to those of the source data. The ImageNet is used as an effective proxy source dataset. Evaluation results show that the proposed method obtains better performances on domain generalization and cross-dataset generalization tasks. ", "questions": "1. In Tab.4, is PromptAlign$^\\dagger$ equivalent to MaPLe+alignment loss? If the alignment loss alone does not improve test-time adaptation, how to validate that narrowing the distribution gap real matters when adapting with entropy loss?\n\n2. How does the proposed method perform on different CLIP backbone architectures?", "limitations": "The authors have included possible limitations in the end of paper. The discussions are constructive.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies test-time adaptation for foundation models. Given the popularity of V-L models and prompt tuning, the paper may be interesting to many researches.\n\n2. Exploring distribution alignment in V-L models at test time is new. The proposed method is easy to implement and can be generalized to different model architectures. \n\n3. The paper is well structured and easy to follow. Experiments are extensive enough. PromptAlign obtains consistently better accuracies than previous methods in most cases.", "weaknesses": "1. The method highly relies on the ImageNet statistics. In the supplementary, the authors show that using ImageNet statistics gives better performance than using other small datasets. It is unclear if this also applies to V-L models other than CLIP.\n\n2. The alignment loss, in my understanding, is a regularization term on the visual prompts to avoid large devation during adaptation. A simple manner is to regularize the difference between initial prompt values (or statistics from the test image) and adapted values. Such comparison is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Effect of proxy source dataset on other VL models**\n\nWe evaluate our method by implementing the MaPLe architecture in the recently introduced EVA-CLIP(CVPR 2023) V-L model [1]. We conduct the same experiment reported in Table 6 of the paper on the EVA-CLIP model. We show that the use of ImageNet as the training dataset to compute source statistics is still relatively better than the use of the training data statistics for distribution alignment. \n\n### Comparison of the effect of the proxy dataset against, the training dataset of the dataset for EVA-CLIP.\n\n| Method | Caltech | Cars | UCF101 | Aircraft | DTD |\n|-----|-----|---|---|---|---|\n| TrainSet | 95.66 | 79.44 | 65.39 | 23.48  | 50.06  |\n| ImageNet | 95.69 | 79.50 | 65.41  | 23.57  | 50.06  |\n\n\n## **Replacing PromptAlign with naive regularization technique**\nAs suggested we experiment with the simpler regularization of prompts based on the previous update of the prompts. This results in a continuous test-time adaptation setting for which we choose a lower learning rate to ensure the prompts do not diverge. While effective, we notice that such a simpler regularization is not consistent and overall drops in performance in comparison to baseline MaPLe (predominantly due to the large drop in the EuroSAT dataset). Regardless, our proposed alignment strategy is better across all datasets. PromptAlign* in the below Table is the suggested naive prompt regularization method.\n\n### Comparison against prompt regularization using continuous adaptation.\n\n| Method|Flowers|DTD|Pets|Cars|UCF|Caltech|Food|SUN   |Aircraft|Eurosat| Average |\n|-----|-----|---|---|---|---|-----|----|---|-----|-----|----|\n| MaPLe | 72.39  | 47.24  | 90.76  | 68.5  | 69.47  | 94.01  | 86.65  | 67.54  | 24.80  | 47.88  |66.92 |\n| PromptAlign*  | 72.05 | 46.24 | 90.60 |\t65.02 |\t68.23 |\t93.71 |\t86.05 |\t65.46 |\t22.32 | 19.03 | 62.87 |\n| PromptAlign  | 72.38  | 47.97  | 90.76  | 68.11  | 70.2  | 94.32 |  87.26 | 68.02  |  24.78 | 47.88 |67.17 |\n\n## **Mutual synergy between Distribution alignment and Entropy objectives**\nYes, PromptAlign^{dagger} in Table 4 in the paper refers to MaPLe with only the distribution alignment loss (i.e. without the entropy minimization objective). The proposed alignment strategy treats each token as a random variable, where the distribution of the source variable is matched for the test sample's distribution. This regularization works in synergy with the entropy minimization objective and improves the learning objective. Unlike entropy minimization, the alignment objective does not promote any discriminative learning, and thus cannot be solely used for test-time adaptation.\n\n## **Different CLIP backbone architecture**\nWe train MaPLe on the ViT-B/32 architecture and evaluate our method on the cross-dataset and domain generalization settings. The results are consistent, showing a similar trend in performance. We thank the reviewer for the suggestion, and we will include this in the final draft of the paper. \n\n### Cross Dataset benchmark evaluation on ViT-B/32 backbone\n\n| Method|Flowers|DTD|Pets|Cars|UCF|Caltech|Food|SUN   |Aircraft|Eurosat| Average |\n|-----|-----|---|---|---|---|-----|----|---|-----|-----|----|\n| MaPLe | 65.33 |\t41.70 |\t88.13 |\t59.93 |\t63.63 |\t92.50 |\t81.00 |\t65.00 |\t17.53 |\t40.80 |\t61.56 |\n| MaPLe+TPT  | 66.08 |\t40.01 |\t88.47 |\t59.35 |\t64.40 |\t91.44 |\t82.08 |\t66.07 |\t18.71 |\t39.67 |\t61.63 |\n| PromptAlign  | 66.14 |\t42.54 |\t88.44 |\t63.48 |\t65.57 |\t92.10 |\t82.07 | 66.08 |\t18.76 |\t39.68 |\t62.49 |\n\n###  Domain Generalization evaluation on ViT-B/32 backbone\n| Method | ImageNet | ImageNet A | ImageNet R | ImageNet K | ImageNet V2 | OOD Average |\n|-----|-----|----|---|-----|-----|----|\n| MaPLe | 65.27 |32.12 |67.64 |42.15|\t57.63 |\t49.89 |\n| MaPLe+TPT  | 67.16 | \t37.52 |\t71.11 |\t43.77 |\t60.01 |\t53.10 |\n| PromptAlign  | 67.42 | 38.02 | 71.44 |44.24 | 60.43 |\t53.53 |\n"}]

Author rebuttal:
We sincerely thank the reviewers (wdgw, H4M3, ftF2, JL6L) for their detailed and positive feedback. The idea of improving test-time adaptation is interesting (ftF2), holds practical value (H4M3), and the proposed distribution alignment in V-L models at test time is novel (JL6L) and technically sound (wdgw). The proposed method is extensively evaluated on an array of benchmarks (wdgw, H4M3, JL6L) and achieves good results on various settings (H4M3, ftF2, JL6L). The paper is well-written and easy to read (wdgw, H4M3, ftF2, JL6L).

We validate the distribution alignment strategy further in terms of the statistical measures used for alignment, and the effect of improved statistics using a bag of samples to estimate test-sample statistics. Table 1 provides the results of using a bag of samples for our test-time adaptation strategy. We also analyze the effect of the proxy source dataset compared to the use of LAION400M and validate the choice of ImageNet as a valid proxy source dataset. The extended cross-dataset evaluation including LAION statistics for alignment is reported in Table 2. In addition, we extend our method to the base-to-novel benchmark setting and show consistent improvement in this setting as well, in Table 3. We also report the computational cost in terms of time and FLOPS for comparison with the baseline Test-Time Prompt Tuning [1] in Table 4. Our code and pre-trained models will be publicly released. All suggested changes will be reflected in the paper.

### Table 1. Bag of Samples Comparison

| Method|Flowers|DTD|Pets|Cars|UCF|Caltech|Food|SUN   |Aircraft|Eurosat| Avg |
|-----|-----|---|---|---|---|-----|----|---|-----|-----|----|
| PrompAlign | 72.39  | 47.24  | 90.76  | 68.5  | 69.47  | 94.01  | 86.65  | 67.54  | 24.8  | 47.88  |66.92 |
| PromptAlign*  | 74.49  | 51.52  | 91.60  | 74.00  | 74.81  | 96.12 |  87.67 | 68.83  |  25.68 | 50.86 |69.56 |

### Table 2. Cross-dataset benchmark evaluation.

| Method | Caltech | Pets | Cars | Flowers | Food | Aircraft | SUN | DTD  | Eurosat |UCF| Average |
|-----|-----|---|---|---|---|-----|----|---|-----|-----|----|
| CLIP |  93.35 | 88.25 | 65.48 | 67.44 | 83.65 | 23.67 | 62.59 | 44.27 | 42.01 | 65.13 | 63.58 |
| CLIP+TPT | 94.16 | 87.79 | 66.87 | 68.98 | 84.67 | 24.78 | 65.50 | 47.75 | 42.44 | 68.04 | 65.10 |
| CoOp | 93.70 | 89.14 | 64.51 | 68.71 | 85.30 | 18.47 | 64.15 | 41.92 | 46.39 | 66.55 | 63.88 |
| CoCoOp | 93.79 | 90.46 | 64.90 | 70.85 | 83.97 | 22.29 | 66.89 | 45.45 | 39.23 | 68.44 | 64.63 |
| ProDA | 86.70 | 88.20 | 60.10 | 77.50 | 80.80 | 22.20 | - | 50.90 | 58.50 | - | 65.62 |
| MaPLe | 93.53 | 90.49 | 65.57 | 72.23 | 86.20 | 24.74 | 67.01 | 46.49 | 48.06 | 68.69 | 66.30 |
| MaPLe+TPT | 93.59 | 90.72 | 66.50 | 72.37 | 86.64 | 24.70 | 67.54 | 45.87 | 47.84 | 69.19  |66.92 |
| PromptAlign | 94.01 | 90.76 | 68.50 | 72.39 | 86.65 | 24.80 |67.54 | 47.24 | 47.86 | 69.47 | 66.92 |
| PromptAlign*  | 94.32 | 90.76 | 68.11 | 72.38 | 87.26 |24.78 | 68.02 | 47.97 | 47.88 | 70.20 | 67.17 |

PromptAlign* refers to LAION400M being used as the proxy source dataset

### Table 3. Base to novel benchmark evaluation

|   Dataset   | Split | ProDA | MaPLe |MaPLe+TPT| PromptAlign |
|------|------|------|-----|-----|------|
| ImageNet | Base  | 75.40 | 76.67 |	77.73 |	78.26  |
|  | Novel  | 70.23 | 70.54 |	72.24 |	72.59  |
| Flowers | Base  | 97.70 | 95.93 |	96.24 |	96.61 |
|  | Novel  | 68.68 | 72.40 |	72.10 |	72.34  |
| DTD | Base  | 80.67| 80.30 |	82.20 |	82.60  |
|  | Novel  | 56.48 | 59.23 |	59.91 |	60.55 |
| Pets | Base  | 95.43 | 95.43 | 95.23 | 95.38  |
|  | Novel  | 97.83 | 97.80 |	97.37 |	97.56  |
| Cars | Base  | 74.70 | 72.90 | 74.00 |	75.02  |
|  | Novel  | 71.20 | 73.97 |	75.20 |	75.71  |
| UCF101 | Base  | 85.23 | 82.97 | 82.23 |	84.11  |
|  | Novel  | 71.97 | 78.57 | 77.34 |	79.30  |
| Caltech101 | Base  | 98.27 | 98.00 |	98.54 |	98.60  |
|  | Novel  | 93.23 | 94.27 | 94.29 |	94.50  |
| Food101 | Base  | 90.30 | 90.70 |	91.13 |	91.63  |
|  | Novel  | 88.57 | 92.07 |	92.03 |	92.68  |
| Sun397 | Base  | 78.67 | 80.80 |	81.15 |	81.57  |
|  | Novel  | 76.93 | 78.70 |	79.18 | 79.48  |
| Aircraft | Base  | 36.90 | 37.27 |	34.31 |	37.21  |
|  | Novel  | 34.13 | 35.53 |	35.81 |	37.27 |
| Eurosat | Base  | 83.90 | 93.63 |	91.02 |	94.10  |
|  | Novel  | 66.00 | 72.87 |	68.96 |	72.71  |
| Average | Base  | 81.56 | 82.24 |	82.16 |	83.19  (+0.95)|
|  | Novel  | 72.30 | 75.09 |	74.95 |	75.88 (+0.79)|

### Table 4. Comparison of FLOPS and inference time. 

|   Method   | GFLOPS | Inference time (secs) |
|------|------|------|
| MaPLe+TPT | 1751  | 0.197 |
| PromptAlign | 1752  | 0.216 |

[1]: Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." Advances in Neural Information Processing Systems 35 (2022): 14274-14289.

Meta-review: This paper received slightly diverging reviews. Reviewer ftF2 expressed some concerns from different aspects whereas the rest 3 reviewers acknowledge the contribution of this work overall. Some of the concerns from ftF2 were addressed after rebuttal but the main concern on novelty (similarity to TPT and the reuse of existing components) remains. Upon reading through the paper and all the discussions, the AC feels that this work contains enough contributions to be accepted despite not being groundbreaking. 1) The authors have done a decent set of experiments, and the results look promising. The effectiveness of the proposed method is thus sufficiently validated. 2) Although this is not the first work on TPT, the AC considers TPT a general problem setting rather than a specific method. Method wise, this work seems a novel application of CMD/Deep Coral to TPT. In light of this consideration, the AC recommends acceptance.

The decision of this paper was communicated with the SAC, and was made based on the discussion.
Decision: Accept (poster)