SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Table 13 we present the performance of our method when using the training set of the respective dataset as the proxy source data to compute the training statistics used for the token distribution alignment loss. We choose 5 datasets: DTD [6], StanfordCars [20], UCF10 1[35] Caltech101 [10], and FGVC Aircrafts [24] for the analysis. We observe that using ImageNet as the proxy source dataset gives better performance across all datasets, except in UCF101, where the performance is the same. Experiments in different Vision-Language models such as EVA-CLIP [9], with different training recipes has a similar trend validating the choice of ImageNet as a proxy dataset.
PromptAlign (Ours)
Figure 7: Comparison of attention map visualizations. PromptAlign focuses on highlighting the discriminative areas while reducing attention on background regions as opposed to TPT.
PromptAlign (Ours)
* Due to compute and time limitations we use a subset of LAION400M dataset
Table 12: Experiment with different source dataset statistics. PromptAlign† refers to source dataset statistics computed using a subset of LAION400M for distribution alignment.
Caltech101	OxfordPnts	StanfordCars	Flowers102	Food101	Aircraft	SUN397	DTD	EuroSAT	UCF101	Average
PromptAlign	94.01	90.76	68.50	72.39	86.65	24.80	67.54	47.24	47.86	69.47	66.92
PromptAlign†	94.32	90.76	68.11	72.38	87.26	24.78	68.02	47.97	47.88	70.20	67.17
Table 13: Comparison of the effect of proxy source dataset in cross-dataset evaluation for different V-L models. PromptAlign† refers to PromptAlign with the training set of the respective test dataset used as the proxy source dataset to compute statistics for the alignment loss.
Model	Method	Target
Caltech101	StanfordCars	UCF101	Aircraft	DTD
CLIP	PromptAlign‡	93.98	68.44	69.47	23.85	47.10
PromptAlign	94.01	68.50	69.47	24.80	47.24
EVA-CLIP	PromptAlign‡	95.66	79.44	65.39	23.57	50.06
PromptAlign	95.69	79.50	65.41	23.48	50.06
Our hypothesis on the effectiveness of ImageNet as a proxy source dataset is two-fold. Firstly, the token distribution alignment loss intends to align the tokens to that of the CLIP pre-training dataset token distributions. The larger scale of ImageNet in comparison to the training set of the above datasets, and the CLIP pre-training strategy, finetuning it to improve performance on the ImageNet benchmark [1] renders ImageNet a better proxy source dataset for CLIP. Furthermore, using ImageNet acts as a regularizer, maintaining the generalization capability of CLIP while performing test-time prompt learning using entropy minimization. Similarly, since LAION400M is a better representation of the CLIP training dataset, the subset of LAION400M used as the proxy shows an improvement over ImageNet in being used as a proxy source dataset.
D.2 Effect of test sample statistics
To analyze the effect of the test sample statistics quality on the distribution alignment strategy, we experiment with the use of more samples for alignment. For each test example, we use a bag of samples  $(N = 5)$  obtained from the same class (while the class label is kept unknown) to estimate the test sample statistics. We present the results in Table 14 which shows that test sample statistics can be better approximated with more samples, improving the performance through the alignment objective. However, in the actual setting of test-time adaptation, only a single sample is available.
D.3 Trend across parameter choices
We present ablations on the distribution loss scaling factor  $\beta$ , the number of augmented views, and the number of prompt update steps across diverse datasets. We choose 5 datasets: DTD [6], StanfordCars [20], UCF101 [35] and Caltech101 [10], and ImageNet-A [16], and compare the average accuracy across them as shown in Figure 8.
Table 14: Evaluation of the effect of more samples for distribution alignment. PromptAlign* utilizes a bag-of-samples from the same class for the distribution alignment.
Caltech	Pets	Cars	Flowers	Food101	Aircraft	SUN397	DTD	EuroSAT	UCF101	Average
PromptAlign	94.01	90.76	68.50	72.39	86.65	24.80	67.54	47.24	47.86	69.47	66.92
PromptAlign*	96.12	91.60	74.00	74.49	87.67	25.68	68.83	51.52	50.86	74.81	69.59
The loss scaling factor  $\beta$  plateaus close to 100, whereas the performance increases with the increasing number of augmented views and prompt update steps. As discussed in Section 5, the token distribution alignment scaling factor  $\beta$  is set to 100 using the ImageNet validation set, which is used for all experiments.
Figure 8: Effect of loss scaling factor  $\beta$  (left), number of augmented views (center), and number of prompt update steps (right) averaged across the 5 datasets. The loss scaling factor trend is similar to that on ImageNet validation set, and the average accuracy increases consistently with the number of augmented views and the number of prompt update steps.
E Analysis of losses for distribution alignment
The comparison of different alignment loss functions is presented in Fig 4 of the main paper. All three losses improve on top of the test-time prompting approach. In terms of a test-time setting, the L1 loss appears better suited to equally valuing all errors as opposed to L2 which primarily penalizes outliers. Moreover, the KL divergence loss does not enforce strict distribution alignment hence leading to sub-par performance. We further show a comparative analysis of alignment loss gradients in Table 15. Each row indicates the ratio of the average gradient in comparison to the gradient when using each of the other losses. In our experiments we notice that the shift in the statistical measures are small, which would lead to very small gradients in the case of L2 and KL divergence losses. This can be observed from Table 15. Specifically, the gradient of the L1 loss is almost 80 times greater than that of the L2 loss and approximately 4 times greater than that of the KL divergence loss. It is evident that the L1 Loss results in improved alignment, primarily due to its higher gradient values, indicating that prompt updates with an L1 penalty can yield better alignment.
Table 15: Comparison of different alignment losses. L1 Loss yields better alignment due smaller deviation in statistical measures, thus higher gradient values.
L1-Loss	L2-Loss	KL-Divergence
L1-Loss	1	81.40	3.95
L2-Loss	0.0123	1	0.048
KL-Divergence	0.25	20.5	1
F Limitations and Future Directions
As a test-time adaptation strategy, PromptAlign improves CLIP performance but relies on the proxy source dataset statistics for aligning the train data distribution with the test sample statistics. Further, it involves a single update step during runtime to optimize the prompts. A potential future direction would be to explore the adaptation of PromptAlign to other downstream tasks such as object detection and segmentation.