SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation Details. In all experiments, we use the publicly available CLIP model with the ResNet-50 [39] visual encoder as the backbone model. Unless otherwise specified, the prompt is initialized randomly with 4 learnable tokens in SwapPrompt, UPL and CoOp. As for TPT, the prompt is initialized as the default one "a photo of a". When comparing the performance with baselines, we select the top 16 test data with the highest confidence to train SwapPrompt and UPL. For SwapPrompt, the decay rate of target prompt is 0.99, both  $\alpha$  and  $\beta$  are 1. We use the same image augmentation
Table 1: Comparison of test-time adaptation methods on 14 datasets.  $\Delta$  denotes SwapPrompt's gain over the better one of UPL and TPT.  $^+$  Online' denotes SwapPrompt with online test data.
Method	Caltech101	DTD	Flowers102	Oxford-Pets	UCF101	StanfordCars	Food101	EuroSAT	SUN397	ImageNet	ImageNet-V2	ImageNet-A	ImageNet-R	ImageNet-Sketch
CoOp [17]	88.76	54.62	83.98	87.44	66.71	61.83	73.79	61.68	64.33	61.23	55.29	23.41	56.96	35.64
CLIP [16]	85.13	42.16	65.40	83.05	61.15	55.65	74.23	37.60	58.55	58.18	51.36	21.69	55.98	33.33
UPL [24]	86.37	45.04	67.11	88.53	63.63	58.46	74.38	41.40	61.07	61.19	52.07	23.59	57.09	36.40
TPT [19]	87.22	42.17	65.42	84.60	61.18	58.49	74.88	43.82	61.46	60.74	54.35	26.24	58.72	35.02
SwapPrompt Δ	89.90	47.34	70.22	89.14	65.66	59.60	75.08	46.64	63.93	61.80	53.94	24.46	60.88	38.21
+2.68	+2.30	+3.11	+0.61	+2.03	+1.11	+0.20	+2.82	+2.47	+0.61	-0.41	-1.78	+2.16	+1.81
+ Online	89.69	46.40	68.12	88.97	64.52	58.88	75.66	42.45	63.36	61.41	52.93	24.42	60.25	38.13
method as SimCLR [40] to generate two different augmented images for an image. We optimize the prompts for 50 epochs with SGD optimizer and a cosine decay learning rate scheduler, the initial learning rate is 0.002. The batch size of images is 32 on all datasets.
We do all experiments on a workstation with an RTX 3090 GPU, a 3.5-GHZ Intel Core i9-11900K CPU and 64GB of RAM.
First, we compare our SwapPrompt with the baseline methods over fourteen benchmark datasets. The classification accuracy is listed in Table 1. It should be noted that CoOp is trained with labeled target domain data, and we use 4-shot data per category. From the results, it can be observed that our proposed SwapPrompt provides superior test-time adaptation performance than baselines on most datasets. We not only outperform the better baseline in UPL and TPT, but also very close even outperform CoOp on many datasets (e.g., Caltech101, Oxford-Pets, Food101, ImageNet, ImageNet-R and ImageNet-Sketch). Figure 3(a) shows that the average accuracy over 14 datasets for all baselines. SwapPrompt outperforms TPT and UPL by  $2.31\%$  and  $2.17\%$  accuracy, respectively.
Strong performance in online test-time adaptation setting. We also add results of SwapPrompt with online test data. Under this setting, we received mini-batches which only have a small part of data. For example, in DTD, a mini-batch has only 64 test data samples, less than  $4\%$  of the entire dataset. On most datasets, there is a slight decrease in accuracy because we cannot hold enough test data at the beginning of training to learn an appropriate prompt to classify the test data that come first. However, online SwapPrompt still outperforms UPL and TPT on most of datasets, as well as the average accuracy on all datasets in Figure 3(a). It should be noted that on Food101, the accuracy of online SwapPrompt is better. It is because the prompt in the intermediate stage is better than the one in the final stage. This could be attributed to the presence of high-confidence noise in pseudo labels during the final stage, which is discussed in section 4.3.
In this subsection, detailed analyses are shown to help understand the superiority of our SwapPrompt method, including the trade-off between accuracy and efficiency, analysis on objective functions  $L_{\mathrm{swap}}$  and  $L_{\mathrm{pseudo}}$ , the decay rate  $\epsilon$  of target prompt, the effect of  $K$  value in data selection, and the effect of prompt's context length and initialization.
The Trade-Off between Accuracy and Efficiency. The main factor which affects the efficiency of SwapPrompt is the prompt adaptation epoch. Figure 3(b) shows the relationship between the epoch and the average accuracy of SwapPrompt on 5 datasets (Caltech101, DTD, Flowers102, Oxford-Pets and UCF101). It can be seen that SwapPrompt's accuracy increases quickly in the first 3 epoch, then
(a) Average accuracy on 14 datasets
(b) Accuracy of different epochs on 5 datasets
Figure 3: (a) The average accuracy on all 14 datasets, CoOp is compared as an upper bound. (b) The average accuracy of SwapPrompt on 5 datasets with different adaptation epochs, the accuracy of UPL and TPT is the final epoch average accuracy.
reaches its highest value around epoch 20 and stabilizes there. Thus, when the test time is limited, SwapPrompt can do a trade-off between the accuracy and adaptation epochs, e.g., only train 3 epochs for a quick inference. Noteworthy, SwapPrompt outperforms the final epoch accuracy of UPL at only epoch 2, and the accuracy of TPT at epoch 1.
Analysis on Objective Functions. We evaluate the two objective functions  $L_{\mathrm{swap}}$  and  $L_{\mathrm{pseudo}}$  of SwapPrompt on 5 datasets. Results in Table 2 gives a clear ablation study to demonstrate the effectiveness of our proposed objective functions. First, We use the UPL as the basic baseline, which has the confident test data selection and objective function  $L_{\mathrm{pseudo}}$ . Then, UPL+AUG means that only adds image augmentation to the baseline, so that the  $L_{\mathrm{pseudo}}$  applies on 2 augmented image views. It can be observed that the accuracy has improved on all datasets, which demonstrates the benefits of data augmentation. Compare with SwapPrompt, UPL+Aug do not has the  $L_{\mathrm{swap}}$  function, and the performance is poor than SwapPrompt. Finally, the case of using all two objective functions, i.e., the complete SwapPrompt, has the best performance, which demonstrates that the loss function  $L_{\mathrm{swap}}$  for the swapped prediction mechanism can further improve prompt.
Table 2: Analysis of objective functions.
ImageNet	Caltech101	DTD	Flowers102	Oxford-Pets	UCF101	Average
UPL [24]	61.19	86.37	45.04	67.11	88.53	63.63	68.65
UPL+AUG	61.30	87.75	46.04	68.43	87.67	65.15	69.39
SwapPrompt	61.80	89.90	47.34	70.22	89.14	65.66	70.68
It should be noted that we do not include the result of using only  $L_{\mathrm{swap}}$ . Because considering the inherent generalization ability of CLIP, it is unfair to directly compare the performance of only  $L_{\mathrm{swap}}$  with  $L_{\mathrm{pseudo}}$ . The standalone application of  $L_{\mathrm{swap}}$  does not sufficiently leverage the rich pretrained knowledge embedded within CLIP. This can be seen as disregarding the pseudo-labels, which encapsulate the most pre-trained knowledge. Our proposed method, SwapPrompt, combines  $L_{\mathrm{swap}}$  and  $L_{\mathrm{pseudo}}$  resulting in improved performance compared to using  $L_{\mathrm{pseudo}}$  alone.
Analysis on the Decay Rate of Target Prompt. SwapPrompt updates the target prompt by a slow-moving average of online prompt, thus the target prompt represents a delay and more stable version of the online prompt, a higher decay rate indicates the retention of a greater amount of historical information. As is shown in Eq. 3, when the decay rate  $\epsilon$  is 0, the target prompt is instantaneously updated to the online prompt at each step. It should be noted that using only one online prompt on  $L_{\mathrm{swap}}$  loss (i.e.,  $L_{\mathrm{swap}}(x_i) = \ell(\mathbf{p}_i^1, \mathbf{q}_i^2) + \ell(\mathbf{p}_i^2, \mathbf{q}_i^1)$ ) cannot work because the gradient will collapse, the alternative is to maintain a target prompt that remains identical to the online prompt. (i.e.,  $\epsilon = 0$ ). When the decay rate  $\epsilon$  is 1, the target prompt is never updated, and remains at a constant value corresponding to its initialization. In this case we initialize the target prompt as "a photo of a", thus it still has basic zero-shot generalization ability. There is a trade-off between updating the targets too often and updating them too slowly.
Table 3 shows the results of different decay rates on 6 datasets. When  $\epsilon = 0$  and  $\epsilon = 1$ , the performance is poor than the other three eclectic values. All values of the decay rate between 0.9 and 0.999 yield have its best applicable datasets and 0.99 decay rate has a highest average accuracy. Besides that, compared with maintaining a fixed target prompt ( $\epsilon = 1$ ), the EMA strategy can increase the average accuracy by  $1.35\%$ .
Table 3: Results for different decay rates  $\epsilon$  .
ε value	ImageNet	Caltech101	DTD	Flowers102	Oxford-Pets	UCF101	Average
1	60.87	88.49	45.21	68.57	87.01	65.85	69.33
0.999	61.88	89.71	47.58	68.70	88.74	66.93	70.59
0.99	61.80	89.90	47.34	70.22	89.14	65.66	70.68
0.9	61.76	87.99	47.64	70.65	87.47	65.00	70.09
0	61.29	87.38	47.22	69.63	87.71	64.68	69.65
Analysis on the swapped prediction mechanism. Inspiring from the idea of self-supervised contrastive learning, SwapPrompt uses both image augmentation and EMA update strategies on target prompt. Both methods are effective and indispensable, and the combination of the two methods naturally leads to "swap prediction" (i.e.,  $L_{\mathrm{swap}}$ ).
There are two other loss functions which are similar to  $L_{\mathrm{swap}}$ :  $L_1(x_i) = \ell(\mathbf{p}_i^1, \mathbf{p}_i^2) + \ell(\mathbf{q}_i^2, \mathbf{q}_i^1)$  and  $L_2(x_i) = \ell(\mathbf{p}_i^1, \mathbf{q}_i^1) + \ell(\mathbf{p}_i^2, \mathbf{q}_i^2)$ . However, compared with  $L_{\mathrm{swap}}$ , both  $L_1$  and  $L_2$  lost one of advantage. For  $L_1$ , the two prompts will adapt independently without any interaction so the online prompt cannot be updated under the guidance of target prompt. The advantage of historical information will be lost. For  $L_2$ , the contrastive phase is removed due to the lack of the approaching process between two augmented images. The advantage of image augmentation will diminish. Table 4 provides the performance of using those three loss functions on 5 datasets. The experimental setting is the same as Table 1 except for the part of  $L_{\mathrm{swap}}$ . The results indicate that  $L_{\mathrm{swap}}$  outperforms  $L_1$  and  $L_2$ .
Table 4: Results for using different loss function.
Caltech101	DTD	Flowers102	Oxford-Pets	UCF101	Average
L1	87.38	47.22	69.63	87.71	64.68	71.32
L2	88.45	46.69	69.28	87.30	64.46	71.24
Lswap	89.90	47.34	70.22	89.14	65.66	72.45
Analysis on the Top- $K$  Confident Data Selection. We have a data selection before the prompt adaptation to filter out potential noise pseudo labels. Only top  $K$  confident test data will be used in test-time adaptation. Too much data not only will decrease model performance, but also slightly increased training time. Table 5 provides the performance of SwapPrompt with different  $K$  values on 5 datasets, the result without data selection is also included. In general, larger values of  $K$  show better performance, the highest accuracy across all 5 datasets is achieved when  $K$  is set to 8 or 16, with  $K = 16$  having the best average accuracy. On the other hand, the performance using the entire test data does not surpass the performance when data selection is employed. Data selection can increase the average accuracy by  $2.5\%$ . This indicates that the negative impact of noisy pseudo labels outweighs the positive effects.
Analysis on the Context Length and Initialization of Prompt. To explore whether our SwapPrompt works equally well on prompts in different context lengths and initialization, we repeat experiments on 5 datasets by varying the context length from 4 to 8 to 16, and initializing with three different type.
The results of different context lengths are shown in Table 6(a), which indicates that having more context tokens sometimes leads to slightly decrease on accuracy. This is probably due to shorter prompt learned less overfitting from selected samples. SwapPrompt only uses a part of test data to do adaptation, too much parameters in prompts may cause overfitting on those data. Nevertheless,
Table 5: Results for different  $K$  in data selection. 'None' denotes no data selection.
Kvalue	Caltech101	DTD	Flowers102	Oxford-Pets	UCF101	Average
1	87.51	41.13	65.04	87.83	62.20	68.74
2	88.72	43.50	64.66	87.42	62.97	69.45
4	89.49	44.15	66.63	88.98	62.76	70.40
8	90.14	47.70	66.30	88.43	63.96	71.31
16	89.90	47.34	70.22	89.14	65.66	72.45
None	88.03	43.97	68.41	86.68	62.64	69.95
SwapPrompt still maintains advanced performance on different context lengths. In Table 6(b), the three different initialization are hand-craft: "a photo of a [CLS]", Pre-ImageNet: the prompt which is trained on 16-shot ImageNet with CoOp, and random initialization. We find that different initialization only slightly affects the final accuracy. Because of the effective adaptation on prompt, SwapPrompt demonstrates robust performance, which indicates that our method does not rely on any prior source domain prompt.
Table 6: Analysis on the context length and initialization of prompt.
(a) Results for different context lengths.
Length	Caltech101	DTD	Flowers 102	Oxford-Pets	UCF101	Average
4	89.90	47.34	70.22	89.14	65.66	72.45
8	88.24	47.46	70.77	88.79	65.62	72.18
16	87.71	46.39	70.12	88.43	66.48	71.83
(b) Results for different initialization.
Initialization	Caltech101	DTD	Flowers102	Oxford-Pets	UCF101	Average
Hand-craft	89.53	46.92	70.32	88.92	65.98	72.33
Pre-ImagNet	89.65	47.15	70.44	89.03	65.34	72.36
Random	89.90	47.34	70.22	89.14	65.66	72.45
Analysis on the sensitivity of hyper-parameters  $\alpha$  and  $\beta$ . To explore the sensitivity of SwapPrompt about hyper-parameters  $\alpha$  and  $\beta$ , we conduct experiments with different values of  $\alpha$  and  $\beta$  on 5 datasets, other experimental settings are the same as Table 1. The results are shown in Table 7. It can be seen that SwapPrompt is not sensitive to the choice of hyperparameters  $\alpha$  and  $\beta$  in most cases. Results of SwapPrompt with different hyper-parameters settings in Table 7 still outperforms baselines in Table 1.
Table 7: Analysis on the sensitivity of hyper-parameters  $\alpha$  and  $\beta$  .
Caltech101	DTD	Flowers102	Oxford-Pets	UCF101	Average
α = 0.6, β = 1.4	88.56	46.80	70.04	88.24	65.21	71.77
α = 1.0, β = 1.0	89.90	47.34	70.22	89.14	65.66	72.45
α = 1.4, β = 0.6	89.45	47.10	70.32	87.97	65.77	72.12
In this paper, we have investigated a novel test-time prompt adaptation method, SwapPrompt, to learn the prompt adapted to the test domain for pre-trained vision-language models. Specifically, we maintain an online prompt and an EMA updated target prompt which interact and learn from each other. A swapped prediction mechanism is designed to train the online prompt, enabling it to predict the target prompt's class assignment of the same image under a different augmented view. Without any other requirement, SwapPrompt can be easily deployed on the test-time of vision-language models. Extensive empirical experiments have been conducted over various datasets to verify the effectiveness and superior performance of SwapPrompt.
This research was supported by fundings from the Key-Area Research and Development Program of Guangdong Province (No. 2021B0101400003), Hong Kong RGC Research Impact Fund (No. R5060-19, No. R5034-18), Areas of Excellence Scheme (AoE/E-601/22-R), General Research Fund (No. 152203/20E, 152244/21E, 152169/22E, 152228/23E), Shenzhen Science and Technology Innovation Commission (JCYJ20200109142008673).
[1] J Quinonero Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. The MIT Press, 1:5, 2009.
[2] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[3] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[5] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. Pmlr, 2018.
[6] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pages 9229–9248. PMLR, 2020.
[7] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[8] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 295-305, 2022.
[9] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8344-8353, 2022.
[10] Fatemeh Azimi, Sebastian Palacio, Federico Raue, Jorn Hees, Luca Bertinetto, and Andreas Dengel. Self-supervised test-time adaptation on video data. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3439-3448, 2022.
[11] Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen. Test-time adaptation to distribution shift by confidence maximization and input transformation. arXiv preprint arXiv:2106.14999, 2021.
[12] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pages 6028-6039. PMLR, 2020.
[13] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
[30] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[31] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[32] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
[33] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[34] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
[35] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
[36] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
[37] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[38] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[40] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.