SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaosong Ma
Department of Computing  
The Hong Kong Polytechnic University  
Hong Kong, China  
xiaosong16.ma@connect.polyu.hk
Jie Zhang *
Department of Computing  
The Hong Kong Polytechnic University  
Hong Kong, China  
jie-comp.zhang@polyu.edu.hk
Song Guo
Department of Computer Science and Engineering Hong Kong University of Science and Technology Hong Kong, China songguo@cse.ust.hk
Wenchao Xu
Department of Computing  
The Hong Kong Polytechnic University  
Hong Kong, China  
wenchao.xu@polyu.edu.hk
Test-time adaptation (TTA) is a special and practical setting in unsupervised domain adaptation, which allows a pre-trained model in a source domain to adapt to unlabeled test data in another target domain. To avoid the computation-intensive backbone fine-tuning process, the zero-shot generalization potentials of the emerging pre-trained vision-language models (e.g., CLIP, CoOp) are leveraged to only tune the run-time prompt for unseen test domains. However, existing solutions have yet to fully exploit the representation capabilities of pre-trained models as they only focus on the entropy-based optimization and the performance is far below the supervised prompt adaptation methods, e.g., CoOp. In this paper, we propose SwapPrompt, a novel framework that can effectively leverage the self-supervised contrastive learning to facilitate the test-time prompt adaptation. SwapPrompt employs a dual prompts paradigm, i.e., an online prompt and a target prompt that averaged from the online prompt to retain historical information. In addition, SwapPrompt applies a swapped prediction mechanism, which takes advantage of the representation capabilities of pre-trained models to enhance the online prompt via contrastive learning. Specifically, we use the online prompt together with an augmented view of the input image to predict the class assignment generated by the target prompt together with an alternative augmented view of the same image. The proposed SwapPrompt can be easily deployed on vision-language models without additional requirement, and experimental results show that it achieves state-of-the-art test-time adaptation performance on ImageNet and nine other datasets. It is also shown that SwapPrompt can even achieve comparable performance with supervised prompt adaptation methods.
When there is a discrepancy between the distribution of the training data and the testing data, the generalization performance of deep neural networks can be compromised [1, 2, 3]. The focus of domain adaptation is to construct models that can adapt to variations in data distribution by
(b) TPT
Figure 1: Comparison on V-L model architectures. (a) CoOp adapts prompt on labeled data. (b) TPT optimizes prompt by minimizing marginal entropy. (c) Ours SwapPrompt leverages self-supervised contrastive learning to facilitate test-time prompt adaptation.
transferring knowledge from a source domain to a new related target domain, which usually requires both the source and target domain data during the training phase [4, 5]. However, in practical scenarios, it is common to have only the model after it has been trained in the source domain, while without the access to the source data, or the authorization to alter the original training procedure [6, 7, 8]. To address this problem, Test-time adaptation (TTA) [9, 10] has been proposed and shown its potential to adapt models to target/unseen domains by only leveraging the unlabeled test data streams. Existing works have developed techniques such as entropy minimization [7, 11], class prototypes [12, 13], image generation [14, 15], and self-supervised training [10], which have already demonstrated superior performance.
Although traditional model-based TTA methods are shown effective, they typically rely on computationally intensive tuning to the parameters of the model backbone. The situation would be even worse with the advent of vision-language pre-trained models (e.g., CLIP [16], CoOp [17] and CoCoOp [18]), that have a massive number of parameters and are difficult to optimize. Therefore, it is promising to explore efficient techniques to fine-tune only a small set of parameters for adapting models to novel domains during testing while keeping the backbone fixed. Pre-trained vision-language models, which are trained on a significant amount of image-text pairs, have introduced a powerful paradigm that provides fresh insights for tackling this issue. A straightforward way is to utilize the strong zero-shot capabilities of the pre-trained vision-language models to discriminate various domains of test data via the fine-tuning over the labeled data of downstream tasks, however such way may not be feasible in TTA scenarios where the labeled downstream data is unavailable. Shu et al. [19] propose the test-time prompt tuning (TPT) to address the label scarcity problem in test-time. Nevertheless, it may lead to a risk of over-trust in the model (i.e., generating high confidence for a wrong result) from directly minimizing the entropy to tuning instance-specific prompts. The prediction confidence of TPT and proposed SwapPrompt can be found in appendix.
To this end, we propose SwapPrompt, a novel test-time prompt adaptation method as illustrated in Figure 1. Unlike previous approaches, SwapPrompt leverages a self-supervised contrastive learning strategy in the test domain, which consists of two key components: exponential moving average (EMA) prompt and prompt swapped prediction mechanism. The EMA mechanism employs a dual prompts paradigm: the target prompt and the online prompt. We optimize the online prompt while the target prompt is gradually updated through a slow-moving average process, which incorporates past information to increase stability and effectiveness. The prompt swapped prediction mechanism is inspired by the unsupervised learning method SwAV [20]. Based on an augmented view of image and the online prompt, SwapPrompt predicts the class assignment of an augmented view of the same image. This enables the online prompt to learn more representation knowledge. The rationale behind the swapped prediction strategy is that two different augmentations of the same image should have similar class assignments. The contrastive representation learning approach is leveraged to generate better decision boundaries [21].
In addition to the loss function of self-supervised contrastive learning, we employ the conventional cross-entropy loss as in CLIP and CoOp, which tune the prompt with high-confidence pseudo labels generated by the zero-shot CLIP. Our approach can also be employed for online test-time scenarios, where test data arrive in a flow of mini-batches. We break down the operation performed on all test data into multiple mini-batches, which is discussed in detail in the experimental section. We evaluate our method on various test-time adaptation benchmarks, including ImageNet and four natural
distribution shift datasets based on it, as well as nine fine-grained classification datasets. Experiment results show that our method achieves state-of-the-art test-time adaptation performance. We present our main contributions as follows:
- We propose SwapPrompt, a novel test-time prompt adaptation method that employs a self-supervised contrastive learning strategy, enabling prompts to better adapt to downstream image classification tasks.
- To the best of our knowledge, we are the first to apply unsupervised representation learning in prompt adaptation for pre-trained vision-language models. We introduce EMA prompt and prompt swapped prediction strategies, which enable the prompt to learn more knowledge from the powerful representation capabilities of pre-trained models.
- We conduct extensive experiments on ImageNet and its four variants, as well as nine other image classification datasets. The empirical evaluation shows that our approach significantly outperforms current TPT methods and can even compete with supervised prompt adaptation methods on most datasets.
Test-Time Adaptation. Test-time adaptation refers to reducing the performance gap when a source model is deployed on a different target domain of test data. The challenge of this issue is that only the source model and unlabeled test data are available, the training process and source data should not be accessed. Many solutions have been proposed to solve this problem, i.e., minimizing the entropy of the model's predictions [7, 11], maintaining a set of dynamically prototypes and measuring the similarity between test samples to each prototype [12, 13], generating new data similar to the target domain to assist model adaptation [14, 15] and utilizing the idea of self-supervised training to improve the generalization capability [10]. However, test-time adaptation in vision-language model is under-explored. Recently, Shu et al. [19] propose test-time prompt tuning (TPT) to extend the old entropy minimization method to vision-language model, but it is limited in practice due to the potential obvious over-confidence risk on predictions.
Prompt Learning in Vision-Language Models. Prompt learning first emerged in the field of natural language processing (NLP), aiming to enhance the performance of pre-trained models by utilizing different prompts. With the advent of vision-language models that integrate both visual and textual modalities, inspired by prompt learning in NLP, CoOp [16] is proposed for prompt learning, which transforms hand-crafted prompts into learnable continuous prompts and tunes them to adapt to downstream tasks. CoCoOp [17], an improvement upon CoOp, employs a meta-net and image features to generate individual prompts for each image. Additionally, there are also some other methods such as CLIP-adapter [22] and Tip-adapter [23] that do not modify the prompts but instead add additional classification layers after the backbone models. What they have in common is that all of them heavily rely on a set of labeled data, making them unsuitable for test-time settings. Another line of work focuses on enabling prompt learning in an unsupervised manner during training process, i.e., unsupervised prompt learning (UPL) [24]. However, it simply extends pseudo-labeling methods to vision-language models without fully leveraging the powerful representation capabilities of pre-trained models.
In this section, we first introduce the preliminary and problem definition of test-time prompt adaptation, then elaborate the proposed SwapPrompt framework that leverages self-supervised contrastive learning to facilitate prompt adaptation, which is shown in Figure 2. Finally, we present the workflow in Section 3.3.
3.1 Preliminary and Problem Definition
We focus on test-time prompt adaptation for pre-trained vision-language models (e.g. CLIP), where the model is trained on the source domain, but the test data belongs to the target domain. In this scenario, zero-shot CLIP with a general prompt (e.g., "a photo of a [CLS]") has shown barely acceptable zero-shot generalization ability. However, these hand-crafted prompts cannot fully extract
Figure 2: Framework of the proposed SwapPrompt. We use the text features generated by the target prompt as prototypes and assign the image feature of an augmented view of an image to these prototypes to obtain a soft class assignment. The online prompt is trained to predict this class assignment with a different augmented view of the same image. The EMA of online prompt is used to update the target prompt.
the rich knowledge learned by CLIP from large-scale and diverse pre-training datasets. Optimized prompts can further improve CLIP's ability to retrieve knowledge about the target domain. There have been some related works on supervised target domain data, including the well-known and effective CoOp method, part of our approach also incorporates the ideas of CoOp. Then we will briefly review CoOp as well as define the notation used in this paper.
Context Optimization (CoOp). CoOp is a prompt adaptation method based on CLIP. Similar to CLIP, CoOp includes an image encoder and a text encoder, which are denoted as  $f(\cdot)$  and  $g(\cdot)$ , respectively. Let  $\mathcal{D}_{\mathrm{target}} = \{(x_i,y_i)\}_{i=1}^N$  be the dataset on the target domain, where  $x_i$  is the  $i$ -th input data sample,  $y_i$  is the corresponding label and  $y_i \in \{1,2,\dots,C\}$  for a  $C$ -class classification problem,  $N$  is the size of the dataset. Let  $\mathbf{t}$  represent the learnable continuous prompt and  $\{\mathbf{t};c\}$  be the input of  $c$ -th class for the text encoder. We then define  $\mathbf{z}_i = f(x_i)$  and  $\mathbf{w}_c = g(\{\mathbf{t};c\})$  as the output features embedding of image encoder and text encoder, respectively. The probability of  $c$ -th class for  $x_i$  is computed as:
$$
p (c \mid x _ {i}) = \frac {\exp (\operatorname {s i m} \left(\mathbf {z} _ {i} , \mathbf {w} _ {c}\right) / \tau)}{\sum_ {j = 1} ^ {C} \exp (\operatorname {s i m} \left(\mathbf {z} _ {i} , \mathbf {w} _ {j}\right) / \tau)}, \tag {1}
$$
where  $\tau$  denotes the temperature parameter,  $\mathfrak{sim}(\cdot)$  denotes the cosine similarity. For all training data, CoOp calculates the probabilities of all classes by Eq. 1 and minimizes the cross-entropy loss to tune the prompt.
Test-Time Prompt Adaptation. In the test-time scenario, labeled data from the target domain is not available, thereby the prompt cannot be optimized as in CoOp. Consider a test dataset  $\mathcal{D}_{\mathrm{test}} = \{x_i\}_{i=1}^N$  without label information. The objective of test-time prompt adaptation can be formulated as:
$$
\mathbf {t} ^ {*} = \underset {\mathbf {t}} {\arg \min } \mathcal {L} (f, g, \mathbf {t}, \mathcal {D} _ {\text {t e s t}}), \tag {2}
$$
where  $\mathcal{L}$  is the cross-entropy loss function, our purpose is to design an unsupervised prompt adaptation method, facilitating prompt  $\mathbf{t}$  compatibility with the target domain of  $\mathcal{D}_{\mathrm{test}}$ , eliciting more target domain knowledge for CLIP.
3.2 Overview of SwapPrompt
In this section, we present our proposed test-time adaptation method SwapPrompt, which contains two key insights: Exponential Moving Average (EMA) Prompt and Prompt Swapped Prediction. As illustrated in Figure 2, SwapPrompt employs a dual prompts paradigm: the online and target prompts. These two types of prompts will be interacted and learnt from each other to adapt to the target domain by applying an EMA update strategy. Moreover, unlike most previous frameworks that only match one prompt to one image, SwapPrompt adopts a prompt swapped prediction mechanism to establish self-supervised representation learning in prompt adaptation with the goal of assigning similar classes for two different augmentations of the same image.
3.2.1 Exponential Moving Average Prompt
SwapPrompt's goal is to learn an online prompt  $\mathbf{t}_o$  which can be used on the test dataset. As described previously, SwapPrompt has a target prompt  $\mathbf{t}_t$  to guide the update of online prompt  $\mathbf{t}_o$ . The main motivation behind this design is: from a given target prompt  $\mathbf{t}_t$ , we can train a new and potentially improved prompt  $\mathbf{t}_o$  through predicting the representation knowledge generated by the  $\mathbf{t}_t$ . By repeatedly using subsequent online prompts as new target prompts for further training, we can create a sequence of prompts that improves in quality over time. Practically, SwapPrompt uses a slowly moving exponential average of the online prompt as the target prompt, and we perform the following update after each training step:
$$
\mathbf {t} _ {t} = \epsilon \mathbf {t} _ {t} + (1 - \epsilon) \mathbf {t} _ {o}, \tag {3}
$$
where  $\epsilon \in [0,1]$  is the decay rate of target prompt.
3.2.2 Prompt Swapped Prediction
In the field of self-supervised learning, cross-view prediction has been widely used in many existing works. These methods usually cast the prediction problem into a representation space, and then learn representations by predicting different augmented views generated from the same image. It is assumed that the different augmented views of an image should be relatively close in the representation space.
Among these methods, SwAV [20] proposes a different approach. Instead of directly enforcing consistent mapping between the image features in the representation space, SwAV clusters a set of image features, uses the cluster centroids as prototypes, and matches different augmentations of an image to these prototypes to compute its cluster assignment. By comparing their cluster assignments rather than their features, SwAV performs contrastive learning across multiple image views.
Inspired by SwAV, we assign an image's augmented view with prototypes to obtain its soft class assignment and predict this class assignment with another augmented view of the same image. This approach is well-suited for CLIP, as it has natural prototypes: text features outputted by the text encoder. Specifically, for all classes  $\mathcal{V} \in \{1,2,\dots,C\}$  in the test dataset  $\mathcal{D}_{\mathrm{test}}$ , target prompt  $\mathbf{t}_t$  will form  $C$  inputs  $\{\mathbf{t}_t;c\}_{c = 1}^C$  for text encoder, which will generate  $C$  text features  $\mathbf{w}_c^t = g(\{\mathbf{t}_t;c\})$  of different classes. Due to the supervised contrastive pre-training on large-scale image-text pairs, the text features generated by the CLIP have high similarity with the image features of the same class and low similarity with the text features of different classes. Therefore, these  $C$  text features  $\{\mathbf{w}_c^t\}_{c = 1}^C$  can serve as a set of high-quality prototypes.
For an image  $x_{i}$ , we use two different data augmentation methods  $A_{1}$  and  $A_{2}$  to obtain two augmented views  $A_{1}(x_{i})$  and  $A_{2}(x_{i})$ . The image features generated by the image encoder are  $\mathbf{z}_i^1 = f(A_1(x_i))$  and  $\mathbf{z}_i^2 = f(A_2(x_i))$ , respectively. By applying Eq. 1 on prototypes  $\{\mathbf{w}_c^t\}_{c=1}^C$ , we can acquire the corresponding class assignments of two augmented image views:
$$
\mathbf {q} _ {i} ^ {1} = \left\{p _ {t} \left(c \mid A _ {1} \left(x _ {i}\right)\right) \right\} _ {c = 1} ^ {C} \quad \text {a n d} \quad \mathbf {q} _ {i} ^ {2} = \left\{p _ {t} \left(c \mid A _ {2} \left(x _ {i}\right)\right) \right\} _ {c = 1} ^ {C}, \tag {4}
$$
where the  $p_t(c|A_1(x_i))$  and  $p_t(c|A_2(x_i))$  can be expressed as:
$$
p _ {t} \left(c \mid A _ {1} \left(x _ {i}\right)\right) = \frac {\exp \left(\operatorname {s i m} \left(\mathbf {z} _ {i} ^ {1} , \mathbf {w} _ {c} ^ {t}\right) / \tau\right)}{\sum_ {j = 1} ^ {C} \exp \left(\operatorname {s i m} \left(\mathbf {z} _ {i} ^ {1} , \mathbf {w} _ {j} ^ {t}\right) / \tau\right)} \quad \text {a n d} \quad p _ {t} \left(c \mid A _ {2} \left(x _ {i}\right)\right) = \frac {\exp \left(\operatorname {s i m} \left(\mathbf {z} _ {i} ^ {2} , \mathbf {w} _ {c} ^ {t}\right) / \tau\right)}{\sum_ {j = 1} ^ {C} \exp \left(\operatorname {s i m} \left(\mathbf {z} _ {i} ^ {2} , \mathbf {w} _ {j} ^ {t}\right) / \tau\right)}. \tag {5}
$$
Similarly, the predictions generated by online prompt are defined as:
$$
\mathbf {p} _ {i} ^ {1} = \left\{p _ {o} (c | A _ {1} (x _ {i})) \right\} _ {c = 1} ^ {C} \quad \text {a n d} \quad \mathbf {p} _ {i} ^ {2} = \left\{p _ {o} (c | A _ {2} (x _ {i})) \right\} _ {c = 1} ^ {C}. \tag {6}
$$
We establish the prompt swapped prediction loss function for image  $x_{i}$  as:
$$
\boldsymbol {L} _ {\text {s w a p}} \left(x _ {i}\right) = \ell \left(\mathbf {p} _ {i} ^ {1}, \mathbf {q} _ {i} ^ {2}\right) + \ell \left(\mathbf {p} _ {i} ^ {2}, \mathbf {q} _ {i} ^ {1}\right), \tag {7}
$$
where  $\ell$  is the function that measures the difference between the predictions and the class assignments. In this paper, we utilize the cross-entropy loss function. i.e.,
$$
\ell \left(\mathbf {p} _ {i} ^ {1}, \mathbf {q} _ {i} ^ {2}\right) = - \sum_ {c = 1} ^ {C} p _ {t} \left(c \mid A _ {2} \left(x _ {i}\right)\right) \log p _ {o} \left(c \mid A _ {1} \left(x _ {i}\right)\right). \tag {8}
$$
The online prompt  $\mathbf{t}_o$  will be optimized by Eq. 7 while the target prompt  $\mathbf{t}_t$  will not be updated by this loss function.
3.2.3 Prompt Optimization by Pseudo Label
In addition to utilizing self-supervised representation learning methods to optimize prompts, we also employ a set of labeled data, similar to CoOp, to optimize online prompts. However, unlike CoOp where the labels are available for target domain, the test data is unlabeled in the test-time scenario. Therefore, we first perform inference on the test data with hand-crafted prompts (e.g., "a photo of a [CLS]", obtaining their pseudo-labels  $\hat{\mathcal{Y}}_{\mathrm{test}} = \{\hat{y}_i\}_{i=1}^N$ , and then employ Eq. 6 and cross-entropy loss as:
$$
\boldsymbol {L} _ {\text {p s e u d o}} \left(x _ {i}\right) = \ell_ {c e} \left(\mathbf {p} _ {i} ^ {1}, \hat {y} _ {i}\right) + \ell_ {c e} \left(\mathbf {p} _ {i} ^ {2}, \hat {y} _ {i}\right). \tag {9}
$$
It should be noted that the pseudo-labels obtained through this approach may contain noise. Therefore, we cannot apply Eq. 9 to all test data. The process of data selection is discussed in Section 3.3.
3.3 Algorithm Workflow
In this subsection, we illustrate the overall prompt adaptation process of SwapPrompt and the training method when facing online test samples.
Before training with Eq. 7, Eq. 9 and pseudo labels  $\hat{y}_{\mathrm{test}}$ , we need to perform data selection to filter out potential noisy pseudo labels. Specifically, we first employ the zero-shot CLIP and a hand-crafted prompt to obtain pseudo labels and classification confidences for the test data. Then, for each class, we only select the top  $K$  test data with the highest confidence. These selected test data form the adaptation set  $\mathcal{D}_{\mathrm{adapt}}$ , which is a subset of  $\mathcal{D}_{\mathrm{test}}$ . For all  $x_{i} \in \mathcal{D}_{\mathrm{adapt}}$ , given trade-off hyper-parameters  $\alpha$  and  $\beta$ , the following loss function will be used to do prompt adaptation:
$$
\boldsymbol {L} _ {\text {a d a p t}} \left(x _ {i}\right) = \alpha \boldsymbol {L} _ {\text {s w a p}} \left(x _ {i}\right) + \beta \boldsymbol {L} _ {\text {p s e u d o}} \left(x _ {i}\right). \tag {10}
$$
When target images arrive in a flow of mini-batches, i.e., the test data is online, we cannot sort the confidences of the entire test dataset. However, we can still perform confidence-based sorting on mini-batches to select the top  $k$  ( $k < K$ ) test data with the highest confidence, while keeping the rest of the training process unchanged. When new mini-batch test data arrives, the available test data are sorted by confidence again to obtain new  $\mathcal{D}_{\mathrm{adapt}}$  for prompt adaptation.
Dataset. We evaluate the proposed SwapPrompt over fourteen datasets, including ImageNet [25] and its four variants: ImageNet-V2 [26], ImageNet-A [27], ImageNet-R [28] and ImageNet-Sketch [29], and nine other publicly available image classification datasets used in CLIP: Caltech101 [30], DTD [31], Flowers102 [32], Oxford-Pets [33], UCF101 [34], StanfordCars [35], Food101 [36], EuroSAT [37] and SUN397 [38]. These datasets encompass a diverse range of visual classification tasks, including general objects, fine-grained categories, and even texture classification, forming a comprehensive benchmark. We only use the test data to do adaptation and also evaluate models with them.
Baselines. We compare the performance of SwapPrompt with the state-of-the-art methods. In addition to zero-shot CLIP [16], we also include TPT [19], a test-time prompt tuning method that minimizes the marginal entropy of test data; UPL [24], an unsupervised prompt learning approach and we make some modifications on it to suit the test-time setting; CoOp [17], a supervised few-shot prompt tuning method. We use some labeled data from the same domain as the test data during training this baseline, in order to use it as an upper bound performance of test-time prompt adaptation.