SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haoran Chen $^{1,2}$  Xintong Han $^{3}$  Zuxuan Wu $^{1,2\dagger}$  Yu-Gang Jiang $^{1,2}$
$^{1}$ Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University  $^{2}$ Shanghai Collaborative Innovation Center of Intelligent Visual Computing  $^{3}$ Huya Inc
Most existing methods for unsupervised domain adaptation (UDA) rely on a shared network to extract domain-invariant features. However, when facing multiple source domains, optimizing such a network involves updating the parameters of the entire network, making it both computationally expensive and challenging, particularly when coupled with min-max objectives. Inspired by recent advances in prompt learning that adapts high-capacity models for downstream tasks in a computationally economic way, we introduce Multi-Prompt Alignment (MPA), a simple yet efficient framework for multi-source UDA. Given a source and target domain pair, MPA first trains an individual prompt to minimize the domain gap through a contrastive loss. Then, MPA denoises the learned prompts through an auto-encoding process and aligns them by maximizing the agreement of all the reconstructed prompts. Moreover, we show that the resulting subspace acquired from the auto-encoding process can easily generalize to a streamlined set of target domains, making our method more efficient for practical usage. Extensive experiments show that MPA achieves state-of-the-art results on three popular datasets with an impressive average accuracy of  $54.1\%$  on DomainNet.
Deep learning has achieved remarkable progress in various computer vision tasks [15, 11, 31, 24]. However, the success usually relies on supervised training using a massive amount of manually labeled data, which are often expensive and time-consuming to collect. Furthermore, current deep models are brittle to the presence of domain shift [27, 34, 47] in the forms of different image styles, varied lighting conditions, diverse viewpoints, etc., between training and testing distributions.
Unsupervised domain adaptation (UDA) is a popular strategy that mitigates domain discrepancies by transferring knowledge learned from a well-labeled source domain to an unlabeled target domain [25, 4, 39, 42, 23]. While significant advances have been achieved, current approaches focus on the single source setting, where all the labeled training data share the same distribution. In practice, however, it is more common for the labeled data to be collected from multiple sources that are diverse in distribution. Naturally, one could still tackle this problem by straightforwardly combining all the data into one single source and applying off-the-shelf UDA methods. However, directly applying single-source UDA methods often results in limited performance, as domain shift also exists among different source domains.
The integration of multiple source domains for improved adaptation results on the unlabeled target domain is generally known as multi-source unsupervised domain adaptation. Inspired by the theoretical analysis of Ben-David et al. [2], learning domain-invariant feature representations has become a
(a) Method comparison
(b) Performance comparison on DomainNet
Figure 1: (a) Most conventional multi-source UDA methods use a common feature extractor with domain-specific classifier heads, while we introduce prompt learning to multi-source UDA. (b) MPA outperforms all other multi-source UDA methods by a large margin on the DomainNet dataset with roughly one-third of tunable parameters. We also introduce an LST strategy for continuous adaptation to a streamlined set of target domains that further reduces the number of tunable parameters and still achieves high accuracy compared with MPA. See texts for more details.
prevailing paradigm for multi-source UDA. One typical approach is to jointly learn a common feature extractor together with domain-specific classifier heads. Various feature distance metrics [22, 33, 14] or domain adversarial training [35] can serve as a preliminary alignment between source and target domains, followed by different auxiliary losses carefully designed to further reduce the domain shift. Although these methods offer decent results in the single source setting, as the number of source domains increases, using only a single shared feature extractor to obtain domain-invariant features is inevitably difficult to optimize [44]. Such a problem is further amplified in the modern large model era if we wish to apply more advanced backbones for improved performance.
In this paper, we introduce prompt learning [16, 41], which has been designed to transfer knowledge learned from large pre-trained vision language models like CLIP [28], to multi-source UDA. In prompt learning, image representations are learned contrastively with a piece of language text termed "prompt". Consequently, prompts are tuned with the backbone network fixed for a more efficient adaptation to downstream tasks (See Figure 1 for a comparison). While recent studies [10, 1] suggest that learnable prompts can be used for UDA, they are restricted to the single source scenario and directly generalizing them to the multi-source setting produces limited results.
In light of this, we present a surprisingly simple framework, Multi-Prompt Alignment (MPA), for multi-source UDA. MPA is composed of two steps—one to learn an individual prompt by tuning a small set of parameters for each source and target domain pair, while the next to mine the relationships among the learned prompts by deriving a shared embedding space that is domain-invariant. More specifically, given a source domain and a target domain, we use CLIP as our backbone and learn one prompt tailored for such a pair. Then, inspired by the intrinsic dimensionality of data [40], we further reconstruct all the learned prompts using a simple auto-encoder network, aiming at removing redundant information potentially stemmed from the discrepancies among all the source domains. Finally, given the denoised prompts, an  $\mathcal{L}_1$  constraint is incorporated as our alignment strategy so that the prompts agree on the classification of target images.
We conduct extensive experiments on multiple benchmarks and the results clearly show that our approach outperforms state-of-the-art methods in the multi-source setting. In particular, on DomainNet [26], the most challenging dataset for multi-source UDA to date, MPA surpasses all state-of-the-art methods. Additionally, as the latent space is optimized with prompts from multiple sources, it encodes knowledge shared by different domains and could potentially generalize to multiple target domains by traversing the space. Consequently, we show how to tune the learned low-dimensional embedding for efficient deployment to a streamlined set of target domains, a strategy we name Latent Subspace Tuning (LST). In summary, our contributions are three-fold:
- We introduce Multi-Prompt Alignment (MPA) for multi-source UDA. MPA takes advantage of prompt learning and thus is capable of achieving a balance between performance and efficiency compared with alternative methods.
- MPA learns a latent space by maximizing the consensus of multiple learned prompts. Based on this, we introduce Latent Subspace Tuning (LST) that is able to continuously adapt to a streamlined set of target domains.
- MPA achieves state-of-the-art results on several popular benchmarks while LST offers comparable results with tunable parameters that are one order of magnitude less.
2.1 Multi-Source Unsupervised Domain Adaptation
First studied by Yang et al. [46], multi-source UDA has drawn increasing attention in the community. Throughout the years, various methods have been studied. For example, in MDAN [48] and DCTN [43], a discriminator applied with adversarial losses is trained so that the features from source and target domains are aligned. MFSAN [51] calculates and aligns the maximum mean discrepancy for each source and target pair. Similarly,  $\mathbf{M}^3\mathbf{SDA}$  [26] aligns the moment distance for both target and source domains. All these methods require a shared feature extractor to obtain domain-invariant features. On the contrary, Rakshit et al. [29] adopt one domain-specific encoder for each source and target pair while Zhao et al. [49] pre-trains a classifier for each source domain and then adversarially maps the target images into each trained feature space. The better alignment of these methods is at the cost of a significantly increased number of parameters needed for training. More recently, Xu et al. [44] propose to combine joint alignment and separate alignment for a better adaptation, and Deng et al. [5] propose a method that detects and alleviates the negative impact from pseudo-labels for a better self-training scheme.
2.2 Prompt Learning
Traditionally, given a pre-trained language model, a common approach in deep learning is fine-tuning the whole model or its task-specific heads to adjust to downstream tasks. While effective, however, two main drawbacks exist. First of all, as the model size keeps increasing, pre-training and fine-tuning are becoming more and more expensive. Secondly, for each new task, fine-tuning needs to be repeatedly conducted. Recently, researchers have shown that pre-trained large-scale language models can handle a wide range of downstream tasks with only a few or even no samples by pre-pending instructions to the input text [20]. Such instruction texts are called prompts. Consequently, prompts can be tuned instead of the entire network for a more efficient adaptation to downstream tasks. Originally, prompts are essentially sequences of manually designed language tokens that are mapped to an embedding space. To date, extensive research has demonstrated that training soft prompts, i.e., prompts with their own parameters learned by deep models, are more effective [18, 16]. The success of prompt learning in NLP has also garnered attention in the vision community which motivated the establishment of many related works. To name a few, Zhou et al. [50] are the first to apply soft prompt learning to the image recognition task. Ju et al. [13] explore prompt learning for efficient and lightweight video understanding. While promoting in these studies is limited to the input of text encoders, Jia et al. [12] preprocess tokens directly to the image patches.
Our goal is to use multiple labeled source domains for improved performance on target domains. To this end, we leverage prompt learning, which is an effective strategy by learning a small set of parameters to adapt a pretrained model to different downstream tasks. In the following, we first review prompt learning in CLIP in Sec. 3.1 and then elaborate in Sec. 3.2 and Sec. 3.3 our proposed MPA and LST method respectively.
3.1 An Overview of Prompt Learning in CLIP
CLIP consists of an image encoder and a text encoder that are jointly trained with a contrastive loss on 400M image and text pairs. The image encoder  $f$ , which can either be a ResNet [11] or a Vision
Figure 2: Each source and target pair prompt  $P_{i}$  is the concatenation of a "source prompt" segment and a "target prompt" segment, both composed of domain-invariant and domain-specific features. Therefore, the size of  $P_{i}$  is  $\mathbb{R}^{2K \times (M_1 + M_2) \times 512}$ . During our prompt training step, the text encoder and the image encoder of CLIP are both frozen.
Transformer [7], maps raw images to an embedding space, and the text encoder  $g$  is a Transformer [36] that projects an input text sequence to the same embedding space. A prompt in CLIP usually exists in the form of "a photo of [CLS]" where [CLS] is a class token that can be replaced by a certain class name. This sequence of tokens is first converted into a lower-cased byte pair encoding (BPE) representation, which is essentially a unique numeric ID [50]. Then the numeric IDs are embedded in a 512 dimension vector that is further passed to the Transformer text encoder. In our work, instead of using manually crafted prompts, we train soft prompts that are directly embedded by the text encoder. Given an image  $x$  and a text embedding  $w_{k}$  for class  $k \in \{1,2,\dots,K\}$ , where  $K$  is the total number of categories, CLIP aligns them in a contrastive manner so that
$$
p (y = k | \boldsymbol {x}) = \frac {\exp \left(<   \boldsymbol {w} _ {k} , f (\boldsymbol {x}) > / T\right)}{\sum_ {i = 1} ^ {K} \exp \left(<   \boldsymbol {w} _ {i} , f (\boldsymbol {x}) > / T\right)} \tag {1}
$$
is maximized when the input image  $\pmb{x}$  indeed belongs to class  $k$ . Here  $<\cdot, \cdot>$  denotes the cosine similarity and  $T$  is a learnable temperature parameter.
3.2 Multi-Prompt Alignment
Let  $N$  denote the total number of domains, where the first  $N - 1$  domains are source domains and the  $N$ -th domain is the target domain. For multi-source UDA, our goal is to learn a domain-invariant latent space so that the domain shift among different source domains as well as the discrepancies between all the source and target domain pairs can be minimized. To achieve such a goal, we introduce the framework MPA. Specifically, we design prompts to contain domain-invariant and domain-specific feature following Ge et al. [10] and train such a prompt for each source and target domain pair. Then, an auto-encoder structure is introduced to our method to denoise the acquired prompts, followed by an  $\mathcal{L}_1$  constraint for further alignment.
Prompt Design. Following Ge et al. [10], our prompt for multi-source UDA includes a set of class-specific context tokens  $\pmb{v}_i^k, i \in \{1, 2, \dots, M_1\}, k \in \{1, 2, \dots, K\}$  and another set of domain-specific tokens shared across all classes  $\pmb{d}_j^d, j \in \{1, 2, \dots, M_2\}, d \in \{s, t\}$ . See Figure 2 for an overview. Here,  $M_1$  and  $M_2$  represent the number of tokens,  $K$  is the number of classes,  $s$  is short for source and  $t$  is short for target, resulting in a total of  $2K$  categories for training with contrastive loss. Therefore, the prompt for each source and target pair can be derived as:
$$
\boldsymbol {P} _ {i} = \left[ \boldsymbol {t} _ {1} ^ {s _ {i}}, \dots , \boldsymbol {t} _ {K} ^ {s _ {i}}, \boldsymbol {t} _ {1} ^ {t _ {i}}, \dots , \boldsymbol {t} _ {K} ^ {t _ {i}} \right] ^ {\top}, i \in \{1, 2, \dots , N - 1 \}. \tag {2}
$$
These prompts serve as learnable parameters that bridge the domain gap between a source domain and a target domain through a contrastive loss, as will be introduced next.
(a) Alignment in MPA on the Office-Home dataset with Rw as target domain
(b) Illustration of LST
Figure 3: (a) Example of prompt alignment on the Office-Home dataset. Here,  $P_{1}, P_{2}, P_{3}$  are prompts for domain Ar-Rw, Cl-Rw and Pr-Rw respectively. All prompts are projected into the same latent space for alignment by an auto-encoder structure. (b) When facing a new target domain, tuning the latent subspace learned by the auto-encoder in MPA can allow quick adaptation that is more computationally efficient.
Prompt Learning for Multi-source UDA. To apply prompt learning to multi-source UDA, we first train individual prompts for each source and target pair using the image and text encoders of CLIP. Given an image  $\boldsymbol{x}^s$  sampled from the source domain  $\mathcal{D}_s$  whose label is  $y^*$ , we optimize the prompts so that the outputs from the image and text encoder are aligned. For an image  $\boldsymbol{x}^t$  from the target domain  $\mathcal{D}_t$  whose label is unknown, we first leverage the strong zero-shot ability of CLIP to generate a static pseudo-label  $\hat{y^*}$  for image-text alignment. Pseudo-labels are only generated for images whose maximum probability is larger than a fixed threshold  $\tau$  using Equation 1. While more sophisticated approaches like self-training could be leveraged to generate pseudo-labels [52, 19], we find that pseudo-labels from CLIP are simple and effective. Finally, prompts are trained with cross-entropy loss functions and Figure 2 gives an overview of the process. More formally, for a prompt  $P_i, i \in \{1, 2, \dots, N - 1\}$ , the objective function for optimization follows:
$$
\min  _ {\boldsymbol {P} _ {i}} - \frac {1}{n _ {s}} \sum_ {\boldsymbol {x} ^ {s} \sim \mathcal {D} _ {s}} \log P (y = y ^ {*} | \boldsymbol {x} ^ {s}; \boldsymbol {P} _ {i}) - \frac {1}{n _ {t}} \sum_ {\boldsymbol {x} ^ {t} \sim \mathcal {D} _ {t}} \log P (y = \hat {y} ^ {*} | \boldsymbol {x} ^ {t}; \boldsymbol {P} _ {i}). \tag {3}
$$
Here, the probability  $P(y = k|\pmb{x}^d; \pmb{P}_i)$  of an image sample belonging to the  $k$ -th class is derived from a contrastive loss:
$$
P (y = k) = \frac {\exp \left(<   g \left(\boldsymbol {t} _ {k} ^ {d}\right) , f \left(\boldsymbol {x} ^ {d}\right) > / T\right)}{\sum_ {d \in \{s , t \}} \sum_ {i = 1} ^ {K} \exp \left(<   g \left(\boldsymbol {t} _ {i} ^ {d}\right) , f \left(\boldsymbol {x} ^ {d}\right) > / T\right)}, \tag {4}
$$
where  $d \in \{s, t\}$  is a domain identifier indicating where the image comes from,  $T$  is a learnable temperature parameter, and  $f$  and  $g$  represents the image and text encoder in CLIP respectively, which are kept frozen during training. This specific design can push the prompts to learn disentangled representation of both class-invariant and class-specific semantic information to boost the performance of domain adaptation methods [3, 21].
Even though Eqn 3 allows us to obtain a prompt for each source and target domain pair, the noise level in the learned prompts differs due to variations in the amount of images and the domain gap between each source and target domain. One obvious consequence is that they might produce inconsistent results even for the same target domain image. Therefore, an  $\mathcal{L}_1$  constraint (Eqn 5) can be applied as a strategy for prompt alignment.
$$
\mathcal {L} _ {1} = \frac {2}{(N - 1) \times (N - 2)} \sum_ {j = 1} ^ {N - 2} \sum_ {i = j + 1} ^ {N - 1} | P \left(y = k ^ {t} \mid \boldsymbol {x} _ {k}, \boldsymbol {P} _ {i}\right) - P \left(y = k ^ {t} \mid \boldsymbol {x} _ {k}, \boldsymbol {P} _ {j}\right) | \tag {5}
$$
Better Alignment through Reconstructed Prompts. While directly aligning the learned prompts produce decent results for multi-source UDA, the prompts are high-dimensional and might contain redundant information. Motivated by the theory that high dimensional data usually lies in a lower
dimensional manifold [40], we build upon auto-encoders for better alignment. By using such an architecture, we hope to learn a domain-invariant "latent subspace" of denoised prompts so that redundant information can be removed through the reconstruction of the learned prompts. More formally, we use an auto-encoder consisting of a projection function  $\mathbf{Proj}(\cdot)$  and a back-projection function  $\mathbf{Proj}_b(\cdot)$ . The learned prompts  $\pmb{P_i}$  are first projected into a latent subspace of a lower dimension  $d_I$  by  $\mathbf{Proj}(\cdot)$ , followed by  $\mathbf{Proj}_b(\cdot)$  projecting the vectors back into soft prompts  $\hat{\pmb{P}}_i$ . The  $\mathbf{Proj}(\cdot)$  function is implemented by a one-layer feed-forward network while  $\mathbf{Proj}_b(\cdot)$  is a two-layer nonlinear perceptron:
$$
\mathbf {P r o j} \left(\boldsymbol {P} _ {i}\right) = \boldsymbol {W} _ {1} \left(\boldsymbol {P} _ {i}\right) + \boldsymbol {b} _ {1} \tag {6}
$$
$$
\mathbf {P r o j} _ {b} \left(\boldsymbol {v} _ {I}\right) = \boldsymbol {W} _ {3} \left(\tanh  \left(\boldsymbol {W} _ {2} \boldsymbol {v} _ {I} + \boldsymbol {b} _ {1}\right)\right) + \boldsymbol {b} _ {2} \tag {7}
$$
where  $\pmb{v}_I = \mathbf{Proj}(\pmb{P}_i)$  and we optimize a reconstruction loss:
$$
\mathcal {L} _ {A E} = \frac {1}{N - 1} \sum_ {i = 1} ^ {N - 1} \| \hat {\boldsymbol {P}} _ {i} - \boldsymbol {P} _ {i} \| _ {2} ^ {2}. \tag {8}
$$
We then perform prompt alignment, i.e. the  $\mathcal{L}_1$  constraint, on the reconstructed prompts. Therefore, the overall objective function can be written as:
$$
\mathcal {L} = \mathcal {L} _ {C L S} + \mathcal {L} _ {A E} + \alpha \mathcal {L} _ {1}, \tag {9}
$$
where  $\mathcal{L}_{CLS}$  is the cross entropy loss calculated using the static pseudo-labels that helps with the reconstruction process. Here  $\alpha$  is a hyper-parameter controlling the weight of the  $\mathcal{L}_1$  loss. The whole alignment procedure is depicted in Figure 3a. Finally, for predicting the labels of target samples, we compute the average of the output logits using all  $\hat{P}_i$ 's.
3.3 Latent Subspace Tuning
In real-life applications, it is more practical when adaptation to a streamlined set of targets is needed. While we can repeatedly apply MPA to each target domain, it is computationally inefficient especially when equipped with a large-scale backbone model. To mitigate this issue, we introduce an LST strategy that explores the latent space derived by autoencoders for fast adaptation. The key idea is that since the latent space of the learned auto-encoder from MPA is optimized with prompts from multiple source domains, it alone is capable of encoding domain-invariant knowledge. Therefore, we can adopt MPA on the first target domain, and traverse the subspace learned by MPA to generalize to the following ones.
More formally, given a streamlined set of target domains  $\mathcal{D}_{T_1},\mathcal{D}_{T_2},\dots,\mathcal{D}_{T_L}$ , to continuously adapt to each one of them in a computationally efficient manner, we first conduct MPA on domain  $\mathcal{D}_{T_1}$ . After successfully applying MPA, a low-dimensional embedding space that captures the relationships among different domains is derived and can be leveraged for quick adaptation to domains  $\mathcal{D}_{T_2},\ldots,\mathcal{D}_{T_L}$ . Specifically, for each following target domain  $\mathcal{D}_{T_i}$ , a domain-invariant feature vector  $v_{\mathrm{tune}}^{T_i} \in R^{N \times M_1 \times d_I}$  together with a domain-specific feature vector  $d_{\mathrm{tune}}^{T_i} \in R^{1 \times M_2 \times d_I}$  is randomly initialized and passed to the learned back projection function  $\mathbf{Proj}_b(\cdot)$  from MPA. As a result, an entirely new prompt  $\tilde{\boldsymbol{P}}^{T_i} = \mathrm{Concat}[\mathbf{Proj}_b(v_{\mathrm{tune}}^{T_i}),\mathbf{Proj}_b(d_{\mathrm{tune}}^{T_i})]$  can be constructed. Again with the help of pseudo-labels, we can tune this new prompt by minimizing the following objective function:
$$
\min  _ {d _ {\text {t u n e}} ^ {T _ {i}}, v _ {\text {t u n e}} ^ {T _ {i}}} - \frac {1}{n _ {T _ {i}}} \sum_ {\boldsymbol {x} \sim \mathcal {D} _ {T _ {i}}} \log P (y = \hat {y} ^ {*} | \boldsymbol {x}; d _ {\text {t u n e}} ^ {T _ {i}}, v _ {\text {t u n e}} ^ {T _ {i}}), \tag {10}
$$
which is essentially  $\mathcal{L}_{CLS}$  in Eqn 9 and inference on  $\mathcal{D}_{T_i}$  can be conducted by only using  $\tilde{\boldsymbol{P}}^{T_i}$ . An illustration of LST is shown in Fig 3b. Since the prompt for CLIP is of size  $K\times$  token length  $\times$  embedding size, adopting the LST strategy can both reduce the need of training individual prompts and decrease the number of tunable parameters by a factor of at least  $(N - 1)\times \frac{\text{embedding size}}{d_I}$  times compared with MPA. This is in fact non-trivial when confronting large-scale datasets such as DomainNet, where  $N = 6$ ,  $d_{I} = 250$ , and embedding size  $= 512$ .
Datasets and Metrics. Experiments are conducted on three popular benchmark datasets of UDA to evaluate the effectiveness of MPA, namely ImageCLEF, Office-Home, and DomainNet. ImageCLEF is a small-scaled dataset consisting of 1,800 images from 12 object categories in 3 different domains: ImageNet ILSVRC 2012 (I), Pascal VOC 2012 (P), and Caltech-256 (C). Office-Home is a medium-scaled dataset consisting of about 15,500 images from 65 categories in 4 different domains: Art, Clipart, Product, and Real World. DomainNet is the largest dataset to date, consisting of about 0.6 million images from 345 categories in 6 different domains: Clipart, Infograph, Painting, Quickdraw, Real, and Sketch.