SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen*
School of Electronic Engineering, Xidian University, Xi'an, China
{wds,limiaoge,xinyangatk,msxu}@stu.xidian.edu.cn
bchen@mail.xidian.edu.cn
Hanwang Zhang
School of Computer Science and Engineering, Nanyang Technological University, Singapore
hanwangzhang@ntu.edu.sg
Advancements in prompt tuning of vision-language models have underscored their potential in enhancing open-world visual concept comprehension. However, prior works only primarily focus on single-mode (only one prompt for each modality) and holistic level (image or sentence) semantic alignment, which fails to capture the sample diversity, leading to sub-optimal prompt discovery. To address the limitation, we propose a multi-mode token-level tuning framework that leverages the optimal transportation to learn and align a set of prompt tokens across modalities. Specifically, we rely on two essential factors: 1) multi-mode prompts discovery, which guarantees diverse semantic representations, and 2) token-level alignment, which helps explore fine-grained similarity. Consequently, the similarity can be calculated as a hierarchical transportation problem between the modality-specific sets. Extensive experiments on popular image recognition benchmarks show the superior generalization and few-shot abilities of our approach. The qualitative analysis demonstrates that the learned prompt tokens have the ability to capture diverse visual concepts. The code is available at https://github.com/wds2014/ALIGN.
Recently, prompt tuning has experienced significant advancements in adapting large pre-trained vision language models (PVLs) such as CLIP [1] and BLIP [2] to downstream tasks [3-6]. A typical PVL model consists of two branch networks: the text and image encoders. These networks are used to extract the corresponding modality features. PVLs are often contrastively pre-trained on Web-scale image-text pairs, which encourage the alignment of visual concepts with natural language in the shared semantic space. One of the core ideas behind prompt tuning is to formulate the downstream tasks into the original pre-training pipeline. For example, CLIP designs category descriptions with a manual prompt template "a photo of a {class} ", which works well in generic image recognition. Unlike fine-tuning, where the entire model is tuned using task-specific objectives, demands prohibitive computing cost, and poses a risk of knowledge shift issues [7-9], prompt tuning fixes the model parameters instead and optimizes prompt vectors, which act as demonstrations to help extract task-related features. This significantly benefits the representations via PVLs, even in performing zero-shot inference without training samples.
However, identifying optimal prompts for PVLs is not a trivial task, which usually needs to solve the intricate semantic alignments between the textual and visual modalities. Inspired by the success
Figure 1: The alignment comparison in recent prompt tuning methods. The proposed ALIGN learns multimodal multi-mode prompts at the same time, resulting in comprehensive alignments.
of prompt learning in neural language models (NLP) [10, 7, 11], approaches called textual prompt tuning (TPT) are proposed to learn continuous prompt embeddings for CLIP's text encoder, e.g., "X XXX {class} ", where "X" denotes the learnable vectors [3, 4]. Optimized with a task-specific loss, the learned prompt embeddings distill the pre-trained knowledge encoded in the fixed parameters, achieving better flexibility and efficiency than hand-crafted methods [1]. To improve the generalization of TPT on unseen classes, many studies attempt to give the solutions from gradient flow [12, 13], prototype and composition prompt learning [14-16]. Moving beyond learning a single-mode prompt, which often fails to capture diverse concepts, various methods prefer to explore multiple prompts based on ensemble learning[1], optimal transport [17] and Bayesian inference [18-20], showing robust alignments and better performance.
In parallel with TPT, visual prompt tuning (VPT) focuses on the patch embedding space of the CLIP's image encoder [6]. VPT views images as a patch sequence and introduces visual prompts to enhance the image representations, e.g., "XXXX{image}", where "image" denotes the image patch sequence. VPT provides a simple and efficient idea to extract task-relevant visual features, which has been widely applied to many visual tasks, for example, video understanding [21], domain adaptation [22], transfer learning [23] and image segmentation [24-26]. More recently, there has been a research trend to combine TPT and VPT to learn multi-modal prompts together [27, 28]. However, they currently concentrate on single-mode prompt discovery, i.e., only one prompt for one modality, which may be insufficient to represent a class [17]. This issue is even more acute in multi-modal prompt learning, where both visual and textual concepts and their alignments need to be inferred. Additionally, it is less sound to represent the image and label only with the global features [29, 30], which could lose local region features of the target object, resulting in sub-optimal classification.
To this end, this work develops a comprehensive prompt tuning framework, where multi-modal multi-mode prompts are learned by building the prompt and token-level optimal transport (OT). Formally, after feeding multiple prompt inputs into the modality-specific encoder, our prompt-level OT views each image as a discrete distribution  $P$  over the visual prompt space and views each label as a discrete distribution  $Q$  over the textual prompt space. With such formulation, the classification task becomes to measure the distance between  $P$  and  $Q$ . Moreover, along with the global prompt-level features, the patch (or token) embeddings capture the local region features of the target object (or category description). This motivates the token-level OT, where each prompt output is modeled as a discrete distribution over the token embedding space. The cost matrix is then calculated between the visual patches and textual tokens, enabling the token-level alignments. Crucially, the cost matrix in prompt-level OT that measures the transport cost between prompts from two domains is now converted to integrate the global features and the output of the token-level OT. This hierarchical connection makes it possible to predict the label with the detailed token and patch features, resulting in higher accuracy.
In summary, our method provides a novel prompt tuning framework that incorporates multiple modalities and token-level alignments via the hierarchical OT. The prompt-level OT learns the diverse semantics of a class from both image and language domains, and the token-level OT explores fine-grained alignments between token embeddings. Notably, with different hyperparameter settings, the variants of the proposed model cover many previous works, offering flexibility for easy adaptation across diverse applications. The main contributions of the paper are as follows:
- We propose a multi-mode token-level alignment framework for multi-modal prompts tuning, where multiple prompts are learned to improve the representation for both visual and textual modalities. With special settings, many previous works can be margined into our framework.
- We formulate the prompt tuning task as the distribution matching problem, and develop the prompt and token-level OT to tackle the task with a principle and elegant solution.
- We apply our method to few-shot classification, dataset transfer learning and domain generalization. Experiential results on widely used datasets show the superiority of the proposed model.
2.1 Multi-modal Prompt Tuning
Multi-modal prompt tuning (MPT) [28, 27] is a newly developed task that enables the joint learning of textual and visual prompts for PVLs. Instead of optimizing the unimodal prompts separately, the joint tuning paradigm not only leverages the two branch networks of PVLs, but also allows interactions between two modalities during training, resulting in dynamic alignments. Without loss of generality, we use a vision transformer (ViT) based CLIP for example, which consists of a ViT as an image encoder  $f$  and a transformer as a language encoder  $g$ . Given an input image  $\mathbf{X} \in R^{H \times W \times 3}$  and  $K$  label names  $\{class_k\}_{k=1}^K$ , MPT first incorporates  $b$  learnable tokens as visual prompts  $\{v_i \in R^{d_v}\}_{i=1}^b$ , and another set of  $b$  learnable tokens as textual prompts  $\{t_i \in R^{d_l}\}_{i=1}^b$ . After concatenating them alongside the image patches and class names, one can obtain the output of CLIP as:
$$
\begin{array}{l} [ \boldsymbol {z}, \widetilde {\boldsymbol {e}} _ {1}, \dots , \widetilde {\boldsymbol {e}} _ {O}, \widetilde {\boldsymbol {v}} _ {1}, \dots , \widetilde {\boldsymbol {v}} _ {b} ] = f (<   c l s >, \boldsymbol {e} _ {1}, \dots , \boldsymbol {e} _ {O}, \boldsymbol {v} _ {1}, \dots , \boldsymbol {v} _ {b}), \\ [ -, \widetilde {t} _ {1}, \dots , \widetilde {t} _ {b}, \widetilde {\boldsymbol {w}} _ {k, 1}, \dots , \widetilde {\boldsymbol {w}} _ {k, k _ {l}}, \boldsymbol {h} _ {k} ] = g (<   c l s >, \boldsymbol {t} _ {1}, \dots , \boldsymbol {t} _ {b}, \boldsymbol {w} _ {k, 1}, \dots , \boldsymbol {w} _ {k, k _ {l}}, <   e o s >), \\ \end{array}
$$
where  $< \text{cls} >, < \text{eos} >$  are virtual tokens,  $[e_1, \dots, e_O]$  are  $O$  image patch embeddings, and  $[\pmb{w}_{k,1}, \dots, \pmb{w}_{k,k_l}]$  are token embeddings with length  $k_l$  of  $k$ -th class. After the stacked self-attention layers of  $f$  and  $g$ , CLIP outputs the token embeddings and views  $z$  and  $h_k$  as the prompt-level features of the image and label, respectively. Empirical findings suggest that it is more effective to obtain the vision prompt  $v$  by projecting the language prompt  $t$  through a vision-to-language mapping function, such as  $v = F(t)$ , rather than learning them independently [28, 6]. Finally, MPT estimates the label of  $x$  according to the cosine similarity score:
$$
p (y = k | \boldsymbol {x}) = \frac {\exp \left(\operatorname {s i m} \left(\boldsymbol {z} , \boldsymbol {h} _ {k}\right) / \tau\right)}{\sum_ {k ^ {\prime} = 1} ^ {K} \exp \left(\operatorname {s i m} \left(\boldsymbol {z} , \boldsymbol {h} _ {k ^ {\prime}}\right) / \tau\right)}, \tag {1}
$$
where  $\tau$  is the fixed temperature parameter. MPT unifies the ideas of TPT and VPT by directly tuning the visual prompts  $v$  and textual prompt  $t$  at the same time. Eq. 1 indicates that the text encoder  $g$  takes the category prompts as input and outputs  $h$ , which serves as the corresponding classifier weights. Thanks to the pre-trained knowledge in CLIP, MPT retains the ability to perform open-set classification. Note that both the encoders  $f$  and  $g$  in CLIP are frozen and only the prompt sequences  $v$  and  $t$  are optimized during downstream training. This process can be seen as a bootstrapping step that helps guide the encoders to extract task-relevant features.
2.2 Optimal Transport Distance
Optimal transport (OT) is an efficient tool to measure the distance between two distributions, which is widely used in recent machine learning studies, such as text analysis [31-33], computer vision [34-39] and generative model [40, 41]. Here we review the discrete OT matching and refer readers to [42] for details. Given two sets of data points  $\mathbf{X} = \{x_{i}\}_{i = 1}^{m}$  and  $\mathbf{Y} = \{y_j\}_{j = 1}^n$ , of which discrete distributions are formulated as  $p = \sum_{i = 1}^{m}a_{i}\delta_{x_{i}}$  and  $q = \sum_{j = 1}^{n}b_{j}\delta_{y_{j}}$ , respectively.  $\mathbf{a}\in \Delta^m$  and  $\mathbf{b}\in \Delta^n$ , where  $\Delta^m$  denotes the probability simple of  $R^m$ . We define the cost matrix between  $\mathbf{X}$  and  $\mathbf{Y}$  as  $\mathbf{C} = (C_{ij})\in R_{\geq 0}^{m\times n}$ , where  $C_{ij} = c(x_i,y_j)$  is the transport cost from  $x_{i}$  to  $y_{j}$ , with  $c$  is the cost function. The goal of OT is to optimally transport  $p$  to  $q$  at the smallest cost:
$$
d _ {\mathrm {O T}} (p, q; \mathbf {C}) := \underset {\mathbf {T} \in \Pi (p, q)} {\min } <   \mathbf {T}, \mathbf {C} >, \tag {2}
$$
where  $<\cdot, \cdot>$  denotes the Frobenius dot-product and  $\mathbf{T} \in \mathbb{R}_{>0}^{m \times n}$  denotes the transport plan to be learned. OT distance is then minimized over all the joint probabilities of  $m \times n$  space with two marginal constraints  $\Pi(p,q) := \{\mathbf{T}: \mathbf{T1}_n = \mathbf{a}, \mathbf{T}^T \mathbf{1}_m = \mathbf{b}\}$ , where  $\mathbf{1}_m$  denotes m-dimensional
all-one vector. As directly learning the optimal plan  $\mathbf{T}$  in Eq. 2 can be time-consuming for large-scale problems, Sinkhorn distance from [42, 43] introduces the entropic constraint on the transport plan  $h(\mathbf{T}) = \sum_{m,n} - T_{mn}\ln (T_{mn})$  and thus the resulting algorithm estimates  $\mathbf{T}$  within a few iterations, showing better flexibility and scalability.
3 The Proposed Model
3.1 Overall Method
In this section, we introduce the technical details of our proposed model, named ALIGN, a holistic framework for multi-modal prompt tuning with optimal transport (shown in Fig. 2). Benefiting from the carefully designed multi-mode token-level alignment module, most existing works can be merged into our ALIGN with special settings. Intuitively, humans learn one class with various concepts, which provides sufficient semantic features, such as color, layout, and shape, to distinguish it from others [17]. Inspired by this, one of the goals of this work is to learn  $M$  visual prompt and  $N$  textual prompt simultaneously. Specifically, we first introduce our prompt-level OT, where each image and label are modeled as the discrete distributions  $P$  and  $Q$  over  $M$ -dimensional visual space and  $N$ -dimensional textual space. Moreover, instead of representing the prompt outputs as a single point, e.g., the global features  $z$  and  $h$ , we distill the token-level knowledge implied in CLIP. Recalling that, the  $n$ -th textual prompt output of the  $k$ -th class contains  $b + k_l$  token embeddings and the  $m$ -th visual prompt output of an image contains  $b + O$  patch embeddings, which capture the local-region features of corresponding modalities. This motivated us to develop the token-level OT that makes token-level comparisons for fine-grained alignments. As a result,  $m$ -th and  $n$ -th points in  $P$  and  $Q$  themselves are further modeled as discrete distributions over the shared token embedding space. Due to the compelling two-level OT connections, where the cost matrix in prompt-level OT is obtained by the output of token-level OT, the learned transport plan captures both the prompt and token-level features, which provides a principled and elegant way to estimate the distance between label and image sets.
3.2 Multi-mode Token-level Prompt Alignment
Moving beyond MPT which learns a single-mode prompt to describe the class and estimates the similarity based on prompt-level features, we aim to explore multi-mode representations in the textual and visual domains and make fine-grained alignment to improve the prediction accuracy. Now we have  $M$  groups of visual prompts  $\{\pmb{v}^m\}_{m=1}^M$  and  $N$  groups of textual prompts  $\{\pmb{t}^n\}_{n=1}^N$ , where each  $\pmb{v}^m \in R^{d_v \times b}$  and  $\pmb{t}^n \in R^{d_l \times b}$  are learnable prompt sequences with length  $b$ . Mathematically, we employ two empirical distributions  $P$  and  $Q$  to model the sets of two modalities:
$$
P = \sum_ {m = 1} ^ {M} \frac {1}{M} \delta_ {\boldsymbol {x} _ {m}}, \quad Q = \sum_ {n = 1} ^ {N} \frac {1}{N} \delta_ {\boldsymbol {y} _ {n}}, \tag {3}
$$
where  $\mathbf{x}_m$  and  $\mathbf{y}_n$  denote the  $m$ -th visual output and  $n$ -th textual output in the  $d$ -dimensional latent space. They are further modeled as discrete distributions over token-level embeddings, which will be introduced later. Eq. 3 views each prompt equally and adopts the uniform distribution to model the weights. With those two semantic sets  $P$  and  $Q$ , the distance between images and labels is no longer calculated by first representing each image and label as a single point and then using the cosine similarity. ALIGN prefers to mine multi-mode features to describe various class concepts, resulting in better representations. The distance thus can be formulated as an entropy-regularized prompt-level OT problem [42]:
$$
d _ {\mathrm {O T}} ^ {\lambda} (P, Q; \mathbf {C}) := d _ {\mathrm {O T}} (P, Q; \mathbf {C}) - \lambda h (\mathbf {T}) \tag {4}
$$
where  $\lambda > 0$  is the weight of regularization, and  $\mathbf{C} \in R^{M \times N}$  is the cost matrix between visual set  $x$  and textual set  $y$ .  $\mathbf{T} \in R^{M \times N}$  is the to-be-learned transport plan with the marginal constraint, e.g.,  $\mathbf{T} \mathbb{1}_N = 1 / M$ ,  $\mathbf{T}^T \mathbb{1}_M = 1 / N$ . Note that,  $T_{mn}$  measures the transported probability from  $m$ -th visual prompt to  $n$ -th textual prompt, and a large value means the high semantic connection between two prompts across modalities. Therefore, Eq. 4 estimates the expected transport cost between  $P$  and  $Q$ , which provides a principle solution to calculate the similarity between the images and labels.
Noticeably, the cost matrix  $\mathbf{C}$  in Eq. 4 plays a critical role in the learning of  $\mathbf{T}$ , and intuitively, the larger the transport costs between two points are, the lower the transport probabilities will be. A
Figure 2: (a) The framework of the proposed ALIGN. ALIGN learns multiple prompts for PVLs by aligning modality-specific distributions with hierarchical OT. (b) The t-SNE visualization of image embeddings of ALIGN.
natural choice is to specify  $\mathbf{C}$  with the global features  $C_{mn} = 1 - \mathrm{sim}(z^m, h^n)$ , where  $z^m$  and  $h^n$  denote the prompt-level features of  $m$ -th visual prompt and  $n$ -th textual prompt. However, the above definition primarily emphasizes prompt-level representation and might have a limited capacity to capture the detailed token-level features, e.g., different patches within an image may capture various local region features. Thus, the obtained transport plan may fail to reflect the true relations between  $P$  and  $\bar{Q}$ . To this end, we further introduce the token-level OT that considers token-level alignments between two prompts. Specifically, we specify the visual output  $\pmb{x}$  and textual output  $\pmb{y}$  as two empirical distributions over token embeddings (here we omit the subscript  $m$  and  $n$  for clarity):
$$
\boldsymbol {x} = \sum_ {j = 1} ^ {J} \frac {1}{J} \delta_ {\boldsymbol {r} _ {j}}, \quad \boldsymbol {y} = \sum_ {l = 1} ^ {L} \frac {1}{L} \delta_ {\hat {\boldsymbol {s}} _ {l}},
$$
where  $\boldsymbol{r} = [\widetilde{\boldsymbol{e}}_1, \dots, \widetilde{\boldsymbol{e}}_O, \widetilde{\boldsymbol{v}}_1, \dots, \widetilde{\boldsymbol{v}}_b]$  is the output visual patches with length  $J = b + O$ , and  $\boldsymbol{s} = [\widetilde{\boldsymbol{t}}_1, \dots, \widetilde{\boldsymbol{t}}_b, \widetilde{\boldsymbol{w}}_{k,1}, \dots, \widetilde{\boldsymbol{w}}_{k,k_l}]$  is the output textual tokens with length  $b + k_l$ . Unlike  $\boldsymbol{z}$  and  $\boldsymbol{h}$  that agent the prompt-level features,  $\boldsymbol{x}$  and  $\boldsymbol{y}$  collect the token-level features in the shared embedding space of CLIP. Naturally, the cost matrix  $\hat{\mathbf{C}} \in R^{J \times L}$  in the token-level OT is defined as  $\hat{C}_{jl} = 1 - \mathrm{sim}(\boldsymbol{r}_j, \boldsymbol{s}_l)$ , which measures the transport cost between the visual patches and textual tokens. As a result, the distance between  $\boldsymbol{x}$  and  $\boldsymbol{y}$  is the total transport cost of the token-level OT:
$$
d _ {\mathrm {O T}} ^ {\lambda} (\boldsymbol {x}, \boldsymbol {y}; \hat {\mathbf {C}}) = d _ {\mathrm {O T}} ^ {\lambda} (\boldsymbol {x}, \boldsymbol {y}; \hat {\mathbf {C}}) - \lambda h (\hat {\mathbf {T}}), \tag {5}
$$
where the transport plan  $\hat{\mathbf{T}}\in R^{J\times L}$  denotes how likely is that the  $j$  -th visual patch transports to the  $l$  -th token feature, providing a principle solution to align token-level features. This motivated us to develop a combined cost matrix that considers prompt and token-level features together:
$$
C _ {m n} = 1 - \operatorname {s i m} \left(\boldsymbol {z} ^ {m}, \boldsymbol {h} ^ {n}\right) + \beta d _ {\mathrm {O T}} ^ {\lambda} \left(\boldsymbol {x} _ {m}, \boldsymbol {y} _ {n}; \hat {\mathbf {C}} ^ {m n}\right), \tag {6}
$$
where  $\beta$  is a trade-off parameter that controls the weight of token-level cost. The first two terms are the cosine distance between prompt-level features, and the last term is the OT distance between the token-level sets. In this way, Eq. 6 combines the pre-trained knowledge from two levels: the prompt-level features and the token-level embeddings. This enables the learned transport plan  $\mathbf{T}$  in prompt-level OT to make fine-grained matching between  $M$  visual and  $N$  textual features, resulting in detailed alignments and better representations.
Once Eq. 4 is computed, we follow previous work [17] and predict the label of image  $\mathbf{X}_j$  as:
$$
p (y = k | \mathbf {X} _ {j}) = \frac {\exp \left(\left(1 - d _ {\mathrm {O T}} ^ {\lambda} \left(P _ {j} , Q _ {k} ; \mathbf {C} ^ {j k}\right)\right) / \tau\right)}{\sum_ {k ^ {\prime} = 1} ^ {K} \exp \left(\left(1 - d _ {\mathrm {O T}} ^ {\lambda} \left(P _ {j} , Q _ {k ^ {\prime}} ; \mathbf {C} ^ {j k ^ {\prime}}\right)\right) / \tau\right)}, \tag {7}
$$
where  $\mathbf{C}^{j,k}$  denote the cost matrix of  $j$ -th image and  $k$ -th label. Note that the weight of the classifier  $Q_{k}$  in our model can be viewed as a discrete uniform distribution over  $N$  textual prompts of label  $k$ , which contains multiple class-related semantics, improving the classification results. Thanks to the
differentiable Sinkhorn algorithm, all parameters of the proposed model can be optimized end-to-end by minimizing the following cross-entropy loss:
$$
L = - \frac {1}{| \mathcal {X} |} \sum_ {\mathbf {X} \in \mathcal {X}} \sum_ {k = 1} ^ {K} y _ {\boldsymbol {x}, c} p (y = k | \boldsymbol {x}). \tag {8}
$$
where  $y_{\mathbf{X}}$  is the one-hot label vector of image  $\mathbf{X}$ . Due to the OT formulation, our proposed ALIGN aims to learn  $M$  visual prompt sequences and  $N$  textual prompt sequences without introducing any neural networks. We describe our proposed model in the Appendix Algorithm 1.
Single-modal prompt tuning: There are two storylines of single-modal prompt tuning, TPT and VPT. The former focuses on the language branch of a PLV and is interested in prompt learning in continuous embedding space. As one of the representative works, CoOp [3] models a prompt's context using a set of learnable vectors and shows great improvement over intensively-tuned manual prompts. To solve the weak generalizability on unseen category, CoCoOp [4] extends CoOp by explicitly conditioning prompts on image instances, which shifts the concentrations away from a specific set of classes to each input instance, enabling a stronger generalization performance. Instead of single-mode prompt learning, PLOT [17] learns multiple textual prompts by adopting the OT distance between prompts and image patches, achieving diverse prompt tuning. ProDA [19] first maturely designs multiple prompts and then models the uncertainty of prompts by employing the Gaussian distribution to model prompt embeddings. Correspondingly, VPTs refer to prepending visual patches to the image input space, which also shows impressive results in adapting PVLs into downstream tasks. For example, Jia et al. [6] introduces trainable visual prompt vectors into the image patch sequence of each Transformer layer and learns them along with a linear head. Despite the promising performance on various visual tasks, those models are designed to learn single-modal prompts, which fails to make use of the pre-trained multi-modal knowledge.