Tuning Multi-mode Token-level Prompt Alignment across Modalities
Authors: Dongsheng Wang
Miaoge Li
Xinyang Liu
MingSheng Xu
Bo Chen
Hanwang Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-modal prompt learning; Optimal transport
TL;DR: We in this paper propose a novel multi-mode token-level prompt across modalities under the optimal transport theory

Abstract: Advancements in prompt tuning of vision-language models have underscored their potential in enhancing open-world visual concept comprehension. However, prior works only primarily focus on single-mode (only one prompt for each modality) and holistic level (image or sentence) semantic alignment, which fails to capture the sample diversity, leading to sub-optimal prompt discovery. To address the limitation, we propose a multi-mode token-level tuning framework that leverages the optimal transportation to learn and align a set of prompt tokens across modalities. Specifically, we rely on two essential factors: 1) multi-mode prompts discovery, which guarantees diverse semantic representations, and 2) token-level alignment, which helps explore fine-grained similarity. Consequently, the similarity can be calculated as a hierarchical transportation problem between the modality-specific sets. Extensive experiments on popular image recognition benchmarks show the superior generalization and few-shot abilities of our approach. The qualitative analysis demonstrates that the learned prompt tokens have the ability to capture diverse visual concepts.

Tuning Multi-mode Token-level Prompt Alignment across Modalities
Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen" School of Electromic Engineering, Xidian University, Xi'an, an, China { (wds, limiaoge, xinyangatk, ms msxui @stu. xidiaan. edu, en bchen&mail 1. xidi an. edu. cn
Hanwang Zhang
School of Computer Science and Engineering, Nanyang Technological University, Singapore hanwangzhang@@tuu. edu.
Abstract
Advancements in prompt tuning of vision-language models have underscored their potential in enhancing open-world visual concept comprehension. However, prior works only primarily focus on single-mode (only one prompt for each modality) and holistic level (image sentence) semantic alignment, which fails to capture the sample diversity, leading to sub-optimal prompt discovery. To address the limi- tation, we propose multi-mode leven-level tuning framework that leverages the optimal transportation to learn and align set of prompt tokens across modalities. Specifically, we rely two essential factors: 1) multi-mode prompts discovery, which guarantees diverse semantic representations, and 2) token-level alignment, which helps explore fine-grained similarity. Consequently, the similarity be calculated as a hierarchical transportation problem between the modality-specific sets. Extensive experiments on popular image recognition benchmarks show the superior generalization and few-shot abilities of our approach. The qualitative anal- ysis demonstrates that the learned prompt tokens have the ability capture diverse visual concepts. The code is available at https://github..cm https://gihubbcoooooo cwwm///////oo/ce hwtpm//ds211/A...... 1/ALIGN.
1 Introduction
Recently, prompt tuning has experienced significant advancements in adapting large pre-trained vision language models (PVLs) such as CLIP [1] and BLIP [2] to downstream tasks [3-6]. A typical PVL model consists of two branch networks: the text and image encoders. These networks are used to extract the corresponding modality features. PVLs are often contrastively pre-trained on Web-scale image-text pairs, which encourage the alignment of visual concepts with natural language in the shared semantic space. One of the core ideas behind prompt tuning is to formulate downstream tasks into the original pre-training pipeline. For example, CLIP designs category descriptions with manual prompt template "a photo of a {class)" )" which works well in generic image recognition. Unlike fine-tuning, where the entire model is tuned using task-specific objectives, demands prohibitive computing cost, and poses a risk of knowledge shift issues [7-9], prompt tuning fixes the model parameters instcad and optimizes prompt vectors, which act as demonstrations to help extract teskerelated features. This significantly benefits the representations via PVLs, even in performing zero-shot inference without training samples.
However, identifying optimal prompts for PVLs is not trivial task, which usually needs to solve the intricate semantic alignments between the textual and visual modalities. Inspired by the success
"Correspondigg author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

deg
Kpop>
phsto a dog
CeoSP
dogg
Keoph
dog
dog
TPT
(b)
(c) PLOT
(c) MPT
(e) (e)AN (ours)
Figure I:
The alignment comparison in recent prompt tuning methods. The proposed ALIGN learns multi- modal multi-mode prompts at the same time, resulting in comprchensive alignments.
of prompt learning in neural language models (NLP) [10, 7, 11], approaches called textual prompt tuning (TPT) are proposed to learn continuous prompt embeddings for CLIPP's text encoder, e.g., "X XXX {class}". where "X" denotes the learnable vectors [3, 4]. Optimized with a task-specific loss, the learned prompt embeddings distill the pre-trained knowledge encoded in the fixed parameters, achieving better flexibility and efficiency than hand-crafted methods [1]. To improve the generaliza- tion of TPT unseen classes, many studies attempt to give the solutions from gradient flow [12, 13], prototype and composition prompt learning [14-16]. Moving beyond learning a single-mode prompt, which often fails to capture diverse concepts. various methods prefer to explore multiple prompts based on ensemble learning[1]] J. optimal transport [17] and Bayesian inference [18-20], showing robust alignments and better performance.
In parallel with TPT, visual prompt tuning (VPT) focuses on the patch embedding space of the CLIP's image encoder [6]. VPT views images patch sequence and introduces visual prompts to enhance the image representations, egg.. "X X {image]", where "image" denotes the image patch sequence. VPT provides simple and efficient idea to extract task-relevant visual features, which has been widely applied to many visual tasks, for example, video understanding [21], domain adaptation [22], transfer learning [23] and image segmentation [24-26]. More recently, there has been a research trend to combine TPT and VPT to learn modal prompts together [27, 28]. However, Langle-odee prompt discovery, i.e., only one prompt for modality, which may be insufficient to represent a class [17]. This issue is even more acute in multi-modal prompt learning, where both visual and textual concepts and their alignments need to be inferred. Additionally, it is less sound to represent the image and label only with the global features [29, 30], which could lose local region features of the target object, resulting in sub-optimal classification.
they currently concentrate
To this end, this work develops a comprehensive prompt tuning framework, where multi-modal multi-mode prompts are learned by building the prompt and token-level optimal transport (OT). Formally, after feeding multiple prompt inputs into the modality-speciffc encoder, our prompt-level OT views each image discrete distribution P over the visual prompt space and views each label as a discrete distribution Q over the textual prompt space. With such formulation, the classification task becomes to measure the distance between P and Q. Moreover, along with the global prompt-level features, the patch (or token) embeddings capture the local region features of the target object (or category description). This motivates the token-level OT, where each prompt output is modeled as discrete distribution over the token embedding space. The cost matrix is then calculated between the visual patches and textual tokens, enabling the token-level alignments. Crucially, the cost matrix in prompt-level OT that measures the transport cost between prompts from two domains is now converted integrate the global features and the output of the token-level OT. This hierarchical connection makes it possible to predict the label with the detailed token and patch features, resulting in higher accuracy.
In summary, our method provides a novel prompt tuning framework that incorporates multiple modalities and token-level alignments via the hicrarchical OT. The prompt-level OT learns the diverse semantics of class from both image and language domains, and the token-level OT explores fine- grained alignments between token embeddings. Notably, with different hyperparameter settings, the variants of the proposed model cover many previous works, offering flexibility for easy adaptation across diverse applications. The main contributions of the paper are follows:
We propose multi-mode token-level alignment framework for multi-modal prompts tuning, where multiple prompts are learned to improve the representation for both visual and textual modalities. With special settings, many previous works can be margined into our framework.

---

We formulate the prompt tuning task as the distribution matching problem, and develop the prompt and token-level OT to tackle the task with principle and elegant solution.
We apply our method to few-shot classification, dataset transfer learning and domain generalization. Experiential results on widely used datasets show the superiority of the proposed model.
2 Background
2.1 Multi-modal Prompt Tuning
Multi-modal prompt tuning (MPT) [28, 27] is newly developed task that enables the joint learning of textual and visual prompts for PVLs. Instead of optimizing the unimodal prompts separately, the joint tuning paradigm not only leverages the two branch networks of PVLs, but also allows interactions between two modalities during training, resulting in dynamic alignments. Without loss of generality, we use a vision transformer (ViT) based CLIP for example, which consists of a ViT as an image encoder f and a transformer as language encoder g. Given an input image X E # x3 and K label names {class;, } 1 MPT first incorporates learnable tokens as visual prompts {u; E R and another set of learnable tokens as textual prompts {t, E R R } After concatenating them alongside the image patches and class names, one can obtain the output of CLIP as:
[2, (1, Co: U1 ... T6 [(< cls 2, E1, eo, U1 0b);
[-, to, 1 Wk hR] = g(< cls >, t1, ..., to Wk.1, We.ki < EOS >)
where < cls >.< cos > are virtual tokens, [el.... co] are image patch embeddings, and (,,,1,...., Wk.k, are token embeddings with length ky of k-th class. After the stacked self-attention layers of f and g. CLIP outputs the token embeddings and views z and h& as the prompt-level features of the image and label, respectively. Empirical findings suggest that it is more effective to obtain the vision prompt v by projecting the language prompt through a vision-to-language F(t). rather than learning them independently [28, 6]. Finally, MPT
mapping function, such estimates the label æ according to the cosine similarity score:
ply k/wr exp(sim(2 z, h2)/T)
k''m exp(sim(z, hw )/T)
where 7 is the fixed temperature parameter. MPT unifies the ideas of TPT and VPT by directly tuning the visual prompts and textual prompt t at the same time. Eq. 1 indicates that the text encoder g takes the category prompts as input and outputs h. which serves as the corresponding classifier weights. Thanks to the pre-trained knowledge in CLIP, MPT retains the ability to perform open-set classification. Note that both the encoders f and g in CLIP are frozen and only the prompt sequences and t are optimized during downstream training. This process can be seen bootstrapping step that helps guide the encoders to extract task-relevant features.
2.2 Optimal Transport Distance
Optimal transport (OT) is efficient tool to measure the distance between two distributions, which is widely used in recent machine learning studies, such as text analysis [31-33], computer vision [34- 39] and generative model [40, 41]. Here we review the discrete OT matching and refer readers to [42] for details. Given two sets of data points X = {m and Y = {リ,) of which discrete distributions are formulated as = Σm= 0g5g; and = Ej=1 وyررر= respectively. a E Am and b E A", where M denotes the probability simple of R. We define the cost matrix between X and Y C- (Ctz) E R" where Ci c(24, 3j) the transport cost from I; yj. with c is the cost function. The goal of OT is to optimally transport p to q the smallest cost:
dor T C) TEll(p-g)
min
< <T,C C>
where <., denotes the Frobenius dot- dot-product and T E I denotes the transport plan to be learned. OT distance then minimized over all the joint probabilities of m x n space with two marginal constraints 1I(p, g) := (T T1, a, T b). where 1m, denotes m-dimensional
3

---

all-one vector. As directly learning the optimal plan T in Eq. 2 can be time- consuming for large-scale problems, Sinkhorn distance from [42, 43] introduces the entropic constraint on the transport plan ((T) = Em, -7 In (Tma) and thus the resulting algorithm estimates T within a few iterations, showing better flexibility and scalability.
3
The Proposed Model
3.1 Overall Method
In this section, we introduce the technical details of our proposed model, named ALIGN, a holistic framework for multi-modal prompt tuning with optimal transport (shown in Fig. 2). Benefiting from the carefully designed multi-mode token-level aligmment module, most existing works can be merged into our ALIGN with special settings. Intuitively, humans learn one class with various concepts, which provides sufficient semantic features, such as color, layout, and shape, to distinguish it from others [17]. Inspired by this, one of the goals of this work is to learn M visual prompt and N textual prompt simultaneouslly Specifically, we first introduce our prompt-level OT, where each image and label are modeled as the discrete distributions P and Q over -ddimensional visual space and N- N-dimensional textual space. Moreover, instead of representing the prompt outputs as single point, egg., the global features z and h, we distill the token-level knowledge implied in CL.P. Recalling that, the n-th textual prompt output of the k-th class contains b + ky token embeddings and the m-th visual prompt output of an image contains + O patch emheddings, which capture the local-region features of corresponding modalities. This motivated us to develop the token-level OT that makes token-level comparisons for fine-grained alignments. As a result, m-th and n-th points in and Q themselves are further modeled as discrete distributions over the shared token embedding space. Due to the compelling two-level OT connections, where the cost matrix in prompt-level OT is obtained by the output of token-level OT, the learned transport plan captures both the prompt and token-level features, which provides a principled and elegant way to estimate the distance between label and image sets.
3.2 Multi-mode Token-level Prompt Alignment
Moving beyond MPT which learns single-mod prompt to describe the class and estimates the similarity based on prompt-level features, we aim to explore multi-mode representations in the textual and visual domains and make fine-grained alignment to improve the prediction accuracy. Now we have M groups of visual prompts { Jm=1 and N groups of textual prompts ((") where each n E Rd xb and tn E Rd are learnable prompt sequences with length 6. Mathematically, we employ two empirical distributions P' and Q model the sets of two modalities:
LM
M
N
P
Sam. M
Q
N YN.
3))
m=1
==1
where xm and yn denote the m-th visual output and n-th textual output in the d-dimensional latent space. They are further modeled as discrete distributions over token-level embeddings, which will be introduced later. Eq. views each prompt equally and adopts the uniform distribution to model the weights. With those two semantic sets P and Q, the distance between images and labels is no longer calculated by first representing each image and label as a single point and then using the cosine similarity. ALIGN prefers to mine multi-mode features to describe various class concepts. resulting in better representations. The distance thus can be formulated as an entropy- regularized prompt-level OT problem [42]:
(=) dor C) AT(
where À > 0 is the weight of regularization, and C E RMXN is the cost matrix between visual set and textual set y. T E RMX is the to-be-learned transport plan with the marginal constraint, egg.TTy = 1/M, 1 = 1/N. Note that, T.n measures the transported probability from m-thh visual prompt to n-th textual prompt, and large value means the high semantic connection between two prompts across modalities. Therefore, Eq. 4 estimates the expected transport cost between P and Q. which provides principle solution to calculate the similarity between the images and labels.
Noticeably, the cost matrix C in Eq. 4 plays critical role in the learning of T, and intuitively, the larger the transport costs between two points are, the lower the transport probabilities will be. A

---

don
dog
textual prompts
Base 94.03
C.
Token-level
M visual prompts
New 74.9
Prompt-level OT
(a) Overview of the pruposed ALIGN
(b) t-SNE Eurosat
Figure 2: (a) The framework of the proposed ALIGN. ALIGN learns multiple prompts for PVLs aligning modelity-specific distributions with hierarchical OT. (b) The t-SNE visualization of image embeddings of ALIGN.
natural choice to specify C with the global features Can =1 denote the prompt-level features of m-th visual prompt and n-th textual prompt. However, the above definition primarily emphasizes prompt-level representation and might have a limited capacity to capture the detailed token-level features, e.g., different patches within image may capture various local region features. Thus, obtained transport plan may fail to reflect the true relations between P and Q. To this end, we further introduce the token-level OT that considers token-level alignments between two prompts. Specifically, we specify the visual output x and textual output as two empirical distributions over token embeddings (here omit the subscript and n for clarity):
sim(2"". h²),, where nMh and h
y
j=1
where [e..... ēo: 01 Ub the output visual patches with length [t, [...... La, Wk,,1, ...; Wg,s, the output textual tokens with length b ++ hq. Unlike and h that agent the prompt-level features, a and collect the token-level features in the shared embedding space of CLIP. Naturally, the cost matrix Ć E R'x in the token-level OT is defined as C₁ - sim(r,; (r,, 8r), which measures the transport cost between the visual patches and textual tokens. As a result, the distance between and y is the total transport cost of the foken-level OT:
O, and
dar (2, y: C) = dor (x. y: C) AT(T).
(5)
where the transport plan T J denotes how likely that the j-th visual patch transports the I-th token feature, providing principle solution to align token-level features. This motivated us to develop a combined cost matrix that considers prompt and token-level features together:
Cmn sim(=" h²") + (do yni CTmn)
where B is a trade-off parameter that controls the weight of token-level cost. The first two terms are the cosine distance between prompt-level features, and the last term the OT distance between the token-level sets. In this way, Eq. 6 combines the pre-trained knowledge from two levels: the prompt-level features and the token-level embeddings. This enables the learned transport plan T in prompt-level OT to make fine-gre grained matching between M visual and N textual features, resulting detailed alignments and better representations.
Once Eq. 4 is computed, we follow previous work [17] and predict the label of image X, as:
exp((1 - drr (P; Qx: //T)
ply py=
ΣK=1 exp((1 (P,, QR: COk ()7))
where C denote the cost matrix of j-th image and k-th label. Note that the weight of the classifier Qs our model can viewed as discrete uniform distribution over N textual prompts of label which contains multiple class-related semantics, improving the classification results. Thanks to the

---

differentiable Sinkhorn algorithm, all parameters of the proposed model can be optimized end-to- end by minimizing the following cross-entropy loss:
1
Yu Ju.e k/x²).
(8)
X
X=
where yx is the one-hot label vector of image X Due to the OT formulation, our proposed ALIGN aims to learn M visual prompt sequences and N textual prompt sequences without introducing any neural networks. We describe our proposed model in the Appendix Algoritm. 1.
4 Related Work
Single-modal prompt tuning: There are two storylines of single-modal prompt tuning, TPT and VPT. The former focuses on the language branch of a PLV and is interested in prompt learning in continuous embedding space. As one of the representative works, CoOp [3] models a prompt's context using a set of learnable vectors and shows great improvement over intensively-ture sttuned manual prompts. To solve the weak generalizability on unseen category. CoCoOp [4] extends CoOp by explicitly conditioning prompts on image instances, which shifts the concentrations away from a specific set of classes to each input instance, enabling a stronger generalization performance. Instead of single-mode prompt learning, PLOT [17] learns multiple textual prompts by adopting the OT distance between prompts and image patches, achieving diverse prompt tuning. ProDA [19] first maturely designs multiple prompts and then models the uncertainty of prompts by employing the Gaussian distribution to model prompt embeddings. Correspondingly, VPTs refer to prepending visual patches to the image input space, which also shows impressive results in adapting PVLs into downstream tasks. For example, Jia et al. [6] introduces trainable visual prompt vectors into the image patch sequence of each Transformer layer and learns them along with a linear head. Despite the promising performance various visual tasks, those models are designed learn single-modal prompts, which fails to make use of the pre-trained multi-modal knowledge.
Multi-modal prompt tuning: Moving beyond single- modal prompt tuning, MPT is recently introduced task that learns textual prompts and visual prompts at the same time. This jointly tuning strategy not only distills the multi multi-modal knowledge but enables the dynamic alignments between prompts across modalities, showing better generalization. Zang et al. [27] propose a unified prompt tuning framework (UPT) [27] that shares initial prompt across different modalities and designs tiny network to generate the modality prompts together. Almost parallel to UPT, Khattak al. [28] proposed multi-modal prompt tuning (MaPLe) and adopted projection matrix condition vision prompts on their language counterparts explicitly allowing mutual propagation of gradients to promote synergy. In comparison, this work aims to learn multi-modal multi-mode prompts to better meet the requirement of diverse comprehensive representations. Besides, unlike measuring the similarity between images and labels by the global prompt-level features, we model each prompt as an empirical distribution over the token-level embedding space, and the similarity score is calculated by combining the prompt and token-level features under hierarchical OT framework, which provides novel and elegant tool to adapt PVLs into downstream tasks.
5 Experiments
5.1 Experimental Setup
To make comprehensive evaluation, we performed extensive experiments on 4 task settings, such as few-shot image recognition, base-to-new generalization, cross-dataset transfer learning, and domain generalization. Those experiments are conducted on 15 widely used image datasets, varying in scale and domains, including ImageNet [44], Caltech 101 [45], OxfordPets [46], StanfordCars [47]. Flowers [48], Food101 [49], FGVCAircraat [50], EuroSAT [51], UCF101 [52], DTD [53], SUN397 [54], ImageNetV2 [55], ImageNet-Skett [56], ImageNet-A [57], and ImageNet- R [58]. The details of each dataset are provided in the Appendix Table. B. 1.
Datasets
Baselines We compare ALIGN with the state-of-the-art methods, including: CLIP [1], which provides the base results without prompt tuning; the single-modal prompt tuning methods,e.g..

---

TPTs: CoOP [3], Cp [4] and PLOT [17], and VPTs: VPT [6], and multi-modal prompt tuning
methods: UPT [27] and MaPLe [28]. Note that we modified the official code PLOT and changed the backbone to ViT-B/16 for fair comparison.
Implementation Details Following previous MaPLe [28]. we load the pre-trained Vit-B/16 CLIP model as our backbone, where di 512, dy 768 and d 512. We the number of textual and visual prompts M = N = 4, the length of prompt tokens b = 2, the hyperparameter À = 0.1, and 1. The maximum iteration number in the Sinkhorn algorithm train our model with a batch-size of 4, a learning rate of 0.0035, and an optimizer as SGD. For each task, we optimize the number of epochs. Following MaPLe we run 2 epochs to train ImageNet as a source model with a learning rate of 0.0026. The reported results are the averaged value over seeds. Please refer to the Appendix Sec. B for more details. For all baselines, we set the length of prompts as 4 and collect their results according to the original papers or previous works. Thus, some experimental results may be missing.
100. For all tasks,
Average 11 datasats
Caltech101
Oxfardlitts
StanfordCars
Melte
RoY
M.N-
0.1
2.1
Shote
Shots
0.22
Shote
2b
Food101
EurOSAT
UCF101
30000
12
72.
70.
Eiiy
UPT
Odty
ALVX
2010 2901
112 4
612
0
26
Figure 3: The few-shot learning results on 7 datasets (more detailed results of other datasets can be found in the Appendix Table. D. 1.). The red solid line denotes our ALIGN method, and the dotted lines represent various baselines. All results are reported the mean value over three seeds.
5.2 Evaluation with the Standard Setting
Few-Shot Learning. We first evaluate our model on few-shot classification, where models are trained on 1,2,4,8, and 16 shots and then applied to the test sets. We report the accuracy scores of all models across 11 datasets at Fig. 3. Overall, most cases and achieves consistent improvement over CoOp on all datasets. Among the multi- method shows superior performance compared UPV and
proposed ALIGN outperforms others in
modal prompt tuning methods, MaPLe in general, except for the EuroSAT datasets. This demonstrates that ALIGN has the abil- ity to distill the across-modes moods--modaliiies knowledge and efficiently adapt PVLs into downstream tasks. Although there is a small margin between our model and those two models on some datasets, the competing models usually cannot achieve high performance over all datasets, e.g., ALIGN exhibits /0..6000...... 61/0.78(% ()) accuracy improvements compared with UPT on DTD datasets and achieves 0..56688.57777 w1..66666666 /......6..6.677666 64/4.47(%) improvements compared with MaPLe. In addition, we also find that our ALIGN performs better on 1/2/4 shots settings, showing the efficiency of our fine- tinengrained alignments, which provide the token-level comparison during prediction. This capability contributes to more accurate classification, even with limited training samples.
Base-to-New Generalization. To assess the generalizability of our model, we follow CoCoOp [4] to equally split the classes into base and new sets, and models are only trained on the base classes while tested on the new sets. Table. I reports the results, and we have the following observations: First, our proposed ALIGN surpasses previous baselines by achieving the highest average scores,
7

---

Table I: Base-to-New 1I datasets. The prompts learned from the 16-shots base We report
the classification accuracy base set (Base), new set (New), and their harmonic mean (H), where (2 x Base x (Base + New). The best results are highlighted.
Average New
ImageNet New
Caltech 101
Oxford Pets
Base
H
Base
Base
New
H
Base
New
CLIP
69.34
74.22
71.69
72.34
68.14
70.21
96.84
94.00
95.39
91.17
97.26
94,
CoOp
82.66
63.22
71.65
76.14
67.88
71.77
98.00
89.81
93.72
93.67
95.29
94.47
CoCoOp
80.47
71.69
75.83
75.98
70.43
73.10
97.96
93.81
95.84
95.20
97.69
96.43
PLOT
77.20
60.38
67.76
75.97
69.23
72.44
96.53
82.86
89.17
93.45
79.76
86.06
MaPLe
82.28
75.14
78.55
76.66
70.54
73.47
97.74
94.36
96.02
95.43
97.76
96.58
ALIGN
83,38
75.51
79.25
76.89
72.15
74.45
98.37
94.70
96.50
95.67
97.93
96.79
Stanford Cars
Flowers 102
Food 101
FGVC, Aircraft
Base
New
H
Base
New
Base
New
H
Base
New
H
CLIP
63.37
74.89
68.65
72.08
77.80
74.83
90.10
91.22
90.65
27.19
36.29
31.08
CoOp
78.12
60.40
68.12
97.60
59.67
74.06
88.33
82.26
85.18
40.44
22.30
28.74
CoCoOp
70.49
73.59
72.10
94.87
71.75
81.71
90.70
91.29
90.99
33.41
23.71
27.74
PLOT
61.41
42.69
50.37
95.26
56.03
70.56
88.45
85.28
86.84
29.63
16.17
20.92
MaPLe
72.94
74.00
73.47
95.92
72.46
82.56
90.71
92.05
91.38
37.44
35.61
36.50
ALIGN
77.24
76.38
76.80
97.70
73.3
83.75
90.77
92.07
91.42
37.56
36.97
37.26
SUN 397
DTD
EuroSAT
UCF 101
Base
New
H
Base
New
Base
H
Base
New
CLIP
69.36
75.35
72.23
53.24
59.90
56.37
56,48
64.05
60.02
70.53
77.50
73.85
CoOp
80.60
65.89
72.50
79.44
41.18
54.24
92.19
54.74
68.69
84.69
56.05
67.45
CoCoOp
79.74
76.86
78.27
77.01
56.00
87.49
71.21
82.33
73.45
77.64
PLOT
78.56
72.34
75.32
69.87
53.63
60.68
87.39
67.63
74.30
72.71
41.51
52.84
MaPLe
80.82
78.70
79.75
80.36
59.18
68.16
94.07
73.23
82.35
83.00
78.66
80.77
ALIGN
82.47
79.68
81.05
82.13
54.17
65.28
94.03
74.9
83.38
84.43
78.33
81.27
Table 2: Cross-dataset transfer learning accuracy results. Here use the key letters denote the datasets. The best results are highlighted.
Source
Target
ImageNet
EuroSAT
UCF101
Average
FGCV
SUNER
Cal
Pets
Cars
Flo
Food
DTD
Method
CoOp
71.51
93.70
89.14
65.41
68.71
85.30
18.47
64.15
41.92
46.39
66.55
63.88
CoCoOp
71.02
94.43
90.14
65.32
71.88
86.06
22.94
67.36
45.73
45.37
68.21
65.74
MaPLe
70.72
93.53
90.49
65.57
72.23
86.20
24.74
67.01
46.49
48.06
68.69
66.30
ALIGN
72.03
93.91
90.55
65.84
73.75
86.40
24.95
67.59
46.75
47.25
67.15
thereby illustrating the superiority of the proposed framework. Second, ALIGN outperforms others in terms of H score across all datasets, except for the DTD dataset which indicates our method offers a more favorable trade-off between the base and new sets. We attribute this success the token-leve multi-mode prompt tuning strategy, where the multi-mode prompts enhance the ability to identify diverse visual concepts, which plays an essential role in unseen category prediction. Furthermore, for datasets that have small intra-class variances, such as Stanford Cars and FGVCAircraft, AL. achieves a noticeable improvement over MaPLe. The token-level alignment in AL GN might account for this improvement, as it makes it more effective for fine-grained image classification.
Transfer Learning and Domain Generalization. To investigate the generalizability across-datasets or across-done drross-dommins, we first train our model on ImageNet, utilizing all 1,000 classes, and subsequently apply to I) other 10 datasets and 2) other 4 domain shift datasets. We report those results at Table. and 3, respectively. Based on those results, we find that our approach outperforms the baseline methods on 8/10 datasets with the best average accuracy score on dataset transfer learning task and 3/4 datasets on domain shift setting. These overall improvements highlight that ALIGN is less susceptible to the distribution shift between the source and target domains, thus revealing robust generalizability of our model. Despite the marginal performance gain of ALIGN in contrast to MaPLe and UPT, our method outperforms them most cases terms of all four tasks and provides novel multi-mode loken-level alignment alternative for prompt tuning.
8

---

P_1
P_2
P_3
P_4
p_tt2
P_1
P_2_0_3
p_4_t_1
P_1_11
P_2_L_
p_2_t_
P_3_t_1
Figure 4: Visualization of the learned prompts and tokens. p_m m denotes the m-th prompt and p_m. __mmm denotes the I-th token of m-th prompt.
Table 3: Cross-domain generalization accuracy results. The best results are highlighted.
Source
Target
Method
Learnable
ImageNet
ImageNetV2
ImageNet-Sketch
ImageNet-A
ImageNet-R
CLIP
X
66.73
60.83
46.15
47.77
73.96
CoOp
71.51
64.20
47.99
49.71
75.21
CoCoOp
71.02
64.07
48.75
50.63
76.18
VPT
70.57
63.67
47.66
43.85
74.42
UPT
72.63
64.35
48.66
50.66
76.24
MaPLe
70.72
64.07
49.15
50.90
76.98
ALIGN
72.03
64.64
49.96
50.94
76.16
Qualitative Analysis Besides the extensive quantitative results, are also interested in the learned visual concepts of the proposed token-level alignments. Fortunately, the transport plan learned in our token-level OT provides us with access convenient tool visualize the most related visual patches. We report the qualitative analysis at Fig. 2(b) and Fig. 4. From Fig. 2(b), we find that our model prefers to learn separable representations in both base and new classes. Recalling that h" denotes the global feature of n-th prompts, to visualize the learned prompt and obtain the attention map over the patch space, we calculate the cosine similarity between h" and all patch embeddings è Rdx0 We then view the normalized cosine similarity as the attention map and visualize the learned
prompts at the top row of Fig. 4. We observe that different prompts tend align different patch regions, each of which contributes to the final prediction. This finding also meets with the motivation of the multi-moot prompt tuning, where each prompt aims to learn specific visual semantics.
Moving beyond the prompt-level visualization, we also visualize the token-level concepts. Specif- ically, for I-th column of the learned transport plan T, E Rd in token-level OT, it measures how likely the /-th token is transported to J = + O patches. Here we focus the O image patches and visualize the transport plan an image sampled from the base set and new set the second and third row in Fig. 4, respectively. We find that 1) different tokens within prompt can capture various patches with similar visual concepts. For example, both p_l_t_ and p___3 attend to the head of the cat; 2) Learning from the base set, the prompt tokens prefer to align the similar patches in the new set, which reveals that our token-level alignment module has the ability to transfer from the base set to the new set, rather than over-fitting to the base categories.
Complexity Analysis As discussed above, one of the key ideas of the proposed ALIGN is to learn multiple prompts for vision and language inputs and explore the token-level alignments under the hierarchical OT framework.
demonstrate the computation cost, we report the complexity analysis at Table. 4, where we focus on the number of trainable parameters (#Paras) and inference speed (fps). We find that 1) Overall, the multimodel prompts tuning methods (last three) require more trainable parameters and inference time than single-modal methods. 2) The proposed ALIGN requires

---

Table 4: Complexity analysis over various baselines. report the number of trainable parameters (#Paras) and frames per second(fps). We can not report the fps result of UPT because of unrelcased code.
Methods #Paras
CoOp 2,048
CoCoOp 35,360
VPT
PLOT
UPT 3,555,072
MAPLE 3,555,072 282
ALIGN 3,582,720
13,824
8,192
fps
645
37
152
583
62
slightly more training parameters than UPT and MAPLE because of the multiple prompts. And it also requires more inference time than MAPLE, due to the hierarchical OT operations. 3) Thanks to the independent OT operations, which be calculated parallelly with the GPU, ALIGN has faster testing time than CoCoOp, and achieves 62 fps at the test stage.
Conclusion
This paper introduces novel multi-mode token-level alignment framework for multi-modal prompt tuning under optimal transport. We first employ the prompt-level OT to model the multi-mode prompts across modalities, and then introduce the token-level OT by viewing each prompt itself as a set over token embedding space. By coupling those two-level OT via the cost matrix, the final prediction is obtained by combining the prompt-level features and the token-level embeddings. enabling fine-gre ane-ggrined alignments. Extensive experiments have been conducted, showing that our proposed model achieves competing performance on four settings. terms of the limitations, the users may still need large GPU memory to load the pre-trained weights of PVLs to apply the proposed model to the test process. One potential solution is to combine prompt tuning with knowledge distillation. We leave it as future study. Thanks to the open-world visual concept understanding of PVLs, our model shows promising zero-shot/few-shh ability for image recognition, which has the potential to encourage researchers to derive new and better methods for prompt tuning. Our work may indirectly lead to a negative impacts if there is sufficiently malicious ill-informed choice of few-shot classification task.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant U21B2006; in part by Shaanxi Youth Innovation Team Project; in part by the Fundamental Research Funds for the Central Universities QTZX23037 and QTZX22 160; in part by the 111 Project under Grant B B18039, in part by the Fundamental Research Funds for the Central Universities; in part by the Innovation Fund of Xidian University under Grant YJSJ23016;
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 8748-8763, PMLR, 2021.
[2] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA.
PMLR, 2022.
[3] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasiun-l models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[4] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
10

---

[5] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 139-149, 2022.
[6] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII. Springer, 2022.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[8] Florian Wenzel, Andrea Dittadi, Peter V. Gehler, Carl-Johann Simon-Gabricn Max Horn, Do- minik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, Bernhard Schölkopf, and Francesco Locatello. Assaying out-of-distribution generalization in transfer learning. In NeurLPS, 2022.
[9] Korawat Tanwisuth, Shujian Zhang, Huangjie Zheng, Pengcheng He, and Mingyuan Zhou POUF: trompt--oientee unsupervised fine-tuning for large pre-trained models. In ICML 2023: International Conference on Machine Learning. July 2023.
[10] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438, 2020.
[11] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[12] Chengeheng Ma, Yang Liu, Jiankang Deng, Lingxi Xie, Weiming Dong, and Changsheng Xu. Understanding and mitigating overfitting in prompt tuning for vision-language models. IEEE Transactions on Circuits and Systems for Video Technology, pages 1-1, 2023. doi: 10.1 10.1109/TCSVT.22233325555
[13] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. arXiv preprint arXiv:2205. 14865, 2022.
[14] Yue Zhang, Hongliang Fei, Dingcheng Li, Tan Yu, and Ping Li. Prompting through prototype: prototype-based prompt learning on pretrained vision-language models. arXiv preprint arXiv:2210 10. 10841, 2022.
[15] Feng Wang. Manling Li. Xudong Lin, Hairong Lv, Alex Schwing, and Heng Ji. Learning to decompose visual features with latent textual prompts. In The Eleventh International Con- ference on Learning Representations, 2023. URL https: /ttp:////peeeeeeeee/mme id=wtcud6Hroza
/ wtwpi/ewwwweeeeeeeeeeeeeeeeeeeee
[16] Nihal V. Nayak, Peilin Yu, and Stephen Bach. Learning to compose soft prompts for composi- tional shoo-shot learning. In The Eleventh International Conference on Learning Representa- tions, 2023, URL https:: http://pppneeeieeeeeeeee.. net forum? ? id 88 A2FXnIh.
[17) Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. PLOT: Prompt learning with optimal transport for vision-language models, In The Eleventh International Conference Learning Representations, 2023, URL https: httpttps:://ppeeevviieee 2qwryBoXYnh.
net/ coformmmiiii 1et:
[18) Xinyang Liu, Dongsheng Wang, Miaoge Li, Zhibin Duan, Yishi Xu, Bo Chen, and Mingyuan Zhou. Patch-token aligned bayesian prompt learning for vision-language models, arXiv preprint arXiv:2303.0972 09000, 2023.
[19] Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the IEEF/VV Conference on Computer Vision and Pattern Recognition, pages 5206-5215, 2022.
11

---

[20] Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor Guilherme Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Variational prompt tuning improves generalization of vision-language models. arXiv preprint arXiv:22 2022.
[21] Roei Herzig. Ofir Abramovich, Elad Ben-Avraham, Assaf Arbelle, Leonid Karlinsky, Ariel Shamir, Trevor Darrell, and Amir Globerson. Promptonomyvit: Multi-task prompt learning improves video transformers using synthetic scene data. arXiv preprint arXiv:22 12.0221,, 2022.
[22] Yunhe Gao, Xingjian Shi, Yi Zhu, Hao Wang. Zhiqiang Tang. Xiong Zhou, Mu Li, and Dimitris N Metaxas. Visual prompt tuning for test-time domain adaptation. arXiv preprint WrXiv:2210004811, 2022.
[23] Kihyuk Sohn, Yuan Hao, José Lezama, Luisa Polania, Huiwen Chang, Han Zhang, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. arXiv preprint arXiv:2210.00990 2022.
[24] Weihuang Liu, Xi Shen, Chi-Man Pun, and Xiaodong Cun. Explicit visual prompting for low-level structure segmentations. arXiv preprint arXiv:2303. 2023.
[25] Lingho Liu, Bruce XB Yu, Jianlong Chang, Qi Tian, and Chang- Wen Chen. Prompt-matched semantic segmentation. arXiv preprint arXiv:2208. 10159, 2022.
[26] Senqiao Yang. Jiarui Wu, Jiaming Liu, Xiaoqi Li, Qizhe Zhang, Mingjie Pan, and Shanghang Zhang. Exploring sparse visual prompt for cross-domain semantic segmentation. arXiv preprint arXiv:2303.09792, 2023.
[27] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. CoRR, abs/2210.07225, 2022.
[28] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi-modal prompt learning. arXiv preprint 17.0:12210.03117, 2022.
[29] Hyunjin Choi, Judong Kim, Seongho Joe, and Youngjune Gwon. Evaluation of bert and albert sentence embedding performance on downstream nlp tasks. In 2020 25th International conference on pattern recognition (ICPR), pages 5482-5482 IEEE, 2021.
[30] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joins Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3980-3990. Association for Computational Linguistics, 2019.
[31] Viet Huynh, He Zhao, and Dinh Phung. Ould: A geometry-aware optimal transport approach for topic modeling. Advances in Neural Information Processing Systems, 33:18573-18582, 2020.
[32] He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray L. Buntine. Neural topic model via optimal transport. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
[33] Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou. Representing mixtures of word embeddings with mixtures of topic embeddings. In The Tenth International Conference on Learning Representations, ICLR 2022, Viriual Event, April 25-29, 2022.
[34] Nicholas Kolkin, Jason Salavon, and Gregory Shakhnarovich. Style transfer by relaxed optimal transport and self-similarity. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 10051-10060, 2019.
12

---

[35] Dandan Guo, Zhuo Li. He Zhao, Mingyuan Zhou, Hongyuan Zha, et al. Learning to re- weightt examples with optimal transport for imbalanced classification. Advances in Neural Information Processing Systems, 45:25511-25530, 2022.
[36] Tuan Nguyen. Trung Le, He Zhao, Quan Hung Tran, Truyen Nguyen, and Dinh Phung. Most: sulti-source domain adaptation via optimal transport for student-teache learning. In Proceed- ings of the Thirty-Severe Seventh Conference on Uncertainty Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 225-235, 27-30 Jul 2021.
[37] Dandan Guo, Long Tian, He Zhao, Mingyuan Zhou, and Hongyuan Zha. Adaptive distribution calibration for few-shot learning with hierarchical optimal transport. In Advances in Neural Information Processing Systems, pages 6996-7010, 2022.
[38] Zheng Gc, Songtao Liu, Zeming Li. Osamu Yoshie, and Jian Sun. Ota: Optimal transport assignment for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 303-312, 2021.
[39] Miaoge Li, Dongsheng Wang, Xinyang Liu, Zequn Zeng. Ruiying Lu, Bo Chen, and Mingyuan Zhou. Patchet: Aligning patch set and label set with conditional transport for multi-label image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[40] Long Tung Vuong, Trung Le. He Zhao, Chuanxia Zheng, Mehrtash Harandi, Jianfei Cai, and Dinh Q. Phung. Vector quantized wasserstein auto-encoder. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 35223-35242. PMLR, 2023.
[41] Viet Huynh, Dinh Q Phung, and He Zhao. Optimal transport for deep generative models: State of the art and research challenges. In Proceedings of the Thirlieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event Montreal, Canada, 19-27 August 2021, pages 4450 4457,445,420 2021.
[42] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[43] Dan dan Guo, Long Tian, Minghe Zhang, Mingyuan Zhou, and Hongyuan Zha. Learning prototype-oriented set representations for meta-learning. In International Conference on Learn- ing Representations, 2022.
[44] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. lece, 2009.
[45] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[46] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pages 0599-3505... IEEE, 2012.
[47] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[48] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Gruphics & Image Processing, pages 722-729. IEEE, 2008
[49] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 10nd-inn discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446-461. Springer, 2014.
13

---

[50] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint 1rXiv:1306.51551 2013.
[51] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-222 2019.
[52] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[53] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[54] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 20/0 IEEE computer society conference on computer vision and pattern recognition, pages 3485 3492. IEEE, 2010.
[55] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference Machine Learning, pages 5389-5400. PMLR, 2019.
[56] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global repre- sentations by penalizing local predictive power. Advances Neural Information Processing Systems, 32, 2019.
[57] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15262-15271, 2021.
[58] Dan Hendrycks, Steven Basart, Norman MU, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of dior-iistribttion generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340 8349, 2021.
14

---

Appendix
A Discussion
We this paper propose ALIGN, a unified framework for multi-modal prompt tuning, where multi- mode modality-specifie prompts are learned via the token-level alignment strategy. Moving beyond the single-model methods, which focus on textual prompt tuning or visual prompt tuning, ALIGN allows one learn textual and visual prompts simultaneously, resulting in better representations in the shared vision-text embedding space. Compared to recent multi-modal methods, such as UPT [27] and MaPLe [28]. ALIGN prefers learn multi-mode prompts to capture diverse class attributes and develop the token-level alignment for fine- fine-grained comparisons. This provides ALIGN with an efficient tool calculate the similarity between prompts. We find that many previous works can be merged into our ALIGN framework with special hypermeter settings. We summarize this relationship Table. The N/A in Table. means that PLOT calculates the similarity between the prompt-level label embeddings and the visual patch embeddings, which is not the case in ALIGN, where we calculate the similarity prompt-level OT between textual label embeddings and visual image embeddings. and calculate the similarity of token-level OT between token embeddings and patch embeddings.
Table Most previous works can be merged into our ALIGN framework. M: Number of visual prompts. N: Number of textual prompts. 3: Weight of token-level OT in Eq.6 in the manuscript.
Methods
Type
M
N
CoOp [3]
Textal Prompt Tuning
0
VPT [6]
Visual Prompt Tuning
PLOT [17]
Textual Prompt Tuning
0
2
1
N/A
UPT [27]
Multi-modal Prompt Tuning
MaPLe [28]
Multi-modal Prompt Tuning
1
0
ALIGN(Ours)
Multi-mode Prompt Tuning
2
2
0
2
B Data statisties and Hyperparameter setting
We thoroughly evaluate proposed ALIGN framework across four distinct tasks: few-shot recog- nition, base-to-new generalization, cross cross-dataset transfer, and cross-domain generalization. These extensive experiments are conducted on diverse set of fifty commonly used vision datasets, covering various contexts. These datasets include ImageNet [44] and Caltech101 [45] for generic image classi- fication, OxfordPets [46], StanfordCars [47], Flowers 102 [48], Food101 [49], and FGVCAireraft [50] for fine-grained image recognition, SUN397 [54] for scene recognition, UCF101 [52] for action recognition, DTD [53] for texture classification, and AuroSAT [5]] for satellite imagery recognition. In the case of the cross-domain generalization task, our model is trained on ImageNet and subse- quently tested ImageNetV2 [55], Imagehet-Sketch [56], ImageNet-A [57], and ImageNet-R [58]. We summarize data statistics at Table. B.
The evaluation pipeline for each task follows the approach employed by previous works [3, 28]. The specific details of this pipeline are summarized below:
Few-shot Recognition. To evaluate the efficiency of the proposed ALIGN the few-shot case, we follow CoOp [3], and first partition the dataset into base and novel sets. Those two sets share the categories. Models are trained the base set using variety of shot settings, including 1, 2, 8, and 16 shots per class, and then tested on the full novel set. The accuracy scores are reported to compare the performance. The training epoch is set as 10 for 1, 2, and 4 shots and 40 for 8 and 16 shots.
Base-to-New Generalization. To show the Generalizability of unseen categories, we first divide the dataset into two separate subsets: the base subset and the new subset. Importantly, these subsets do not share the same categories. The base subset contains a specific set of categories used for model training, while the new subset consists of previously unseen categories that the model has not
15

---

been exposed to during training. Besides reporting the accuracy score on base and novel sets, we also calculate the harmonic mean (2 Base New) (Base + New), which acts as trade-off between Base and New, providing comprehensive measure of overall model performance. The training epoch is set
Cross- Mooss-Dataset Transfer. To determine the transferability of our model across different datasets, we first train our model on the source dataset (ImageNet) and then evaluate on 10 different target datasets. The training epoch is set as 2 and the learning rate is set 0.0026.
Cross-Domain Generalization. determine the robustness our model on the distribution-shin setting, trained model the source dataset (ImageNet) and then assess 4 domain-shifted datasets, including ImageNetV2, ImageNet-Skent epoch is set and the learning rate is set 0.0026.
ImageNet-A, and ImageNet-R, The training
The other training hyperparameters in the previous experiments are set according to MaPLe [28], which are detailed listed at Table B. 2.
Table B. Statistics validation sets.
datasets, denotes that not use the corresponding training or
Dataset
Domains
#Classes
#Train
#Val
#Test
ImageNet
generic object
1000
1.28M
N/A
50,000
Caltech101
generic object
4,128
1,649
2,465
OxfordPets
fine-grained object
37
2,944
736
3,669
StanfordCars
fine-grained object
196
6,509
1,635
8,041
Flowers 102
fine-grained object
102
4,093
1,633
2,463
Food 101
fine-grained object
101
50,500
20,200
30,300
FDVCAircraft
fine-grained object
100
3,334
3,333
3,333
SUN397
scene recognition
397
15,880
3,970
19,850
UCF101
action recognition
101
7,639
1,808
3,783
DTD
texture recognition
47
2,820
1,128
1,692
EuroSAT
satellite object
10
13,500
5,400
8.100
V2
generic object
1000
N/A
N/A
10,000
ImageNet-Sketch
sketch object
1000
N/A
N/A
50,889
ImageNet-A A ImageNet-R
generic object
200
N/A
N/A
7,500
generic object
200
N/A
N/A
30,000
C Training Algorithm
Given the training datasets D = (X, vx,) No our method aims to learn M visual and N textual prompts simultaneously. All parameters ALIGN are optimized by minimizing the cross-entropy loss end-to-end. We summarize the training algorithm at Algorithm. 1.
16

---

Table B. 2: Hyperparameter setting used in the previous experiments.
Hyperparameters
Values
Batch Size
4
Input Size
224 x 224 "Bicubic"
Input Interpolation
Input Pixel Mean
00.48145466, 0.4578275, 0.40821073
Input Pixel STD
00.26862954, 0.26130258, 0.27577711]
Transforms
["random resized crop". "random filp", "normalize"]
Optimizer
SGD
Learning Rate
0.0035
LR Scheduler
"cosine"
Warmup Epoch
Warmup Type
"constant"
Warmup L.R
le-5
Backbone
VIT-B/16
Number of Textual Prompts
4
Number of Visual Prompts
4
Learnable Prompt Length Fixed Prompt Length weight of token-level cost 1
2
2
weight of regularization in OT
0.1
Prompt Initialization Precision
"a photo of a"
"fp16"
Algorithm Training algorithm of ALIGN.
Input:
Training dataset D, pre-trained vision-language model, class name set, number of visual prompts M, number of textual prompts N, and the training epoch.
Output: The learned ALIGN, which discovers multi-mode multi-mode prompts downstream tasks.
Initialize: The M and N multi-modal prompt embeddings.
Preprocess: Built N x K textual token inputs according to Sec 2.1 in the manuscript. for iter 1,2.3.... do Feed the textual input into the encoder g and collect the outputs with the corresponding prompt-level K,N
K.N K,N representation [^2] 1= and token embeddings [ [=1]k=1,m=1 where each sf is the output token embeddings re-th prompt k-th label with length + kz.
2. Sample a batch of / images. Built x B visual patch inputs according to Sec2.1 in the manuscript. Feed the visual input into the visual encoder f and collect the outputs with the corresponding prompt-level representation {=5" " J.M
jj=1.m=1 and patch embeddings (r mn J.M m== where each r."" denotes the output patch embeddings of m-th prompt of j-th image with length b +
Two-level OT
3. Caleulate the token-level OT distance between each image and each label in Eq.5 with the collected
patch set and token set.
4. Calculate the cost matrix in prompt-level OT according to Eq.6, and then get the prompt-level OT
distance in Eq.4
Compute the cross-entropy loss L with the obtained prompt-level OT distance according Eq.8 and update all learnable parameters by minimizing L with the stochastic gradient descent algorithm.
end
D Additional Results
We in this section report additional results other datasets on the few-shot task and conduct the ablation studies on the prompt and token-level OT.
17

---

Hamonicc mean (1) with various
Caltech101(H)
OxferdPats/M)
StanfordCaas(4)
Flewer102)H)
77
76.6
96.5
83
75.5
85
82
0
0.2
0.4
0.8
0.8
0.2
0.4
0.8
0.6
0.2
0.4
0.8
0.8
0.2
0.4
0.8
0.8
(ood101(H)
FOVCAircraf(M)
DTD(H)
EureSATIH)
37.2
$1.4
91.2
36.8
85
91
36.6
80
0.5
0.8
0.6
0.2
0.4
0.5
0.8
0.2
0.4
0.6
0.8
Figure DI: Harmonic mean (II) results ALIGN on Base-to-New task under different B.
D.1 Few-shot Results
We report the numerical results of various methods on datasets Table. From the results, we find that our method ALIGN outperforms baselines in most cases, which demonstrates the efficiency of the token-level prompt alignment module.
D.2 Ablation studies
Recall that the proposed model consists of the prompt-level and token-level OT, which align the textual and visual modalities from hierarchical semantics. In the previous experiment, we view the prompt-level and token-level OT equally and set the hyperparameter weight / = 1 in Eq.6 in the manuscript. Here want analyze how those two OT's affect the model performance. To this end, we rewrite the cost matrix in Eq.6 in the manuscript as:
Cmn (1 B) sim(="" h²)) (d (a Uni CTRa).
Note that 9 = 0 and B = 1 denote two of our variants, where the former denotes only prompt-OT works and the latter means we only focus on token-level similarity. report the ablation results of ALIGN on Base-to-New tasks under various settings, e.g., [0, 0.2, 0.5, 0.7. 1.0) Fig. D1. We have the following interesting findings: 1) The combined ALIGN works better than each of them; 2) After finetuning for each dataset, one can obtain better results than the reported values in paper.
18

---

Table D. 1:: The few-shot results of various methods on 11 datasets. We report mean value over 3 different sceds. The best results are highlighted.
Dataset
Methods
shot
shots
4 shots
shots
16 shots
CoOp
92.4
93,2
93.5
94.0
94.8
PLOT
88
89.95
91.50
93.00
93.24
UPT
93.66
94.17
94.09
95.04
95.95
Caltech101
MaPLe
91.73
93.33
94.23
94.43
95.26
ALIGN
93.97
94.13
95.00
95.43
96.00
CoOp
48.4
31.3
59.2
64.4
69.5
PLOT
51.90
58.24
70.52
UPT
45.01
60.74
65.44
70.62
DTD
MaPLe
51.16
54.70
61.63
65.63
70.60
ALIGN
54.07
56,53
63.3
67.6
71.40
CoOp
51.8
60.9
69.0
76.0
84.1
PLOT
60.10
68.45
72.97
79.84
83.12
UPT
66.46
69.07
75.36
85.62
90.77
BuroSAT
MaPLe
66.67
79.26
84.25
89.96
92.14
ALIGN
53.23
71.43
80.93
85,97
90.77
CoOp
24.2
25.8
27.9
32.7
34.2
PLOT
21.50
21.71
23.96
27.02
30.24
29.91
33.34
FGVCAireraft
MaPLe
26.64
27.86
33.56
40.66
49,93
ALIGN
29.57
31.63
34.03
40.95
49,99
CoOp 72.9
80.4
85.7
92.3
96.2
PLOT
70.00
81.34
88,29
92.84
95.10
UPT
74.97
81.81
91.90
95.17
97.41
Flowers102
MaPLe
80.24
88. 14
90.07
95.10
96.34
ALIGN
81.33
88.77
92.53
95.43
96.57
CoOp 81.6
80.9
81.5
82.4
84.9
PLOT
69.10
72.89
74.89
76.70
77.87
UPT
84.21
85.01
85.34
86.16
86.83
FOOD101
MaPLe
78.73
77.30
79.03
80.10
82.43
ALIGN
85.29
86.05
86.66
86.74
86.90
CoOp
68.07
69.26
69.60
71.53
PLOT
67.51
68.80
70.00
70.21
71.40
UPT
69.55
69.88
70.28
71.58
72.6 64
ImageNet
MaPLe
69.56
69.94
70.65
71.80
72.74
ALIGN
69.80
70.02
70.84
71.77
72.45
CoOp
90.0
89.8
92.3
92.0
92.1
PLOT
83.21
85.77
86.02
89.13
89.95
UPT
82,93
85.40
85.97
87.40
88.10
OxfordPets
MaPLe
89.80
86.76
90.76
90.23
91.30
ALIGN
93.4
93.67
94.17
CoOp
70.1
75.2
PI.OT
46.20
$1.67
$4.35
60.52
65.32
UPT
67.60
69.57
75.88
80.19
84.17
StanfordCars
MaPLe
65.96
69.10
75.73
79.76
85.36
ALIGN
68.27
72.84
76.58
81.65
86.75
CoOp 65.2
66.6
68.1
70.5
73.2
PLOT
55.33
60.02
63.21
66.02
67.98
UPT
68.84
69.76
72.12
74.00
75,90
SUN397
MaPLe
61.73
63.23
67.60
69.13
73.00
ALIGN
69.14
69.98
71.88
74.15
76.57
CoOp
70.7
73.8
76.6
79.6
80.4
PLOT
51.42
54.89
61.23
67.45
70.85
UPT
71.98
77.49
80.91
83.86
UCF101
MaPLe
73.23
73.00
77.45
81.2
84.67
ALIGN
74.42
75.87
80.18
81,99
85.69
19


Reviews:
[{"review_id": "FNp6mJIKA4", "reviewer": "Reviewer_TnsF", "summary": "This paper argues that existing prompt turning fails to caption the sample diversity, learning to sub-optimal prompt discovery. To this end, they propose a multi-mode token-level alignment for multi-modal prompt tuning. Specifically, they formulate the prompt turning as the hierarchical optimal transportation problem (distribution matching problem). As a result, the extensive experiments on few-shot image classification, transfer learning, and domain generalization, show the superiority of the ALIGN.", "questions": "As shown in weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. **The proposed method is effective and comprehensive**. Learning and aligning a set of prompt tokens across modalities by hierarchical optimal transportation is effective and simple.\n2. **The result is strong and the evaluation is comprehensive**. The extensive experiments on 15 widely used image datasets under the setting of four task settings show the superiority of the proposed method.", "weaknesses": "1. **Time Complexity Analysis.** Since this method introduces hierarchical optimal transmission, I would like to ask whether there is more time overhead in the inference phase than in other methods. Can you provide relevant experimental comparisons in detail?\n\n2. **More Cases for Visualization.** In order to prove that learned prompt tokens have the ability to capture diverse visual concepts, it seems that Fig. 4 cannot explain the above claim well. I am more curious whether there are multiple prompts that will pay attention to the same visual concepts. Can you provide more cases for analysis?\n\n3. **The different with PLOT[1] and MAPLE[2].**  Can you provide a more thorough comparison to illustrate the novelty of your work?\n\n4. **The writing is need to be improved**. It is hard to understand the core idea from the messy introduction. What's more, Sec.2 Background and Sec.4 Related work have a large number of duplicates that exist and can be merged.\n\n\n[1] PLOT: Prompt Learning with Optimal Transport for Vision-Language Models. ICLR 2023.\n\n[2] MaPLe: Multi-modal Prompt Learning. CVPR 2023.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer TnsF for providing positive feedback and helpful suggestions. Below please find a response.\n\n> Time Complexity Analysis\n\nWe have provided a detailed experimental comparison at the global comment. Please refer to the above analysis for more details. \n\n>More Cases for Visualization\n\nOne of the main motivations of this paper is to mine diverse visual concepts, and thus we formulate the priors of both visual and textual prompt embeddings as the Uniform distributions in Eq(3) of the manuscript, e.g., each visual patch has an equal probability of being attended to. This mathematically guarantees that the learned prompts have the ability to align diverse visual concepts. Empirically, we find that most prompts find their concepts and few prompts attend to the same concepts. \n\nFollowing your advice, we added more visualization in the uploaded pdf file. Please have a check. We will add those results in our revision.\n\n>The different with PLOT and MAPLE\n\nCompared to PLOT (multiple modes and single modality) and MAPLE (single mode multiple modalities), our proposed multi-mode multi-modal token-level alignment algorithm acts as a unified framework for prompt tuning, e.g., PLOT and MAPLE can be viewed as our particular cases. We briefly summarize the main differences below. \n\n1) Multi-mode prompt learning for both visual and textual modalities. Our method learns M visual and N textual prompts for two modalities of CLIP, while PLOT learns N textual prompts and MAPLE learns a single visual prompt and single textual prompt. This enables our model to capture diverse visual and textual concepts. \n\n2) Token-level alignments of both visual and textual modalities. The proposed model aligns the vision and language domains by considering token-level features, e.g., the visual patch features and the textual token embeddings, resulting in fine-grained comparison. The MAPLE and PLOT methods either only consider the prompt-level alignments or focus on visual patch features, failing to model the token-level features of both modalities. \n\n3) Hierarchical transport framework for prompt tuning. Technically, we formulate the prompt tuning task as a hierarchical OT problem, where the visual and linguistic representations are modeled as the empirical distributions over the M visual and N textual prompt embeddings (prompt-level OT), and the prompt embeddings are further modeled as the empirical distribution over the patch and token features, respectively.\n\n>The writing is need to be improved\n\nThank you for the writing suggestion, and we will highlight the core idea, clear the Background and Related work section in the revision."}, {"review_id": "lCX0ddiFAF", "reviewer": "Reviewer_S87N", "summary": "The paper proposes a prompt-learning method for adapting CLIP to few-shot classification. The proposed prompt is multimodal, i.e., both vision and language encoder is adaptable, and multimode, i.e., each modality is assigned with several prompts for diverse representation. Experiments are conducted on few-shot and base-to-new transfer settings, showing the method's superiority over previous state-of-the-art.", "questions": "* In Fig. 3, ALIGN performance on UCF is much superior to previous methods, but the superiority on UCF is not obvious in the Base-to-New setting, especially on the base classes.  Why is that case? \n* In Page 3 Line 97, \"Empirical findings\" needs to specify the source of information or citing papers.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Promising results are achieved, both in the few-shot setting and base-to-new transfer setting. \n* The presentation is mostly clear and easy to understand.\n", "weaknesses": "* Incremental novelty of the paper. Visual-language prompt learning with multimode prompts and optimal transport (OT) is explored in PLOT [17]. Also, multimodal prompts are explored in works like [27][28]. This proposed multimode multimodal prompts with OT multimode seems to combine the two types of previous methods and reveals limited insights. \n* Missing comparison on computation complexity. As multimode prompts require quite large parameters and computations compared to a single prompt, I want to see the comparison of parameter efficiency and time efficiency.\n* It is strange that there is no ablation study of each component of the models. It is important to see the individual contribution of the multi-mode prompts and the token-level alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer S87N for the comments and suggestions. Below, we address the concerns raised in your review point by point. Please let us know if you have any further concerns or whether this adequately addresses all the issues that you raised with the paper.\n\n> Incremental novelty of the paper\n\nFirst, we would like to note that building a unified prompt tuning framework that enjoys the good properties of multi-mode and multi-modal methods is novel. It is not the case that it is a trivial combination(e.g., extend MPTs by learning multiple prompts and then apply OT to align two modalities). Technically, the introduced hierarchical OT (prompt-level and token-level OT) in this paper is different from that in PLOT in terms of formulation, motivation, and alignment strategies. \n\n1) The prompt-level OT models the visual and textual representations as two empirical distributions over the M and N modality-specific prompt embeddings, and the token-level OT further views each prompt embedding as a discrete distribution over its token points (patch or token). Those two formulations enable the proposed model to capture hierarchical features and make fine-grained alignments. However, PLOT calculates the OT distance between the N textual prompts and a set of visual patch embeddings, ignoring the textual token-level alignment, which is different from our cases. \n\n2) As discussed above, we aim to align two domains hierarchically, the token-level OT focuses on image patches and sentence tokens, and the prompt-level OT focuses on global image and sentence features. This distinguishes ALIGN from PLOT. PLOT focuses on global sentence features and local image patches. \n\n3) One of the main challenges is to combine the two OTs efficiently. Here we naturally formulate the prompt-level and token-level alignments as a hierarchical OT problem, where the transport distance of the token-level OT acts as the cost matrix of the prompt-level OT, boosting the connection of those two OTs.\n\nOur proposed model belongs to MPTs and provides a new hierarchical OT perspective to improve their performance. In fact, the existent two MPTs [27][28] share a similar idea that is based on traditional continuous prompt tuning. This paper proposes a novel unified framework that can learn multi-mode multi-modal prompts for MPTs and achieves consistent improvement over the existent MPTs.\n\n> Missing comparison on computation complexity\n\nWe have reported the detailed comparison in the global comment section. Please check for more discussion.\n\n> Ablation study of each component of the models.\n\n|  Methods  | UCF 101 | Stanford Cars | Flowers102 | Oxford Pets |\n|:--:|:---:|:--:|:----:|:---:|\n| ALIGN w/o prompt |  80.84  | 74.18 |  82.64 | 96.51  |\n|  ALIGN w/o token |  81.04  |  75.32  |  83.09 | 96.61  |\n|   ALIGN   |  81.27  |  76.80  |  83.75   | 96.79 |\n\nFollowing your advice, we have reported the ablation results of the introduced modules above. We find that both the multi-mode prompts and the token-level alignment contribute to the improvements.\n\n> In Fig. 3, ALIGN performance on UCF is much superior to previous methods.\n\nHere we thank reviewer S87N for pointing out this mismatch case in Fig3. We have checked the numerical results carefully of Fig3 and found that there is typing error on UCF101 16-shot result (85.69 to 95.69). Note that the corrected results still outperform baselines on all few-shot settings.  We apologize again for making you confusion. In terms of the Base-to-New setting, CoCoOp found that CoOp usually overfits the seen set and is not generalizable to the unseen set (higher base-set accuracy and lower new-set accuracy). The proposed model balances the seen and unseen sets well and achieves the highest H score.\n\n> Citing papers at line 97\n\nWe will cite related papers [1,2] in the revision.\n\n[1] Yuhang Zang, et al. Unified vision and\nlanguage prompt learning.\n\n[2] Muhammad, et al.  Maple: Multi-modal prompt learning. In CVPR 2023."}, {"review_id": "I7pLh3TH03", "reviewer": "Reviewer_hEEC", "summary": "This paper introduces a multi-mode token-level alignment framework for multi-modal prompt tuning, which improves the representation of visual and textual modalities and can be used to improve existing methods. The task is formulated as a distribution matching problem, addressed using prompt and token-level optimal transportation (OT), providing a principled and elegant solution. The method is applied to few-shot classification, dataset transfer learning, and domain generalization, showing superior results on widely used datasets.", "questions": "See the comments in weakness.", "limitations": "• As pointed out in the conclusion section, this paper's method still demands substantial GPU memory.\n• The method suggested does not appear to be well-suited for a fully zero-shot scenario where there is no training samples. This scenario is little bit different from the Base-to-New Generalization scenario.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "• The learning of multi-modal, multi-mode prompts is facilitated by establishing optimal transport (OT) at the prompt and token level.\n• The structure of the manuscript is solid and it's well-written overall.\n• The efficiency of the proposed ALIGN method for both few-shot classification and generalization has been confirmed through a series of diverse experiments.", "weaknesses": "• The proposed model might be memory-intensive, however, an analysis of the additional time and memory costs has not been provided.\n• The omission of specific details, particularly regarding the ablation analysis, somewhat undermines the impressive results.\n  - The study does not examine the influence of prompt length and quantity on the experimental outcomes.\n  - It remains unclear whether token-level alignment provides any enhancements compared to prompt-level alignment.\n\n• Miscellaneous issue\n- Figure 1 is not mentioned in the body text.\n- It seems that line 135 is missing a period, and the expression \"maximuming\" appears to be awkward.\n- In Table 1, CoOp shows the best result in 'Stanford Cars'-Base, so it should be highlighted instead of ALIGN", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer hEEC for providing positive feedback and helpful suggestions. Below are our responses.\n\n> Analysis of the additional time and memory costs has not been provided.\n\nWe have reported the detailed comparison in the global comment section. Please check for more discussion.\n\n> Additional results of prompt length and quantity.\n\n| Length | 2  | 4| 8 |16 |\n|:--:|:----:|:--:|:--:|:--:|\n|  UCF 101 | 81.27 | 81.24 | 81.45 | 81.42 |\n|  Flowers102| 83.75 | 83.84 | 83.32 | 83.57 |\n| Stanford Cars | 76.80 | 76.68 | 77.14 | 76.82 |\n\n\n|Number | 1 | 2  |  4  |   8   |\n|:--:|:-----:|:--:|:--:|:--:|\n| UCF 101| 80.84 | 81.13 | 81.27 | 81.42 |\n| Flowers102| 82.64 | 82.97 | 83.75 | 83.43 |\n| Stanford Cars | 74.18 | 75.32 | 76.80 | 76.73 |\n\nFollowing your advice, we have reported the results of prompt length and quantity. We fix the prompt length as 2 and set the number of prompts as 4 according to the previous works in the manuscript. From the added results, we find that the proposed model enjoys good robustness of prompt length and quantity. And one may have better results than we report when finetuning those hyperparameters.\n\n\n> Unclear whether token-level alignment provides any enhancements compared to prompt-level alignment.\n\n|  Methods  | UCF 101 | Stanford Cars | Flowers102 | Oxford Pets |\n|:--:|:---:|:--:|:----:|:---:|\n| ALIGN w/o prompt |  80.84  | 74.18 |  82.64 | 96.51  |\n|  ALIGN w/o token |  81.04  |  75.32  |  83.09 | 96.61  |\n|   ALIGN   |  81.27  |  76.80  |  83.75   | 96.79 |\n\nWe have reported the ablation results of the introduced two modules. Compared ALIGN w/o token and ALIGN, we find that the token-level alignment indeed has a positive improvement for the performance.\n\n> Miscellaneous issus\n\nWe thank the reviewer's careful reading. We will correct those typos in the revision.\n\n> this paper's method still demands substantial GPU memory\n\nWe would like to note that not only the proposed method but also its baseline methods load the pre-trained CLIP as the image and text encoders and thus all compared methods in this paper demand substantial GPU memory. It is a common limitation among the prompt-tuning methods and is beyond the scope of this paper.\n\n>  The method suggested does not appear to be well-suited for a fully zero-shot scenario where there is no training samples\n\nWe agree with the reviewer's concern. In fact, all compared baselines (except for CLIP) introduce to-be-learned embeddings for better contiguous prompt tuning, and thus the training samples are needed to train such embeddings. Besides the few-shot and Base-to-New settings, we conduct the cross-dataset experiment, where the models are trained on a source dataset and tested on a target dataset. This allows no overlap categories between the source and target datasets and somewhat can be viewed as a fully zero-shot setting for the target dataset. Our proposed model outperforms baseline methods in most cases, showing the generalization of the method."}, {"review_id": "DEmUZSRZej", "reviewer": "Reviewer_Uuq5", "summary": "This work aims to overcome the limitations of previous works in prompt tuning for vision-language models. Unlike prior approaches that focus on single modality or holistic prompt alignment, the paper proposes a multi-mode token-level tuning framework that leverages optimal transportation to align prompt tokens across different modalities. The framework introduced in the paper relies on two crucial elements: multi-mode prompt discovery and token-level alignment. By enabling diverse semantic representations, multi-mode prompt discovery ensures a broader range of prompts. On the other hand, token-level alignment allows for a more detailed exploration of similarity between modalities. Extensive experiments demonstrate the effectiveness of the new method ALIGN.", "questions": "1. How about the comparison of computing cost between your method and the baselines?\n2. Is there any experimental results in inference tasks other than classification?", "limitations": "See \"Weaknesses\".", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, this work is well-motivated, and the paper is well-written. It focuses on a practical problem in prompt learning and proposes a novel method called ALIGN.\n2. This work first leverages Optimal Transportation to address the limitations of prompt tuning. Intuitively, the new method can succeed in finding out token level vision-language prompts. Compared with the single-mode or holistic level prompt tuning approaches, this method can better reveal the connection between visual and textual prompts, which would be a very valuable insight for the study of vision-language alignment.\n3. The authors conduct extensive experiments to evaluate their method, and the results demonstrate non-trivial improvements over the existing baseline models. The comparison is clear, and their method ALIGN achieves SoTA performance in most scenarios.\n", "weaknesses": "1. The paper lacks discussion of computing cost. It seems that your ALIGN requires more self-attention computation compared to the baselines such as VPT and TPT. Your improvements might be challenged if the difference in training/inference cost is not provided or is too big.\n2. While the paper demonstrates its merits in token-level prompt learning, more empirical results in fine-grained tasks such as semantic segmentation should be included. However, the paper only gives classification results. I understand that implementing prompts into segmentation tasks is not easy and it merely appears in your baselines, but it would be a very strong support to your ALIGN’s effectiveness and significance.\n3. Some expressions are confusing. For example, in line 249-250, there is “For each task, we optimize the number of epochs”. Do you mean “the same number of epochs”?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Uuq5 for providing positive feedback and helpful suggestions. Below are our responses.\n\n\n> Lacks discussion of computing cost and Q1: How about the comparison of computing cost between your method and the baselines?\n\nWe have reported the detailed comparison in the global comment section. Please check for more discussion.\n\n> Is there any experimental results in inference tasks other than classification? and more empirical results in fine-grained tasks such as semantic segmentation should be included and Q2: Is there any experimental results in inference tasks other than classification.\n\nWe thank the reviewer for pointing out the potential applications of the proposed model. The proposed model proposed a novel token-level alignment for multi-modal prompt tuning tasks. It improves the classification results by aligning the semantics of the visual patches and the textual tokens. Following the previous empirical setting, we conduct extensive experiments on 4 classification tasks on 12 datasets. We appreciate the reviewer's suggestion and will leave the semantic segmentation to future work. Also, we would be very happy if the reviewer could suggest relevant papers.\n\n> For each task, we optimize the number of epochs.\n\nFollowing previous work, we use different epochs for 4 tasks. We will correct the statement in the revision."}, {"review_id": "CfchJNACCn", "reviewer": "Reviewer_dk2b", "summary": "The paper aims at improving few-shot classification based on CLIP-ViT-B/16 using prompt tuning. The authors introduced a multi-modal prompt tuning framework with token-level alignment and distribution matching. The prompts are tuned on ImageNet and then the model is benchmarked on 15 image datasets.", "questions": "Please see the Weaknesses.", "limitations": "The authors have discussed their limitations on the GPU memory during test. It may also be important to discuss their training cost.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The method achieves better results than baselines (e.g., CoOp, MaPLe) on a series of datasets, including both few-shot classification and transfer learning tasks.\n+ The code is attached, making the method reproducible.\n+ The introduced approach should be practical in real-world applications.", "weaknesses": "- The writing appears rushed and some sections are not easy to follow. There are also some typos and grammar issues, e.g., L135-136 \"...CLIP. maximuming...\".\n- The central insight of this work is to introduce token-level alignment in a multimodal prompt-tuning framework. However, the fine-grained alignment with optimal transportation is not new (e.g., Hierarchical Optimal Transport for Multimodal Distribution Alignment, NeurIPS19) and the multimodal prompt-tuning framework is similar to existing works like MPT. It provides limited new knowledge to the community.\n- In the experimental part, there are only comparisons based on base-size models. However, it is pretty essential to verify the scalability of the introduced method on larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer dk2b for the comments and suggestions. Below, we address the concerns raised in your review point by point. Please let us know if you have any further concerns or whether this adequately addresses all the issues that you raised with the paper.\n\n> The writing appears rushed and some sections are not easy to follow. There are also some typos and grammar issues, e.g., L135-136 \"...CLIP. maximuming...\".\n\nWe will carefully improve the writing, reorganize the sections, and correct typos in the revision.\n\n> The central insight of this work is to introduce token-level alignment in a multimodal prompt-tuning framework. However, the fine-grained alignment with optimal transportation is not new (e.g., Hierarchical Optimal Transport for Multimodal Distribution Alignment, NeurIPS19) and the multimodal prompt-tuning framework is similar to existing works like MPT. It provides limited new knowledge to the community.\n\nFirst, we would like to note that one of the main contributions of this paper is to develop a unified multi-model prompt tuning method under the HOT framework, e.g., most previous works can be viewed as a particular case of our framework (Table 1 in the Appendix). \n\nSecond, We agree that the token-level alignment module shares a similar idea with the previous Hierarchical Wasserstein Alignment algorithm Technically. However, this does not imply that the proposed methodology is not innovative. Our model is different from the previous work in terms of both tasks and objective functions. The proposed ALIGN views the transport distance as the similarity of prompts and the model is optimized by the classification loss. While the previous work focused on aligning clustered datasets and trained the model by minimizing the transport distance unsupervised. It is not a trivial case that directly employs the previous method on the prompt tuning task. \n\nThird, compared to existing MPTs, which often learn single modality-specific prompt sentence, our method provides the community with a novel alternative that prefers to learn multiple prompts, resulting in diverse concept discovery. Mathematically, we formulate the prompt tuning task as a HOT problem, which distinguishes the proposed model from previous MPTs. \n\nLast, the transport plan provides users with a visualization tool to explain the learned prompts, while the previous MPTs fail to give such interpretability.\n\n\n\n\n> In the experimental part, there are only comparisons based on base-size models. However, it is pretty essential to verify the scalability of the introduced method on larger models.\n\nTechnically, the proposed method can be easily applied to pre-trained two-tower models. In the manuscript, we follow a series of previous works and load the pre-trained CLIP as our backbone. Extensive experiments are conducted to evaluate the superiority of the proposed model. We believe that those empirical results support this paper. We would appreciate it if the reviewers could provide some references that apply prompt tuning on large-scale vision-language models.\n\n\n\n> The authors have discussed their limitations on the GPU memory during test. It may also be important to discuss their training cost.\n\nWe have reported the detailed comparison in the global comment section. Please check for more discussion."}]

Author rebuttal:
We thank all the reviewers for the time and expertise they have invested in these reviews and for their valuable comments. We are encouraged that all reviewers noted the impressive results achieved by the paper, and reviewers TnsF, hEEC, and Uuq5 praised the novelty and effectiveness of our method. Your comments and suggestions have helped us to improve the paper. We provide a response and clarifications below for each reviewer respectively and hope they can address your concerns.

>Complexity Analysis

A common concern among reviewers is the complexity analysis of the proposed paper. Here we report the comparisons on the number of trainable parameters (#Parameters) and inference time below (FPS):

|  Methods |  CoOp | CoCoOp |   VPT  |  PLOT |    UPT    |   MAPLE   |   ALIGN   |
|:---:|:--:|:--:|:---:|:---:|:--:|:--:|:---:|
| #Parameters | 2,048 | 35,360 | 13,824 | 8,192 | 3,555,072 | 3,555,072 | 3,582,720 |
|  FPS |  645  |   37   |  152  |  583  |  -   | 282  |   62    |

We find that the overall multimodel prompts tuning methods (last three) require more trainable parameters and inference time than single-modal methods.
The proposed ALIGN method aims to learn multi-mode prompts across visual and textual modalities and achieves consistent improvements over baseline methods in most cases. The proposed ALIGN requires slightly more training parameters than UPT and MAPLE because of the multiple prompts. And it also requires more inference time than MAPLE, due to the hierarchical OT operations. The proposed model supports GPU parallel inference and thus has a faster testing time than CoCoOp. 

Here, we would like to note that the main idea is to develop a unified prompt tuning framework and provide a new hierarchical OT view for the community. We thank the reviewers for pointing out the potential future work. We will keep this in mind and Continuously improve our method.

Meta-review: The paper explores a multi-modal prompt learning approach where multiple modes are modeled across modalities. The reviewers have overall appreciated the novelty and experimental gains demonstrated by the authors but also mentioned concerns on compute complexity, experimental comparisons, ablation on components, and clarity on differences with previous works. The author's response helped clarify most of the questions, especially regarding the ablation, compute complexity, and additional results with a larger model. 

AC finds the paper interesting and believes it will be a useful result to the community (though components of the proposed approach have been individually used in literature e.g., the hierarchical OT and multi-modal prompt learning, the overall approach shows promise for multi-modal prompt tuning). Therefore, AC recommends acceptance.

However, AC strongly emphasizes on highlighting/correcting the below points in the revision: a) The complexity of the approach is much higher than competitors due to learning multiple-mode prompts and applying hierarchical OT for each image-text pair. This analysis should be included in the paper. b) The visualization provided with rebuttal is poorly labeled and does not really align with the claim "Empirically, we find that most prompts find their concepts and few prompts attend to the same concepts." Authors should check if this can be improved or clarified by explaining the differences. c) Regarding point 1 under differences between PLOT and MAPLE, note that the statement "MAPLE learns a single visual prompt and single textual prompt" is incorrect. b) The clarity in the introduction needs to be improved and the typos need to be fixed in the final revision.
Decision: Accept (poster)