PLOT: Prompt Learning with Optimal Transport for Vision-Language Models
Authors: Guangyi Chen
Weiran Yao
Xiangchen Song
Xinyue Li
Yongming Rao
Kun Zhang
Venue: ICLR 2023 notable top 25% (2023)

Abstract: With the increasing attention to large vision-language models such as CLIP, there has been a significant amount of effort dedicated to building efficient prompts. Unlike conventional methods of only learning one single prompt, we propose to learn multiple comprehensive prompts to describe diverse characteristics of categories such as intrinsic attributes or extrinsic contexts. However, directly matching each prompt to the same visual feature is problematic, as it pushes the prompts to converge to one point. To solve this problem, we propose to apply optimal transport to match the vision and text modalities. Specifically, we first model images and the categories with visual and textual feature sets. Then, we apply a two-stage optimization strategy to learn the prompts. In the inner loop, we optimize the optimal transport distance to align visual features and prompts by the Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distance from the supervised data. Extensive experiments are conducted on the few-shot recognition task and the improvement demonstrates the superiority of our method. The code is available at https://github.com/CHENGY12/PLOT.

Published as a conference paper at ICLR 2023
PLOT: PROMPT LEARNING WITH OPTIMAL TRANS- PORT FOR VISION-LANGUAGE MODELS
Guangyi Chen'*, Weiran Xiangchen Songl, Xinyue Li,, Yongming Raoi, Kun Zhang?*
CCarnegie Mellon University, Pittsburgh PA, USA *Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE Tsinghua University, Beijing, China
New York University, Abu Dhabi, UAE
ABSTRACT
With the increasing attention to large vision-language models such as CLIP, there has been significant amount of effort dedicated to building efficient prompts. Unlike conventional methods of only learning one single prompt, propose to learn multiple comprehensive prompts to describe diverse characteristics of categories such as intrinsic attributes or extrinsic contexts. However, directly matching each prompt to the same visual feature is problematic, as it pushes the prompts to converge one point. To solve this problem, we propose to apply optimal transport to match the vision and text modalities. Specifically, we first model images and the categories with visual and textual feature sets. Then, we apply two-stage optimization strategy to learn the prompts. In the inner loop, we optimize the optimal transport distance to align visual features and prompts by the Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distance from the supervised data. Extensive experiments are conducted on the few-shot recognition task and the improvement demonstrates the superiority of our method. The code is available at https: / thub. com Camm/CHENYY11 22//111
INTRODUCTION
In the past few years, large-scale vision-language pre-trained (VLP) models, such as CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), and BLIP (Li et al., 2022) have achieved remarkable success in open-world visual concept learning. These methods have brought new light but also pose new question: how to efficiently adapt the knowledge from pretraining to the downstream tasks since these models are typical of massive sizes which are not feasible for normal users to re-train.
One of the conventional paradigms of utilizing pretrained knowledge is "pre-training, fine-tuning", which fixes the architecture of the pre-trained neural network and tunes its parameters using task-specific objective functions. Beyond fine-tuning the parameters, many recent methods (Zhou et al., 2021b; 2022) introduce the concept of prompt learning from the field of NLPP to the vision domain and achieve striking performance gain for the few-shot visual classification. They fix the model parameters and instead learn suitable prompts by turning a template sentence into set of learnable vectors. Then, these prompts are learned by minimizing the distance between the visual features and prompt-based language features.
Brambling
A bird with dark fan-tail
A bird with black crown and eye
A bird that lives in winter wood
A bird with orange and black texture
The motivation that category can be complementar- ily described different views (An example of "Brambling").
Figure 1:
Despite significant improvements over manual prompts, learning only a sentence is intuitively insufficient to represent class. One class be described by many intrinsic characteristics and even extrinsic context relations. Thus, for one object, we may have multiple prompt candidates which focus different attributes. As shown in Figure can describe the class "Brambling" in

---

Published as a conference paper at ICLR 2023
different views: such as the color of the wing, the color of the crown and eyes, the shape and color of the tail, and even the living environment information. It motivates us to learn multiple prompts to comprchensively represent the class and thus facilitate classification.
The most natural solution is to directly learn multiple prompts by respectively matching each prompt with the visual features. However, it is the same matching the mean of prompt features and the visual features. This solution is problematic since all prompts are encouraged to be closer to one single point and thus tend to learn the same characteristics. It contradicts our purpose learn comprehensive prompts. To solve this problem, we tested adding some constraints to push away the prompt from each other, but found that this solution still fails to learn representative and comprehensive prompts. This solution treats the visual representation as one single point, and such unified view of visual features ignores the fact that different prompts may only focus on one or a subset of characteristics.
To address this problem, in this paper, we propose Prompt Learning with Optimal Transport (PLOT), which applies optimal transport (OT) to align the local visual features and multiple textual prompts. Optimal transport can calculate the distance between two distributions under the form of multiple sampling. In our prompt learning framework, we formulate local visual features and multiple prompts as the samplings of two discrete distributions and use OT to encourage fine-grained cross-modal matching. Specifically, to obtain the local visual features with different semantic clues, we extract all feature maps as the visual representation instead of the single global representation. Fortunately, we can easily obtain the visual feature maps from the visual encoder of CLIP by using all outputs of the multi-head self-attention layer (Rao et al., 2021). Then the problem comes down how to calculate the distance between two feature sets.
We solve this problem by introducing the optimal transport theory (Villani, 2009) and formulate the feature sets as a discrete probability distribution where each feature has an equal probability value. Furthermore, to reduce the computational cost and avoid the extra model parameters, we learn the prompts with a two-stage optimization strategy. At the first stage in the inner loop, we fix both visual and text features and optimize the optimal transport problem by a fast Sinkhorn distances algorithm (Cuturi, 2013). Then, in the outer loop, we fix all parameters of optimal transport and back- paoo-prap the gradient to learn the prompts with different characteristies. Compared with conventional distance (such as Euclidean distance of mean features), optimal transport can align different visual features for each local prompt, which is more robust to the visual misalignment and tolerates well feature shift (Rubner et al., 2000). It is because OT learns an adaptive transport plan to align features, which achieves fine-grained matching across two modalities. We conduct experiments on 11 datasets following the standard setting of CLIP (Radford et al., 2021) and CoOp (Zhou et al., 2021b) to evaluate our method. These experiments span the visual classification of generic objects, scenes, actions, fine-grained categories, and so on. The significant result improvement demonstrates that PLOT can effectively learn representative and comprehensive prompts.
2 RELATED WORK
Optimal Transport The Optimal Transport (Monge, 1781) is initially introduced to solve the prob- lem of how to reduce the cost when moving several items simultancously. Recently, OT theory has drawn wide attention in the machine learning and computer vision community by comparing distributions readily available to them under the form of feature sets (Peyre & Cuturi, 2019). Due to the brilliant property of distribution matching, OT has been applied in many theoretic and application tasks including generative models (Arjovsky et al., 2017: Salimans et al., 2018; Zhao et al., 2021a), structural matching (Chen et al., 2019; Xu et al., 2020; Zhao et al., 2021b; Xu et al., 2019) (e.g. sequence matching (Chen et al., 2019) and graph matching (Xu et al., 2019). and image match- ing (Zhang et al., 2020; Liu et al., 2021a; Zhao et al., 2021b)). and other distribution based tasks (such as clustering (I (Laclau et al., 2017). distribution estimation (Boissard et al., 2015), and causal discovery (Tu et al., 2022)). In this paper, we use OT to align the features of vision and language modalities by learning adaptive transport plan (Rubner et al., 2000).
Vision-Language Pre-trained Models Vision-Language Pre-trained (VLP) models aim to explore the semantic correspondence between the vision and language modalities through large-scale pre- training. Recently, VLP models have achieved an exciting performance improvement in shot visual recognition (Radford et al., 2021; Gao et al., 2021; Zhou et al., 2021b: 2022; Zhang et al., 2021b), which shows the great potential to promote open-world visual understanding with the help of language. In terms of objectives, VI.P methods can be divided into reconstruction (Li al.,
2

---

Published as a conference paper at ICLR 2023
2019; Hong et al., 2021; Dou et al., 2021; Kim et al., 2021), contrastive matching (Radford et al., 2021; Jia et al., 2021: Jain et al., 2021), or the combination of both two (Li et al., 2021; Wang et al., 2021b; Kamath et al., 2021). Besides, recent progress in the field of VLP also benefits lot from large-scale pair-wised datasets. For example, CLIP (Radford et al., 2021) applies 400 million image-text pairs for contrastive learning. Beyond recognition, these VI. models also show great potential for other downstream applications, such as dense prediction (Rao et al., 2021; Zhou et al., 2021a), image generation (Nichol et al., 2021; Ramesh al., 2022; Patashnik et al., 2021), and action understanding (Wang et al., 2021a; Tevet et al., 2022).
Prompt Learning Prompt learning introduced from the field of NLP to efficiently adapt the large language model to downstream tasks. Different from the conventional "pre-training fine-tuning" paradigm which initializes the pre-trained model and tunes the parameters of the network using downstream task-specific objective functions, prompt learning applies textual prompt to reformulate the downstream tasks as the original pretrained task (Liu et al., 2021b; Petroni et al., 2019). By the prompt, the domain shift between pretrained task and the downstream application is reduced and thus the pretrained knowledge can be easier adapted to downstream tasks. The concept of prompt learning (Petroni et al., 2019; Radford et al., 2019; Poerner et al., 2019) begins from the success of GPT (Radford et al.. 2019) series. Early prompt learning methods (such as Petroni et al. (Petroni et al., 2019) and Pörner al. (Poerner et al., 2019)) always manually create templates based on human prior knowledge. Furthermore, some mining-based methods (Jiang et al., 2020) and gradient-bass methods (Shin et al., 2020) are proposed to automatically search for appropriate templates. Beyond search in the discrete space, some methods (Li & Liang, 2021; Tsimpoukelli et al., 2021; Liu et al., 2021c) remove the constraint that the prompts are "words" and instead learn prompts in the continuous embedding space. Recently, CoOp (Zhou et al., 202 and its extended version (Zhou et al., 2022) introduce prompt learning into open-world visual understanding to adapt the knowledge from the large-scale visual-language pretrained models and achieve great performance improvement on the few-shot visual recognition. Compared with CoOp, our PLOT method further improves prompt learning by introducing the optimal transport distance to learn multiple local prompts and achieves fine-grained vision-language matching. PDL (Lu et al., 2022) is also motivated by the more diverse prompts, which assumes parametric distribution of prompts and fits the parameters during training. Different from it, PLOT learns multiple prompts without parametric distribution.
APPROACH
In this section we first revisit the baseline CoOp (3.1), review the preliminaries of optimal trans- port (3.2), and then introduce our PLOT (3.3) to show how we learn multiple comprehensive prompts.
3.1 A REVISIT OF CoOP
CoOp (Zhou et al., 2021b) is one of the pioneering methods to learn the prompts for using vision language pretrained knowledge (such CLIP (Radford al., 2021)) for downstream open-world visual recognition. Different from CLIP which manually designs the prompt templates, CoOp sets a part of context words in the template as continuous learnable parameters which can be learned from the few-shot data. Then the classification weights can be represented by the distance between the learned prompt and visual feature.
Specifically, given an image æ, visual feature f = f(x) is obtained by the visual encoder f of CLIP. Then, the textual prompt can be formulated as tt = {wi, w2...., WL, c& where C₁ is the word embedding of the class name, = {wi](=] } are learnable vectors where each vector has the same dimension as the original word embedding and L is the length of context words. With prompt
t as the input, the text encoder g outputs the textual feature gk g(tx). The final prediction probability is computed by the matching score as follows:
exp(sim(f..rg (x))/
ply = k|[)
(1)
ΣK exp(sim (faisim(f.0) gk )/7)
where sim(-..) denotes metric function such as cosine similarity, and 7 stands for the temperature of Softmax. Then we can optimize the parameters of (vecil<;,) with the cross-entropy loss between the prediction and the labeled target.
3

---

Published as a conference paper at ICLR 2023
[XX X CLASS
Prompt set
[X X X X CLASS]
Text Encoder
Optimal Transport Distance
X ... X CLASS]
XX X CLASS]
Brambling, House finch,
Feature set
Image
Encoder
The framework: PLOT first describes each category with multiple prompts and obtains set of prompt features by text encoder. The image is also encoded as a set of local features. Then the optimal transport used as the metric between prompts and visual features.
Figure 2:
3.2 OPTIMAL TRANSPORT
Optimal transport (OT) distance is widely used metric for the comparison of distributions. Here, we only focus on the discrete situation which is more related to our framework. Assuming we have two sets of points (features), the discrete distributions formulated as:
U
U fm
and
V
Un
where and the discrete probability vectors that sum to and oy Dirac delta function placed at support point f in the embedding space. Then, the total distance is written as:
M
T-
Tm. Cm,.-
We call C' the cost matrix in which each point denotes the cost between fm and In. such as Cm,, = 1 sim 9n). While the T' called the transport plan, which learned to minimize the total distance. The optimization problem of optimal transport is formulated as:
dor(u, v v|C) minimize T
< <
(4)
subject T1 = T 1M v, TERMXN
As directly optimizing the above objective is always time-consuming, we apply the Sinkhorn dis- tance (Cuturi, 2013) to use an entropic constraint for fast optimization. The optimization problem with Lagrange multiplier of the entropy constraint
dor,s da C) minimize <TC>> -Xh(T)
(5)
subject to
Tlx = u, T IM
T
where ((-) is entropy and À > 0 is hyper-parameter. Then we can have fast optimization solution
with few iterations as:
T diag(u")) ()) exp( Crppdd(()ddaa((())
(6)
where l denotes the iteration and in each iteration (1) u/ ((exp(-C/\jo((-)) and ((() v/ ((exp(-C/2) ("),, with the initiation (0) 1.
3.3 PROMPT LEARNING WITH OPTIMAL TRANSPORT
In this subsection, introduce the details of PLOT which learns multiple prompts describe different characteristics of the category by minimizing the OT distance.
Specifically, shown in Figure 2, given image æ, first feed it to the visual encoder branch of CLIP. Apart from the global visual feature f. we can also obtain a set local features The visual encoder has multi-head attention pooling layer in which the input is the combination
4

---

Published as a conference paper at ICLR 2023
of the global feature and a set of local features (feature map) and the output is tensor with the shape RR(HxW1))XC, where H and W is the height and width of feature map and C' is the feature dimension. Therefore, we can obtain M H W local features and a global feature. At the same time, for class k, we can initialize N local prompts as (te, i } with learnable vectors {wol I where each is the same as the prompt in CoOp. With both visual and textual encoders, we can obtain local visual features F {f IM E IR M XCC and prompt features G2 {gulm } E R RN
In the inner loop, we learn the transport plan T with these fixed support sets F, Gk. by minimizing the following OT distance to push G1 to F:
dor(k) dor dor(u, u/ F G+),
where C==1FFT G. denotes that we use the cosine distance between and G3 as the cost matrix. Then obtain the solution of transport plan T Eq. 6 and the final OT distance dor(k).
Given the OT distance between G3 and F, we reformulate the prediction probability as:
kt)=)> exp ((1 do)( /T)
Por
(8)
= exp ((1 dor(k') /r)
In the outer loop, fix the transport plan T and optimize (vect,)l=11m=1) with cross entropy:
K
1
LCE
Va k|x),
|2
BEX 1=1
where yr one-hot label vector. The detailed algorithm can be found in Appendix A1.
Though the optimization strategy of the optimal transport and prompts is two-stage, the whole training flow is end-to-end. It is because the transport plan is computed using a small number of matrix multiplications as forward module. The gradients of these matrix multiplications are taped for back- socc-proppggtion for end-to-end optimization, which makes the whole system fully differentiable (including the iterative algorithm) and easy to implement using an autograd library like Py PyTorch. In the experiments, we found that it natural and relatively easy to this optimization strategy.
EXPERIMENTS
Extensive experiments are conducted to evaluate our method, including comparison with CoOp, ablation studies, parameter analysis extensibility analysis, computing cost analysis, and visualization.
4.1 DATASETS
We followed the experimental settings in the CoOp (Zhou et al., 2021b) for the few-shot learn- ing evaluation. The experiments are conducted on the 11 visual recognition datasets, including C1 (Fei-Fei al., 2004), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), FGV- CAircraft (Maji et al., 2013), Flowers 102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), ImageNet (Deng et al., 2009), OxfordPets (Parkhi et al., 2012), StanfordCars (Krause et al., 2013), SUN397 (Xiao et al., 2010), and UCF101 (Soomro et al., 2012). These datasets span visual classification of generic objects. scenes, actions, fine- tneegrrined categories, and on, which constitutes a comprehensive evaluation of our method. All experiments adopted the few-shot evaluation protocol used in CLIP (Radford al., 2021) and CoOp (Zhou et al., 2021b), where we respectively choose 1, 2, 4, 8, and 16 shots for model training and use the original test set for evaluation. Besides, we also evaluated the robustness of our method with domain shift. Following CoOp, we used the ImageNet as the source domain and evaluate our method with ImageNet-basse based robustness evaluation datasets including ImageNetV2 (Recht al., 2019), ImageNet-Skenn -Seechh (Wang et al., 2019), ImageNet- A (Hendrycks et al., 2019), and ImageNet-R (Hendrycks et al., 2020). A detailed introduction of each dataset can be found the appendix.
4.2 IMPLEMENTATION DETAILS
We chose CoOp (Zhou et al., 2021b) with CoOp which only learns a global prompt for one class, our PLOT method learns multiple local prompts and applies the OT distance for fine-grained alignment. Besides, also reported
main competitor to evaluate method. Compared
5

---

Published as a conference paper at ICLR 2023
Averaag 11 datasets
Ca tech101
Imagelte
OxfordPess
Scorett
Scorento
Ours
Curs
Durs
Curs
coop
Colip
Eacp
CoOp
CocoOp
CotaOp
CocoOp
CaCarOp
Linear frobe
Probe
Linear
Linear Frobe
Shots per class
Shats class
Shots per class
Shoos per class
StanfordCars
Flowers102
Food101
FGVCAincraft
Scorer00
ScoretS0
Ours
Gurs
Durs
Coop
CoOp
Coop
CoOp
CoCoOp
CoCuOp
CoCoOp
CoCaOp
Linear Probe
Frobe
Linear
Linear Frobe
Shots per class
Shots class
Shots per class
Shots per class
DTD
EurISAT
UCF301
sur1907
Score
Scoreht
Outs
Gurs
Durs
CoOp
CoOp
CoOp
CoOp
CoCoOp
CoCoop
CoCoOp
CoCoop
Lincar Probe
Probe
Lincar
Linear Probe
Shots per class
Shotz per class
Shots per class
Shots per class
Figure 3: The few-shot learning results on 11 datasets. We compare our PLOT with CoOp, CoCoOp. and the Linear Probe method and observe the consistent and significant performance improvement on most datasets. (The average accuracy on all datasets is shown on the left top.)
the performance of training a linear classifier with the CLIP (Radford et al., 2021) features and the conditional version of CoOp, called CoCoOp (Zhou et al., 2022). They are also widely-used methods to adapt the pretrained knowledge for the downstream task. Please note that we evaluate CoCoOp in the same setting for fair comparison (the base-to-new setting can be found in the appendix). The original CoOp method has different versions with different class token positions and parameter initialization strategies. For easy comparison, we directly chose one of them as our baseline with "end" token position, "random" initialization, 16 context tokens, and RN50 backbone. More implementation details can be found in Section A2.
4.3 COMPARISON WITH CoOp
In this subsection, we compare our PLOT with the baseline CoOp on the few-shot recognition and domain generalization tasks.
Few-Shot Learning We summarized the experimental results in Figure where the red line denotes our PLOT method, the blue one denotes CoOp, the purple line denotes CoCoOp (Zhou et al., 2022), and the green one the CLIP linear probe. As the settings in the CoCoOp and CoOp are different, we re-run the CoCoOp method in the setting of CoOp. We observed that all prompt learning methods outperform the linear probe method by a large margin. Besides, PLOT can further outperform CoOp and CoCoOp on most of the datasets. Taking the average accuracy (at the left top) as the example, PLOT respectively gained 3. 3.03%. 3.45%, 2. 2.13%, 1.38%. 0.61% performance boost over CoOp at 1, 2, 4, 8, 16 shots. Among all datasets, PLOT achieves larger improvement over CoOp the 100 and DTD datasets and achieves comparable performance on the StanfordCars datasets. It may be because the discriminative characters in StanfordCars coincide with each other, such that one global prompt and one global visual feature can work well. Note that we don't use the class-specific context, thus the performance on the ine-ggrined classification datasets lower, e.g. the performance of both CoOp and PLOT without class-specife context is lower than the linear probing on FGVCAircraft. All these performance comparisons can serve as experimental evidence to demonstrate that multiple local prompts and the OT distance facilitate the prompt learning of vision- laiiunlangguae models. The detailed accuracy can be found the appendix.

---

Published as a conference paper at ICLR 2023
Table Ablation studies on few-shot recognition. PLOT our defined model with N 4. CoOp: the baseline method. G: respectively matching the global visual feature and multiple textual prompts V: applying a constraint to add the variance of prompts. E: using different initializations as the ensemble: M: using the visual feature map instead of the global visual feature. More details of different variants can be found in Section A2.4 in the appendix.
Dataset
Settings
shot
2 shots
4 shots
8 shots
16 shots
PLOT
89.83 + 0.33
90.67 + 0.21
90.80 + 0.20
91.54 + 0.33
92.24 = 0.38
CoOp
87.51 + 1.02
87.84 1 1.10
89.52 1 0.80
90.28 + 0.42
91.99 1 0.31
88.13 0.36
86.98
1.25
88.45 + 0.79
90.16
0.22
90.72 0.18
Caltech101
G+V
88.28 1 0.43
87.72 1: 1.25
88.45 1 0.30
89.82 0.20
92.00 1 0.13
G+E
88.91 + 0.38
90.01 + 0.22
90.41 0.200 0.20
90.60 0.10
91.74 0.42
M
69.78 + 1.75
71.57 1.59
77.18 = 2.16
81.77
86.21 + 0.20
M+V
66.11 = 8.29
71.45 3.98
79.30 = 3.96
0.78 86.96
89.80 0.17
PLOT
46.55 + 2.62
51.24 + 1.95
56.03 + 0.43
61.70 + 0.35
65.60 + 0.82
CoOp
43.62
1.96
45.35 0.31
53.94 = 1.37
59.69 = 0.13
62.51 0.25
45.12 1.69
$3.9+2.00
54.75 = 0.48
60.15
63.59 + 0.76
DTD
G+V
45.90
48.50
53.96 0.48
59.69
63.51 + 0.66
G+E
46.39 +
49.31 0.56
$2.99
6..4
63.97 + 0.48
M
13.18 4.57
12.25
13.00
5.44
26.99 1.98
M+V
12.61 5.93
15.11 1.81
20.35 = 1.33
44.13 2.39
56.85 + 0.54
PLOT
77.74 0.47
77.70
0.02
77.21 + 0.43
75.31 + 0.30
77.09 + 0.18
CoOp
74.25 1 1.52
72.61
+ 1.33
73.49 1 2.03
71.58 1 0.79
74.48 1 0.15
G
74.63 +.13+00111
70.15
0.49
70.41 0.46
70.72
73.68 + 0.46
FOODIO1
G+V
74.83 #.31
0.85
70.86 0.22
70.80
73.93 + 0.35
G+E
75.77 0.62
0.88
75.82 0.44
7.50
75.52 0.33
52.02
4.86
46.12 1.46
46.86 - 1.39
53.43 0.88
61.28 0.23
M+V
16.52 + 1.15
15.95 + 2.66
53.57 + 0.83
62.95 + 0.37
67.63 + 1.11
Table Parameter analysis for the number of prompts
Dataset
Settings
shot
2 shots
4 shots
shots
16 shots
N=1
88.47 1.15
89.19 + 0.39
89.70 + 0.38
90.45 + 0.24
91.56 0.14
N=2
88.86 + 0.51
89.60 + 0.10
90.60 = 0.17
91.25
0.65
91.89 + 0.36
Caltech101
N=4
89.83 + 0.33
90.67
+ 0.21
90.80
0.20
91.54 0.33 0.33
92.24 + 0.38
N=8
89.74 + 0.30
90.18 - 0.46
91.02 0.18
91.28 0.28
92.04 0.29
N=1
43.91 + 0.65
48.21 2.20
53.69 + 1.10
58.90
0.19
62.85 + 0.74
N=2
45.59 + 2.46
48.06 + 1.92
55.58 + 1.71
61.56+0.11
64.60 0.92
DTD
N=4
46.55
2.62
51.24 + 1.95
56.03 0.43
61.70 0.35
65.60 - 0.82
N-8
46.89
1.94
51.87 + 2.06
54.45 - 0.48
62.20 0.56
65.25 1 0.38
N=1
75.96 + 0.48
76. + 0.59
0.59
77.11 - 0.41
76.56
77.43 + 0.80
N=2
77.12 0.49
76.89
+ 0.23
76.16 = 0.52
75.23
76.81 81 + 0.50
FOODI01
N=4
77.74 0.47
77.70 0.02
77.21 = 0.43
75.31 = 0.30
77.09 + 0.18
N-8
78.05 + 0.15
78.19 +
0.07
78.12 0.17
76.63 + 0.22 77.48 0.12
Table 1: Comparisons on robustness to domain shift.
Domain generalization The robustness also plays a critical role in model applications since the real-world environment may have large domain shifts with the training data. Therefore, conducted robustness eval- uation to investigate the transferability of models learned by PLOT Table summarizes the results of PLOT method and CoOp four ImageNet-basse based robustness evaluation datasets. For both methods, we trained the models on Ima- geNet with 16 shots per class. For PLOT, we the number of prompts N 4. We can observe that PLOT outperforms CoOp consistently on both source and target domains. These experimental results demonstrate that the performance improvement of learning multiple prompts doesn't 't rely on single-domain overfitting.
Source ImageNet
Target
Method
-V2 -Sketch
-R
CLIP + CoOp
61.91
54.26 32.47 21.78 54.21
CLIP + PLOT 44) 63.01
55.11
33.00 25.611
4.4 ABLATION STUDIES AND MORE ANALYSIS
In this subsection, we conducted the ablation studies to investigate the effectiveness of different components, in order to answer the following questions.
7

---

Published as a conference paper at ICLR 2023
Table 4: The few-shot accuracies of Tip-adapter-F and our adapter-based PLOT on 11 datasets.
Dataset
Methods
shot
shots
4 shots
8 shots
16 shots
Tip-Adapter-F + PLOT
89.33
90.87
90.87
92.29
93.15
Caltech101
Tip-Adapter-F
89.33
89.74
90.56
91.44
92.86
Tip-Adapter-F PLOT Tip-Adapter-F
51.12
52.42
59.81
63.71
67.79
DTD
49.65
53.72
57.39
62.71
66.55
Tip-Adapter-F FFppperr-+PPoTT +PLOT
64.37
76.53
79.51
79.17
85.75
EuroSAT
Tip-Adapter-F
59.53
66.15
74.12
77.93
84.54
Tip-Adapter-F +PLOT
19.89
22.20
26.22
30.69
36.21
FGVCAircraft
Tip-Adapter-F
20.22
23.19
25.80
30.21
35.55
Tip-Adapter-F PLOT Tip-Adapter-F
77.59
81.98
89.32
93.75
96.10
Flowers102
79.98
82.30
88.83
91.51
94.80
Tip-Adapter-FF PLOT Tip-Adapter-F
78.71
78.52
77.90
76.93
78.36
FOOD101
77.51
77.81
78.24
78.64
79.43
Tip-Adapter-F PLOT
62.27
64.31
63.89
65.04
66.17
ImageNet
Tip-Adapter-F
61.32
61.69
62.52
64.00
65.51
Tip-Adapter-F PLOT
87.16
87.68
88.63
89.78
87.54
OxfordPets
Tip-Adapter-F
87.00
87.03
87.54
88.09
89.70
Tip- Adapter- + PLOT
59.12
62.32
67.50
70.64
76.00
StanfordCars
Tip-Adapie--F
58.86
61.50
64.57
69.25
75.74
Tip- Adapter- + PLOT
64.26
64.91
67. 29.
69.87
71.64
SUN397
Tip-Adapter-F
62.50
63.
66.21
68.87
71.47
Tip-Adapter-F PLOT
65.69
70.22
72.56
76.53
79.51
UCF101
Tip-Adapter-F
64.87
66.43
70.55
74.25
78.03
Tip- Ap--Adater-FFa PLOT
65.45
68.63
71.23
73.49
76.20
Average
Tip-Adapter-F
64.62
66.65
69.67
72.45
75.83
Q: Can we directly learn multiple prompts by matching the prompt ensemble with the global visual feature? A: No. As shown in Table 2, we report the performance of directly matching the prompt ensemble with the global visual feature (notated as "G") on three datasets including Caltech101, DTD, and FOOD101. The performance improvement of this method over CoOp is limited and far lower than PLOT It may be because this "G" method incentivized to learn the indistinguishable prompts, which contradicts our purpose to learn multiple comprchensive prompts.
Q: Can ensemble methods that encourage the variety of prompts work well? A: Not really. As shown in Table 2, we further apply two methods to encourage the variety of prompts and then use the ensemble to match the global feature. In method "V",, we add an objective function to add the distance between every two prompts different initializations to replace the random initializations, such as "a photo of a", "this is a photo", "this is a", and "one picture of a". However, "G+V" did not achieve consistent improvement over the "G". Despite the clear improvement brought by "G+E", our PLOT showed consistent superiority over "G+E", which further demonstrates the effectiveness of the OT distance.
regularization term. In method "E", we use predefined
Q: Does the improvement mainly come from using all feature maps? A: No. In PLOT, we apply all feature maps of the visual encoder branch, where each feature a local embedding one spatial position. However, we demonstrate that the improvement of PLOT does not only rely on using all feature maps. On the contrary, directly using the feature map to replace the global feature causes large performance drop. For example, on all three datasets, directly using the feature map ("M" or "M+V"" has around 20% shot accuracy drop over using the global visual feature. is not surprising since the original CLIP model is trained by matching the global visual feature and language feature. Without using the OT method, the distance between the feature map and multiple textual prompts degenerates to the mean distance of each feature centuree-romppttt pair.
Q: How many prompts are needed? A: 4 prompts are enough One important in PLOT is the number of prompts. To analyze the effect of the number of prompts, we conducted the experiments on three datasets with 1, 2, 4, prompts. The results are summarized in the white part of Table 3. We observe that the performance obviously increases when adding the number
8

---

Published as a conference paper at ICLR 2023
Aicnatt Canfer
Brambling
Rooster
Dog Sled
Figure 4: Visualization, We provide the heatmaps of transport plan T related to each prompt on categories in ImageNet. Different transport plans focus on different attributes of the object.
of prompts from 4. For example, PLOT (N=4) respectively obtains 1.36%, 2.64%, and 1.68% 1-shot accuracy improvement over PLOT (N=1) on three datasets. Besides, when we further increase the number of prompts, the improvement not consistent. To balance the improvement and cost, we set N = 4 as the default configuration of our PLOT model. In the experiments, we tuned this hyper-parameter on the 11 dataset and applied it to other datasets.
Q: Can PLOT benefit Adapter-based methods? A: Yes. Adapter-based methods (Gao et al., 2021; Zhang et al., 2021a) is another research direction of the efficient adaptation of pre-trained vision-language models. Different from prompt learning that fixes the model parameters and tunes the language prompt, adapter-base methods (Gao et al., 2021; Zhang et al., 2021a) allow for fine-tuning part of the network or adding extra model for training. Recently, slaater-basee methods also achieved good performance on few-shot visual recognition. Therefore, we would like to explore whether our PLOT approach can benefit them, and how.
We apply the Tip-adapter-F (Zhang et al., 2021a) as our baseline method, which learns a Linear(d, Nete x K shots) model to describe one image by the similarity with all training sam- ples, where d is the dimension of visual feature, Nets is the number of categories (e.g. 1000 in ImageNet), and Kshots is the number of shots. Then, the final similarity consists of the original distance between the visual feature and prompt ensembling and the new distance calculated by the learned feature and one-hot vector of labels (whose dimension (Neis X K shots: Nets)). Please find details in Tip-adapter-F (Zhang et al., 2021a). To introduce PLOT to this framework, we first used the feature map to replace the global feature and then learned multiple linear models. As result, with different local features and different linear models, we can obtain M x N distance matrix and apply the Sinkhorn algorithm (Cuturi, 2013) to calculate the OT distance. Furthermore, can apply the learned prompts as co- co-partner of the ensembling prompt refine the final similarity. Table 4 summarizes the few-shot recognition results the original Tip-Adapter-P method and our adapter- based PLOT methods all 11 datasets.
Q: Can PLOT benefit zero-shot learning? A: No. The detailed analysis and discussions can be found in the appendix.
Q: What is the extra computation time cost of PLOT over CoOp baseline? A: Around 10% inference speed and 5% training time. Please the detailed analysis in the appendix.
4.5 VISUALIZATION
In this subsection, we provide some visualization examples of the transport plans T related to different prompts (N=4) in Figure 4. A detailed analysis of these visualization examples and further visualization results including the interpretation of the learned prompt, a T-SNE visualization of prompts, and the visualization of the false case can be found in Section A3
CONCLUSION
In this paper, we present a method, named PLOT, to learn multiple comprehensive prompts to describe diverse characteristics of one category. To avoid convergence to one point, we propose to apply the optimal transport to achieve the fine-grained alignment between both vision and language domains. We apply two-stage optimization strategy where the inner loop fixes the prompts and learns the transport plan to calculate the cross cross-modalln modality distance, and the outer loop uses this distance to optimize the prompt learner. We build our method the base of CoOp and achieve significant improvement on the few-shot recognition task in various datasets, which demonstrates the advantage to learn multiple prompts instead of a single one.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This project was partially supported by the National Institutes of Health (NTH) under Contract ROIHL by the NSF-Convergence Accelerator Track-D award #2134901, by grant from Apple Inc., a grant from KDDI Research Inc, and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, pp. 214-223, 2017.
Emmanuel Boissard, Thibaut Le Gouic, and Jean-Michel Loubes. Distribution's 's template estimate with wasserstein metrics. Bernoulli, 21(2):440-759 2015.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 modd-00--miiin discriminative compo- nents with random forests. In ECCV, pp. 446-461, 2014.
Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrenee Carin. Improving sequence-ter sequence learning via optimal 2019.
transport. arXiv preprint 19Xiv:/901.000
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, pp. 3606-3613, 2014.
Marco Cuturi. Sinkhorn distances: lightspeed computation of optimal transport. In NeurIPS, volume 2, pp. 4, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255, 2009.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang. Lijuan Wang, Chenguang Zhu, Zicheng Liu, Michael Zeng, et al. An empirical study of training end-to-end vision-anddlanguage transformers. arXiv preprint urXiv:2111.02333 2021.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, pp. 178-178, 2004.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint 10.iv:2110.04544. 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07174 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint 2rXi::2006.16211, 2020.
Yicong Hong. Qi Wu, Yuankai Qi. Cristian Rodriguez-Opazo. and Stephen Gould. VIn bert: A recurrent vision-and- aasion-and-language bert for navigation. In CVPR, pp. 1643-1653, 2021.
Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia. Yinfei Yang, and Jason Baldridge. Mural: multimodal, multitask retrieval across languages. arXiv preprint 109.v2209.05/// 2021.
10

---

Published as a conference paper at ICLR 2023
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-lane Saiionlanguage representation learning with noisy text supervision. In ICML, pp. 1904-4916. 2021.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? TACL, 8:423-453 2020.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-moda understanding. In ICCV, pp. 1780-1790, 2021.
Wonjae Kim, Bokyung Son, and IIdoo Kim. Vilt: Vision-and-language transformer without convolu- tion or region supervision. In ICML, pp. 5583-5594, 2021.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, pp. 554-561. 2013.
Charlotte Laclau, levgen Redko, Basarab Matei, Younes Bennani, and Vincent Brault. Co-clustering through optimal transport. In ICML, pp. 1955-1964, 2017.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS, 34.202. 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- linguae understanding and generation. arXiv preprint arXiv:2201.12086, 2022.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 17X5v.190..0355,, 2019.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 1xXiv2101.000190 2021.
Benlin Liu, Yongming Rao, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Multi-proxy wasserstein classifier for image classification. In AAAI, volume 35, pp. 8618-8626, 2021a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenghao Jiang. Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2 13586, 2021b.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint 1rXiv:2103.10388 2021c
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In CVPR, pp. 5206-5215, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1rXiv:1306.5151, 2013.
Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Histoire de l'Académie Royale des Sciences de Paris, 1781
Alex Nichol, Prafulla Dhariwal. Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 / /2. 10741, 2021.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pp. 722-729, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar, Cats and dogs. In CVPR, pp. 2012.
11

---

Published as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen- -r, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In ICCV, pp. 2085-2094, 2021.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 11909000066 2019.
Gabriel Peyre and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning, 11(5-6)3555607, 2019.
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. Bert is knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. arXiv preprint arXiv:1911.0363) 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 20022003000000 00020, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-gun guideed dense prediction with context-aware prompting. arXiv preprint arXiv:21 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint urXiv://922 108/1, 2019.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance metric for image retrieval. IJCV, 40(2):99-121, 2000.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport. ICLR, 2018.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.1199 15980, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 12/2.4022 2012.
Guy Tevet, Brian Gordon, Amir Hertz, Amit Bermano, and Daniel Cohen-Os Or. Motionclip: Exposing human motion generation to clip space. arXiv preprint arXiv:2203.08063 2022.
Matthew Thorpe. Introduction to optimal transport. Lecture Notes, 2019.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NewrIPS, 34:200-212, 2021.
Ruibo Tu, Kun Zhang, Hedvig Kjellström, and Cheng Zhang. Optimal transport for causal discovery. ICLR, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Haohan Wang. songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurlPS, 32, 2019.
12

---

Published as a conference paper at ICLR 2023
Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021a.
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-language pre-training with -ooxuureof-moddalt--eeppep
arXiv preprini arXi::2111.023555 2021b.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, pp. 3485-3492, 2010.
Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov- learning for graph matching and node embedding. In ICML, pp. 6932-6941, 2019.
Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. Vocabulary learning via optimal transport for neural machine translation. arXiv preprint arXiv:20/2.1567), 2020.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's 's distance and structured classifiers. In CVPR, pp. 12203- 12213, 2020.
Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111.03930, 2021a.
Renrui Zhang, Longtian Qiu, Wei Zhang, and Ziyao Zeng. Vt-clip: Enhancing vision-language models with visual-guided texts. arXiv preprint arXiv:2112.02399 2021b.
He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. Neural topic model via optimal transport. ICLR, 2021a.
Wenliang Zhao, Yongming Rao, Ziyi Wang, Jiwen Lu, and Jie Zhou. Towards interpretable deep metric learning with structural matching. In ICCV, pp. 9887-9896, 2021b.
Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv preprint 1 2021a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. arXiv preprint arXiv:2109.01 2021b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. CVPR, 2022.
13

---

Published as a conference paper at ICLR 2023
Appendix for
"PLOT: Prompt Learning with Optimal Transport for Vision-Language Models"
Appendix organization:
A1 Method Details
14
A2 Experimental Details
16
A2.1 Dataset Details
16
A2.2 Implementation Details
16
A2.3 Few-shot Recognition Accuracy
17
A2.4 Ablation Studies Details
18
A2.5 Base-to-New Results
18
A2.6 Zero-shot Setting Analysis
18
A2.7 Computation Cost Evaluation
19
A3 Visualization
19
A3.1 More Analysis on Visualization
19
A3.2 Visualization of Failure Cases
20
A3.3 Interpretation of Text Prompts
20
A3.4 T-SNE of Prompts
20
A1
METHOD DETAILS
The Optimal Transport (Monge, 1781) is initially introduced to find transportation plan to move simultancously several items at minimal cost, such as moving a pile of sand to fill all the holes. Recently, it widely used for the comparison of distributions. Mathematically, given two probability density function U and V over space X' and ), the OT (Wasserstein) distance (Thorpe, 2019) can be defined as
f C(x, (dr,,))
Dor (U, V) = inf
Jxxxy T
(10)
where C(æ,, y is the cost between two points in the space X' x and I denotes the set of transport plans between support points æ and y (e.g. ((2,)) We can regard two probability density functions U and as piles and holes and C' is the function of moving unit of sand.
In our problem of multiple prompts learning, we formulate the sets of visual features and prompt features discrete distributions
U
& mm
and
V
U gm.
(11)
where and the discrete probability vectors that sum to and dy Dirac delta function placed at support point f in the embedding space. Given two support points fm and gn. the cost For simply, in this discrete situation, C XM is matrix in which each point denotes the cost between fm and g..
function is written as C(fm- gx.) sim(fm- 9n) = IT I1
f
14

---

Published as a conference paper at ICLR 2023
Algorithm A1: The training process of Prompt Learning with Optimal Transport
Input: Training few-shot image data: X {æ),, pretrained CLIP model f and g, number of prompts
N, entropy parameter X, maximum number of iterations in inner and outer loops Tin Tout-
Output: The parameters of prompts {w,, h=1)
1: Initialize (w,, (
2: for tout 1, ..,,,..,, Tout in the outer loop do
3: Obtain a visual feature set F E IR M XC with the visual encoder f(r);
4: Generate prompt feature set G E R N xC of each class with the textual encoder {g(tg) Im
5: Calculate the cost matrix Cj 1
F Gy RMxN of each class
6:
Calculate the OT distance with an inner loop: Initialize the v = 1, å = 0.01 and A, = oo
7:
for lin = 1,2..... Tin do
8:
Update u. (fi)) u//(pep(( C/A))o((n Update (Pin.)) = v/((exp(- ( CC/ "((in)) Update A, = Σ (o((tia)) vltin 111/N
9:
10:
if A, < then
12:
break
13:
end
14:
end for
15:
Obtain optimal transport plan as T = diag(u(!)) exp(- ( C ()diag(b()))
16:
Calculate the OT distance dor (k) =< Th: Cs
17:
Calculate the classification probability por(y = kt(z)) with the OT distance
18:
Update the parameters of prompts {wm, ( with cross-entropy loss LCE.
19:
end for
return (w
Then, the total distance of these two distributions is written as:
M
T-C>=
Tm, Cm,
(12)
where the T XNN matrix of transport plan, which is learned minimize the total distance. Each point Tm., in T is a weight of local cost Cm,.-
The optimization problem of optimal transport is formulated as:
dor(u, vlC) minimize < <TC>>
T
(13)
subject T1N u. T 1M U. TERMXN
These constraints of T are used to match its marginal distributions and original discrete distributions in Eq. 11. our framework, we treat visual features fm and prompt features gn equally and thus = 1 /M and v INx1/N.
As directly optimizing the above objective is always cime-consuming, we apply the Sinkhorn dis- tance (Cuturi, 2013) to use an entropic constraint for fast optimization. The optimization problem with Lagrange multiplier of the entropy constraint is:
dor,a (2, a vlC) = minimize < <TC>> -Xh(T)
(14)
subject to
Tl = u, T 1M T TER MXN
where h(-) is entropy and À > 0 is with few iterations as:
Then can have fast optimization solution
T diag(u")) (0) exp cxp((-//)ddiag((())
(15)
where denotes iteration and in each iteration (t) u/ ((exp(-C/\)o(t-I)) and (it)
v/ ((exp(-C/X) (!))), with the initiation (0) = 1. The detailed algorithms the training
and testing processes shown Algorithms Al and A2
15

---

Published as a conference paper at ICLR 2023
Algorithm A2: The inference process of Prompt Learning with Optimal Transport
Input: Testing image data: X = (æ), number of prompts N, number of classes K, learned prompts {th ( ==1==1n=== a frozen pretrained CLIP model including image encoder and text encoderg
KNN
Output: The classification of each image
1: for x in X do
2: Obtain a visual feature
E IR MxC with the visual encoder fx):
3: Generate prompt feature set G5 E R.NC of each class with the textual encoder {s((z)) h
4: Calculate the cost matrix C₁ = 1 GR E RR of each class
5: Calculate the OT distance with an inner loop: Initialize the (0)) 1, 0.01 and A, = 0o
6:
for lin 1,
Update (tin) Tin do (((exp(-C/A)ollee Update (((nn) // (exxp(- C/A) T (
7:
8:
Update A, = Σ (o((in)) (f 1D1/N
9:
10:
if A. < o then
11:
break end if end for
12:
13:
14:
Obtain optimal transport plan T diag(u(!)) exp(- C+/A)diag(b()))
15:
Calculate the OT distance dor (k) =< Th Cx
Calculate the classification probability por kl(z) with the OT distance
17:
return k max Por = k(())
&
18:
end for
A2
EXPERIMENTAL DETAILS
A2.1 DATASET DETAILS
The datasets we used in the experiments follow CoOp (Zhou et al., 2021b), which include 11 datasets for few-shot visual recognition and ImageNet- based datasets for generalization (robustness) evaluation. The details of each dataset are shown in Table A1, including the number of classes, the sizes of training and testing sets, and the original tasks.
A2.2 IMPLEMENTATION DETAILS
The original CoOp method has different versions with different class token positions and parameter initialization strategies. As the performance gap among different versions is limited, we directly chose of them our baseline, where the token position is "end", the parameter initialization strategy is "random", and the length of learnable context tokens is set as 16. Following the widely used setting in (Zhou al., 2021b; 2022; Gao et al., 2021: Zhang et al., 2021a), 2016) as the backbone network of the visual branch. All the code of our method is based on CoOp, which adopted the SGD optimizer with 0.002 initial learning rate, ConianAnnea schedule, and a warmup trick with le-5 learning rate. We also followed the epoch strategy to train more epochs for more shots. For small datasets such FGVCAircraft, OxfordFlowers, and StanfordCars, the batch size is set as 32, while for the larger dataset such as Imagenet and SUN397, the batch size is set as 128.
also chose RN50 (He et al.,
We apply N = 4 prompts for each category and use M = 7 x 7 due to the feature map size. We set the hyper-parameters in the Sinkhorn distances algorithm (Cuturi, 2013) as 0.1 for all the datasets. We set the maximum iteration number of the inner loop 100 and will early stop the iteration when the average absolute update value A < 0.01. We initialize all values in the vector u and μ as 1/N and 1/M respectively. All models are conducted the Pytorch (Paszke et al.. 2019) 1.7.1 and trained on NVIDIA A100 GPUs. We repeated the experiments three times with different seeds and reported the average.
16

---

Published as a conference paper at ICLR 2023
Table Al: The detailed statistics of datasets used in experiments.
Datasct
Classes
Training size
Testing size
Task
Caltech101 (Fei-Fei al., 2004)
100
4,128
2,465
Object recognition
DTD (Cimpoi et al,, 2014)
47
2,820
1,692
Texture recugmition
EuroSAT (Helber ct al., 2019)
10
13,500
8,100
Satellite image recognition
FGVCAíreraft (Maji ct al., 2013)
100
3,334
3,333
Fine-grained aircraft recognition
Flowers 102 (Nilshack & Zisserman, 2008)
4,093
2,463
Fine-grained flowers recognition
Food101 (Bossard et al. 2014)
50,500
30,300
Fine- Fine-grained food recognition
ImageNer (Deng al., 2009)
1,000
1.28M
50,000
Object recognition
OxfordPets (Parkhi et al., 2012)
37
2,944
3,669
Fine-grained pets recognition
StanfordCars (Krause et al., 2013)
196
6,509
8,041
Fine-graine ain--graindd car recugnition
SUN397 (Xiao et al., 2010)
397
15,880
19,850
Sccne recognition
UCF101 (Soomro al., 2012)
7,639
3,783
Action recognition
Im (Recht el al,, 2019)
1,000
10,000
Robusiness of collocation
ImageNet-Skette (Wang al., 2019)
1000
50,889
Robustness of sketch domain
ImageNet-A (Hendryeks al., 2019)
200
7,500
Robustness adversarial attack
ImaageNetRR (Hendrycks al., 2020)
200
30,000
Robustness of mulii-domains
Table A2: The few-shot visual recognition accuracy on 11 datasets.
Dataset
Methods
shot
shots
shots
8 shots
16 shots
PLOT
89.83
0.33
90.67 + 0.21
90.80 + 0.20
91.54 + 0.33
92.21 + 0.38
Caltech101
CoOp
87.51 = 1.02
87.84 1.10
89.52 + 0.80
90.28
0.42
91.99 0.31
PLOT
46.55
= 2.62
51.24 + 1.95
56.03 + 0.43
61.70
0.35
65.60 0.82
DTD
CoOp
43.62 1.96
45.35 + 0.31
53.94 1.37
59.69 + 0.13
62.51
0.25
PLOT
54.05
5.95
64.21 + 1.90
72.36
2.29
78.15
2.65
82.23 0.91
EuroSAT
CoOp
52.12 = 5.46
59.00 1 3.48
68.61 + 3.54
77.08 + 2.42
83.69 1 0.47
PLOT
17.90 = 0.09
18.94 0.44
22.36 = 0.42
26.17 = 0.29
31.49 + 0.89
FGVCAircraft
CoOp
8.59 + 5.79
16.52 + 2.38
20.63 + 2.46
26.63 + 0.86
31.43 + 0.96
PLOT
71.72 = 0.97
81.19 1 0.79
87.82 1 0.20
92.43 1 0.25
94.76 1 0.34
100
CoOp
67.98
1.98
77.58 1 1.46
86.10 1 1.05
91.27 1 0.83
94.49 1 0.40
PLOT
77.74 = 0.47
77.70 0.02
77.21
75.31 = 0.30
77.09 + 0.18
FOOD101
CoOp
74.25
1.52
72.61 + 1.33
73.49 2.03
71.58 = 0.79
74.48 0.15
ImageNet
PLOT
59.54 0.16
60.64
0.06
1.49++0.2
61.92 + 0.09
63.01 + 0.13
CoOp
56.99 1.03
56.40 0.87
58.48
60.39 = 0.57
61.91 = 0.17
PLOT
87.49 0.57
86.64
0.63
88.63 + 0.26
87.39 + 0.71
87.21 + 0.40
OxfordPets
CoOp
0.28 85.99
82.22 + 2.15
86.65 + 0.97
85.36
1.00
87.02
0.89
PLOT
56.60
0.36
0.75
03.41
67.03 = 0.50
72.80 0.75
StanfordCars
CoOp
55.81
1.67
58.41 1 0.43
62.74
67.64 = 0.06
73.60 0.19
PLOT
62.47 0.43
61.71
+ 0.65
65.09 0.43
67.48 0.04
69.96 + 0.24
SUN397
CoOp
60.12 = 0.82
59.60 0.76
63.24 0.63
65.77 + 0.02
68.36 + 0.66
PLOT
64.53
0.70
66.83
0.43
69.60 0.67
74.45
0.50
77.26
11:26100 0.64
UCF101
CoOp
62.13 - 1.14
64.05
1 0.99
67.79 0.71
72.71 + 0.50
76.90 1.900100 0.50
PLOT
62.59 = 1.13
2.72
68.60 0.52
71.23 = 0.51
73.94
Average
CoOp
59.56 = 2.06
61.78 1 1.39
66.47 1 1.29
69.85 1 0.69
73.33 1 0.42
A2.3 FEW-SHOT RECOGNITION ACCURACY
In Section 1. we provide line chart to show and compare the performance of PLOT and CoOp. Here, we provide detailed performance results on all 11 few-shot recognition datasets in Table A2, where we use gray for our method and white for CoOp. To highlight, we respectively use dark cyan and light cyan to represent the performance of PLOT and CoOp on the average all 11 datasets. We repeat all experiments times and report the mean and standard deviation in the table.
17

---

Published as a conference paper at ICLR 2023
A2.4 ABLATION STUDIES DETAILS
In this section, we provide more details about the different variants in Table 2. We compare PLOT with the other 6 baseline methods briefty described below:
CoOp: CoOp is the baseline method that only learns single prompt and matches this single prompt and the global visual feature. We apply the officially released code to reproduce this method.
"G": In this paper, we propose to explore whether we can learn multiple prompts for more comprehensive textual representation and fine-grained visual-textual alignment. "G" denotes that we build multiple prompts (similar to our PLOT and learn them by matching them with the single global visual feature.
Matching all local prompts to a single visual feature will reduce the diversity of the learned prompts. To improve the variety of learned prompts, "G+V" further adds an objective function increase the distances between every two prompts.
"G+V":
"G+E": "G+E" is also method to increase the variety of prompts by separated initializations. It applies predefined different initializations to replace the random initialization, such as "a photo of a", "this is photo", "this is a", and "one picture of a".
"M": One key difference between PLOT and CoOp is utilize the feature map for more fine- one-grained information. To evaluate whether our improvement mainly comes from using feature map, design method "M", which removes the OT distance of PLOT and matches local visual features and multiple textual prompts by the average distance of each visual-textual pair.
"M+V": Similar to "G+V", we add objective function to increase the distances between every two prompts to the method "M" to increase the variety of prompts.
A2.5 BASE-TO-NEW RESULTS
To investigate the generalization of our method for other baseline prompt-learning-aatt methods, we apply our PLOT to CoCoOp (Zhou et al., 2022), by learning multiple textual prompts (e.g. N=4) instead of the single prompt in CoCoOp. We name it CoPLOT. Specially, we learn multiple prompts and use the same neta-ke for all local prompts. Then apply the Optimal Transport to calculate the distance between multiple local prompts and local visual features. We evaluate both CoCoOp and CoPLOT in the setting of "base-to-new" and implement them using the same RN50 backbone. The results on the 11 datasets with 16 shots are provided in Table A3. We observe that PLOT achieves improvement applied to different prompt-learning-ue bbasdd methods. For example, on average,PIOT achieves almost 3% improvement on the "new" side without reducing "base" performance. suggests that these two methods are complementary: CoCoOp proposes conditional formulation that uses each image feature as the context condition to refine the single prompt, while PL.OT aims learn multiple prompts.
most datasets and on average, which demonstrates that it can be
A2.6 ZERO-SHOT SETTING ANALYSIS
PLOT can not benefit in the setting of zero-shot learning. Below we provide some experimental details and corresponding analysis. CLIP shows that manually designing the prompts can still achieve good performance. We obtain 7 prompts by prompt engineering on the ImageNet dataset and can further ensemble them to obtain 60.38% top accuracy. In this section, we replace the cosine distance between the global visual feature and prompt ensemble with the OT distance between the feature map and all 7 prompts. However, without any learning, the OT distance only obtains 58.78% accuracy. It is a limitation of the PLOT to still need few-shot data for optimization, which cannot be directly applied in the zero-shot setting. We argue there are two reasons why the OT distance does not work without learning: 1) prompt engineering selects prompts based the global feature and cosine distance, instead of OT distance with feature map; 2) all these selected prompts are close to the global feature and lack the complementarity.
18

---

Published as a conference paper at ICLR 2023
Table A3: Comparison of CoCoOp (Zhou al., 2022) and CoPLOT(ours) in the base-to-new generalization setting. All methods are implemented with RN50 backbone and evaluated with 16 shots. We report the performance of the base classes, new classes, and the mean of them. We show that PLOT can be applied to CoCoOp and achieve improvement.
(a) Average
(b) ImageNet.
(c) Caltech101.
Base
New
H
Base
New
H
Base
New
H
CoCoOp 75.7
64.6
70.2
CoCoOp
68.3
63.1
65.7 65.7
CoCoOp 95.0
90.0
92.5 93.2
CoPLOT 75.9 67.6
71.8
CoPLOTT 68.2
63.1
CoPLOTT 95.4
90.9
(d) OxfordPets.
(c) StanfordCars.
Flowers102.
Base
New
Base
New
Base
New
H
CoCoOp 92.3
94.6
93.5
CoCoOp CoPLOT 63.2 66.5
65.3
63.6 64.9
CoCoOp 91.2
67.5
79.4 79.4
CoPLOT 92.1
95.9
94
CoPLOT 89.6
69.2
(g) Food101.
(h) FGVCAireraft.
(i) SUN397..
Base
New
Base
New
Base
New
H
CoCoOp 85.0
86
85.5
CoCoOp
25.5
25.7
25.6 26.1
CoCoOp
75.1
73.6
74.4 74.2
CoPLOT 85.0
85.2
85.1
CoPLOT 25.6
26.6
CoPLOT 75.2
73.2
(i) DTD.
(k) EuToSAT.
()) UCFI01.
Base
New
H
Base
New
H
Base
New
H
CoCoOp
73.1
50.0
61.6
CoCoOp 88.9 33.5
61.2
CoCoOp
76.5
61.6
69.1
CoPL.OT 72.6
51.4
62.0
CoPLOT 91.0
55.3
73.2
CoPLOT 77.4
66.2
71.8
Table A4: The training and inference time comparison.
Settings
CoOp
PLOT (N=1)
PLOT (N=2)
PLOT (N=4)
PLOT (N=8)
Training Time (s)
1.127
1.135 714.4
1.148 690.7
1.182
1.267
Inference Time (images/s)
719.1
653.0
519.8
A2.7 COMPUTATION COST EVALUATION
As shown in Table A4, provide the comparison of the training time and inference seed of the baseline method CoOp (Zhou et al., 2021b) and our PLOT with the different number of prompts. We report the one-epoch time training the 1-shot setting of the Food101 (Bossard et al., 2014) dataset and the number images processed by the model second. Taking N as an example, PLOT only reduces the 9.2% inference speed and requires extra 4.9% training time, which is acceptable given the performance improvement.
A3
VISUALIZATION
A3.1 MORE ANALYSIS ON VISUALIZATION
In this section, we provide some visualization examples of the transport plans T related to different prompts (N=4), We translate each transport plan into colorful heatmaps and resize them to their original size and combine them with the raw image. As shown in Figure 4, provide the heatmaps of 4 categories ImageNet. We observe that different transport plans highlight different regions of the image, which demonstrates that the learned multiple prompts "Brambling". the prompts respectively focus on the head, tail, wing, and environment. For "Dog Sled", the prompts related to dogs, the sled, ties, and the snow environment.
complementary. For the class
19

---

Published as a conference paper at ICLR 2023
Table A5: The nearest words for 16 context vectors of all N prompts learned by PLOT N/A means non-Latin characters.
Number
Prompt
Prompt 2
Prompt 3
Prompt
ag pa
trying
gaz
2
flint leaving wit sot tint tar
field
white
3
N/A
4
icons
ario
eclub safe
6
yi indiffe class
7
represented attend vie N/A brought band recognized old late stel head ag industry awhile
attn
N/A ts
8
2
job cold
rollingstones
built yeah
10
11
N/A or love hel
12
13
artifact an
bad N/A ded
15
chie across these
16
5
in actual
visiting
A3.2 VISUALIZATION FAILURE CASES
To better understand the method and further discover the reason for the failure cases, we visualize the attention maps of some failure cases. As shown Figure A1, we showed two failure examples with class "2000 AM General Hummer" in the StanfordCars dataset. During the training, we set the number of prompts as 4, but in these visualization results, we found that some of the learned prompts remarkably coincide with each other. These prompts can be roughly divided into two classes: Foreground and Background. For example, in both images, prompts (right top) and (left down) focus the foreground car, while the others focus the background. It demonstrates that not all classes have multiple complementary attributes, which motivates us to go further to learn the dynamic local prompts numbers to reduce the computational load in the future.
A3.3 INTERPRETATION OF TEXT PROMPTS
The learned prompts are difficult to be understood by humans since the parameters are optimized in the continuous space (Zhou al., 2021b). CoOp proposes use the word which nearest to learned prompts in the embedding space to visualize the prompts. Following this manner, we show nearest words of our learned prompts Table A5. Similar to CoOp, most words can not be directly understood by human logic. However, still find the relations between the learned prompts and the corresponding optimal transport plan. As shown in Figure in the main paper, we can observe that the optimal transport plan for Prompt always focuses the "head", such the head of "brambling". the head of "rooster", and even the head of "aircraft carrier". is because the word "head" is in Prompt 1. Similarly, as the white environment in the image of "brambling" and the snow in the image of "dog sled". It demonstrates that the learned multiple prompts focus on different characteristics of categories.
find that Prompt 4 prefers the white part of images, such
A3.4 T-SNE OF PROMPTS
To better understand the learned prompts, provide visualization with T-SNE (Van der Maaten Hinton, 2008) for the learned textual prompts. Specifically, we randomly select 10 classes from ImageNet and generate the textual embedding with our learned prompts. Then, embeddings with dimension = 1024. Then we apply the T-SNE to reduce the dimension and visualize the embeddings. As shown in Figure A2, the textual embeddings of the same class with different prompts are clustered well. Besides, despite being well clustered, we found that the textual embeddings also have intra-diversitie..
obtain x 10
20

---

Published as a conference paper at ICLR 2023
2000 AM General Hummer SUV
2000 General Hummer SUV
Figure A1: Failure Visualization. We provide the heatmaps of transport plan T related to each prompt on 2 failure examples in the StanfordCars dataset.
T-SNE projection of prompt
-1.
-1.5
-2.0
-25
-3.0
-3.
4.0
-2
Figure A2: T-SNE Visualization of 10 classes with different prompts. We apply the T-SNE for the embeddings of 10 randomly selected classes in ImageNet with different prompts. Different color denotes different classes. We observe that the textual embeddings cluster well.
21


Reviews:
[{"review_id": "iqLVJnLeyox", "reviewer": "Reviewer_FMp9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focused on adapting large-scaled pre-trained vision-language model (i.e., CLIP) to downstream datasets under the few-shot setting. Compared to the recent related work CoOp, this work (1) learns multiple prompts, (2) uses both local features and global features, and (3) measures the similarity of prompts and visual features using optimal transport. Experiments on 11 benchmark visual recognition datasets show that the proposed method outperforms the baseline CoOp. Ablation studies further more analysis on the contribution of each component.", "summary_of_the_review": "Overall, considering both strengths (simple but effective pipeline, comprehensive experimental analysis) and weakness (generalizability), my initial rating is accept.\n\nReasons to accept:\n\n\\+ The motivation for this work is clear. Single prompts may lose the details of local attributes or contexts.\n\n\\+ The idea of applying optimal transport to prompt learning makes sense and meets well with the motivation.\n\n\\+ The experiments and analysis are comprehensive. The improvement is significant.\n\nReasons to reject:\n\n\\- This work is only applied to CoOp. It is not clear whether the method can be extended to other prompt-learning-based methods.", "strength_and_weaknesses": "Strengths:\n\n\\+ The problem of adapting CLIP under few-shot setting is recent. Compared to the baseline method CoOp, the improvement of the proposed method is significant.\n\n\\+ The motivation that (1) single prompt is insufficient to represent a class, (2) directly learning multiple prompts reduce to one single point, are straightforward and make sense. The proposed optimal transport based method is consistent with the motivation.\n\n\\+ The explanation of the comparison between optimal transport and other distances is clear, which supports the motivation.\n\n\\+ The ablation studies and analysis in Section 4.4 is well organized and clearly written. It is easy to follow the analysis and figure our the contribution of each component. Also, Figure 2 is well designed and clear to illustrate the pipeline.\n\n\\+ The experimental analysis is comprehensive. The analysis on computation time and inference speed is also provided.\n\nWeakness:\n\n(As I reviewed this paper before and all my previously concerns are addressed, I do not have much comments about the weakness.)\n\nThis work is built on top of CoOp. It would be interesting to know whether this proposed method can be extend to other prompt-learning-based methods, especially CoCoOp. If not, it would be great to discuss the reasons. Somehow, the method is only applied to CoOp, which is a limitation of this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZJWrpmtRCLg", "reviewer": "Reviewer_qGjf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes PLOT, a prompt learning method for CLIP based on CoOp. PLOT aims to diversify the textual prompts to better depict the category for both its intrinsic and extrinsic contexts. By applying optimal transport, PLOT shows favorable performance over different few-shot benchmarks.", "summary_of_the_review": "The reviewer expects the authors' response to the experiment weakness.", "strength_and_weaknesses": "Strength:\n1) The motivation of enriching prompts for different image regions is practical and well illustrated in the paper. The usage of optimal transport is also novel, which might be inspiring to future works.\n2) The ablation study with discussion clearly tackles most my confusion with interesting points.\n\nWeakness:\n1) The major concern is the classification performance. As shown in Figure 3, despite the better overall performance, PLOT performs worse than previous prompting methods or linear probe on some of the datasets. Considering the much more complicated training, PLOT is expected to achieve higher accuracy.\n2) The authors should provide complete comparisons between PLOT (adapter-based PLOT) and existing adapter-based methods, e.g. Tip-Adapter-F on all 11 datasets, not just ImageNet. As the adapter-based methods are SOTA for CLIP few-shot learning, this can better emphasize the contribution of PLOT.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NMym3dMSTqG", "reviewer": "Reviewer_1oAQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presented a new prompt learning approach for vision-language models, namely PLOT. The main idea of this paper is to learn multiple prompts per class and use optimal transport to match them with visual features of images. The whole optimization process contains inner and outer loops to optimize the matching and prompts, respectively. The experiment results show that the proposed approach can achieve state-of-the-art performance in the few-shot image recognition task with a slight increasement in training time.", "summary_of_the_review": "This paper addressed prompt learning for vision-language models which is an interesting and important problem in few-shot image classification tasks. The proposed PLOT is reasonable and technically sound to me. However, I still have some concerns on the experiment part of the paper. Therefore, it leads me to borderline acceptance as my initial rating. I will be happy to increase my rating if the concerns could be addressed in the discussion period.", "strength_and_weaknesses": "__Strength__\n\n1. This paper is overall well-structured and easy to follow.\n\n2. The idea of learning multiple comprehensive prompts to describe the image categories is reasonable to me, and using optimal transport for optimization prompt learning is novel.\n\n3. The ablation studies are comprehensive which clearly demonstrate the design choices of the proposed method.\n\n__Weakness__\n\n1. In Figure 3, the improvement achieved by the proposed method over CoCoOp is marginal. I wonder if the conditional formulation presented in CoCoOp can be easily extended to PLOT? If so, at least more detailed discussions should be provided by the authors.\n\n2. Figure 4 shows the visualization results of attended regions of images from different prompts. It is also interesting to see how learned inter-class and intra-class prompts are different from each other by using t-sne or PCA.\n\n3. It is unclear to see the differences among model variants in Table 2 just from the caption. It will be good if an individual section on introducing the detailed design/training setup of each model in the paper (or in the appendix). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AnYgpOo3eFK", "reviewer": "Reviewer_RWPs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for prompt learning in vision and language models that aims at producing a variety of templates that can be used by the text encoder to provide better classification weights in a few-shot scenario. \n\nIn particular, the paper observes that learning a single prompt is not sufficient to represent a class, and proposes a variety of prompt candidates that can be used to cover a wider variety of concepts within the same image. The authors note that using a distribution of prompts that are learned independently using a standard contrastive learning will result in some of the prompts to collapse to the same centroid, and propose an optimal transport scheme to avoid such behaviour. \n\nThe methodology goes as follows: 1) a set of different visual features are computed for a given image (using different backbone exits); then 2) the scores for the pairs visual/text features are computed using the learnable prompts; then 3) a transport matrix is computed to minimize the total cost of assigning a feature vector a weighted combination of the text features (i.e. the transport matrix is the soft-extension of the permutation matrix that would assign vision/text features through the typical CLIP matching). Finally, for a fixed transport matrix 4) the prompts are learned using the standard cross-entropy where the class assignment is given by the computed combination of the weights produced by the prompts. \n\nThe use of a transport matrix allows to assign to each feature vector a weighted combination of the prompts, thus making the latter have a broader \"semantic\" capacity. The use of different feature vectors for each image rather than a global CLIP-based visual feature vector allows the prompt to describe different concepts that might co-occur in the same image, making the prompts having more ``local\" capacity. \n\nThe paper is validated in the same setting as CoOp for few-shot learning, using 11 visual recognition datasets, delivering consistently better results than the competing method. \n", "summary_of_the_review": "In summary, the paper proposes an interesting approach to prompt learning that is technically sound and well motivated. However, the clarity and the lack of an exhaustive comparison against state of the art works ([Lu et al.] and CoCoOp) both experimentally and conceptually is needed. Should the authors be able to bring this up would further increase the quality of the paper. ", "strength_and_weaknesses": "Strengths: \n\nThe paper is technically solid and contributes enough novelty to the domain of prompt learning which is of growing interest to the research community devoted to exploring pre-trained vision/language models. The use of a diverse set of prompts to generate different text classifiers that can all contribute to a single class prediction is of interest and the methodology is well developed. \n\nThe results seem to consistently outperform CoOp, showing the need of having a large set of prompts that require proper use to be able to generalize better to new classes.\n\nWeaknesses:\n\nWhile the core methodology is new, the idea of learning a distribution of prompts is not new, and indeed the authors do not include a discussion on why their proposed approach would be a better choice than CoCoOp which produces prompts for each image. Additionally, the authors need to consider, discuss, and compare, against \n\n[Lu et al.] Prompt Distribution Learning. CVPR 2022\n\nwhich is a method that learns a collection of prompts that produce a distribution of per-class weights, increasing their expressivity. Some analysis in terms of methodology and computation against [Lu et al.] should be included in the paper in my opinion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%