Prompt Learning with Optimal Transport for Vision-Language Models
Authors: Guangyi Chen
Weiran Yao
Xiangchen Song
Xinyue Li
Yongming Rao
Kun Zhang
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Prompt learning
Few shot learning
Vision-language pretrained model
Optimal transport

Abstract: With the increasing attention to large vision-language models such as CLIP, there has been a significant amount of effort dedicated to building efficient prompts. Unlike conventional methods of only learning one single prompt, we propose to learn multiple comprehensive prompts to describe diverse characteristics of categories such as intrinsic attributes or extrinsic contexts. However, directly matching each prompt to the same visual feature is problematic, as it pushes the prompts to converge to one point. To solve this problem, we propose to apply optimal transport to match the vision and text modalities. Specifically, we first model images and the categories with visual and textual feature sets. Then, we apply a two-stage optimization strategy to learn the prompts. In the inner loop, we optimize the optimal transport distance to align visual features and prompts by the Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distance from the supervised data. Extensive experiments are conducted on the few-shot recognition task and the significant improvement demonstrates the superiority of our method.

Prompt Learning with Optimal Transport for Vision-Language Models
Anonymous Author(s)
Affiliation Address email
Abstract
With the increasing attention to large vision-language models such as CLIP, there has been significant amount of effort dedicated to building efficient prompts. Unlike conventional methods of only learning one single prompt, we propose learn multiple comprehensive prompts describe diverse characteristics of categories such as intrinsic attributes or extrinsic contexts. However, directly matching each prompt the same visual feature problematic, as pushes the prompts to converge to one point. To solve this problem, we propose to apply optimal transport to match the vision and model images and the categories with visual and textual feature sets. Then, we apply two-stage optimization strategy to learn the prompts. the inner loop, we optimize the optimal transport distance to align visual features and prompts by the Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distance from the supervised data. Extensive experiments are conducted on the few-shot recognition task and the improvement demonstrates the superiority our method.
modalities. Specifically, we first
10
11
12
14
Introduction
In the past few years, large-scale vision-language pre-trained (VLP) models, such as CLIP [39] ALIGN [17]. and BLIP 23 have achieved remarkable success open-world visual concept learning. These methods have brought new light but also pose question: how to efficiently adapt the 19 knowledge from pretraining to the downstream tasks since these models are typical of massive sizes which are not feasible for normal users to re-train.
17
21 One of the conventional paradigms of utilizing pretrained 23 knowledge is "pre-training, fine-tuning", which fixes the architecture of the pre-trained neural network and tunes its parameters using task-specific objective functions. Beyond fine-tuning the parameters,
Brambling
bird with dark fan-tail
bird with black crown and eye
25
bird that lives in winter wood
bird with orange and black texture
27
29
Figure
The motivation that one category can complementar- ily described in different views (An example of "Brambling"),
30 many recent methods [63 64
31 introduce the concept of prompt learning from the field of NL.P the vision domain and achieve striking performance gain for the few-shot visual classification. They fix the model parameters and instead learn suitable prompts by turning template sentence into of learnable vectors. Then, these prompts are learned by minimizing the distance between the visual features and prompt-based language features.
33
34
35
Despite significant improvements insufficient to represent a class. One class can be described by many intrinsic characteristics and
manual prompts, learning only sentence is intuitively
37
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

38 even extrinsic context relations. Thus, for one object, we may have multiple prompt candidates which focus on different attributes. As shown in Figure [I] we can describe the class "Brambling" in 40 different views: such as the color of the wing, the color of the crown and eyes, the shape and color of the tail, and even the living environment information. It motivates us to learn multiple prompts to comprehensively represent the class and thus facilitate classification.
39
41
43
The most natural solution is to directly learn multiple prompts by respectively matching each prompt with the visual features. However, it is the same as matching the mean of prompt features and the visual features. This solution problematic since all prompts are encouraged to be closer to one single point and thus tend to learn the same characteristics. It contradicts our purpose to learn comprehensive prompts. To solve this problem. we tested adding some constraints to push away the prompt from each other, but found that this solution still fails to learn representative and comprehensive prompts. This solution treats the visual representation as one single point, and such unified view of visual features ignores the fact that different prompts may only focus on one or a subset of characteristics.
44
45
46
47
48
49
50
51
To address this problem, in this paper, we propose Prompt Learning with Optimal Transport (PLOT), which applies optimal transport (OT) to align the local visual features and multiple textual prompts. Optimal transport can calculate the distance between two distributions under the form of multiple sampling. In our prompt learning framework, we formulate local visual features and multiple prompts as the samplings of two discrete distributions and use OT to encourage fine- mneegraaddd mooal matching. Specifically, to obtain the local visual features with different semantic clues, we extract all feature maps as the visual representation instead of the single global representation. Fortunately, we can easily obtain the visual feature maps from the visual encoder of CLIP by using all outputs of the multi-head self-attention layer 442. Then the problem comes down to how to calculate the distance between two feature sets.
52
53
54
55
58
57
58
59
60
61 We solve this problem by introducing the optimal transport theory [5] and formulate the feature sets as a discrete probability distribution where each feature has an equal probability value. Furthermore, reduce the computational cost and avoid the extra model parameters, we learn the prompts with a wto-staee optimization strategy. At the first stage in the inner loop, we fix both visual and text features and optimize the optimal transport problem by fast Sinkhorn distances algorithm [6]. Then, in the outer loop. we fix all parameters of optimal transport and back propagate the gradient to learn the prompts with different characteristics. Compared with conventional distance (such as Fuclidean distance of mean features), optimal transport can align different visual features for each local prompt, which is more robust to the visual misalignment and tolerates well feature shift (44). It is because OT learns an adaptive transport plan to align, features, which achieves fine- mne-grained matching across two modalities. We conduct experiments on 11 datasets following the standard setting of CLIP [39] and CoOp 663 to evaluate our method. These experiments span the visual classification of generic objects,
62
63
64
65
66
67
68
69
70
71
75 scenes, actions, fine-grained categories, and so on. The significant result improvement demonstrates that PLOT can effectively learn representative and comprehensive prompts.
74
2 Related Work
78
Optimal Transport The Optimal Transport 33 is initially introduced to solve the problem of how
77 to reduce the cost when moving several items simultaneously. Recently, OT theory has drawn wide
78 attention in the machine learning and computer vision community by comparing distributions readily available to them under the form of feature sets [37]. Due to the brilliant property of distribution
79
80 matching, OT has been applied in many theoretic and application tasks including generative models [I]
81
45 60), structural matching 4 57 61 (56) (e.g. sequence matching 4 and graph matching (56)),
and other distribution- based tasks (such as clustering (22), distribution estimation (2), and causal
83 discovery (50). In this paper, we use OT to align the features of vision and language modalities
84
which represents the data structure by learning an adaptive transport plan [44).
85
Vision-Language Pre-trained Models Vision-Language Pre-trained (VLP) models aim to explore the semantic correspondence between the vision and language modalities through large-scale pre- training. Recently, VLP models have achieved an exciting performance improvement in the zero-shot and few-shot visual recognition [39 10 open-world visual understanding with the help of language. One key part of learning VLP models is the self-supervised learning objective on two modalities. The popular VLP objectives can be divided
86
87
88
63 64 59), which shows the great potential to promote
89
90
91
into reconstruction 25 15 g 20 contrastive matching [39 17 16, or the combination of both
two 24 54 19]. Besides, recent progress in the field of VLP also benefits a lot from large-scale
2

---

93
applies 400 million image-text pairs for contrastive learning, while ALIGN even exploits 1.8 billion data pairs. Beyond recognition, these VLP models also show great potential for other downstream applications, such as dense prediction 42 62
pair-wised datasets. For example, CLIP [9
94
95
image
96
generation 31 41 and action understanding [5 48
87
Prompt Learning Prompt learning is introduced from the field of NLP to efficiently adapt the large language model downstream tasks. Different from the conventional "pre-training, fine-tuning" paradigm which initializes the pre-trained model and tunes the parameters of the network using downstream task-specific objective functions, prompt learning applies textual prompt to reformulate the downstream tasks as the original pretrained task [6). By the prompt, the domain shift between pretrained task and downstream application is reduced and thus the pretrained knowledge can be
98
99
100
101
102
103
casier adapted to downstream tasks. The concept of prompt learning [6 40, 38] begins from the
104
success of GPT 40) series. Farly prompt learning methods (such as Petroni et al. [36] and Pörner er
105
al.
al always manually create templates based on human prior knowledge. Furthermore, some methods [46] are proposed to automatically search for appropriate templates. Beyond search in the discrete space, some methods [26] 49 28 remove the
38)
106
mining- based methods [8] and gradient-
108
constraint that the prompts are "words" and instead learn prompts in the continuous embedding space,
Recently, CoOp [63] and its extended version [64] introduce prompt learning into open- weenwworld visual understanding to adapt the knowledge from the large-scale visual- risual-language pretrained models and achieve great performance improvement on the few-shot visual recognition. Compared with CoOp, our PLOT method further improves prompt learning by introducing the optimal transport distance to learn multiple local prompts and achieves fine- ane-grained vision-language matching.
110
112
3 Approach
114
115
In this section we will first revisit the baseline method CoOp 3.1 review the preliminaries of optimal transport and then introduce our proposed 3.3 to show how we can learn multiple comprehensive prompts.
116
118
3.1
A Revisit of CoOp
119
CoOp
(63] is one of the pioneering methods to learn the prompts for using vision language pretrained knowledge (such as CLIP (39) for downstream open-world visual recognition. Different from CLIP
120
121
which manually designs the prompt templates, CoOp sets a part of context words in the template as continuous learnable parameters which can be learned from the few-shot data. Then the classification
123
weights can be represented by the distance between the learned prompt and visual feature.
124
Specifically, given an image æ, visual feature f f(x) is obtained by the visual encoder f of CLIP. Then, the textual prompt can be formulated as t {veci vec2..... vecL, c where dj is the word embedding of the class name, {veci / } are Icarnable vectors with the same dimension as the original word embedding and L. is the length of context words. With prompt t. as the input, the
125
128
127
128
text encoder outputs the textual feature as 9b: = g(t)). The final prediction probability is computed
129
by the matching score as follows:
ply = k/() Σ#=1 exp(sim (fo ffg.)//))))))
(I)
130 where sim(.,) denotes a metric function such as cosine similarity, and 7 stands for the temperature of Softmax. Then we can optimize the parameters of (vecilf with the cross-entropy loss between the prediction and the labeled target.
132
3.2 Optimal Transport
Optimal transport (OT) distance is widely used metric for the comparison of distributions. Here, we only focus on the discrete situation which is more related to our framework. Assuming we have two sets of points (features), the discrete distributions formulated as:
135
N
U
Σ
Unn 5
and
V
Σ
137 where u and v are the discrete probability vectors that sum to 1, and 8y a Dirac delta function placed at support point f in the embedding space. Then, the total distance of these two distributions
3

---

[ XX CLASS
Prompt set
[X ... X CLASS)
Text Encoder
Optimal Transport Distance
[X ... X CLASS]
I.. XX CLASS]
Brambling, House finch,
Feature set
Image
Encoder
Figure 2: The framework of PLOT. PLOT first describes each category with multiple prompts and obtains a set of prompt features by text encoder. The image is also encoded as set of local features. Then the optimal transport used the metric between prompts and visual features.
139
written as:
N Σ TTm...m.pr
T,C Σ
(3)
We call C' the cost matrix in which each point denotes the cost between fm and In, such as 141 Cm,. = 1 sfm(fm: 9n). While the T is called the transport plan, which learned to minimize the total distance. The optimization problem of optimal transport is formulated as:
dor x|C) minimize < T C
subject to
T1 = u, T
143 As directly optimizing the above objective is always time-consuming, we apply the Sinkhorn dis- tance [6] to use entropic constraint for fast optimization. The optimization problem with 145 Lagrange multiplier of the entropy constraint is:
dor.x(u, v|C) = miiimmiee < <T,C>
T
(5)
subject to
T1 u. T 1
146
where h(()) is entropy and 2 a hyper-parameter. Then we can have fast optimization solution with few iterations as:
T diag(u') (eap()"))(CC/))daag/o
(6)
where t denotes iteration and in each iteration u² u/((er((CC/\)ot1 and ot C/X)F with the initiation v
149
150
3.3
Prompt Learning with Optimal Transport
151
In this subsection, we introduce the details our PLOT, which learns multiple prompts to describe different characteristics of the category by minimizing the OT distance.
153
Specifically, as shown in Figure 2 given an image æ, we first feed it to the visual encoder branch of CLIP. Apart from the global visual feature f. we can also obtain a set of local features {f M mn The visual encoder has multi-heat attention pooling layer which the input the combination of the global feature and a set of local features (feature map) and the output is tensor with the shape ( [ +1)xC' where H and W is the height and width of feature map and C is the feature dimension. Therefore, we can obtain M 1l W local features and global feature. At the same time, for 1 where each the same as the prompt in CoOp. With both visual and textual encoders, we can obtain local visual features F {f... I E IR M XC and prompt features G = [g E NCC
154
155
156
157
158
IV class k we can initialize N local prompts (tv.nIN with learnable vectors {vec,,, L.N
159
160
In the inner loop, learn the transport plan T with these fixed support sets F. Gk. by minimizing the following OT distance to push G to F:
163
dor = dor (uuuvil v Gk).
(7)
184 where C FT denotes that we use the cosine distance between and G3 as the cost matrix. Then obtain the solution of transport plan T Eq (6) and the final OT distance doT(k).

---

166
Given the OT distance between G₁, and F, we reformulate the prediction probability as:
cxp((1 dor(k))/TT)
Porly = kl/= Σ exp((1 - dor (k'))7)
(8)
167
In the outer loop, we fix the transport plan T and apply the cross entropy loss to optimize the L.N as:
188
K
1
LCF =
Un. k|xz),
[28
=EX
189
where Yn is a one-hot label vector. The detail algorithm can be found in the supplementary materials.
170
Though the optimization strategy of the optimal transport and prompts is two-stage, the whole training flow is end-to-end It is because that the transport plan is computed using a small number of matrix multiplications as one forward module of the neural network. The gradients of these matrix multiplications are taped for backpropagation for end-to-end optimization, which makes the whole system fully differentiable (including the iterative algorithm) and easy to implement using an autograd library like PyTorch In the experiments, we found that it is natural and relatively easy to this optimization strategy.
171
172
175
175
178
177
3.4
Inference strategy
178
the inference, given one query image and the learned prompts, we first obtain the visual feature set containing M = H x W vectors and a prompt feature set containing N x C vectors. Then, we calculate the distance between the visual feature set and the prompt feature set of each class by OT as 6 After obtaining the OT distance for each class, we sort the distance and classify the image.
180
4 Experiments
182
Extensive experiments are conducted to evaluate method, including comparison with CoOp. ablation studies, parameter analysis extensibility analysis, computing cost analysis and visualization.
184
4.1
Datasets
186
We followed the experimental settings in the CoOp 663 for the few-shot learning evaluation. The
187
experiments are conducted on the 11 visual recognition datasets, including Caltech101 91 DTD
[5]
188
EurOSAT [12], FGVCAircraft
[29]. Flowers 33 Food101 [3] ImageNet (7). OxfordPets 33
189
StanfordCars 21 SUN397 (55] and UCF101
These datasets span visual classification of
190
generic objects, scenes, actions, fine-grained categories, and so on, which constitutes a comprehensive evaluation our method. All experiments adopted the few-shot evaluation protocol used in CLIP [39
191
and CoOp 163), where we respectively choose 1, 2, 4, 8, and 16 shots for model training and use the original test set for evaluation. Besides, we also evaluated the robustness of our method with domain shift. Following CoOp, we used the ImageNet as the source domain and evaluate our method with ImageNet- based robustness evaluation datasets including ImageNetV2 [43), ImageNet-Skeet
193
195
[52), ImageNet-A A [14], and ImageNet-R [13]. A detailed introduction of each dataset can be found in the supplementary materials.
197
4.2
Implementation details
199
We chose CoOp 6x as our main competitor to evaluate our method. Compared with CoOp which only learns a global prompt for one class, our PLOT method learns multiple local prompts and applies the OT distance for better fine-grained alignment. Besides, we also reported the performance of training a linear classifier with the CLIP [39] features. It is also a widely-used strategy to adapt the pretrained knowledge for the downstream task [46). We reproduced the performance of CoOp and the CLIP linear probe with the released official code.
200
202
204
205
The original CoOp method has different versions with different class token positions and parameter initialization strategies. We applied the default model that fixes the class token positions in the end due to the limited performance gap between two different ways of positioning the class token. Besides, we
206
207
208
used the random parameter initialization strategy but not the class-specific context version. Following
the widely used setting in 63 64
10 588 we also chose RN50 [1]] as the backbone network of the visual branch and set the length of learnable context tokens as 16. All the code of our method based
210
on CoOp, which adopted the SGD optimizer with 0.002 initial learning rate, CosineAnnealingL..
5

---

Average over 11 detasets
30
Inogefitt
Oxfiriihess
Scoret)
ScoreNo
Unee Probe
Uinea Probe
Starto
Rowerc10
Peod100
PovEAkrwt
Carr
Dars
Clars
Loop
Coop
CoCl
diss
Shots clss
clos
Shots
Euroder
Ucr301
sund07
Sgrato
Ours
Duts
Qut's
Cars
CoOp
CoOp
CoOp
CoCo
CaCoOp
CoCoOp
CaCodo
CoCoOp
Unear Probs
Lincrr Phse
Unser frobe
Proba
Shots perdisss
Shots per cass
Shots per closs
Shott per clsss
Figure 3: The few-shot learning results on 11 datasets. We compare our PLOT with CoOp, CoCoOp. and the Linear Probe method and observe the consistent and significant performance improvement on most datasets. (The average accuracy on all datasets is shown on the left top.)
Table 1: Comparison with CoOp on robustness to domain shift.
Source
Target
Method
ImageNct
-V2
-Sketch
-A
-R
CLIP + CoOp
61.91
54.26 55.11
32.47
21.78 21.86
54.21
CLIP + PLOT (N=4)
63.01
33.00
55.61
212 schedule, and a warmup trick with le-5 learning rate. Besides, we also followed the epoch strategy to train more epochs for more shots.
We apply N 4 prompts for each category and use M 7 x 7 due to the feature map size. We set the hyper-parameters in the Sinkhorn distances algorithm [6] as À = 0.1 for all the datasets. We set the maximum iteration number of the inner loop as 100 and will early stop the iteration when the
215
216
217
average absolute update value A < 0.01. We initialize all values in the vector V and as 1 I/N and 1 1/M respectively. All models are conducted on the Pytorch [34] 1.7.1 and trained on 4 NVIDIA
218
219
A100 GPUs. We repeated the experiments three times with different seeds and reported the average.
220
4.3
Comparison With CoOp
221
In this subsection, we compare our PLOT with the baseline CoOp on the few-shot recognition and domain generalization tasks.
222
223
Few-Shot Learning We summarized the experimental results in Figure 3 where the red line denotes our PLOT method, the blue one denotes CoOp, the purple line denotes CoCoOp, and the green one the CLIP linear probe. The detailed accuracy can be found in the supplementary materials. We observed that both prompt learning methods (PL.OT and CoOp) outperform the linear probe method
224
225
226
227
by a large margin. Besides, PLOT can further improve the performance of CoOp and CoCoOp on most of the datasets. Taking the average accuracy (at the left top) as the example, Plot respectively gained 3.03%, 3.45%, 2.13%, 1.38%, 0.61% performance boost over CoOp at 1, 2, 4, 8, 16 shots. We
228
229
230
found the performance gap will reduce when shots increase. It is not surprising since both CoOp and PLOT focus on utilizing the pre-trained knowledge, and the effect of pre-training diminishes given more training data. Among all datasets, PLOT achieves a larger improvement over CoOp on
231
232
233
the FOOD101 and DTD datasets and achieves comparable performance only on the StanfordCars datasets. For the FGVCAircraft dataset in which the CoOp only obtains 7.77% accuracy, our PLOTT can achieve an accuracy of 17.79%, twice as high as that of the CoOp. Note that we don't use the class class-specific context, thus the performance on the fine- fine-grained classification datasets is lower, e.g.
235

---

Table 2: Ablation studies on few-shot recognition. PLOT is our defined model with N 4, CoOp the baseline method, M denotes that we respectively match the global visual feature and multiple textual prompts, V denotes that we apply constraint add the variance of prompts, M indicates using the visual feature map instead of the global visual feature.
Dataset
Settings
1 shot
2 shots
4 shots
8 shots
16 shots
PLOT
89.83 + 0.33
90.67 + 0.21
90.80 + 0.20
91.54 + 0.33
92.24 + 0.38
CoOp
87.51 + 1.02
87.84 = 1.10
89.52 = 0.80
90.28 = 0.42
91.99 + 0.31
88.13 1 0.36
86.98 + 1.25
88.45 0.79
90.16 0.22
90.72 0.18
Caltech101
G+V
88.28 0.43
87.72
1.25
88.45 + 0.30
89.82 + 0.20
92.00 0.13
M
69.78 + 1.75
71.57 + 1.59
77.18 + 2.16
81.77 + 0.47
86.21 + 0.20
M+V
16.11:8.20
71.45 3.98
79.30 = 3.96
86.96 + 0.78
89.80 + 0.17
PLOT
46.55 + 2.62
51.24 + 1.95
56.03 + 0.43
61.70 + 0.35
65.60 + 0.82
CoOp
43.62 + 1.96
45.35 + 0.31
53.94 + 1.37
59.69 + 0.13
62.51 + 0.25
G
45.12 + 1.69
+6.39
54.75 + 0.48
6.70
63.59 0.76
DTD
G+V
45.90 2.00
0.99
53.96 + 0.48
59.69
63.51 + 0.66
M
13.18 + 4.57
12,25 3.86
13.00 = 4.73
1.42
26.99 + 1.98
M+V
12.61 :: 5.93
15.11 1.81
20.35 = 1.33
44.13
56.85 1 0.54
PLOT
77.74 0.47
77.70 + 0.02
77.21 + 0.43
75.31 + 0.30
77.09 + 0.18
CoOp
74.25 + 1.52
72.61 + 1.33
73.49 = 2.03
71.58 + 0.79
74.48 + 0.15
74.6310.11
1..15
70.41 1 0.46
1
73.68 1 0.46
FOODI01
G+V
74.83 0.31
70.09 0.85
70.86 1 0.22
70.80 0.68
73.93 0.35
M
52.02 1: 4.86
46.12 1 1.46
46.86 1 1.39
53.43 1 0.88
61.28 1 0.23
M+V
46.52 + 1.15
45.95 + 2.66
53.57 = 0.83
62.95 + 0.37
67.63 + 1.11
Table 3: Parameter analysis for the number of prompls
Dataset
Settings
1 shot
2 shots
4 shots
8 shots
16 shots
N=1
88.47 + 1.15
89.19 + 0.39 0.39
89.70 = 0.38
90.45 = 0.24
91.56 + 0.14
N=2
88.86 + 0.51
$9.60
90.60 = 0.17
91.25 0.65
91.89 0.36
Caltech101
N=4
89.83
0.33
90.67 0.21
90.80 0.20
91.54 + 0.33
92.24 - 0.38
N-8
89.74 4: 0.30
90.18 + 0.46
91.02 0.18
91.28 + 0.28
92.01 + 0.29
N=1
43.91 0.65
48.21 + 2.20
53.69 = 1.10
58.90
62.85 + 0.74
N=2
45,59 + 2.46
48.06 + 1.92
55.58 = 1.71
11.56:0.11
64.60 + 0.92
DTD
N=4
46.55 2.62
51.24 1.95
56.03 0.43
61.70
65.60 0.82
N=8
46.89 + 1.94
51.87 + 2.06
51.45 + 0.48
62.20
65.25 + 0.38
N=1
75.96 + 0.48
7
77.11 + 0.41
76.56
77.43 0.80
N=2
77.12 :: 0.49
76.89
0.23
76.16 = 0.52
0.52
75.23
76.81 1 0.50
FOOD101
N=4
77.74 0.47
77.70 0.02
77.21 0.43 0.43
75.31 0.30
77.09 1 0.18
N=8
78.05 + 0.15
78.19 + 0.07
78.12 + 0.17
76.63 + 0.22
77.48 + 0.12
the performance of both CoOp and PLOT without class-specific context is lower than the linear probing on FGVCAircraft. All these performance comparisons can serve as experimental evidence to demonstrate that multiple local prompts and optimal transport distance facilitate the prompt learning of asioonllanguag models. On StanfordCar, learning multiple prompts didn't 't significantly improve the performance over a single prompt. It may be because the discriminative characters in this dataset coincide with each other, such that one global prompt and one global visual feature can work well.
236
240
242
Domain generalization The robustness also plays critical role in model applications since the real-world environment may have large domain shifts with the training data. Therefore, we conducted robustness evaluation to investigate the transferability of models learned by PLOT.
244
246
Table D summarizes the results of PLOT method and CoOp on four ImageNet-based robustness evaluation datasets. For both methods, we trained the models on ImageNet with 16 shots per class. For PLOT, we set the number of prompts as N 4. We can observe that PLOT outperforms CoOp consistently on both source and target domains. These experimental results demonstrate that the performance improvement of our learning multiple prompts doesn' 't rely on single-domant overfitting.
247
248
250
251
4.4
Ablation Studies and More Analysis
252
In this subsection, we conducted the ablation studies to investigate the effectiveness of different components, in order to answer the following questions.
7

---

254
Q: Can we directly learn multiple prompts by respectively matching each prompt with the global visual feature? A: No. As shown in T we report the performance of directly matching the global visual feature (notated as "G") and compare it with the baseline CoOp and our PLOT on three datasets including Caltech101, DTD, and FOOD101. We observe that there is no improvement
255
256
257
258
over the baseline on some datasets (such as Caltech101 and FOOD101) if we only directly match prompts and global features. Though "G" obtained the improvement on the DTD dataset, this improvement is still less than that of PLOT. It is because this "G" method is incentivized to learn the indistinguishable prompts, which contradicts our purpose to learn multiple comprehensive prompts. We further add some constraints to push away the prompt from each other. For example, we add an objective function to add the distance between every two prompts as a regularization term, which is notated as "V". However, comparing "G" and "G+V", we do not find significant and consistent improvement when using variance loss.
259
261
262
263
265
266
Q: Does the improvement mainly come from using all feature maps? A: No. In PLOT, we apply all feature maps of the visual encoder branch, where each feature a local embedding at one spatial position. Compared with the global feature, these local features are more informative and contain fine-grained clues. However, we demonstrate that the improvement of PLOT does not only rely on using all feature maps. On the contrary, directly using the feature map to replace the global feature causes a large performance drop. For example, on all three datasets, directly using the feature map ("M" "M+V") has around 20% shot accuracy drop over using the global visual feature. It is not surprising since the original CLIP model is trained by matching the global visual feature and language feature. Without using the OT method, the distance between the feature map and multiple textual prompts degenerates to the mean distance of each feature-promp prompt pair. Besides, when using the feature map, adding the variance loss works well, especially for more shots. For example, the accuracy on 16 shots DTD is improved by a large margin (from 26.99 to 56.85).
267
269
271
273
275
276
277
Q: How many prompts are needed? A: 4 prompts are enough One important hyper- hyper--arameter in PLOT is the number of prompts. To analyze the effect of the number of prompts, we conducted the experiments on three datasets with 1, 1,2, 4, 8 prompts. The results are summarized in the white part of 3 We can observe that the performance obviously increases when adding the number of prompts from to 4. For example, PLOT (N=4) respectively obtains 1.36%, 2.64%, and 1.68% 1 shot accuracy improvement over PLOT (N=1) on three datasets. Besides, when we further increase the number of prompts, the improvement is not consistent. To balance the improvement and cost, we set N = 4 as the default configuration of our PLOT model. In the experiments, we tuned this hyper-parameter on the Caltech101 dataset and applied it to other datasets.
279
281
282
283
284
285
286
287
Q: Can PLOT benefit zero-shot learning? A: No. CLIP (39] shows that manually designing the prompts can still achieve good performance. We obtain 7 prompts by prompt engineering on the ImageNet dataset and can further ensemble them to obtain 60.38% top I accuracy. In this section, we replace the cosine distance between the global visual feature and prompt ensemble with the OT distance between the feature map and all 7 prompts. However, without any learning, the OT distance only obtains 58.78% accuracy. It is limitation of the PLOT still need few-shot data for optimization, which cannot be directly applied in the zero-shot setting. We argue there are two reasons why the OT distance does not work without learning: 1) prompt engineering selects prompts based on the global f feature and cosine distance, instead of OT distance with feature map; 2) all these selected prompts are closed to the global feature and lack the complementarity.
289
291
292
293
294
295
296
297 Q: Can PLOT benefit Adapter- methods? A: Yes. Adapter-based methods 10 58 is another research direction of the efficient adaptation of pre-trained vision-language models. Different from the prompt learning that fixes the model parameters and tunes the language prompt, adapter-base methods 110 (58 allow for fine-tuning part of the network or adding extra model for training. Recently, adapter- based methods also achieve good performance on shw-shot visual recognition. Therefore, we want explore whether our PLOT method can benefit them, and how.
299
301
302
303
We apply the Tip-adapter-F [88 as our baseline method, which learns a Linear(d,Na) model to describe one image by the similarity with all training samples, where is the dimension of
Kshots)
304
305
visual feature, Not. is the number of categories (e.g. 1000 in ImageNet), and Kshots is the number
of shots. Then, the final similarity consists of the original distance between the visual feature and
307
prompt ensembling and the new distance calculated by the learned feature and one-hot vector of
labels (whose dimension is (Nele x K Kahotes Net.)). Please find details in Tip-adapter-F [58). To
309
introduce PLOT to this framework, we first used the feature map to replace the global feature and
8

---

Brambling
Rooster
Aircraft
Carrier
Dog Sled
Figure 4: Visualizations. We provide the heatmaps of transport plan T related each prompt on categories in ImageNet. Different transport plans focus on different attributes of the object.
Table 4: Comparison with Adapter- based method.
Dataset
Methods
shot
shots
shots
8 shots
16 shots
Tip-Adapter-F
61.32
61.69
62.52
64.00
65.51
ImageNet
Tip-Adapter-F OT Tip-Adapter-F PLOT 62.27
61.44
61.98
62.86
64.13
65.76
64.31
63.89
65.04
66.17
310
then learned multiple linear models. As a result, with different local features and different linear models, obtain a M x N distance matrix and apply the Sinkhorn algorithm [6] to calculate the OT distance. Furthermore, we can apply the learned prompts as co-partner of the ensembling prompt refine the final similarity.
312
313
314
Table summarizes the few-shot recognition results the original Tip-Adapter-F method and our adapter- baapee PLOT methods on ImageNet. From this table, We observe that using the OT distance can improve the performance the adapter-based method. Using the learned prompts, we can further promote the accuracy of all settings.
316
318
Q: What is the extra computation time cost of PLOT over CoOp baseline? A: Around 10% inference speed and 5% training time. Despite the performance improvement, the extra computation cost still a limitation of PLOT. Please see the detailed analysis in the supplementary materials.
320
4.5
Visualization
322
In this subsection, we provide some visualization examples of the transport plans T related to different prompts (N=4). We translate each transport plan into colorful heatmaps and resize them into their
324
original size and combine them with the raw image. As shown in Figure 4 we provide the heatmaps of 4 categories in ImageNet. We observe that different transport plans highlight different regions of
326 the image, which demonstrates that the learned multiple prompts are complementary. For the class Brambling", the prompts respectively focus the head, tail, wing, and environment. For "Dog Sled", the prompts are related to dogs, the sled, some ties, and the snow environment.
328
Conclusion
329
330
In this paper, present a method, named PLOT, learn multiple comprehensive prompts to describe diverse characteristics of one category. To avoid convergence to one point, we propose to apply the optimal transport achieve the fine-grained alignment between both vision and language domains. We apply a two-stage optimization strategy where the inner loop fixes the prompts and
332
333
334
learss the transport plan to calculate the cross-modality distance, and the outer loop uses this distance
to optimize the prompt learner. We build our method on the base of CoOp and achieve significant improvement on the few-shot recognition task in various datasets, which demonstrates the advantage
396
to learn multiple prompts instead of a single one.

---

References
338
339
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML. pages 214-223, 2017.
340
341
[2] Emmanuel Boissard, Thibaut Le Gouic, and Jean- Michel Loubes. Distribution's 's template estimate with wasserstein metries. Bernoulli, 21(2):740- 759, 2015.
343
[3] Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. fo0d--00-mining discriminative components with random forests. In ECCV, pages 446-461, 2014.
345
[4] Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. Improving sequence-to-sequeecee learning via optimal transport. arXiv preprint arXiv: 09029000,,,,,,.. 2019.
346
348
[5] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pages 3606 (000--663, 2014.
350
[6] Marco Cuturi. Sinkhorn distances: lightspeed computation of optimal transport. In NeurlPS, volume 2, page 4, 2013.
352
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: A large-scale hicrarchical image database. In CVPR, pages 248-255, 2009.
353
354
[8] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang. Chenguang Zhu, Zicheng Liu, Michael Zeng, et al. An empirical study of training end-to-end vision-and and--nd--aad -an-lagg transformers. arXiv preprint arXiv:2111 023387 2021.
355
357 [9] Li Fei-Fei. Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPRW, pages 178-178, 2004.
359
[10] Peng Gao, Shijic Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 2021.
380
382
[1I]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778 2016.
384
[12] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
365
366
387
[13] Dan Hendrycks. Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli. Mike Guo, al. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint (XXi:2000.16244 2020.
389
[14] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.017
371
2019.
372
[15] Yicong Hong. Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. VIn bert: A recurrent vision-and-language bert for navigation. In CVPR, pages 1643-1653, 2021.
373
374
[16] Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. Mural: multimodal, multitask retrieval across languages. arXiv preprint 205.5.209 2021.
378
377
[17] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoe Le, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation lcarning with noisy text supervision. In /CML, pages 4904-4916, 2021.
Sung,
378
379
[18] Zhengbao Jiang. Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? TACL, 8:423-438, 2020.
381
[19] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end end-to-end multi-modal understanding, In ICCV, pages 1780-1794 2021.
383
[20] Wonjac Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languun transformer without convolution or region supervision. In ICML, pages 5583-5594, 2021.
385
[21] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained calegorization. In ICCVW, pages 554-561, 2013.
387
388
[22] Charlotte Laclau, levgen Redko, Basarab Matei, Younes Bennani, and Vincent Brault. Co-clustering through optimal transport. In ICML, pages 1955-19666 2017.
390
[23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- ve-training for unified rision-languan understanding and generation. arXiv preprint arXiv:2201.12086, 2022.
10

---

382
[24]
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong
Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurlPS,
394
34, 2021.
385
[25]
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint 1908:1908.03333 2019.
397
[66]
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint (ix::2011.00// 2021.
398
399
127
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint
400
401
2021.
402
[88] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103. 10385, 2021.
403
404
[29] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aireraft. arXiv preprint 1306 2013.
406
[30 Gaspard Monge, Mémoire sur la théorie des déblais et des remblais. Histoire de l'Académie Royale des Sciences de Paris, 1781.
407
408
[31] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1072 2021.
409
410
[32] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of
412
classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729, 2008.
413
[33] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR. pages 3498-3505, 2012.
415
416
[34]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
418
419
[35] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV, pages 2085-2094, 2021.
420
421
[36] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv: 01066, 2019.
422
428
[37] Gabriel Peyre and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine
424
Learning, 11(5-6):355 607, 2019.
425
[38] Nina Poemer, Ulli Waltinger, and Hinrich Schütze. Bert is not knowledge base (yet): Factual knowledge vs. name-based reasoning in unsupervised qa. arXiv preprint arXiv: /tt/://////00///// 2019.
427
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint axXiv:2103.00020 2021.
428
429
430
[40] Alec Radford, Jeffrey Wu, Rewon Child, David Luaa, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)9, 2019.
431
432
[41]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0611 06125, 2022.
433
434
[42] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: sanguage-guided dense prediction with context-aware prompting, arXiv preprint arXiv:211.005518 2021.
436
437
[43] Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenct classifiers generalize to imagenet? arXiv preprint arXiv: 108/1, 2019.
439
[44] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as a metric for image retrieval. IJCV, 40(2)99-111, 2000.
440
[45] Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport.
442
ICLR, 2018
443
[46] Taylor Shin, Yasaman Razeghi, Robert L Logan IV. Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint 20100.000000088 2020.
445
448
[47]
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 122 2012.
11

---

448
[48]
Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation clip space. arXiv preprint arXiv:2203.0000 2022.
450
[49]
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal sew-shot learning with frozen language models. NeurlPS, 34:200-212, 2021.
451
452
[50]
Ruibo Tu, Kun Zhang, Hedvig Kjellström, and Cheng Zhang. Optimal transport for causal discovery. ICLR, 2022.
453
[51]
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
495
[52]
Haohan Wang. Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. NeurIPS, 32, 2019.
456
457
[53] Mengmeng Wang. Jiazheng Xing, and Yong Liu. Actionelip: A new paradigm for video action recognition.
458
arXiv preprint arXiv:2109.0844
[54]
Wenhui Wang. Hangbo Bao, Li Dong, and Furu Wei. Vimo: Unified vision-language pre-training with ooaeiiett-mmdlllyyy-ooo
460
arXiv preprint arXiv:2111.02356 2021.
461
[55]
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey 200. In CVPR. pages 1485-3492 2010.
465
[56]
Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov-wasserstenn learning for graph matching and node embedding. In ICML, pages 6932-6941, 2019.
464
465
[77]
Jingjin Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. Vocabulary learning via optimal transport for neural machine translation. arXiv preprint WXXv:2012.15671, 2020.
466
467
[58]
Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adupter for better vision-language modeling. arXiv preprint arXiv:2111. /1.03.0, 2021.
468
469
[59| Renrui Zhang, Longtian Qiu, Wei Zhang, and Ziyao Zeng. Vt-clip: Enhancing vision-language models with visual-guided texts. arXiv preprint arXiv:2112.02339 2021.
472
[60]
He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. Neural topic model via optimal transport. ICLR, 2021.
473
474
[61]
Wenliang Zhao, Yongming Rao, Ziyi Wang, Jiwen Lu, and Jie Zhou. Towards interpretable deep metric learning with structural matching. In ICCV, pages 9887-9896, 2021.
475
[62]
Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv preprint arXiv:2112.01071 2021.
477
478
[63]
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint PrXii:2109.0113: 2021.
480
[64] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vasion-language models. In CVPR, 2022.
481
Checklist
482
483
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
485
486
(b) Did you describe the limitations of your work? [Yes] See our analysis in Section 4.4 1) Our method is still need few-shot data for optimization, which cannot be applied in zero-shot setting.
487
488
2) The method needs more computing cost than CoOp.
489
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We propose a general framework for using the vision-language pre-trained model. It is not for specific applications, which does not directly involve societal issues.
490
491
492
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
483
2. If you are including theoretical results...
494
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
495
(b) Did you include complete proofs of all theoretical results? [N/A]
496
3. If you ran experiments...
487
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
499
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
12

---

(c) Did you report enror bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]] See Section 4.2
505
4. II you are using existing assets (e.g., code, data, models) or curating/reeasingg new assets...
(a) your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
509
(d) Did you discuss whether and how consent was obtained from people whose data you' re us- ing/eurating? [N/A]
(e) Did discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A]
513
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(h) Did you descrihe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
519
13


Reviews:
[{"review_id": "pk-p9nwMETe", "reviewer": "Reviewer_fGe6", "summary": "This paper focused on adapting large-scaled pre-trained vision-language model (i.e., CLIP) to downstream datasets under the few-shot setting. Compared to the recent related work CoOp, this work (1) learns multiple prompts, (2) uses both local features and global features, and (3) measures the similarity of prompts and visual features using optimal transport. Experiments on 11 benchmark visual recognition datasets show that the proposed method outperforms the baseline CoOp. Ablation studies further more analysis on the contribution of each component.\n\n----------------------\n\nUpdated after rebuttal:\n\nReasons to accept:\n\n+ The motivation for this work is clear. Single prompts may lose the details of local attributes or contexts.\n\n+ The idea of applying optimal transport to prompt learning makes sense and meets well with the motivation.\n\n+ The experiments and analysis are comprehensive. The improvement is significant.\n\nI am satisfied with the authors' responses to my comments and concerns, and I increased my rating from 6 to 7. I have read other reviewers' concerns on (1) two-stage training, (2) sharing of prompts, and (3) failure analysis and visualization. I think these questions are not severe for rejection and the authors have responded to these concerns.", "questions": "Please see weaknesses in the above.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, considering both strengths (simple but effective pipeline, comprehensive experimental analysis) and weakness (insight and contribution), my initial rating is borderline. \n\nStrengths:\n\n\\+ The problem of adapting CLIP under few-shot setting is recent. Compared to the baseline method CoOp, the improvement of the proposed method is significant.\n\n\\+ The ablation studies and analysis in Section 4.4 is well organized and clearly written. It is easy to follow the analysis and figure our the contribution of each component. Also, Figure 2 is well designed and clear to illustrate the pipeline.\n\n\\+ The experimental analysis is comprehensive. The analysis on computation time and inference speed is also provided.\n \nWeakness:\n\n\\- (major concern) The contribution is somehow limited. The main contribution is applying optimal transport for few-shot adaptation of CLIP. After reading the paper, it is not clear enough to me why Optimal Transport is better than other distance. Especially, the insight behind the application of Optimal Transport is not clear. I would like to see more analysis and explanation on why Optimal Transport works well. Otherwise, it seems that this work is just an application work on a specific model and a specific task, which limits the contribution.\n\n\\- The recent related work CoCoOp [1] is not compared in the experiments. Although it is a CVPR'22 work that is officially published after the NeurIPS deadline, as the extended version of CoOp, it is necessary to compare with CoCoOp in the experiments.\n\n\\- In the approach method, there lacks a separate part or subsection to introduce the inference strategy, i.e., how to use the multiple prompts in the test stage.\n\n\\- Table 2 mixed different ablation studies (number of prompts, visual feature map, constraint). It would be great if the table can be split into several tables according to the analyzed component.\n\n\\- The visualization in Figure 4 is not clear. It is not easy to see the attention as it is transparent.\n\nReferences\n\n[1] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.\n\n\n------------\n\nAfter reading the authors' response and the revised version, my concerns (especially the contribution of introducing the optimal transport distance for fine-tuning vision-language models) are well addressed and I am happy to increase my rating.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GbKOG8TMbYp", "reviewer": "Reviewer_bMB5", "summary": "The authors propose a new prompt learning method, which learns multiple prompts to describe diverse characteristics of categories. To optimize the model effectively, they apply optimal transport to match vision and language modalities. The experiments demonstrate its effectiveness and achieve improvement on few-shot learning tasks.", "questions": "Questions\nAfter optimization, for different images and classes, are they sharing the same prompts? If so, there might be issues. Because different images have different local patterns, like 'a dog lying on the ground'  v.s. 'a bird in the forest'. They have totally different local patterns (positions).\n\nTypos: \n\t1. Line 55, prompts learning -> prompt learning\n\t2. Line 75, simultaneously several items -> several items simultaneously\n\t3. Line 90, filed -> field\n\t4. Line 110, 111: improve -> improves, achieve -> achieves\n\t5. Line 135, vectorss -> vectors\n\t6. Line 158, is same as -> is the same as\n\t7. Line 232, summarize -> summarizes\nLine 274, we obtain -> We obtain", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\t1. The motivation is innovative. Conventional methods use a single prompt which is not able to capture multi-grained features. The authors therefore aim to learn multiple comprehensive prompts to describe different categories. \n\t2. An interesting combination between optimal transport theory and prompt learning.\n\t3. The experiments are comprehensive and demonstrate the effectiveness of introduced method.\n\n\nWeaknesses:\n\t1. The proposed method is a two-stage optimization strategy, which is a bit difficult to balance the two steps optimization. Could it be end-to-end training? \n\t2. Although it is intuitive that including multiple local prompts helps, for different categories, the features and their positions are not the same.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-LeYm67kwv", "reviewer": "Reviewer_c3r2", "summary": "This paper focuses on the few-shot prompt learning for vision-language model CLIP.\nOn the top of the recent work CoOp who only learns one single prompt, this work proposes a novel method called PLOT, which learn multiple prompts to leverages more fine-grained information from image's feature map. The key motivation of PLOT is to generate a set of prompts so that each prompt can help the model to focus on some specific areas on the feature map. To realize the idea, optimal transport method is used to contrast the effects of each prompt.\nThe experiment shows that PLOT can achieve better performance over CoOp in most of the datasets.", "questions": "None", "limitations": "The authors do not include the discussion on the limitation of the method. We hope the authors can provide more analysis on its weakness in discusssion.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n+ The motivation of PLOT is strong. Figure 1 clearly states the main idea.\n+ Using optimal transport to learn distinguishable prompt is reasonable and simple.\n+ The experiment shows the effectiveness of the proposed methods.\n+ Implementation details are clearly stated.\n+ Analysis is comprehensive and strong.\n\n\nCon:\n- Lack of explanation of failure cases, such as StanfordCar in Fig.3. A good illustration and discussion on failure case can help the community know better about your method and the problem.\n- Lack of visualization of the learnt prompt. Although it is understandable that it is difficult to print the learnt prompt out into a completed sentence, some words in the prompt can still reveal what the prompt is about. For example, in CoOp, the learnt prompt in OxfordPets has some words like fluffy, paw, etc. Hope PLOT can show more clues about its intrinsic mechanism.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject