Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focused on adapting large-scaled pre-trained vision-language model (i.e., CLIP) to downstream datasets under the few-shot setting. Compared to the recent related work CoOp, this work (1) learns multiple prompts, (2) uses both local features and global features, and (3) measures the similarity of prompts and visual features using optimal transport. Experiments on 11 benchmark visual recognition datasets show that the proposed method outperforms the baseline CoOp. Ablation studies further more analysis on the contribution of each component.

----------------------

Updated after rebuttal:

Reasons to accept:

+ The motivation for this work is clear. Single prompts may lose the details of local attributes or contexts.

+ The idea of applying optimal transport to prompt learning makes sense and meets well with the motivation.

+ The experiments and analysis are comprehensive. The improvement is significant.

I am satisfied with the authors' responses to my comments and concerns, and I increased my rating from 6 to 7. I have read other reviewers' concerns on (1) two-stage training, (2) sharing of prompts, and (3) failure analysis and visualization. I think these questions are not severe for rejection and the authors have responded to these concerns.
strengths_and_weaknesses: Overall, considering both strengths (simple but effective pipeline, comprehensive experimental analysis) and weakness (insight and contribution), my initial rating is borderline. 

Strengths:

\+ The problem of adapting CLIP under few-shot setting is recent. Compared to the baseline method CoOp, the improvement of the proposed method is significant.

\+ The ablation studies and analysis in Section 4.4 is well organized and clearly written. It is easy to follow the analysis and figure our the contribution of each component. Also, Figure 2 is well designed and clear to illustrate the pipeline.

\+ The experimental analysis is comprehensive. The analysis on computation time and inference speed is also provided.
 
Weakness:

\- (major concern) The contribution is somehow limited. The main contribution is applying optimal transport for few-shot adaptation of CLIP. After reading the paper, it is not clear enough to me why Optimal Transport is better than other distance. Especially, the insight behind the application of Optimal Transport is not clear. I would like to see more analysis and explanation on why Optimal Transport works well. Otherwise, it seems that this work is just an application work on a specific model and a specific task, which limits the contribution.

\- The recent related work CoCoOp [1] is not compared in the experiments. Although it is a CVPR'22 work that is officially published after the NeurIPS deadline, as the extended version of CoOp, it is necessary to compare with CoCoOp in the experiments.

\- In the approach method, there lacks a separate part or subsection to introduce the inference strategy, i.e., how to use the multiple prompts in the test stage.

\- Table 2 mixed different ablation studies (number of prompts, visual feature map, constraint). It would be great if the table can be split into several tables according to the analyzed component.

\- The visualization in Figure 4 is not clear. It is not easy to see the attention as it is transparent.

References

[1] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.


------------

After reading the authors' response and the revised version, my concerns (especially the contribution of introducing the optimal transport distance for fine-tuning vision-language models) are well addressed and I am happy to increase my rating.
questions: Please see weaknesses in the above.
limitations: The authors have adequately addressed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new prompt learning method, which learns multiple prompts to describe diverse characteristics of categories. To optimize the model effectively, they apply optimal transport to match vision and language modalities. The experiments demonstrate its effectiveness and achieve improvement on few-shot learning tasks.
strengths_and_weaknesses: Strengths:
	1. The motivation is innovative. Conventional methods use a single prompt which is not able to capture multi-grained features. The authors therefore aim to learn multiple comprehensive prompts to describe different categories. 
	2. An interesting combination between optimal transport theory and prompt learning.
	3. The experiments are comprehensive and demonstrate the effectiveness of introduced method.


Weaknesses:
	1. The proposed method is a two-stage optimization strategy, which is a bit difficult to balance the two steps optimization. Could it be end-to-end training? 
	2. Although it is intuitive that including multiple local prompts helps, for different categories, the features and their positions are not the same.

questions: Questions
After optimization, for different images and classes, are they sharing the same prompts? If so, there might be issues. Because different images have different local patterns, like 'a dog lying on the ground'  v.s. 'a bird in the forest'. They have totally different local patterns (positions).

Typos: 
	1. Line 55, prompts learning -> prompt learning
	2. Line 75, simultaneously several items -> several items simultaneously
	3. Line 90, filed -> field
	4. Line 110, 111: improve -> improves, achieve -> achieves
	5. Line 135, vectorss -> vectors
	6. Line 158, is same as -> is the same as
	7. Line 232, summarize -> summarizes
Line 274, we obtain -> We obtain
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the few-shot prompt learning for vision-language model CLIP.
On the top of the recent work CoOp who only learns one single prompt, this work proposes a novel method called PLOT, which learn multiple prompts to leverages more fine-grained information from image's feature map. The key motivation of PLOT is to generate a set of prompts so that each prompt can help the model to focus on some specific areas on the feature map. To realize the idea, optimal transport method is used to contrast the effects of each prompt.
The experiment shows that PLOT can achieve better performance over CoOp in most of the datasets.
strengths_and_weaknesses: Pro:
+ The motivation of PLOT is strong. Figure 1 clearly states the main idea.
+ Using optimal transport to learn distinguishable prompt is reasonable and simple.
+ The experiment shows the effectiveness of the proposed methods.
+ Implementation details are clearly stated.
+ Analysis is comprehensive and strong.


Con:
- Lack of explanation of failure cases, such as StanfordCar in Fig.3. A good illustration and discussion on failure case can help the community know better about your method and the problem.
- Lack of visualization of the learnt prompt. Although it is understandable that it is difficult to print the learnt prompt out into a completed sentence, some words in the prompt can still reveal what the prompt is about. For example, in CoOp, the learnt prompt in OxfordPets has some words like fluffy, paw, etc. Hope PLOT can show more clues about its intrinsic mechanism.
questions: None
limitations: The authors do not include the discussion on the limitation of the method. We hope the authors can provide more analysis on its weakness in discusssion.
review_title: null

Meta-review: metareview: This paper presents a novel perspective of prompt tuning for few-shot visual recognition: a dynamic matching algorithm between the prompt candidate and the visual features. Compared to the existing CoOp and CoCoOp algorithm, the proposed "Optimal Transportation" idea definitely sounds better and indeed achieves better performance. All the reviewers acknowledge the merits of the paper. 

Though AC also acknowledges the merits of the paper, there are two unaddressed demerits: 

1) As the proposed method is essentially an ensemble method, comparisons with prompt ensemble should be conducted. Unfortunately, the reported "G" in Table 2 and Line 254-265 is not a proper ensemble, because the "G" baseline may degenerate into many duplicate single CoOp models with different random seeds. AC conjectures that this is the reason why G's performance is only slightly different from CoOp. By "proper ensemble", the authors may want to try initializations by "this is a photo"+"a picture of" + "there is an xx of" etc, or, augmenting each image by random crops, each of which corresponds to a "G", and then ensemble. 

2) This paper lacks an important "Base to New" setting as proposed by CoCoOp. As the proposed PLOT in this paper has significantly more tunable parameters, AC doubts that it may lead to the overfitting of the training classes but ruins (or forgets) other classes which were used to have good zero-shot performance without training.

Unfortunately, AC regrets to recommend reject and wishes the best of luck in re-submitting the paper to other venues.
confidence: Certain
Decision: Reject