Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focused on adapting large-scaled pre-trained vision-language model (i.e., CLIP) to downstream datasets under the few-shot setting. Compared to the recent related work CoOp, this work (1) learns multiple prompts, (2) uses both local features and global features, and (3) measures the similarity of prompts and visual features using optimal transport. Experiments on 11 benchmark visual recognition datasets show that the proposed method outperforms the baseline CoOp. Ablation studies further more analysis on the contribution of each component.
strength_and_weaknesses: Strengths:

\+ The problem of adapting CLIP under few-shot setting is recent. Compared to the baseline method CoOp, the improvement of the proposed method is significant.

\+ The motivation that (1) single prompt is insufficient to represent a class, (2) directly learning multiple prompts reduce to one single point, are straightforward and make sense. The proposed optimal transport based method is consistent with the motivation.

\+ The explanation of the comparison between optimal transport and other distances is clear, which supports the motivation.

\+ The ablation studies and analysis in Section 4.4 is well organized and clearly written. It is easy to follow the analysis and figure our the contribution of each component. Also, Figure 2 is well designed and clear to illustrate the pipeline.

\+ The experimental analysis is comprehensive. The analysis on computation time and inference speed is also provided.

Weakness:

(As I reviewed this paper before and all my previously concerns are addressed, I do not have much comments about the weakness.)

This work is built on top of CoOp. It would be interesting to know whether this proposed method can be extend to other prompt-learning-based methods, especially CoCoOp. If not, it would be great to discuss the reasons. Somehow, the method is only applied to CoOp, which is a limitation of this work.
clarity,_quality,_novelty_and_reproducibility: The quality, clarity and originality of the work look good.
summary_of_the_review: Overall, considering both strengths (simple but effective pipeline, comprehensive experimental analysis) and weakness (generalizability), my initial rating is accept.

Reasons to accept:

\+ The motivation for this work is clear. Single prompts may lose the details of local attributes or contexts.

\+ The idea of applying optimal transport to prompt learning makes sense and meets well with the motivation.

\+ The experiments and analysis are comprehensive. The improvement is significant.

Reasons to reject:

\- This work is only applied to CoOp. It is not clear whether the method can be extended to other prompt-learning-based methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes PLOT, a prompt learning method for CLIP based on CoOp. PLOT aims to diversify the textual prompts to better depict the category for both its intrinsic and extrinsic contexts. By applying optimal transport, PLOT shows favorable performance over different few-shot benchmarks.
strength_and_weaknesses: Strength:
1) The motivation of enriching prompts for different image regions is practical and well illustrated in the paper. The usage of optimal transport is also novel, which might be inspiring to future works.
2) The ablation study with discussion clearly tackles most my confusion with interesting points.

Weakness:
1) The major concern is the classification performance. As shown in Figure 3, despite the better overall performance, PLOT performs worse than previous prompting methods or linear probe on some of the datasets. Considering the much more complicated training, PLOT is expected to achieve higher accuracy.
2) The authors should provide complete comparisons between PLOT (adapter-based PLOT) and existing adapter-based methods, e.g. Tip-Adapter-F on all 11 datasets, not just ImageNet. As the adapter-based methods are SOTA for CLIP few-shot learning, this can better emphasize the contribution of PLOT.
clarity,_quality,_novelty_and_reproducibility: The paper is of good quality concerning writing and figures. The idea is novel, but the experiments can be further improved as above.
summary_of_the_review: The reviewer expects the authors' response to the experiment weakness.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presented a new prompt learning approach for vision-language models, namely PLOT. The main idea of this paper is to learn multiple prompts per class and use optimal transport to match them with visual features of images. The whole optimization process contains inner and outer loops to optimize the matching and prompts, respectively. The experiment results show that the proposed approach can achieve state-of-the-art performance in the few-shot image recognition task with a slight increasement in training time.
strength_and_weaknesses: __Strength__

1. This paper is overall well-structured and easy to follow.

2. The idea of learning multiple comprehensive prompts to describe the image categories is reasonable to me, and using optimal transport for optimization prompt learning is novel.

3. The ablation studies are comprehensive which clearly demonstrate the design choices of the proposed method.

__Weakness__

1. In Figure 3, the improvement achieved by the proposed method over CoCoOp is marginal. I wonder if the conditional formulation presented in CoCoOp can be easily extended to PLOT? If so, at least more detailed discussions should be provided by the authors.

2. Figure 4 shows the visualization results of attended regions of images from different prompts. It is also interesting to see how learned inter-class and intra-class prompts are different from each other by using t-sne or PCA.

3. It is unclear to see the differences among model variants in Table 2 just from the caption. It will be good if an individual section on introducing the detailed design/training setup of each model in the paper (or in the appendix). 

clarity,_quality,_novelty_and_reproducibility: Overall, I think the presentation of this paper is clear and easy to understand. The proposed method is novel and technically sound.
summary_of_the_review: This paper addressed prompt learning for vision-language models which is an interesting and important problem in few-shot image classification tasks. The proposed PLOT is reasonable and technically sound to me. However, I still have some concerns on the experiment part of the paper. Therefore, it leads me to borderline acceptance as my initial rating. I will be happy to increase my rating if the concerns could be addressed in the discussion period.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for prompt learning in vision and language models that aims at producing a variety of templates that can be used by the text encoder to provide better classification weights in a few-shot scenario. 

In particular, the paper observes that learning a single prompt is not sufficient to represent a class, and proposes a variety of prompt candidates that can be used to cover a wider variety of concepts within the same image. The authors note that using a distribution of prompts that are learned independently using a standard contrastive learning will result in some of the prompts to collapse to the same centroid, and propose an optimal transport scheme to avoid such behaviour. 

The methodology goes as follows: 1) a set of different visual features are computed for a given image (using different backbone exits); then 2) the scores for the pairs visual/text features are computed using the learnable prompts; then 3) a transport matrix is computed to minimize the total cost of assigning a feature vector a weighted combination of the text features (i.e. the transport matrix is the soft-extension of the permutation matrix that would assign vision/text features through the typical CLIP matching). Finally, for a fixed transport matrix 4) the prompts are learned using the standard cross-entropy where the class assignment is given by the computed combination of the weights produced by the prompts. 

The use of a transport matrix allows to assign to each feature vector a weighted combination of the prompts, thus making the latter have a broader "semantic" capacity. The use of different feature vectors for each image rather than a global CLIP-based visual feature vector allows the prompt to describe different concepts that might co-occur in the same image, making the prompts having more ``local" capacity. 

The paper is validated in the same setting as CoOp for few-shot learning, using 11 visual recognition datasets, delivering consistently better results than the competing method. 

strength_and_weaknesses: Strengths: 

The paper is technically solid and contributes enough novelty to the domain of prompt learning which is of growing interest to the research community devoted to exploring pre-trained vision/language models. The use of a diverse set of prompts to generate different text classifiers that can all contribute to a single class prediction is of interest and the methodology is well developed. 

The results seem to consistently outperform CoOp, showing the need of having a large set of prompts that require proper use to be able to generalize better to new classes.

Weaknesses:

While the core methodology is new, the idea of learning a distribution of prompts is not new, and indeed the authors do not include a discussion on why their proposed approach would be a better choice than CoCoOp which produces prompts for each image. Additionally, the authors need to consider, discuss, and compare, against 

[Lu et al.] Prompt Distribution Learning. CVPR 2022

which is a method that learns a collection of prompts that produce a distribution of per-class weights, increasing their expressivity. Some analysis in terms of methodology and computation against [Lu et al.] should be included in the paper in my opinion.
clarity,_quality,_novelty_and_reproducibility: The method is novel and the code is provided, hence reproducibility and novelty are properly covered in the manuscript.

The paper could benefit from better writing and clarity in the presentation; in my humble opinion, writing the paper top to bottom (i.e. presenting first optimal transport and then the problem) is more confusing than writing it bottom to top (i.e. the problem of prompt learning and the intuition behind the optimal transport in ''assigning" prompts to different visual features from the same image, there is barely discussion around it). 
summary_of_the_review: In summary, the paper proposes an interesting approach to prompt learning that is technically sound and well motivated. However, the clarity and the lack of an exhaustive comparison against state of the art works ([Lu et al.] and CoCoOp) both experimentally and conceptually is needed. Should the authors be able to bring this up would further increase the quality of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a novel method for prompt learning in vision-language models. The idea is to create a diverse set of prompts (instead of a single prompt as used in previous methods) to better model local attributes or context, while applying optimal transport to match the vision and text modalities. All reviewers recommend acceptance and the AC agrees with this decision. The motivation of this work is clear and the experimental analysis is comprehensive. Initially, the reviewers raised concerns about the generality of the approach and requested comparisons with additional methods such as CoCoOp. The author response was well received and included additional experiments showing the effectiveness of the proposed method.
justification_for_why_not_higher_score: While the paper is easy to follow, the write up could still be improved as pointed out by one of the reviewers. In addition, it would be important to see the experiments with PDL in all datasets.
justification_for_why_not_lower_score: All reviewers agree the proposed method is novel, clearly motivated, and outperforms strong baselines.
Decision: Accept: notable-top-25%