Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper point out two challenges in the downstream inference of pre-training vision-language models: expressive sensitivity and conceptual sensitivity.  To handle the problems, the paper proposes a new dual-model feature prompting methd, named as Decomposed Feature Prompting (DeFo). By providing an independent set of learnable embedding and tuning an additional layer for classification, authors claim that the model trained by DeFo significantly addresses the sensitivity challenges of CLIP-like models.

strength_and_weaknesses: Strength：
1. The two problems pointed out by the paper: do exist and deserve the attention of the community.
2. The authors take time to implement and evaluate several prominent baselines. Experimental evaluation shows competitive performance.
3. This paper is well written and easy to follow.

Weakness:
1. The paper does not discuss the computational complexity of the proposed methods. How to efficiently complete the fine-tuning of the pre-trained model is also a direction worthy of attention. I look forward to seeing the authors discuss a comprehensive comparison of DeFo's training time and other methods, such as CoOp and CLIP-adapter.
2. Some important ablation study may be missing.  Since the authors point out that using class labels to generate text embeddings may bring challenges with expressive sensitivity. So a very straightforward idea is that we can directly set an independently learnable parameter as the prototype of each category to calculate the cosine similarity with image embeddings. or adopt the exponential-moving-average (EMA) manner [3]. These above-mentioned methods do not use text encode. Therefore, it is not necessary to carry out the forward of the text encoder every iteration during training. If these method can also achieve very good results, then I feel that the novelty and effectiveness of DeFo may be challenged. Therefore, I think it is very necessary to supplement this experiment. Look forward to the author discussing in following version.
3. The comparison of some other important baseline is missing, such as Tip-adapter [1] and CoCoOp [2].
4. I look forward to the author's discussion of the additional learnable parameters introduced in addition to CLIP's pre-trained model, and compare the number with other methods. Because if too many parameters are introduced, the performance improvement may come from overfitting of too many parameters. If the authors would like to compare the number of additional parameters of DeFo with CoOp and CLIP-adapter, I think it may be very helpful for us to comprehensively evaluate and compare these methods.
5. We expect that the model can not only achieve good performance on a single dataset, but also have the potential to transfer beyond a single dataset. I suggest authors to add discussion about the perfomance of DeFo for domain generalization. Maybe the setting in Section 4.2 of [3] is a good formulation. This may strengthen the contribution of the paper.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow.
summary_of_the_review: The proposed approach is shown to be effective, but the lack of some experiments may lead to the limited contribution of the proposed method. If the author adds more meaningful experiments, I think it will make the paper more interesting, and I am very happy to revise my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a simple method of prompt learning for Vision-Langauge model (in particular the CLIP). It is an extension based on existing work CoOp, where it leverages learnable embeddings as textual input and performs classification on this basis. Empirically, DoFo shows great improvement, where it reaches 73.2% on ImageNet 1K benchmark. In overall, it is a simple yet effective way for the VL prompt learning.
strength_and_weaknesses: Strength:
1. Good and clear motivation, good paper writing with easy understanding scope&contribution.
2. Great empirical results. 


Weaknesses:
1. Limitation of technical novelty but it may not be a big concern from my perspective.
clarity,_quality,_novelty_and_reproducibility: Good clarity. I have no comments on reproducibility as source codes are not released.
summary_of_the_review: In overall this is a good extension on the previous CoOp, where the authors aim to improve the prompt learning for a pre-trained VL model, in particular the CLIP. The proposed method is simple: by leveraging set of learnable parameters as textual embedding, it assist the classification task especially on ImageNet chanllenge. In essense, DeFo is largely based on the exisitng technique of CoOp, yet it's presented empirical results largely exceed the previous. My biggest concern arises from the aspect of technical contribution and the larger impact of CoOp alike work: 

1. The reviewer understands that simple technique is possibly to lead to bigger impact, so it is indeed not an issue for a intuitive&direct solution as a good submission. Yet my hesitation comes from the consideration that the presented insight of DeFo remains less significant: it seems that the biggest modification is making the learnable embedding as input for textual encoder yet it seems the important baseline ``Target Optimization'' also reaches a good results in such a similar schema. But again, I understand and also agree that the complexity of technique does not reflect contribution so I prone to suggest acceptance for the submission. 

2. Impact of CoOp alike work: another of my concern comes from the real world impact of prompt learning and its variations in recent years. Indeed, prompt learning, especially for LLMs show promissing applications for language comprehending tasks, yet this seems to be less important on the Vision side, which is mainly because that the pre-trained Vision model is that large enough that requires prompt learning. E.g., tiny model like MobileNet can even perform better than CLIP or other Vision tasks much more efficiently and better. A more desperate need of prompt engineering is for the Vision-Language Tasks that require more comprehending and high-level understanding, e.g., image/video question answering, captioning and cross-modal retrieval. In such context, prompt engineering for CLIP is rather limited and does not reflect much extension yet as current efforts seem more focusing on ImageNet challenge for simple classification task. But admittedly, the reviewer still agree that DeFo is in overall a good submission and has it's contribution for the community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aims to address two issues regarding the Clip-like model: 1) degraded accuracy and robustness when inferring by retrieving textual class names (the zero-shot protocol); 2) breaking the well-established vision-language alignment (linear probing). To combine the best of both worlds, this paper proposes Decomposed Feature Prompting (DeFo), which maintains the dual-model architecture yet leverages learnable embeddings as textual input and performs classification with an additional linear layer. The empirical study shows DeFo’s performance in improving the vision-language models.
strength_and_weaknesses: Strengths:
1）Prompts Learning is a meaningful direction, and this paper provides a valuable discussion of this direction to some extent.
2) The authors propose a novel and efficient method to address two issues regarding the Clip-like model.
3) The DeFo achieves 73.2% test accuracy on ImageNet with a ResNet-50 backbone without tuning any pretrained weights of both the vision and language encoder, outperforming zero-shot CLIP by a large margin of 15.0%, and outperforming state-of-the-art vision-language prompt tuning by 7.6%.
4）This paper is easy to follow and the motivation is well explained.

Weaknesses:
There are some issues that need to be improved: 1) The introduction and related work section lack a detailed discussion of the recent method CoCoOp. 2) A comparison with the SOTA method CoCoOp is lacking in the experimental section. For example, in Tab1, Tab2, Tab3, Tab4 and Fig2.

clarity,_quality,_novelty_and_reproducibility: Overall, this paper is well written and the method is easy to follow. The motivation is well explained and the contributions is novel. 
summary_of_the_review: In general, this paper is well written. I suggest that this paper can be accepted after supplementing some contrastive results against SOTA method and more discussions in the experimental part.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposed a new vision-language prompting method called dual-model feature prompting to address two issues of the Clip-like models: 1) degraded accuracy and robustness when inferring by retrieving textual class names (the zero-shot protocol); 2) breaking the well-established vision-language alignment (linear probing), and empirically validates DeFo’s performance in improving the vision-language models. Reviewers generally found this work is novel, well-motivated, empirical results are solid and the paper is well-written. There are some concerns raised by the reviewers and authors provided a proper response. Overall, this is an interesting paper with solid contribution which is recommended to publish. 


justification_for_why_not_higher_score: interesting and solid work but not particularly novel
Decision: Accept: poster