Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work addresses VLM few-shot learning with prompt tuning. It addressed the overfitting of the learned prompts by regularizing the direction of gradients. The direction is enforced not in the opposite direction of the gradient computed with the human-selected prompt. The paper provides a theoretical justification to show its regularized gradients lead to a smaller error bound. The experiments in few-shot image classification, domain generalization, and unseen class classification show that the proposed method achieves better accuracy in all benchmarks.
strength_and_weaknesses: - Strength
1. The proposed method looks novel and makes much sense. Using a generalizable prompt to regularize the learning of a domain-specific prompt is a simple and effective strategy to avoid overfitting in few-shot learning.
2. The experiment is comprehensive. It includes three aspects: standard few-shot image classification, domain generalization, and unseen class classification. The comparison to KD (Table 4) and the cosine classifier (Table 5) directly support its claims that its regularized gradient direction is the key to improving the results.
- Weaknesses
1. The improvement of ProGrad looks marginally better and may be affected by the choice of hyperparameters. What is the upper bound result this approach can get? How about simulating the upper-bound case using many training data to get an optimized prompt instead of using the man-made prompt to create G_g?
clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is easy to follow.
- Quality: Its claim is well supported
- Novelty: The proposed ProGrad is novel
- Reproducibility: The method is clearly described, and its hyperparameters are provided.
summary_of_the_review: The method is well-motivated and described. Its experiments cover a wide range of settings and show improvements. Overall it looks like a well-executed paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work highlights an intriguing problem regarding the tuning of soft prompts in vision and language models. This research asserts that current soft-prompt tuning papers have a tendency to overfit on training data while losing generalization ability on test data. To avoid this, they strive to optimize soft prompts in directions so that they do not clash with general information encoded in vision and language models. This eliminates contradictory directions and helps prevent forgetting. The method proposed in this paper is not novel in general and has been explored in multi-task learning and continual learning. However, this is the first time it is used for soft prompt tuning and I believe it is a appropriate contribution and is aligned with ICLR. In terms of experiments and comparisons, I think it is thorough and authors evaluate their model from different view points. 
strength_and_weaknesses: **Strength:**
- Paper is clearly written and organized.
- Theoretical justification and derivation are correct as far as I know unless other reviewers raise important concerns. For me, this paper address the same question as [2,3] did in order to improve generalization. I would suggest authors to elaborate these similar papers in their related work.
- The idea is novel in the context of soft-prompt tuning, but it has been already explored in multi-task learning and continual learning [1]. It would be even nice if authors discuss a bit more about this prior work or similar ones in the related work. 
- Ablations and experiments are complete in the main body of the paper as well as the appendix part.

**Weakness:**

According to my experience working with **CoOp** and **CoCoOp**, the two baselines of this paper, in CoCoOp work, all results are reported on **4 learnable tokens** initialized with "A photo of a {class}",  **16 shots** and **ViT/B-16 backbone** on three different random seeds for three different tasks: **base-to-new-generalization**, **cross-dataset transfer learning**, and **domain generalization**. Regarding this information, I would appreciate if the authors answers following questions:

1. Do authors train all baseline models themselves or they adopt the results from the main paper?
2. Are learnable prompts initialized with "A photo of a {class}" or they are trained from scratch?
3. I would ask authors why the number of shots are different in tasks  **base-to-new-generalization** and **domain generalization** in Table 1 and 2. Are there any specific reasons? Can they provide the performance with same hyper-parameters? I would suggest authors to be consistent with the baselines in terms of hyper-parameters to conclude a fair comparison. To me, Table 1 and Table 2 looks a bit wired as you change the training hyper-parameters. Maybe for 16 shots, CoCoOp performs better. 
4. I noticed that authors did not provide results for cross-dataset transfer learning (Table 2 of CoCoOp paper). I would suggest to report these numbers as well since it makes the paper more complete. 
5. Regarding ProGrad++, where authors use the idea of prompt ensemble, would it be possible to clarify how the prompt design changes in terms of number of learnable tokens. To me, since the input prompts changes, the length of learnable prompt would change accordingly and it is a bit unclear how many learnable parameters exist. 



[1]. Farajtabar et al., Orthogonal Gradient Descent for Continual Learning, 2019

[2]. Lu et al. Prompt Distribution Learning, CVPR 2022

[3]. Derakhshani et al., Variational Prompt Tuning Improves Generalization of Vision-Language Models, Arxiv 2022
clarity,_quality,_novelty_and_reproducibility: The work's quality, clarity, and originality are satisfactory to me. As stated, the formulation to prevent forgetting is not new; however, in the context of soft-prompt tuning, no one has previously considered it. Importantly, the results demonstrate that this formulation actually aids according to various evaluation protocols.
summary_of_the_review: This paper addresses the issue of reducing the over-fitting problem for downstream adaptation of vision and language models as a whole. I vote for "marginally above the acceptance threshold," unless other reviewers indicate critical issues and I will decrease or increase my score in case it is needed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: ### Problem
The paper tackles the problem of prompt tuning for large scale vision-language models. This is a very recent direction.

### Proposed method
The method is built on top of (i) zero-shot CLIP with hand-crafted prompts and (ii) CoOP which proposed to fine-tune context vectors as tunable prompts. The method proposes a modification on CoOP where the gradients are slightly modified.

### Experimental validation
Authors perform experiments on commonly used datasets.
strength_and_weaknesses: ###STRENGTHS
1. Wide range of experiments

### WEAKNESSES
1. Very incremental contribution
- The delta over CoOP is very small. Infact the method is a very simple extension of CoOP. I don't know why the authors made a whole new name.
2. Results not convincing
- The reason is that the improvement over CoOP seems very negligible in more important datasets like ImageNet and Table2. Can you aadd a table which shows results on individual datasets?
- Some tables have ProGrad and others have ProGrad++. If ProGrad++ is also your method, add it into the method section. 
- Standard deviation is missing. So it is very difficult to draw any conclusion. 0.27 improvement on ViT-B in Table 2 can easily go away with multiple runs.
3. Not properly motivated
- The method does not have a sound technical motivation. It seems like a heuristic. 
- Phenomenon shown in Fig.2 is not enough. It could be a cherry-picked exception.
- Starting of sec3.2 should motivate the challenge with previous methods. However, it says that it has been discussed in Introduction. Can you discuss it properly?
4. Poor Writing
- Although the method section is decently written, the introduction is hard to follow.
clarity,_quality,_novelty_and_reproducibility: Clarity of Introduction is very low, but for method section it is decent. 

Novelty is very low. The method is very incremental. 
summary_of_the_review: I am voting for rejection because:
1. The delta over CoOp is very incremental.
2. The method seems like a heuristic without any proper technical motivation. Even more heuristics have been added into experiment section which are missing from method section. It is not enough for a conference paper.
3. The improvement over CoOP is small and standard deviation has not been reported. This makes it difficult to draw a conclusion.


======================= 
POST REBUTTAL
=======================
After carefully going through the authors comments and the other reviews, I have decided to keep my rating the same. My reasons remain the same, which have not been addressed in the authors response:
1. The method is an incremental extension of CoOP. The fact that the authors have done serious re-branding is not so convincing. If it is a simple extension, saying so is something I would have preferred.
2. Standard deviation is missing from tables. It is good Machine Learning practise to have this. Despite me asking for this, the authors instead of suggesting that 'we will add this to the final version....', the authors chose to disagree with it. 
3. I felt that the introduction is not well written. Authors 'disagree with it, saying that other reviewers did not feel this way, so my claim is wrong'. There are multiple reviewers to get different opinions. 
4. I still feel that the motivation of the method is heuristics.

While these factors might be small alone, but all of them combined together make it a weak paper in the current format. These are changes that the authors can do to make the paper stronger. But the authors did not even accept these weaknesses, let alone proposing to work on them in the final version. Therefore, it is a reject from my side.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work addresses the challenge of prompt engineering under few-shot setting. The key idea is to treat conventional zero-shot CLIP transfer inference as general knowledge, and enforce that the learned prompt will not diverge much from such general knowledge. In implementation, the gradient at each optimization step will be determined according to the divergence of two gradient direction. Comprehensive experiments are conducted to validate the effectiveness of the proposed idea.
strength_and_weaknesses: Strength:

- The work addressed some key challenging in prompt engineering for vision-language models.
- The idea is computationally simple and empirically validated to bring superior performance.

Weakness:

- The proof of generalization bound is not much informative. It is a direct extension of the theoretical results in prior works (e.g. Zhang and Ye's 2012 paper).
- More insights are required for the readers fully understanding the empirical superiority of the proposed idea. In specific,  the "general knowledge" induced by standard zero-shot CLIP transfer does not provide ground truth. The gradient-calculating rule as proposed in Eq. 4 is essentially a fusion of two kinds of cues: one is from zero-shot extension, and the other is data-driven. The trick is to cleverly choose the way for the fusion. In the practice of multi-source fusion, the final results tend to be improved given that all sources are informative and complementary to one another. Section 4.5 provides some analysis on the failure case, which is unfortunately somewhat superficial. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is clearly written. All notations, concepts, ideas, experiments are properly defined or described.
Quality: This is an interesting work on an important research topic. Though there are still a few weaknesses, the overall quality is arguably above the average of all submissions that I have reviewed this year.
Novelty: The idea is novel, investigating of CLIP's few-shot generalization from an unexplored aspect.
Reproducibility: The computation trick is fully in Eq. 4 and one can easily re-implement the key experiments.
summary_of_the_review: Overall I would regard this is an interesting albert simple work toward more robust prompt engineering, under the usual setting of few-shot samples. It will be a good addition to the conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper propses a method for prompt tuning that is less sensitive to overfitting compared to the CoOp approach.  The main idea is simple and effective, and is based on the observation that CoOp starts overftting when the training is not stoped early,  outpefroming Zero-shot CLIP. The authors propose to requalarize the CoOp objective via preserving the Zero-shot CLIP prediction, similar to learning without forgetting[1]. Moreover, they propse to modify the CoOp gradient in the direction of the regulariztion term gradient during the training. The proposed methods outperofrmrs not only CoOp approach but also more adnvced approaches like CoCoOp on the standrd zero-shot learning benchmarks.

[1] Zhizhong Li and Derek Hoiem, Learning without Forgetting, 2016
strength_and_weaknesses: Strengths:

- The paper is based on a simple and effective idea and is executed well.
- The paper is accurately and clearly written.
- The performance is satisfactory.

Weaknesses :
- I see a lot of similarities between the proposed method with [1]. If I am not mistaken, the auxiliary loss function prevents forgetting the general knowledge is the same as [1]. However, this paper's gradient projection approach and theoretical analyses are new. While the authors limit their application to zero-shot learning, I believe the contributions of this paper hold in a general setting where some previous knowledge has to be preserved for training on a new task.

A discussion on the differences with [1] and direct comparisons with their method would be interesting and strengthen the paper, in my opinion. A quick experiment that can be done during the rebuttal is to show the performance of the proposed method without the gradient projection, i.e. adding the KL loss to CoOp and optimizing with the standard SGD. A similar experimental setup as [1] and director comparisons with them would be even more interesting.

[1] Zhizhong Li and Derek Hoiem, Learning without Forgetting, 2016
clarity,_quality,_novelty_and_reproducibility: This paper is clear and of acceptable quality. The novelty might not be its main strength, but it is not a deal breaker either. 
summary_of_the_review: Overall, the idea of this paper is simple, effective, and executed well. More comparisons with [1] would be interesting to include. I'd be happy to discuss this matter during the rebuttal.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new prompt learning method, Prompt-aligned Gradient, for more generalizable prompt tuning under few-shot learning scenarios. In particular, it addresses the overfitting issue of the learned prompts by regularizing the direction of gradients. The method is evaluated on multiple image classification datasets across three different tasks: base-to-new-generalization, cross-dataset transfer learning, and domain generalization. Although the technical idea has been explored in continual learning in the vision and language research area, it is new in the context of prompt learning. However, the result improvements aren't very clear compared to CLIP, CoOp, and CoCoOp, as not all the results are included and compared in the paper (in the author's response, e.g., A 2.5, the improvement over CoCoOp seems pretty marginal). Reviewers have concerns regarding incremental contributions over CoOp, unconvincing results, etc.
justification_for_why_not_higher_score: Multiple reviewers raise concerns about the experimental results. For example, the result improvements aren't very clear compared to CLIP, CoOp, and CoCoOp, as not all the results are included and compared in the paper (in the author's response, e.g., A 2.5, the improvement over CoCoOp seems pretty marginal).
Decision: Reject