Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed to leverage variational inference to learn the underlying distribution of the prompts with or without image feature as the condition. The effectiveness of the proposed method is verified on several benchmarks.
strength_and_weaknesses: Pros:
1. The proposed idea is intuitive and straightforward.
2. The improvement is shown by combing the proposed method with two popular methods for visual prompt tuning.

Cons:
1. It is unclear why it is needed to learn the distribution of prompts. Unlike where variational inference was usually used in computer vision, e.g., VAE, the possible distribution of the prompts is actually not that untractable. 
a. Simple baselines like ensemble learning the soft prompt from different ways of initialization should be provided. It is possible that when ensembling several prompts learned from different initializations, the performance of the model is already somewhat achieved. Because the possible way of introducing an image in caption is rather limited. Probably a few main ways of the prompt initialization can already cover.
b. Any interpretation of the learned/sampled prompts could be provided? This may help to understand why the learning process is necessary.
c. Although the authors claim on the improvement of generalization to new classes, the performance gain on cross-dataset setting is still quite limited. Therefore, the model may still learns pretty domain specific distribution of prompts.

2. In Table1, more than half pf the results sacrifice the base class accuracy for improvement of the average. Therefore, it is also important to provide simple baselines like early stopping or adding some common regularization terms on the prompts.

clarity,_quality,_novelty_and_reproducibility: Clarity, quality and reproducibility look overall good. The novelty is rather limited.
summary_of_the_review: Overall, the reviewer thinks this paper solves an incremental problem following a rather standard approach without providing in-depth analysis and insights on why solving this problem.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary: 
The authors propose a variational prompt generator conditioned on an input instance. This is learned by adding a residual (which is input conditioned) to a fixed set of learnt prompts. During inference, multiple residuals can be sampled to generate different prompts. The authors show that the proposed approach surpasses existing methods on 16/20 datasets for classification

strength_and_weaknesses: 
Strengths:
- The method is intuitive and show strong empirical results on a bunch of tasks. The ablations are reasonable too! 
The model maintains generalization capability for both new  and old classes. 

Weaknesses:
- The paper combines two existing ideas in prompt learning (the residual addition, and the variational formulation). This limits the technical novelty of the paper a little bit. 
- The whole motivation of the paper is about preserving the generalization capability for downstream tasks, yet we see that the model perfoms worse on base classes than existing methods.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, and the experiments are easy to follow. The paper has limited technical novelty because it combines two existing ideas of residual connection and variational formulation of prompt learning
summary_of_the_review: I think the paper has merits, but technical novelty is limited. I am on the fence about the paper because it has strong empirical results but the approach combines existing ideas in unsurprising ways. I will look at the author response and read other reviews to decide my final rating.

Update after rebuttal: I think the authors took effort in addressing all my concern and most of other revierwer's concerns during the rebuttal. I am still on the fence due to limited novelty and tradeoff in performance between base and new classes.  But I think the new analysis makes the paper stronger. To reflect this I am updating my score to 6 (marginally above acceptance threshold).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes an instance-specific variational prompt-tuning framework for image-language downstream tasks. Here, the experiments are performed with a frozen CLIP model while fine-tuning the parameters used for prompt creation. It combines two ideas from existing publications on prompt-tuning with learnable prompt embeddings:

1. Adding a residual vector derived from input image features to the learnable prompt embeddings
2. A variational framework for prompt tuning, which this work implements by assuming a Gaussian distribution on residual vectors that are then added to the prompt embeddings.

Implementation of this framework is discussed in both the conditional (instance-specific) setting, as described above, and the unconditional setting of the previous work CoOpOp. Evaluation is performed on 15 image recognition and classification datasets via experiments on the tasks of base-to-new generalization, cross-dataset transfer, and domain generalization.
strength_and_weaknesses: Pros

- Increase in generalization performance against CoCoOp and ProDA is shown empirically for most datasets, with state-of-the-art results in prompt learning for image recognition tasks with generalization requirements.
- Either of the two mechanisms described - instance specific (conditional prompt learning) and instance independent (standard prompt learning), can be implemented as a simple change to the existing methods with a boost in generalization performance.
- This boost enables higher-performance for few-shot and zero-shot learning for downstream image-text tasks, possibly increasing resilience against distribution shift and adversarial attacks.
- Ablation studies provide evidence of the usefulness of the proposed mechanism.

Cons

- While the proposed model is simple to implement and integrate with existing systems, and has shown improved performance, especially for usage with different domains, there is a concern of lack of enough novelty since the ideas of residual vector and variational prompt learning were already introduced in previous publications.
- The increase in generalization performance comes at a cost to performance of in-domain/seen samples.
- A minor con is that prompt initialization is still manually specified.

Minor Edits

- More details can be specified about implementation in the zero-shot learning scenarios.
- More clear details can be added about the implementation of this method in the unconditional setting of CoOp.
- Can discuss ensembling interpretation of sampling
clarity,_quality,_novelty_and_reproducibility: The paper was well written and was easy to follow overall. More details about zero-shot learning and unconditional setting scenarios would have been useful.

The major ideas used here have already been proposed in existing works, though their adaption and combination is done with simplicity of implementation in mind.

The authors did mention that the code will be released.
summary_of_the_review: Borderline accept. The paper presents useful results about the implementation of simple changes to existing systems and the resultant performance improvement in certain scenarios. However, since existing ideas are heavily used and combined, there is an overall lack of novelty, so I would hesitate from a strong accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes a probabilistic framework for modeling the underlying distribution of prompts, which acts as an additive module on top of existing prompt-learning approaches, and seeks to enable improved domain generalization.
strength_and_weaknesses: (Strengths)

The manuscript proposes a simple, yet principled approach.

The manuscript is well-written and easy to follow.

The manuscript provides numerous experiments, against several strong baselines.

(Weaknesses)

Section 4. The phrase 'Domain Generalization' is slightly misleading, as the section seeks to highlight properties of unseen generalization through zero-shot transfer. The discussion could either be reframed under 'Cross-domain Generalization'.

Section 4. I would like to see further analyses/illustrations of the prompt distribution obtained by VPT. Do prompts that are sampled at the distributional modes correlate with features that well-characterize any known factors of variation (e.g., sub-domains) in the dataset(s)? What further evidence can be provided to show that the prompt distribution approaches reasonable coverage of the underlying distribution? Can equivalent results be obtained through other forms of density estimation?
clarity,_quality,_novelty_and_reproducibility: Clarity: the manuscript is well-written and the majority of claims are well-justified

Quality/Novelty: provides new problem formulation, compared to existing works, albeit somewhat iterative

Reproducibility: no issues
summary_of_the_review: Missing some intuition regarding the prompt distribution, beyond empirical results
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a variational learning method for prompt tuning for few-shot image classification. Like any variational Bayes method, it assumes that the visual prior can be encoded into a Gaussian prior and so the objective is to minimize the ELBO. The method can be plugged into existing prompt tuning methods like CoOp and CoCoOp. 

Strength:
Clear writing and comprehensive experiments.

Weakness:
The paper doesn't provide insights into why such a variational layer can improve the overall generalization. In fact, a critical concern, as raised by all the reviewers, the significant loss of in-domain (base) performance is somewhat unacceptable, which may imply severe theoretical flaws of the proposed method. 
justification_for_why_not_higher_score: The paper lacks an explanation of why the in-domain (base) performance is significantly dropped after applying the proposed method. Therefore, the paper is not yet ready for publication.
Decision: Reject