Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The method proposes entropy-based optimization of prompt tokens during test time. To this end, the image is augmented with different views to make a batch of 64 samples. Noisy samples are filtered based on the entropy score. The Adam-based optimizer steers the tokens' representation with one step. 
An ensemble with a previous baseline (e.g., CoCoOp) significantly improves various classification tasks. 



strengths_and_weaknesses: Strengths: The work proposes a new approach (TPT) for zero-shot image classification that does not require pre-training and work during inference. The method is plausible and intuitive and suggests some novel elements, such as the confidence selection term. One benefit of this approach compared to CoCoOp is that it doesn't require additional training. In most experiments that check classification capabilities, an ensemble of TPT with  CoCoOp shows a significant improvement in several presents compared to the CoCoOp baseline. 

Weaknesses: [W1] The method alone is on par with previous works (i.e., approximately 1% difference between a naive "hand-crafted" ensemble). The most significant gap comes from the ensemble of CoCoOp with TPT. However, as an ensemble, in general, is known to improve performance, the method should check other ensembles, e.g., CoCoOp (with different seeds), CoCoOp + CoOp, CoCoOp + Ensemble,  etc. 
[W2] The work lacks discussions and qualitative study. The study does not contribute much to the reader in its current form. Yes, metrics are improved. But why? Which classes improved? I suggest the authors follow this work in the analysis [1]. 
[W3] I'm not sure the method is significantly new. I would appreciate a better discussion on the differences from MEMO. 

Typos:  Tab. 1 caption requires -> require; L293 is -> are.


Refs: 
[1] - The effects of regularization and data augmentation are class dependent; Balestriero et al.
questions: 
Can the authors provide results of ensemble between different methods? 
What classes are improved by applying TPT?

limitations: yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes test-time prompt tuning (TPT), a method that can learn adaptive prompts on the fly with a single test sample. It optimizes the prompt by minimizing entropy. The authors show TPT can improve 0-shot performance for classification.
strengths_and_weaknesses: ### Pros
- The method achieves good performance compared to other language-vision prompting methods
- The method shows a new way to exploit a text-image model like CLIP for mono-modal tasks.
- This method doesn't require additional data or supervision.

### Cons
- The 0-shot performance is still behind the fine-tuning by a large margin (although understandable)
- The idea is not completely novel since consistency regularization has been explored before.
questions: I'm not sure if the four ImageNet variants are really OOD data? What about other tasks? 
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a test-time tuning method that adapts the pretrained CLIP for each test example using the example itself. 
Such tuning is conducted with the objective of encouraging consistent predictions for different augmented views of the test image, and only soft prompts are updated during this process. The authors demonstrate the effectiveness of this method for image classification task under natural distribution shifts setting and cross-dataset generalization setting, outperforming existing methods that even use additional image classification data for training.

  
strengths_and_weaknesses: Strengths:
- The general idea of test-time tuning by encouraging consistency of different augmented views is novel.
- The performance is surprisingly good, outperforming strong baselines in the two generalization settings.
- The analysis on what parameter groups to optimize at test time is inspiring in that it finds prompt tuning is more effective than tuning other set of parameters or the entire model.
- It's also surprising that only one step of optimization can increase the performance stably.

Weaknesses:
- Although the paper demonstrate the empirical improvement of the proposed method, it's not very clear why encouraging the consistency can lead to such much improvement. I would suggest the author discuss more about the underlying assumption.
- Despite that only one step of backpropagation is required for this test-time training, the method still requires computation on multiple augmented images (8-64), which significantly increases the inference computation (roughly 8-64 x 2 slower).
- The proposed test-time tuning involves several hyper-paramters to be tuned, such as the optimization step or the confidence selection interval. However, it's not clear how they are tuned, which raises the concern of using test data for selecting models.

questions: - The method uses multiple augmented images for one classification, and thus can be regarded as an ensemble method. Did the author compare with a baseline that runs inference on these images and then average the probability? Basically, if we have those augmented images, are there other methods to use them without the optimization step? 
- The 1-step optimization at test time works surprisingly well. I am wondering if the author observed any unstableness of this optimization?
limitations: The author discussed the limitations pretty well.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors introduce test-time prompt tuning to improve CLIP's zero-shot classification accuracy by optimizing the consistency of probability distributions across multiple augmented views. To reduce noise samples from random augmentation, the authors proposed confidence selection to remove samples with low confidence. Numerous experiments show the robustness and generalisation of the proposed method.
strengths_and_weaknesses: Strengths:
1. The proposed method improves the performance of the baselines.
2. This paper is well-organized and easy to follow.

Weaknesses:
1. Although the authors have presented the inference time as a limitation, the paper still lacks a comparison with the time complexity of CLIP, COOP, etc.
2. The authors' motivation is reasonable, but the proposed optimization objective is not. I agree that the predictions should be consistent across perspectives, but it doesn't make sense to keep the predicted class probability vectors consistent. Predictions of non-correct label positions in the probability vector are irrelevant, and it is reasonable to be inconsistent across perspectives.
3. This paper lacks significant contribution and the improvement results from data augmentation.

questions: 1. Compared to the baseline (CLIP, COOP, COCOOP), how much time complexity does the proposed approach increase while improving performance?

2. I think that the optimization objective function (4) proposed by the authors is not reasonable, as I discussed before. Please correct me if I have misunderstood.

limitations: The authors have discussed the limitations of the proposed approach. I do not see the potential social impact.
review_title: null

Meta-review: metareview: This paper proposes a technique for training prompts for open-vocabulary vision models (e.g. CLIP) at test time, i.e. without any labeled data. The model is trained to minimize the entropy of the average prediction of many augmented views of a test image. This improves performance to varying degrees without requiring any additional labeling. The method and approach are interesting and some useful analysis is provided. Reviewers agreed that the paper should be accepted. Beyond the suggested changes made by reviewers, I'd recommend some additional references to better situate the paper with respect to past work on consistency regularization, entropy minimization, and transductive learning.
confidence: Certain
Decision: Accept