Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models
Authors: Shuai Zhao
Xiaohan Wang
Linchao Zhu
Yi Yang
Venue: ICLR 2024 poster (2024)
Keywords: Vision-Language Models
Zero-Shot Generalization
Test-Time Adaptation
CLIP reward
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose to improve the zero-shot generalization capacity  of vision-language models on the fly with CLIP as the feedback source.

Abstract: One fascinating aspect of pre-trained vision-language models (VLMs) learning under language supervision is their impressive zero-shot generalization capability.
However, this ability is hindered by distribution shifts between the training and testing data.
Previous test time adaptation (TTA) methods for VLMs in zero-shot classification rely on minimizing the entropy of model outputs, tending to be stuck in incorrect model predictions.
In this work, we propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident.
Specifically, a CLIP model is adopted as the reward model during TTA and provides feedback for the VLM.
Given a single test sample,
the VLM is forced to maximize the CLIP reward between the input and sampled results from the VLM output distribution.
The proposed \textit{reinforcement learning with CLIP feedback~(RLCF)} framework is highly flexible and universal.
Beyond the classification task, with task-specific sampling strategies and a proper reward baseline choice, RLCF can be easily extended to not only discrimination tasks like retrieval but also generalization tasks like image captioning,
improving the zero-shot generalization capacity of VLMs.
According to the characteristics of these VL tasks, we build different fully TTA pipelines with RLCF to improve the zero-shot generalization ability of various VLMs.
Extensive experiments along with promising
empirical results demonstrate the effectiveness of RLCF.
The code is available at https://github.com/mzhaoshuai/RLCF.

Published as a conference paper at ICLR 2024
TEST-TIME ADAPTATION WITH CLIP REWARD FOR HERO-SHOT GENERALIZATION IN VISION-LANGUAGE MODELS
Shuai Zhao *.+ Xiaohan Wang Linchao Zhu
Yi Yang*
*ReLER Lab, AAII, University of Technology Sydney
RReLER Lab, CCAI, Zhejiang University *Baidu Inc.
(zhacshuaimce, wxh1996111 haai1/immmmm.... com
(zhulinchao,
ABSTRACT
One fascinating aspect of pre-trained vision-language models (VLMs) learning under language supervision is their impressive zero-shot generalization capabil- ity. However, this ability is hindered by distribution shifts between the training and testing data. Previous test time adaptation (TTA) methods for VLMs in zero- shot classification rely on minimizing the entropy of model outputs, tending to be stuck in incorrect model predictions. In this work, we propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confi- dent. Specifically, a CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. Given a single test sample, the VLM forced maximize the CLIP reward between the input and sampled results from the VLM output distribution. The proposed reinforcement learning with CLIP feed- back (RLCF) framework is highly flexible and universal. Beyond the classification task, with task-specific sampling strategies and a proper reward baseline choice, RLCF be easily extended not only discrimination tasks like retrieval but also generalization tasks like image captioning, improving the zero-shot general- ization capacity of VLMs. According to the characteristies of these VL tasks, we build different fully TTA pipelines with RLCF to improve the zero-shot gener- alization ability of various VLMs. Extensive experiments along with promising empirical results demonstrate the effectiveness of RLCF. The code is available at https: //ops:://iiee whw..h..hhte.e.gg www wwwp///wwwssssaa//eoooooooao
INTRODUCTION
Pre-trained vision-language models (VLMs) learning under language supervision (Radford et al., 2021; Jia et al., 2021; Yuan et al., 2021) exhibit promising zero-shot transferability. This encourages researchers to explore the capabilities of VLMs across number of tasks in zero-shot fashion. For example, Hong et al. (2022) employ CLIP for zero-shot text-driven avatar generation, Sain et al. (2023) adapt CLIP for zero-shot sketch-based image retrieval, and Li et al. (2023) achieve zero-shot image captioning without images. Nonetheless, the large domain gap between training and test data is still challenging for VLMs in zero-shot circumstance. In this work, investigate how to fulfill the domain gap during test time in various tasks without task-specific training corpus, namely, test time adaptation (TTA) for VLMs with a zero-shot prerequisite.
One pioneer TTA work in improving the zero-shot classification ability of VLMS is test time prompt tuning (TPT) (Manli al., 2022). Given a single test sample, TPT optimizes the learnable prefix tokens by minimizing the entropy of model outputs to bootstrap its generalization capacity. Never- theless, making the model confident in its predictions is a double-edged sword. It does reduce the test error and close the domain gap at certain level (Wang et al., 2021a), but it makes the model stick its incorrect predictions and unable get out of the dilemma by itself as shown in the top of Figure la. Entropy minimization tends to make the model blindly confident.
Part of this work done during internship Baidu Inc. Yi Yang is the corresponding author.

---

Published as a conference paper at ICLR 2024
GT: stingray
Step
Step 1
Step
Step
(Probability) Top-5
(O.70) (0) sea lon
[C.98) sea lonn
10. (8) sea lion
$1.00) sea lion
1O 16) stingray
[0.03) stingray
100.0 adinggry
10. (0) American wligetor
100 07) American alligator
(00.01) American alligttor
20 American alligstor
submanine
NO 02) submarine
10.000 (0.00) pufferfish
200 (0) armadillo
10: (0) atingray
(0.01) armacilo
(0.00) armaciloo
0.000) puffertish
(0.00) armedilo
(Probability) [Reward] Top-5
NO.70) 70) [+0.028] sea loon
[0.53) [+2 004) sea lion
(0.71) (-0.108) stingray
10 84] [+0.106) 106) atingray
RLCF
10 18) [+0, 108) stingray
[0.44] [+0 007) stinaray
10 (0.29) (+0.025)
10.35) (+0.025) 50a lon
1/
Amarican allggtor
(G.011 1-0 American allgator
(0.00) 040) American alligator
10.
submarlnn
NO 02) [+0.066] submarino
(0.00) [+0.006) submmrriee
(0.00) [+0. (56) submerine
(0) Amer alligator
(0.01) [0.148] armadillo
(0.00) ((00)(10.000 [-0.008] pufteriihh
(0.00) [-0: 149) armadile
30. (0.00) (-0 140) armedillo
(a) Top-5 predictions and probabilities with different TTA steps. Step 0 shows original predictions.
There are three sheeps standing
-0.95
logether on the grass.
group of baseball players is crowded at the mound.
CLIP Reward
+0.03,
Two girls bathe an elephant lying
0.21, -0.2, +0.43
on its side.
(b) Examples of CLIP reward. The average score of image and all sentences is the reward baseline.
Figure 1: Feedback mechanism in zero-shot generalization with CLIP as the reward model.
Inspired by the success of the feedback mechanism in language and vision tasks (Ouyang et al., 2022: OpenAI, 2023; Pinto et al., 2023), we introduce feedback during test time rectify the VLM output as shown in the bottom of Figure la. Previous feedback methods leverage labeled preference data to train a reward model (Ouyang et al., 2022; Lee et al., 2023) or use labels to calculate the reward (Cho et al., 2022; Pinto et al., 2023). Without ground truth, we refer to the well-recognized CLIP (Radford et al., 2021) model as the feedback resource. CLIP shows powerful generalization capacity across many VL tasks. The outputs of CLIP are also well-calibrated (without fine-tuning on specific dataset) (Minderer et al., 2021), i.e., the score from CLIP accurately reflects its uncertainty about the input sample. This makes CLIP a reliable reward model. One more question is why feedback rather than directly tuning with CLIP supervision? Ouyang al. (2022) demonstrate that model learning with feedback has better generalization abilities than a supervised fine-tuning model. We get the same conclusion from our empirical results. Furthermore, CLIP supervision cannot be directly used in generation tasks like image captioning, while the feedback mechanism is versatile.
Our proposed framework, coined as reinforcement learning with CLIP feedback (RLCF), is flexible and universal for TTA with different VLMs in various tasks. With task-specific sampling strategies and a proper reward baseline choice, RLCF is applicable across zero-shot classification, text-image retrieval, and image captioning. In these tasks, the model is given single test sample, we then sample K candidates from the output distribution. For discrimination tasks like classification and retrieval, the top- K sampling is applied; for the caption generalization task, beam search method is adopted. Assuming the input is an image, like Figure 1b, the CLIP model first gives the CLIP- Score (Hessel et al., 2021) between the image and all candidate sentences. As CLIPScore always non-negative, the average score is subtracted from the calculated scores. This average baseline aims distinguish which model behaviors are encouraged and which are discouraged. Then the learnable parameters in the TTA model are optimized by REINFORCE (Williams, 1992) algorithm.
While the reward design and learning algorithm remain consistent across various tasks, the TTA pipelines are tailored to each specific task. For classification, we inherit the data augmentation and confidence selection pipeline from TPT (Manli et al., 2022), making it work for not only prefix tuning but also backbone adaptation. For retrieval, considering a large number of candidate entries, we only update the parameters w.r.f. the query for efficiency. For instance, we only tune the branch w.r.t. the input modality for a two-branch VLM like CLIP. For image captioning, we construct the TTA pipeline with two methods (Mokady et al., 2021; Nukrai et al., 2022) built upon large language models (LLMs). During TTA, we only tune the projector which projects the image into the LLM token embedding space. Plus, several lask-agnostic practical tricks are applied, i.e., multiple reward models, episodic TTA (Wang et al., 2021a), and momentum buffer for incremental learning.
2

---

Published as a conference paper at ICLR 2024
To summarize our contributions: 1) To the best of our knowledge, RLCF is the first universal fully TTA framework for improving the zero-shot generalization capacity of VLMs across different tasks. 2) We develop novel reward function for test time RL with CLIP. It is simple yet effective. Com- pared to previous methods (Cho et al., 2022) in the training stage, it demonstrates that CLIP can be used practical reward model alone, even with single test sample. 3) We design task-specific TTA pipelines for three VL tasks with RLCF, Extensive experiments with promising results validate the effectiveness of RLCF in boosting the zero-shot performance of different VLMs.
2 RELATED WORK
Reinforcement learning in language and vision The most well-known application of RL in natu- ral language process is reinforcement learning with human feedback (RLHF) (Ouyang et al., 2022; OpenAT, 2023). A reward model is trained with preference data collected from humans, and it is used to fine-tune the LLM via proximal policy optimization (PPO, Schulman et al. (2017)). Sim- ilar approaches are applied in (Ziegler et al., 2019; Stiennon et al., 2020; Bai et al., 2022; Glaese al., 2022). In prompt engineering for language models, RLPrompt (Deng et al., 2022) and TEM- PERA (Zhang et al., 2023) search for discrete text prompts by RL. RL has also been widely applied in vision and mulli-modaa research. A comprehensive study of deep RL in computer vision can be found at (Le et al., 2022). Recently, Pinto et al. (Pinto et al., 2023) optimize vision task metrics using RL and achieve promising results, demonstrating the effectiveness of RL vision. In the multi-modal area, ImageReward (Xu et al., 2023) collects preference data and trains reward model for text-to-image generation tasks, similar to RLHF in GPT models. SCST (Rennic et al.. 2017) apply CIDEr metric reward function with REINFORCE (Williams, 1992) algorithm to improve the generation quality in image captioning during training. Cho et al. (Cho et al., 2022) explore the possibility of using CLIPScore (Hessel et al., 2021) the reward in image captioning. Their empir- ical results show that CLIPScore cannot be an independent reward function and should be combined with grammar regularization CIDEr metric. Nevertheless, the training of the grammar head and calculation of CIDEr metric both need reference text, which is unavailable at test time.
Test-time adaptation Test-time adaptation (TTA) aims to address the distribution shift between training and test data during test time (Sun et al., 2020; Liu et al., 2021; Wang et al., 202 Test- time training (TTT, Sun al. (2020)) allows modifications to the training pipeline. In such cases, self-supervised auxiliary tasks are incorporated to help the model adapt to the distribution of test data (Sun al., 2020; Liu et al., 2021; Lin et al., 2023). For example, TTT+ (Liu al., 2021) utilizes instance discrimination tasks from contrastive learning (Chen et al., 2020). On the other hand, fully TTA assumes that the training pipeline cannot be modified as the training data is unavailable (Wang et al., a0) Two popular techniques in fully TTA are normalization layer adaptation and entropy minimization. Normalization layer adaptation updates data statistics or parameters of the normaliza- tion layer based on batched test samples (Wang al., 2021a; Schneider et al., 2020; Niu al., 2023) or augmented data views from a single test sample (Zhang et al., 2022a). Entropy minimization aims to make the model confident in its predictions to reduce generalization error (Wang al., 2021a; Zhang et al., 2022a; Manli et al., 2022; Niu et al., 2022; 2023). There is also a retrieval-augmeent TTA method (Zancato al., 2023), which uses CLIP to retrieve informative data from an external large dataset and update the decision boundary during test time.
3 METHOD
3.1
PRELIMINARIES
Fully test-time adaptation in vision-language tasks Let fo(-) represent a VLM trained on image- text pairs Derain {(t,, x;));²1 with parameter 0, where &, E Terain (training text space) and v; E Virain (training image space). The objective of TTA (Sun et al., 2020; Wang et al., 2021a) is to boost fe(t) or fo(v) on domain-shifted test samples Drest = ) or Drest = {(v, JM where t; E Treat (testing text space), U, E Viest (testing image space), Trest + Trrain, and Viest + Verain. We assume that the VLM takes either text or image as input and outputs the other modality. In fully TTA, the training data are unavailable, and the training pipeline cannot be modified. Following TPT (Manli et al., 2022) and MEMO (Zhang et al., 2022a), the adaptation is conducted with a single test point, Le., the VLM is exposed to only one t, 0-.
3

---

Published as a conference paper at ICLR 2024
Contrastive Language-Imagg Pre-training (CLIP) CLIP (Radford et al., 2021) comprises im- age encoder g(-) and a text encoder h(.). CLIP is pre-trained using a contrastive loss that encourages similarity between feature vectors of paired images and text, aligning them in a shared embedding space. Once pre-trained, CLIP can assess the similarity between the text t. and image v as follows:
Ct, v) cos cos)h((),g(t))
where cos((;)) .) represents the cosine similarity. For image classification with CLIP, the input text consists of the prompt plus the class names, i.e,, = {pr: "dog"), where prompt p; "a photo of a".
3.2 TEST-TIME ADAPTATION WITH CLIP REWARD
3.2.1
REINFORCEMENT LEARNING WITH CLIP FEEDBACK
Without loss of generality, we first consider the case where the VLM fo()) takes an image er as input and maps it to text t.. During TTA, our goal is to learn a conditional distribution P(t|v, 0) = fo( that maximizes a reward function R(.,-). Formally, the optimization problem during TTA is:
max Be- R(t,v).
(2)
Different from previous methods (Rennie et al., 2017: Cho et al., 2022; Pinto et al., 2023) which maximizes the expected reward over batched training samples, here we only maximize the expected reward over a single test sample Viest-
Policy gradient with REINFORCE To compute the gradient of the non-differentiable reward func- tion, REINFORCE (Williams, 1992) is adopted calculate Voles-(R(t,, v)]. It uses the so-called "log-derivative trick" to estimate the gradient of the expected reward for given input:
Vo w)] [ v) Vo log p))
In a VL task, the input and output modalities are closely related, e.g., the input is an image and the output is the description of the image. Therefore, we can use CLIP to evaluate the similarity between the input and output, and the model can maximize this similarity align with task goals. Similar to Cho et al. (2022), we use CLIPScore (Hessel et al., 2021) as the reward:
CLIP ELIP-S((t.) = w max(CLIP(t, v), 0),
where 2.5 is constant. CLIPScore is always non-negative, which means it encourages all model behaviors. However, for an irrelevant sampled image-text pair Figure 1b, we expect the reward model to provide negative feedback to discourage such behavior. Cho al. (Cho et al., 2022) adopt a greedy search bascline which needs to be combined with a grammar regularization or CIDEr metric to be practical reward function. In this work, we demonstrate that with proper sampling strategies and baseline, CLIPScore can also be used as the sole reward function in different VL tasks. Specifically, set the reward baseline as the average CLIPScore of sampled image-text pairs. The reward function with baseline becomes:
Rtt, v) = CLIP--((,, st., v) E v)].
(5)
It is straightforward to get the reward function for VLM which takes text t input and return an image u according to Eq. (5). The sampling strategies will be presented in the next section.
3.2.2 TASK-SPECIFIC FULLY TEST-TIME ADAPTATION
RLCF is flexible and applicable across various VL tasks, and we apply RLCF to three different VL tasks in this work. For all tasks, the VLM solely learns through REINFORCE with Eq. (5) as the reward function during test time. However, VLMs and sampling strategies vary with tasks. Next, we introduce our task-specinf fully TTA pipelines.
Zero-shot image classification on OOD data Figure 2 illustrates the fully TTA pipeline for zero- shot image classification with RLCF Without loss of generality, we also choose CLIP as the classi- fier. The TTA pipelines include two adaptation manners: prompt tuning and image encoder tuning. TPT (Manli et al., 2022) shows that entropy minimization for image encoder tuning results in infe- rior performance compared to prompt tuning. By contrast, RLCF works both with prompt tuning and image encoder tuning, demonstrating its versatility.
4

---

Published as a conference paper at ICLR 2024
frozen parameters
(CLASS)
text enooder
confidence selection
leemnable
class
0.20, 00.16 -0.04
Ps
P(t (v)))
dog, horse
dog
P(t mx(v))
Reward
horse
Image encoder
dog. flower, horse
(v))
0.26, 0.15, 0.11
top-K classes sampling
data augmentations
rewards
TVV
Policy Gradient
Figure 2: Fully TTA for zero-shot image classification with CLIP reward. Left: image encoder tuning, Right: prompt tuning. The pipelines of the two are the same except for the learnable parameters. A single test image is first augmented produce multiple views, then only confident views with low-entropy predictions are selected. For each selected view, we sample the top-K classes, calculate their rewards, and update the parameters using policy gradient.
Top-K sampling image 1 image 2
Too-K sampling
image image
sentence sentence
sentence sentence
image
text encoder
imagen
image K
sentence
sentence K
couple of giraffes that are standing in fence.
text encoder
CLIP Reward
image encoder
CLIP Reward
Policy Gradlent
Policy Gradlent
(a) Text-to-lmage Retrieval
Image-to- Text Retrieval
Figure 3: Fully TTA for zero-shot text-image retrieval with CLIP reward.
In Figure 2, given a test image v, it is first operated with data augmentors {7); .2....: :T) for multiple different views. Following TPT (Manli al., 2022) and SAR (Niu al., 2023), we only reserve the confident samples with low-entropy predictions, namely, the entropy ((((t))(v)) of the selected view should be low. High-entropy predictions are considered unreliable as they lack confidence in their outputs. In practice, views with low entropies as TPT (Manli et al., 2022). For each low-entropy view, class names of the top-K predictions used to calculate their CLIP rewards according to Eq. (5). The learnable parameters are then optimized to maximize the expected reward by gradient descent as Eq. (3).
use the bottom 10th percentile of 64 augmented
One point that needs to be clarified is why using the CLIP reward feedback rather than di- rectly fine-tuning the model with CLIP supervision. For example, methods like knowledge distilla- tion (KD, Hinton al. (2015)) or pseudo-label (Lee et al., 2013). InstructGPT (Ouyang al., 2022) demonstrates that model learning with feedback has better generalization capabilities compared to supervised fine-tuning model. our context, KD or pseudo-label requires weak model (student) to mimic strong model (teacher). However, it is worth noting that the student may be correct while the teacher may be incorrect. For instance, given an image of dog, the top-3 predictions of the student and teacher models are (dog, horse, tree) and (cat, dog, horse), respectively. For KD or pseudo-label, the student will forced follow the incorrect behaviors of the teacher. In contrast, the feedback mechanism only assesses the sampled results from the student, less likely to alter the correct prediction. In such cases, the feedback mechanism combines the merits of both the student and the teacher. Another important reason is that CLIP supervision cannot be directly used in generalization tasks like image captioning, while the feedback mechanism is universal.
Zero-shot text-image retrieval The fully TTA pipeline for zero-shot retrieval with RLCF is pre- sented in Figure CLIP also serves as the zero-shot retrieval model. For retrieval, the number of candidates is usually large, so we only update the parameters with respect to the query for efficiency.
5

---

Published as a conference paper at ICLR 2024
caption
sampling captions search
group zebras swimming body water close up of a shark swimming body water. shark swimming body water. () close up view of a whale swimming in a body of water.
projector
LLM
close up view
CLIP Reward
Policy Gradient
0.12, +0.07, +0.08, -0.11,...
Figure 4: Fully TTA for image captioning with CLIP reward.
For text-to-image retrieval, the image encoder remains fixed, while the text encoder is frozen in the other case. Given a query, top-K sampling is employed to the returned results to calculate the re- ward. Unlike image classification, no augmentations are used for the input query. The retrieval task requires a holistic understanding of the input query rather than identifying a single object. Augmen- tations like crop and flip may lead to corrupt semantics.
Zero-shot and cross-domain image captioning Figure 4 illustrates the fully TTA pipeline for image captioning with RLCF The captioning TTA pipeline is built upon two LLMM-ased methods: CapDec (Nukrai et al., 2022) and CLIPCap (Mokady et al., 2021). CapDec is trained only with text and CLIPCap is trained with images. TTA with CapDec is undertaken with a zero-shot prerequisite, and TTA with CLIPCap is cross-domain. During the test, CapDec and CLIPCap will be given unseen and Hamain-ssiffee images, respectively. Both CapDec and CLIPCap utilize a projector (e.g., an MLP or transformer (Vaswani et al., 2017)) to project CLIP feature vectors into the token embedding space of the LLM. Only the projector is updated through policy gradient, while the LLM remains fixed during TTA. Beam search is employed to sample K generated captions for reward calculation.
3.2.3 TISE-TIME ADAPTATION TRICKS
In this section, we introduce several general TTA techniques applicable across different tasks.
Multiple reward models with weights By default, a single CLIP-ViT-L/IE is used as the reward model. An ensemble of multiple reward models can be used for better feedback. We assign scores based on human preference for different CLIP models: (CLIP- VI----VV---/1-3366: 10, CLIP- ViT- 5, CLIP-RN50x6A4 3). These scores are then normalized to up serving as weights for the ensemble. CLIP-RN uses a ResNet (He et al., 2016) as the image encoder, while CLIP-ViT adopts vison transformer (Dosovitskiy et al., 2021).
Episodic TTA The model is exposed to only a single test sample once, making the learned knowl- edge unreliable for other samples. Hence, after each TTA process, the model parameters 0 reset to the initial state d like (Wang et al., 202 la; Manli et al., 2022). It is called episodic TTA.
Momentum buffer While episodic TTA ensures reliability, it limits the incremental learning ability of the model. To address this issue, we introduce a momentum buffer &, initialized as + 0" After a TTA process, 9 becomes 0, and & is updated by { t- mL + (1 m)8, where m E [0. 1) is momemtum coefficient. Every B, samples, we update l + &. At the start of the next TTA process, +- 0", allowing the utilization of the learned knowledge. The momentum buffer functions similarly to ensemble of different models, resembling model soups (Wortsman et al., 2022).
EXPERIMENTS
This section presents the experimental TTA results in three tasks. For variants our method, RLCF uses CLIP- ViT-L/14 as the reward model, RLCF-S adopts weighted reward sum of (CLIP-VIT. L/14-336, CLIP-VIT-L/I CLIP-RN50x64), and RLCF-S-M adds the momentum buffer.
4.1 ZERO-SHOT IMAGE CLASSIFCCATINN ON OOD DATA
Datasets Following CLIP and TPT, we test RLCF on ImageNet (Deng et al., 2009) and its four variant test sets with distribution shifts: ImageNet-A (Hendrycks et al., 202 lb), ImageNet-V V2 (Recht (Wang et al., 2019).
et al., 2019), ImageNet-R (Hendrycks et al., 2021a), and ImageNet-Sketar ImageNet-A consists 7,500 natural adversarial images misclassified by ResNet-50. ImageNet-V V2 contains 10,000 natural images from different sources. ImageNet-R collects 30,000 images with artistic renditions. ImageNet-Sketch includes 50,000 black and white sketch images.

---

Published as a conference paper at ICLR 2024
Table 1: Top-1 accuracy of zero-shot image classification with TTA on OOD data. KD uses a CLIP-ViT-L/14 as the teacher. The best and second-bes's results are highlighted. Improvement in aceuracy of RLCF compared to the baselines (horo-shot CLIP- B/T--666/ or CoOp) is in (tblue).
Method
ImageNet
ImageNet-AA
ImageNet-V2 V2
ImageNet-R
ImageNet- Sketch
ODD Averge
Zero-shor baxeline
CLIP-VET-B/16 CLIP-VIT-L/14
66.73
60.86
57.20
73:44
68.82
67. 80
85
57.84
69.97
Promp: nuning for CLIP
CoOp (Zhou al, 2021)
71.51
49.71
64, 20
75.21 21
47.5
59.28
CoCoOp (Zhou et al., 2022)
71.02
50.63
59.91
TPT (Manli et al., 2022)
68.98
54.77
63.45
77.06
47.94
60.81
TPT + CoOp (Manli al.)
73.61
57.95
66.83
77.27
49.29
62.84
TPT + CoOp+ KD (Linton et al.)
71.40
63.25
65.28
82.70
53.78
66.75
RLCF
73.23 (e.60)
65.45
65.45
69.77 (8.91)
83.335 83.33
54.74
(4.74
68.33 (111,13)
RLCF + CoOp
76.05 (1
69.744
(0.74,12000 (20.00) no)
70.62 no (n)
84.51
(1.-a (9:300
56.49 (15.00
56.49
70.34 (111
RLCF-S + CoOp
76.50 (4.
71.11
70.92
84.73
(6.
70.93
(111/6))
Image encoder timing for CLIP- -TT--1/
Pscudo-label (Lee
2013)
69.
62,15
63.56
80.03
49,45
63,80
TPT (Manli et al., 2022)
69:42
61.62
63,70
79.74
49.47
63.63
KD (Hinton et al., 2015)
70.92
66.39
65:01
82.12
53.51
66.76
ATKD (Gao et al., 2020)
70.51
70.66
65.54
85.12
53,56
68.72
RLCF
74,85 (8.12)
73.71
69. (reaa))
86, 19/112.1)
57.10/111.0
71 61 (13.44)
RLCF-S
75.34 (8.01)
75.00
127.
70.08 (0:22)
86.97/112.00
(7..59-
72.45 (116.25)
RLCF-S-M
75.48 pau7s)
75. 16,11 127
(0.42
87.23 (11325))
57.73
72.64 (18.44)
Expeetted Callteation Erroe
0.35
Prompt Tuning for CLI-VIIB/L
10p0 Arc.
Top-I and Top-5 Ancuracy TFT
Top-l and Tup-5 Accuracc of RLCP
Sefore
0.30
chdure TTA
0.25
0.20
(ogn) Acc 2
0.18
75
Sbbeore VIA
Cefore VIA
20
Suudoro VIA
TTA Staps
TTA Stage
TEA Staga (samgking K
(a) ECE on IN-A. lower is better.
(b) Acc. of TPT w.t. steps.
(c) Acc. of RLCF w.t.t steps.
Figure 5: ECE and average accuracy on ImageNet-A/V2/R. Prompt tuning with CLIP-ViT-B/16.
Baselines We compare RLCF with few-shot prompt tuning methods for CLIP - CoOp (Zhou et al., 2021) and CoCoOp (Zhou et al., 2022) (16 shots on ImageNet), state-of-the of-thee-rt test-time prompt tuning methods- - TPT (Manli et al., 2022), and knowledge distillation (KD (Hinton et al., 2015), ATKD (Guo et al., 2020)), which use the reward model as the teacher during test time. TPT CoOp means TPT adopts the learned prompts of CoOp as the initialization, otherwise, TPT uses token embedding of a hard prompt "a photo of a" as initial weights. For all prompt tuning methods, the length of learnable prompts is 4. Results of Pseudo-label (Lee et al., 2013) are also presented.
For prompt tuning, the learning rate is 7e-3, the weight decay value is 5e-4, and the optimizer is AdamW (Loshchilov & Hutter, 2019). For image encoder tuning. the learning rate is decreased to le-5. Given a test sample, the parameters will be optimized for steps to maximize the reward of the top-3 (sampling factor K 3) predictions. The momentum coefficient m = 0.9998 and update interval B, = 64 for the momentum buffer.
Implementation details
Results In Table 1, RLCF largely improves the zero-shot generalization capacity of CLIP-ViT- B/16 and outperforms previous methods. Notably, on ImageNet-A/V2n) RLCF with CLI--VI- B/16 surpasses the reward model CLIP-ViT-L/I4, This shows that RLCF effectively combines the capabilities of both the TTA model and the reward model through the feedback mechanism, something that KD or pssudo-label cannot achieve. RLCF significantly outperforms the entropy minimization method - TPT. TPT can only learn from the TTA model itself and lacks awareness of the correctness of its predictions. Figure et al., 2017) of TPT and RLCF. The ECE of the two both increases along with the TTA steps, but the ECE of RLCF is clearly lower. This means the output of RLCF better reflects its uncertainty about the input and is more reliable. In Figure 1a&8, RLCF provides multiple positive scores for various objects, preventing the model from becoming blindly confident. In Figure 5b, the top-5 accuracy of TPT drops with more steps. The model is stuck in its incorrect predictions and pushes away the ground truth as shown in Figure la. By contrast, there is no such issue for RLCF in Figure Sc.
presents the expected calibration error (ECF) (Guo
Ablation study about sampling factors and reward model choices can be found in Appendix B.
7

---

Published as a conference paper at ICLR 2024
Table 2: TTA for zero-shot text-image retrieval. KD uses CLIP-ViT-L/14 as the teacher model.
Improvement in Recall@) I with RLCF compared the CLIP-ViT-B/16 baseline is in (tblue).
MS-COCO (SK test images)
30K (IK images)
Method
lext-lo-image
image-tn-ton
text-lo-image R@5 R@:10
image-lo-lext Io-lext
R@1
R@5 R
R@1
R@5
R
R@1
R
R R
Zem shot hareline
CLIP-VIT-B/A VIT-B/16
33.0
58.2
68.9
52.5
76.8
84.6
62.2
85,7
91.8
81.2
96.4
98.5
CLIP-VEEE VIT-L/14
36.1
60.9
71.1
56.2
78.9
86.9
64.6
87.1
92.1
85.3
97.2
99.1
CLIP LIP-VET-L/14-336
36.6
60.9
71.0
57.3
N0.6
87.8
67.1
88.9
93.2
86.6
98.0
99.1
77A
CIIP-VIT-B/16
Pscudo-label (Lec al.)
33.0
57.8
68.2
52.4
72.4
81.8
62.2
85,3
91.7
81.1
93.2
97.8
KD (Hinton al.) [stops:
87.9
97.5
KD (Hinton et al.) (stoss: 5)
34.6
59.7
69.8
53.7
76.6
84.5
61.4
86.0
91.7
83.1
95.9
97.9
RLCF
37.3
62.7
71.5
59. (16.6)
80.1
86.9
(r..9)
89.1
93.2
87.3
97.2
98.8
RLCF-S
(5.3)
63.4
72.5
60.8 (1:3)
80.8
87.5
68.5 (m0.3)
90.0
93.7
88. (pm.a
97.7
98.9
RLCF-S-M
38.4
63.5
72.6
60.8
(18.8)
80.5
87.6
(1.5
90.2
93.7
(1.1
97.7
98.9
4.2 ZERO-SHOT TEXT-IMAGE RETRIEVAL.
Implementation details For text-image retrieval, we use the test set of Flickr30K (Plummer al., 2015) and test split of MS-COCO (Lin et al., 2014) divided by Karpathy et al. (Karpathy & Fei-Fei, 2015). Each image in the two test sets corresponds retrieval model. The learning rate is lc-6, the weight decay value is 5e-4, and Adam W optimizer used, For MS-COCO, the sampling factor K 12 for eex--to-mmae retrieval, and K 20 for the other case. For Flickr30K, K 12 and K = 16 for text-to-iman image and image-to- te-tettttextttteee retrieval, respectively. The adaptation steps are 8. For the momentum buffer, m = 0.9998 and B, = 64. We also compare RLCF with knowledge distillation (KD) with CLIP-ViT-L/14 as the teacher.
sentences. CLIP-ViT-B/16 is adopted as the
Results Table presents the retrieval results on MS-COCO and Flickr30K. RLCF demonstrates significant improvement compared to the zero-shot bascline and even outperforms the most powerful CLIP-ViT-L/I4-336. Similar phenomena are also observed in zero-shot classification. The feedback mechanism reserves the merits of the TTA model and makes the TTA model improve with the reward model. In contrast, KD or pseudo-label forces the student to mimic the teacher regardless of the correctness of the teacher as discussed in Sec. 3.2.2. In KD for supervised classification (Hinton et al., 2015; Zhao et al., 2022; Wang et al., 2021b), the student is generally worse than the teacher due to their capacity gap and incomplete learning. Nevertheless, RLCF can surpass the powerful reward model with the feedback mechanism during test time in a zero-shot circumstance.
4.3 IMAGE CAPTIONING
Datasets To test. the adaptation ability of RLCF for captioning models in a zero-shot or cross- domain condition, we train the captioning model on MS-COCO train set (Lin et al., 2014) and test it the test set of Flickr3KK (Plummer et al., 2015) and validation set of NoCaps (Agrawal et al., 2019). NoCaps validation set contains three splits according to whether contains MS-COCO objects: in domain contains only MS-COCO objects, near domain contains both MS-COCO and novel objects, and out domain contains only novel objects.
Implementation details CLIPCap (Mokady al., 2021) and CapDec (Nukrai et al., 2022), two LLM-based methods, are chosen as the captioning models. The two have the same architecture, while CLIPCap is trained with CLIP-ViT-B/16 image embedding and CapDec is trained with CLIP- ViT-B/16 text embedding. The projector in Figure 4 is an 8-layer transformer encoder that contains about 43M parameters. The LLM is OPT-125M (Zhang et al., 2022b). During TTA, we only tune the parameters of the projector. For CLIPCap. the learning rate is 2e-6, and sampling factor K = 10; for CapDec, the learning rate is 5e-6 on Flickr30K, 3 on NoCaps, and K = 6. No weight decay is applied. The optimizer is AdamW. The TTA step is 4. After TTA, captions are generated with beam search with width of and the final caption is the one with the highest score.
Results Table 3 presents results for image captioning. A weakly supervised method - MAGIC (Su et al., 2022) and a zero-shot method -DeCap (Li et al., 2023), are included for reference. The reported metrics include BLEU@4 CIDEr, SPICE, and RefCLIPScore (Hessel et al., 2021). Ref- CLIPScore reflects the similarity between generated text and reference captions. The improvements in CIDEr metric (Vedantam et al., 2015) are highlighted. For all metrics, both CapDee and CLIPCap with RLCF significantly improve upon the baselines. This demonstrates the strong generalization ability of RLCF in image captioning, even with single test sample. It is noteworthy that CLIPCap
8

---

Published as a conference paper at ICLR 2024
Table 3: TTA for image captioning, B@4 for BLEU @ 4, C for CIDEr, S for SPICE, and Ref-C for RefCLIPScore. The gain of well-recognized CIDEr metric is in (fblue).
MS-COCO NoCaps
MS-COCO Flickr30K
Method
in domain
near domain
out domain
Karpathy 's test split
B@4
C
S
B64
S
B64
C
S
B@4
C
Ref-C
MAGIC (Su at, 2022)
5.2
18.3
5.7
DeCop (Li et al,, 2023)
72.7
61.9
43,9
17.7
42.0
13.8
TTA for CapDec (zero-shot)
CupDec (Nukrai et al., 2022)
32.4
62.6
10.3
29.2
54.0
9.6
17.2
31.7
6.4
19.3
37.0
11.7
74.1
RLCF + RLCF-S
33.3
(8.0
10.7
30.3
57.9
(1.9)
10.3
17.6
41.9
75.7
34.0
68 (11.77)
10.8
30.3
58.5
10.3
17.7
35.3
6.9
20.1
41.
41.66/43)
12.7
75.7
for CLIPCap (cross-domaln)
CLIPCap (Mokady al., 2021)
36.3
76.9
11.9
34.8
73.5
11.0
22.5
54.6
8.6
21.8
49.3
13.1
76.7
RLCF
38.6
(1.4)
12.5
36.1
79.6 10.11
11.8
24.7
63.8
9.6
23.3
56.6.1..)
14.5
79.4
+ RLCF-S
38.7
84.7 (1:a))
12.6
35.8
(9.7
79.7
11.8
24.2
(3.1
9.5
23.6
57.8
14.6
79.4
steps
stops 4
A is walking greesy field.
dog standing front a people.
is
dog
(31 dog is standing grassy field.
33 A dag is standing the grass with people.
4) man walkin day in feld.
(4) A doy is slanding greesy field with people walching.
0.045, 0.045, 0.006, 0.077.
0.013, -0.018, 0.026, +0.055,
case
sters
sioer 4
11 black motoreydle parked the street 22 motorcycle parked on the side of a road. motorcyde parked on the side of street black motortydle perked on side road.
A black and silver matorcycle parked on street. A black matorcycle parsed on a strest black and silver malorcyca parked street. black and silver motoroycle parked on sidewalk
+0.014, 0.008, 0.007, +0.022
0.003, 0.015, 0.011, - 0.004,...
steps
Aos T B
(1) group paople standing around snow
A group people playing game ice hockey. hoskey.
(2 group people that slanding the covered
standing
A group of people standing in front of a rink.
44 group people standing show covered fleld.
34) A group of people slanding sround a hockey rink.
0.086, -0002, 0.111, 0.052,
40.042, +0.053, 0.003, +0.004,
case
steRe
stops 4
11 women weering and umbrolla.
woman is weariigg hat and umbrala.
woman wearing hat and umbrella. women with a hot and umbralaa in pool woman wearing hat and umiorelia pool.
A Woman in hat and sunglasses in pool.
woman wearrnng hat and susgl in a pool. woman with and umbrelll pool.
0.027, 0.043, 0.016, -..005...
0.036, + 00.100 0.099, 0..044
Figure 6: Intermediate generated captions of CLIPCap and the CLIP reward. The sampling
factor K = 10, only 4 candidates are shown here. The final generated caption is in blue.
achieves greater improvements in CIDEr (up to 9.2) compared to CapDec. CLIPCap can also use a large sampling factor K. This is possible because CLIPCap can generate higher-quality candidate captions. The results of RLCF-S-M are not shown as is no better than RLCF-S.
wwweeeediitegggerrrteee captions and their corresponding rewards. The visualization reveals that the CLIP reward model favors captions that provide holistic description of the image. Through feedback, the generation of such captions is encouraged. During TTA, captions aligned with the preferences of CLIP are given higher priority. Please refer to Figure in Appendix A for more visualization cases.
Qualitative results Figure 6 displays the
5 CONCLUSION
In this work, we introduce reinforcement learning with CLIP feedback (RLCF) to improve the zero- shot generalization ability of VLMs on the fly. A novel reward function with CLIP is developed. We instantiate three TTA pipelines for image classification, text-image retrieval, and image captioning with task-spec sampling strategies and parameter tuning manners. With RLCF, the zero-shot gen- eralization capacity of various VLMs is boosted significantly. We hope RLCF can provide heuristic information for future research that employs TTA with feedback from large foundation models.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the Australian Research Council (ARC) under Grant DP200100938. Thanks Chao Liang for his helpful discussions.
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In ICCV, 2019.
Yuntao Bai. Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:224.00866, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI,, 2020.
Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. Fine-grained image captioning with clip reward. In Findings of NAACL, 2022.
Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In CVPR, 2009.
Mingkai Deng, Jianyu Wang. Cheng-Ping Hsich, Yihan Wang. Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. RLPrompt: Optimizing discrete text prompts with reinforcement learning. In EMNLP, 2022. URL http::://aclaatthhlooo wwtp://////222.mmpmmmmeee.. 222.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. URL https:
https://openrevieeee wptp///eerrreeeeeeeeeeeee net/ www.mmiidYYiioooooo wbw.F wiccbFaNTTy.......
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), ICML, 2017.
Jia Guo, Minghao Chen, Yao Hu, Chen Zhu, Xiaofei He, and Deng Cai. Reducing the teacher- student gap via spherical knowledge disitllation. arXiv preprint arXiv:2010.07485, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. In CVPR, 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of -istriibtittion generalization. In ICCV, 2021a.
Dan Hendryeks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, pp. 15262-15271 b021b.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference-free evaluation metric for image captioning. In EMNLP, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 1503.005 2015.
10

---

Published as a conference paper at ICLR 2024
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu. Avatarclip: zero-shot text-driven generation and animation of 3d avatars. ACM Trans. Graph., 2022.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. In CVPR, 2015.
Ngan Le, Vidhiwar Singh Rathour, Kashu Yamazaki, Khoa Luu, and Marios Savvides. Deep rein- forcement learning in computer vision: a comprehensive survey, Artificial Intelligence Review, 2022.
Dong- Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning. 1CML, 2013.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023.
Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding CLIP latents for zero-shot cap- wttp::/.eeefoouuu
tioning via text-only training. In ICLR, 2023, URL id=Lt 8bM1hiwx2,
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Wei Lin, Muhammad Jehanzeb Mirza, Mateusz Kozinski, Horst Possegger, Hilde Kuchne, and Horst Bischof. Video test-time adaptation for action recognition. CVPR, 2023.
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexan- dre Alahi, Tit++: When does self-supervised test-time training fail or thrive? NeurIPS, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. URL
https: /ttttp:://ppeneee /ttp:///eeeeeeeeeeee/mmeeee .nt/ ftrumorrmmo/ ddBkkggiicyy)
Shu Manli, Nie Weili, Huang De-An, Yu Zhiding, Goldstein Tom, Anandkumar Anima, and Xiao Chaowei. Test-time prompt tuning for zero-shot generalization in vision-language models. In NeurlPS, 2022.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In NeurIPS, 2021.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint
2021.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICML, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. ICLR, 2023.
David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise-injected clip. In Findings of EMNLP, 2022.
OpenAl. Gpt-4 technical report. arXiv, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NewrlPS, 2022.
André Susano Pinto, Alexander Kolesnikov, Yuge Shi. Lucas Beyer, and Xiaohua Zhai. Tuning computer vision models with task rewards. arXiv preprint 202.222302.08222. 2023.
11

---

Published as a conference paper at ICLR 2024
Bryan A Plummer. Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region eegion-to-phreee to-phrase correspondences for richer image- to-sentence models. In ICCV, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ICML, 2019.
Steven Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR, 2017.
Aneeshan Sain, Ayan Kumar Bhunia, Pinaki Nath Chowdhury, Subhadeep Koley, Tao Xiang, and Yi-Zhe Song. CLIP for all things zero-shot sketch-based image retrieval, fine-grained or not. In CVPR, 2023.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. NeurlPS, 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. NeurlPS, 2020.
Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models can see: plugging visual controls in text generation. arXiv preprint arXiv:2205. 220.022500.00 2022.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrlPS, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, 2015.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021a
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. NeurlPS, 2019.
Wenxiao Wang. Minghao Chen, Shuai Zhao, Long Chen, Jinming Hu, Haifeng Liu, Deng Cai, Xiaofei He, and Wei Liu. Accelerate enns from three dimensions: comprehensive pruning framework. In ICML, 2021b.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning, 1992.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2004.05977 2023.
12

---

Published as a conference paper at ICLR 2024
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Lu- owei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. arXiv preprint ar.iv:2 1432, November 2021.
Luca Zancato, Alessandro Achille, Tian Yu Liu, Matthew Trager, Pramuditha Perera, and Stefano Soatto. Train/test-time adaptation with retrieval. CVPR, 2023.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. NeurIPS, 2022a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXi::2205.0006 2022b.
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. Tempera: Test-time prompt editing via reinforcement learning. In ICLR, 2023.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, 2022.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. CoRR, abs/2109.01134 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909.0899, 2019.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A VISUALIZATION RESULTS
case
stops
steps
A couple of people sleeping on bed.
couple of men sleeping on a bed. couple of paople sineping on had.
2) A couple of kids sineping on a bad.
(3) Two people sleeping Ingeter an a had.
Two people slasping together on bod.
A couple of people seeping logetherr on a bed.
A couple people slceping together on a had.
-0.009, 0.031, 0.012, +0.012,
0.009, +0.012, 0.031, +0.012,
steps 3
steps
couple of people siccping on couple men sleeping bed. Two people sleeping logether bed.
couple men sleeping couple people sleeping bed. Two people slaaping souch. couple
A couple of sleeping logether bed.
0.017, +0.037, + 0.005 + +..05....
+-0.023, 0.031, 0.010, +0.049,
CHSP 2
siops
steps
A people walking down street
crowd of people city street night.
(2) A peogle walking down buay streat
group people people walking city streat
33 A crowd of people walking down city street
24) group of people walking down street at right.
group people walking street night.
0.015, 0.005, 10.019, 10.010....
0.030, 0.033, 0.001, 0.009....
steps 3
steps
A crowd people on city street night. crowd paople an crowd people walking down street night. people walking
crowd of people city street night. at night.
night
crowd of paople crowd people walking down of people crosswalk night.
night.
city streat.
+ +0.013, + +00028, +0.012, 0..17....
+ 00.017, 0.027, + 0.016, + +0.044...
case 9
steps
steps
A woman in dress holding a pink vase.
woman wearing a pink dress and white shirt. woman wearing a pirk dress and white hat. woman wearing a pink dress and while vase.
A woman in pink dress holding a while vase. A woman while dress holding pink vase. A woman wearing pink dress
white vase.
woman wearing white dress
-0.030, -0.008 +0.001, +0.012.
+0.018, 00.15 0.001, -0.018..
stops 3
stops
wearing dress
wearing in pink dress and while shirt.
white
woman weering while drees and white shirt. A woman wwaaring while dress and a pink shirt.
woman wearing a white dress and white shirt.
+0.024, +0.016, 0.042, +0.052
+0.024, 0.016, 0.051 -0.42,
ca5e 4
steps 1
stops
window. couple bunk beds
21 two bunk beds and window.
3) A badroom with two bunk beds and window:
M3) couple bunk bads are in room.
4] A couple of bunk hads a room.
14) A couple of bunk bods in room.
-0.011, 00.010, 0.020, +0.004...
-0.015, 0.011, 00.017 0.001.
steps
A coupla of bunk bads ane a room.
couple of bunk bads are in a room.
2 A couple of bunk bacs that are in a room.
Two bunk beds that are in room.
3) A couple of bunk beds in a room.
Two bunk beds are in a room.
4] A pair bunk
room
A couple bunk beds are room.
+0.006, +0.005, 0.011, 0.001....
+ +0.001, 0.022, 0.002, + +0.001
Figure 7: Generated captions at each TTA step of CLIPCap and the CLIP reward. The sam- pling factor K- 10, only 4 candidates are shown here. The final generated caption is in blue.
In Figure 7, we show more captioning examples like that in Figure 6. These samples illustrate how CLIP reward helps the captioning model select description that matches the picture more closely.
14

---

Published as a conference paper at ICLR 2024
GT: Persian cat
Step 0
Step
Stop
Step
(Probsbility) Top:5
TPT
56) umbrella
(00311) umbrella
30 (0.94) umbrella
X1 1000 umbrella
10: 11) sinaping bag
[00 (2) siceping bag
30 (0.02) bubble
30. 00) parathute
06) racket
(0. bubble
(0.01) sleeping beg
20 sllapping
bubbie
10.00
parachute
bubbiee
04) perachule
10. (0) rechel
30 (0.00) racket
30 mongosse
(Probebllitt) [Rewardl
:0.58
umbralla
umbrolla
umbrotla
Parsian
RLCF
(0-11) sleeping beg
[+0.000)
(0,06) [.0030 racket
20-10) [+0 024) lynx
(0.02) [+0 024) lyxx
X0, [+0.024] lymx
10 (0.0 [-0.63) buoble
(0.07) 05
sleeping bag
(0.02) [-00051]
sleeping beg
30. [-0.012] doormal
(0.04) [+0 [-0.334] parachute
parachute
doommat
(0.051) sleoping
goldtinch
(0.39) green iquana
(0.78) green guana
(0.79) green Iquana
(0.74) green iguara
20.13)
American alligttor
10.07) American
20. Americaan allggttor
squimal
(0.04) squirnol
(0. squirol
squiiro!
JO (0.08) American bullhro
(0.03) ganter snake
(0.03) American bullfrog
0 03) Americann bullfog
N0.07) grasohapger
(0,03) American bullinog
10. 03) garter snake
20 garter snake
(0.30) (+0.0133 green iquara
30 (+0 013) green iguana
(0.501 1-0 029) green Iquana
00 52) (+0 00.02 goldfinch
[0:13) 50.0177 garte anako
10 (0.00) (+0 (86) fox squirral
(0.20) (+0 024) fox squimel
(0 [-O (-0.035) American robin
RLCF
30.08)
gerter
garter
20
squámal
(0.08) 500 (0-1011
American bullfrog
(0.04)
0000 grasshoppar
(0.04]
[+8 humminnghir
(0.07) [+0.039]
grasshopper
(0,03) (-0 American bullfrog
(0.03) (+0 063) goldfinch
(0.02) (00.016juumco
Figure 8: Top-5 predictions, probabilities, and rewards with different TTA steps. Prompt tuning for CLIP-ViT-B/16. For RLCF, sampling factor K = 5. We perform TTA steps and step means the original prediction. Images from ImageNet-A.
In Figure 8, we provide visualization of top-5 predictions with different TTA steps like Figure la. In Figure 8, the ground truth is not in the top-5 predictions. In such cases, TPT cannot find the object by minimizing the entropy of model outputs. By contrast, RLCF can discover the ground truth by pushing away the objects with negative feedback.
B ABLATION STUDY
B.1 SAMPLING FACTOR
When using RLCF with TTA for various VL tasks, sample K candidates from output distribution for reward calculations. this section, we exam- ine how K affects different tasks and models.
TTA results CoOp
94
Top-5 Aoc.
90
88
Image classification on OOD data Figure
86
shows the average top-1 and top-5 accuracy on ImageNet-A, ImageNet-V2, and ImageNet-R for different K in image classification. RLCF reduces to pscudo-label (Lee et al., 2013) when K = A larger K improves 5 accuracy, but not top- 1. Too many sampled classes may make the opti- mization process difficult for policy gradient. For example, when K = 5 and only one class gets positive score and other classes get negative scores. pushing away 4 negative classes may cause unpredictable behavior and make the model miss the ground truth after gradient updating.
Top-1 Aec.
80
75
70
before TTA
Sampling Factor K (TTA steps
Figure 9: Different sampling factors K in im- age classification on OOD data.
Zero-shot text-image retrieval Table 4 presents the effect of sampling factor K in zero-shot text- image retrieval. Similar to image classification, a larger generally leads to better Recall@5 and Recall @ 10 compared to smaller K. However, smaller K tends to produce better Recall @ in
15

---

Published as a conference paper at ICLR 2024
Table 4: Different sampling factors K in zero-shot text-image retrieval. K2 for sampling in text-to-image retrieval, and Kize for sampling image-to-text retrieval.
MS-COCO (5K test images)
Flickr30K (IK test images)
Method
text-to-image
umage-to-text
text-to- -mage
image-to-text
R@5 R@ R@1 R@5
R@10 10 R@!
R@5 R@ 10
R@1 R@55 R@ 10
Zemo-shot baseline
CLIP-VIT-B/IE
33.0
58.2
68.9
52.5
76.8
84.6
62.2
85.7
91.8
81.2
96.4
98.5
K22
K12r
TTA for ILIP-VIT-B/16 with RLCF
37.8
61.1
68.1
59.1
86.5
66.6
87.9
91.5
86.7
97.6
16
37.6
62.1
69.6
59.3
80.4
86.8
66.7
89.0
92.1
87.3
97.2
98.8
10
18
37.4
62.4
70.8
59.5
80.1
86.9
67.0
89.0
92.7
87.1
97.2
98.7
12
20
37.3
62.7
71.5
59.1
80.1
86.9
67.1
89.1
93.2
87.2
97.3
98.6
22
37.0
62.5
72.4
59.8
80.2
86.8
66.9
89.0
93.4
87.3
97.3
98.6
36.8
62.4
72.5
59.5
87.3
66.9
93.5
87.6
96.9
98.7
18
26
36.9
62.5
72.5
59.2
80.3
87.0
66.7
88.8
93.5
87.3
96.8
98.6
Table 5: Different sampling factors K in image captioning. Metrics B@4 for BLEU@4, C for CIDEr, for SPICE, and Ref-C for RefCLIPScore.
MS-COCO + NoCaps
MS-COCO = Flickr30K
Method
in domain
near domain
out domain
Karpathy test split
B@4
C
S
B@4
C
S
B@4
C
S
B@4
C
Ref-C
TTA for CapDec (zero-shot)
CapDec (Nukrai al., 2022)
32.4
62.6
10.3
29.2
54.0
9.6
17.2
31.7
6.4
19.3
37.0
11.7
74.1
RLCF (K 4)
32.7
65.5
10.7
30.0
57.5
10.2
17.1
34.6
6.8
20.2
40.8
12.5
75.4
RLCF (K 6)
33.3
68.0
10.7
30.3
$7.9
10.3
17.6
35.5
6.9
20.3
41.9
12.7
75.7
RLCF (K =
32.7
65.5
10.7
30.0
57.5
10.2
17.1
34.6
6.8
20.2
41.8
12.9
75.8
TTA for CLIPCap (cross-domain)
CLIPCap (Mokady al., 2021)
36.3
76.9
11.9
34.8
73.5
11.0
22.5
54.6
8.6
21.8
49.3
13.1
76.7
RLCF (K 8)
38.4
83.3
12.5
36.0
78.5
11.6
24.1
61.4
9.3
22.7
55.8
14.3
79.3
RLCF (K 10)
38.6
84.0
12.5
36.1
79.6
11.8
24.7
63.8
9.6
23.3
56.6
14.5
79.4
RLCF (K =1)
38.5
82.0
12.5
35.9
80.2
11.9
24.4
63.1
9.5
23.1
57.5
14.6
79.4
most cases. In MS-COCO and Flickr30K, one image has reference captions, so the sampling factor for image-to-text retrieval is larger than K for text-to-image retrieval.
Image captioning Table illustrates effect of different values of sampling factor K cross- domain image captioning. The optimal K varies for different image captioning models. CLIPCap has better captioning capabilities than CapDec, so can produce better candidates. Therefore, a larger K is suitable for CLIPCap.
From the ablation study of sampling factor K, we find that the choice of K depends on the tasks and VLMs. Different tasks and models require various sampling strategies.
B.2 DIFFERENT REWARD MODELS
RLCF relies on the good quality of the CLIP reward models. In this section, we show the influence of different CLIP reward models image classification and image captioning.
As shown in Table 6, RLCF is robust to different reward models. Compared to the baseline CoOp, RLCF can achieve improvements even with CLIP-RNSO- as the reward model, which is worse than the prompt tuning model CL.P- ViT-B/6... When the prompt tuning model and the reward model are the same, RLCF is also better than the state-of-the-art test-time prompt tuning method TPT (Manli et al., 2022). With CLIP-VIL ViT-LL4- as the reward model, RLCF with prompt tuning is slightly better than the ensemble result.
worth noting that RLCF with image encoder tuning in Table is obviously better than the ensemble results in the OOD average performance. Compared to the ensemble method, RLCF can adapt to the test distribution with the feedback mechanism. This why RLCF shows better performance than the ensemble results.
We also test different reward models in image captioning. Results are shown Table CapDec and CLIPCap both use ViT-BVi1 the image embedding extractor. RLCF with different reward
16

---

Published as a conference paper at ICLR 2024
Table 6: RLCF with different reward models in image classification. "-" for not available.
Method
Reward Model
ImageNet-A
ImageNet-V2
ImageNet-R
ImageNet-Sketch
OOD Average
Zer--shot baseline
CLIP-RN50X4 CLIP-VVT-B/I6
39.66
58.72
42.73
52.55
47.87
60.86
73.98
46.09
57.20
LLI--VIT-L/14
68.82
67.80
85.40
57.84
69.97
Ensemble (B/16 1214)
65.94
69.02
85,92
57.98
69.72
Prompt nuning for CLIP-VYT-B/16
CoOp (Zhou et al., 2021)
49.71
64,20
75.21
47.99
59.28
TPT CoOp (Manli et al.)
57.95
77.27
RLCF + CoOP
CLIP-RN50x4
52.06
64.62
76,00
48.42
60.28
RLCCF CoOP
CLIP-VIT-R16
61.66
67.04
78.06
49.70
64.12
RLCF+ CoOP
CLIP-VIT-L14
69.74
70.62
84.51
56.49
70.34
Table 7: RLCF with different reward models in image captioning. Metries M for METEOR, C for CIDEr, for SPICE, and Ref-C for RefCLIPScore. We keep the embedding extractor for the two methods as CLIP-ViT-B/16.
Embed. Extractor
Reward Model
MS-COCO = NoCaps
Method
in domain
near domain
out domain
M
Ref-C
M
Ref-C
M
Ref-C
TTA for CupDec (Nukrai et al., 2022) (zero-shot)
CapDec
CLIP-VET-B/16
23.9
75.5
22.3
17.2
6.4
+ RLCF
CLIP VIT-VVG
CLIP-VIT-B/LE B/16
24.3
65.5
10.7
76.6
22.7
57.2
10.0
75.7
17.7
35.0
6.8
72.9
+ RLCF
CLIP-VIT-B/16
CLP-VVT-L/14
24.6
68.0
10.7
76.7
23.0
57.9
10.3
75.7
17.9
35.5
6.9
72.7
TTA for CLIPCap (Mokady at. 2021) (cross-domain)
CLIPCap
CLIP-VTT-B/C
26.4
76.9
11.9
77.8
24.8
73.5
11.0
77.6
20.3
54.6
8.6
75.7
RLCF
CLIP-VET-B/6 CLIP-VIT-B/IC
CLIP-VIT-B16
26.9
25.5
21.2
RLCF
CLI--VIT-L/I
27.2
84.0
12.5
80.3
25.7
79.6
11.8
80.1
21.5
63.8
9.6
78.5
Table 8: Average GPU inference time per sample and GPU memory with different TTA steps. Test on ImageNet-A and ImageNet-V2 V2 with
single NVIDIA 40GB A100 GPU.
CLIP-ViT-B/16
TTA Steps
ImageNet-A
ImageNet-V2 V2
Prompt Tuning
Acc.
Mem. (GB)
Time (s)
Acc.
Mem. (GB)
Time (s)
TPT + CoOp
1
57.95
4.2
0.168
66.83
18.2
0.468
TPT + CoOp
3
60.13
4.2
0.320
66.76
18.2
1.05
RLCF + CoOp
1
63.07
6.2
0.197
69.59
19.8
0.486
RI.F + CoOp
3
69.74
6.2
0.348
70.62
19.8
1.08
Image encoder tuning
Acc.
Mem. (GB)
Time (s)
Acc.
Mem. (GB)
Time (s)
TPT CoOp
1
61.78
8.8
0.208
63.70
8.8
0.272
TPT + CoOp
3
62.07
8.8
0.384
64.02
8.8
0,512
RLCF
71.23
10.8
0.239
67.60
10.8
0.319
RLCF
3
73.71
10.8
0.415
69.77
10.8
0.543
models can always achieve significant improvements the near and out domain data. This shows the robustness of RLCF to open domain scenarios with different CLIP reward models.
B.3 GPU RUNTIME AND MEMORY
Efficiency is also important in TTA. We provide the GPU time and memory in Table 8.
Compared TPT, the inference time of RLCF increases by constant amount for different TTA steps and datasets, iee,, roughly 0.03s per sample. For each sample, the CLIP reward model only needs run the image encoder once. This is the source of the 0.03s increase. The text features of CLIP reward model are always the same because the class names are fixed.
ImageNet-A has 200 classes, while ImageNet-V22 has 1000 classes. For prompt tuning on ImageNet- V2, the input batch size of the CLIP text encoder is 1000, and we need to re-run the text encoder to update the features after each TTA step. This is why prompt tuning is slower and consumes more memory than image encoder tuning. For image encoder tuning, the text features are unchanged and the image encoder only has single image as input.
17


Reviews:
[{"review_id": "BKiayngifM", "reviewer": "Reviewer_9Cat", "summary": "This article suggests a Reinforcement Learning strategy using a CLIP-based model suitable for implementation during test time in the zero-shot learning framework. The suggested concept was tested on three distinct tasks within the zero-shot framework: image classification, image retrieval, and image captioning.", "questions": "Please see the questions raised in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- The paper introduces a novel reinforcement learning-based reward function to enhance the efficacy of the CLIP-based approach, which can be employed during test time for a new dataset.\n\n2- The suggested concept is applied to three distinct tasks: image classification, test to image retrieval, and image captioning, all of which are experimented within the zero-shot framework.\n\n3-  The experiments were conducted on the Imagenet dataset for all three tasks - image classification, image retrieval, and image captioning, demonstrating superior performance compared to recent state-of-the-art methodologies.", "weaknesses": "1- The proposed method is employed within the zero-shot learning framework for all three tasks: image classification, image retrieval, and image captioning. It would indeed be intriguing to explore whether it can also be applied within a few-shot learning framework?\n\n2-  Figure-2 illustrates the application of a data augmentation strategy for image classification. Is this same strategy for augmentation also utilized for textual data?\n\n3-  Figure-6 demonstrates the visual outcomes of step-1 and step-4, depicting the progression from the worst to the best. Including results from all steps, such as step-1, step-2, step-3, and step-4, would offer a more comprehensive understanding of how the transformation from the worst results to the best unfolds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s1tW9BmQxs", "reviewer": "Reviewer_uHKV", "summary": "This paper studies the test time adaptation methods for vision-language models. The authors propose a framework, called reinforcement learning with CLIP feedback  (RLCF).  Specifically, given a test sample, the VLM is optimized to maximize the CLIP reward, which is provided by a CLIP model.  RLCF can be applied to image classification, text-to-image/image-to-text retrieval, and image captioning.  Extensive experiments demonstrate the effectiveness of RLCF.", "questions": "- More advanced KD and the ensemble of the student and teacher model.\n- The results of  CLIPCap and  CapDec with the CLIP-ViT-L/14 architecture.\n- More implementation details of TPT+KD.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-organized and easy to follow.\n\n- The proposed RLCF framework is universal and applicable across various VL tasks.\n\n- The experiments on three tasks show the superior effectiveness of RLCF, compared to TPT, KD, and Pseudo-label.", "weaknesses": "- The proposed RLCF utilizes the CLIP model to provide a reward score, which is very similar to a recent work [1]. The two tweaks are sampling strategies and adding a baseline to the reward function. It weakens the novelty of the proposed method. Besides, the comparison between the proposed RLCH and [1] is missing.\n\n- The authors state that compared to pseudo-label and KD, the feedback mechanism combines the merits of both the student and teacher. However, the authors only investigate the original version of KD proposed in 2015. Moreover, I recommend that the authors evaluate the ensemble of the student and teacher model, which can directly combine their merits.\n\n- For the experiments on image captioning, I suggest that the authors add the results of  CLIPCap and CapDec with the CLIP-ViT-L/14 architecture (the teacher model) for a more comprehensive comparison.\n\n- Could the authors provide more implementation details of TPT+KD?\n\n- Typo: kD->KD on page 8\n\n\n[1] Cho, Jaemin, et al. Fine-grained Image Captioning with CLIP Reward. In Findings of NAACL, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UwMAkiCRxr", "reviewer": "Reviewer_efhx", "summary": "This paper solves test-time adaptation of Vision-Language-Models to improve the zero-shot generalization performances. Unlike previous works that rely on entropy of model outputs, the authors propose to leverage reward from CLIP model as feedback to adapt models. The proposed method based on reinforcement learning with CLIP feedback is evaluated on three tasks, including zero-shot image classification, zero-shot text-image retrieval, zero-shot and cross-domain image captioning, showing improved performances.", "questions": "- In Eq.(4), why need to clip CLIP score to be non-negative? What 'it encourages all model behaviors' means? In Eq.(5), any difference if not subtracting the expectation term?\n- The authors mention RLCF-S adopts weighted reward sum and RLCF-S-M adds a momentum buffer. Could the authors explain more about this? Also as mentioned in the weakness, more comments on using less reliable CLIP like CLIP-RN50, CLIP-B would be appreciated.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The paper solves an important task of test-time adaptation of vision-language models. VLMs have played a critical role in many CV and NLP tasks.  How to improve their zero-shot generalization ability, especially in the challenging task of test-time adaptation, is a worthwhile research problem.\n+ The authors propose to leverage CLIP feedback for adapting models. This seems to be novel and is interesting to me. As many vision-language learning methods are built upon CLIP, the intergration of CLIP reward is natural.\n+ The proposed RLCF method universally applies to different tasks, like zero-shot classification, text-image retrieval, image captioning. Hence, the idea may inspire research in many other VL tasks.\n+ The method is simple yet effective. The experiments and analyses are extensive in the paper. For each task, RLCF is compared with a few  baseline methods, showing significant improvements.\n+ The paper is very easy to follow with a clear motivation and method descriptions. Visualizations (e.g. fig.1) also look nice.", "weaknesses": "- The method relies on good quality of CLIP feedback. This may restrict its applications in tasks other than images with generic objects. For example, CLIP shows less satisfactory accuracies on fine-grained datasets like FGVC Aircraft, EuroSAT, CUB, etc. CLIPScore may be not informative in the fine-grained classification.\n- The authors use CLIP-ViT-L and a few other variants for calculating reward. The commonly used backbone in VL learning is CLIP-ViT-B. It is understandable that CLIP-ViT-L leads to better text-image alignment. But discussions on CLIPScore with CLIP-ViT-B would be helpful to understand how robust RLCF is to unreliable rewards.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)