AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
Authors: Jan Hendrik Metzen
Piyapat Saranrittichai
Chaithanya Kumar Mummadi
Venue: Accepted by TMLR ()

Abstract: Classifiers built upon vision-language models such as CLIP have shown remarkable zero-shot performance across a broad range of image classification tasks. Prior work has studied different ways of automatically creating descriptor sets for every class based on prompt templates, ranging from manually engineered templates over templates obtained from a large language model to templates built from random words and characters. Up until now, deriving zero-shot classifiers from the respective encoded class descriptors has remained nearly unchanged, i.e., classify to the class that maximizes cosine similarity between its averaged encoded class descriptors and the image encoding. However, weighing all class descriptors equally can be suboptimal when certain descriptors match visual clues on a given image better than others. In this work, we propose AutoCLIP, a method for auto-tuning zero-shot classifiers. AutoCLIP tunes per-image weights to each prompt template at inference time, based on statistics of class descriptor-image similarities. AutoCLIP is fully unsupervised, has only a minor additional computation overhead, and can be easily implemented in few lines of code. We show that AutoCLIP outperforms baselines across a broad range of vision-language models, datasets, and prompt templates consistently and by up to 3 percent point accuracy.

Published in Transactions on Machine Learning Research (08/2024)
AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision- Language Models
Jan Hendrik Metzen
bosch.
Bosch Center for Artificial Intelligence, Robert Bosch GmbH
Piyapat Saranrittichai
Bosch Center for Artificial Intelligence, Robert Bosch GmbHl
Chaithanya Kumar Mummadi
Bosch Center for Artificial Intelligence, Robert Bosch LLC
Reviewed on OpenReview: https: // opw.reviee.........mmmm net/ forum? id= jo
Abstract
Classifiers built upon vision-language models such as CLIP have shown remarkable zero-shot performance across a broad range of image classification tasks. Prior work has studied different ways of automatically creating descriptor sets for every class based on prompt templates, ranging from manually engineered templates over templates obtained from a large language model to templates built from random words and characters. Up until now, deriving zero-shot classifiers from the respective encoded class descriptors has remained nearly unchanged, i.e. classify to the class that maximizes cosine similarity between its averaged encoded class descriptors and the image encoding. However, weighing all class descriptors equally can be suboptimal when certain descriptors match visual clues on given image better than others. In this work, we propose AUTOCLIP, a method for aulo-tuning zero-shol classifiers. AUTOCLIP tunes per-image weights to each prompt template at inference time, based on statistics of class descriptor-image similaritics. AUTOCLIP is fully unsupervised, has only a minor additional computation overhead, and can be casily implemented in few lines of code. We show that AUTOCLIP outperforms baselines across a broad range of vision-language models, datasets, and prompt templates consistently and by up to 3 percent point accuracy.
1 Introduction
Classificrs built upon vision-language models (VLMs) such CLIP (Radford et al.) 2021 and CoCa Yu et al. 2022) have shown strong zero-shot transfer capabilities across various tasks. Such zero-shot transfer is appealing since it allows for obtaining high-performing classifiers novel domains without the overhead of data acquisition and labelling. However, it has been observed that prompt engineering plays a crucial role for obtaining strong zero-shot classifiers, that is: zero-shot classifiers derived from VLMs need to be constructed based on a set of prompt templates (parameterized by the class name) that cover potential variation of the domain. These prompt templates can be hand-designed (Radford al. 2021 generated by largo-language 2023
model (Menon Vondrick 2022), randomly generated (Roth al.
Prompts also be learned via test-time prompt tuning (TPT) Shu et al. 2022 Zhao et al. 2023 This
approach makes the zero-shot classifier adaptable to the datum of interest, which is possible by effectively leveraging the knowledge of the general-purpose VLM. Shu el al. (2022) Lune prompts so that the predictive entropy for a single image minimized, while Zhao et al. 2023 maximizes a CLIP reward. These prior TPT methods require the VLM's image encoder to process several augmentations for each image. Morcover, gradients with respect to the prompts require backpropagation through the VLM's text encoder, thereby substantially increasing the overall inference cost.

---

Published in Transactions on Machine Learning Research (08/2024)
Qbike
Dikke
caa
caa
Joint Embedding Space
Joint Embedding Space
Standard CLIP
AutoCLIP
Class queries based on wniform-weighted average descriptor encodings
Class queries based on weighted average
descriptor encodings
Image encoding
Class descriptor encodings q. (car), (bike) for templates ; > A drawing of
Class queries (car/bike)
RED
t, (class)
Prediction
GREEN
t2 (2(cass)>> A pixelated photo of <class name>
BLUE
bg (class)) A photo of <class_name> name>
Acar W1 91 (car) + w2 (c(rrr) + W3 (c(rrr)
Ybike = W1 (bike) + W2 92 (bike) + Wa q3 (bike)
Standard
41
91
W1
qcar
(2
(+)
W2
CLIP
(.)+)
No influence
(+)+
W3
qbike
Avg. Similarity to X
weights
Similarity X
(*)
(1(+)
W1
9car
AutoCLIP
(x)))
(2(+)
W2
43
Influence weights
(*(+)
W3
qbike
Avg. Similarity X
weights
Similarity
X
Figure 1: Conceptual Illustration of AutoCLIP. CLIP's zero-shot classifiers are based on a set of prompt templates (AA photo of <class name>", "A drawing of a <class name>", ...). Inserting class names into these templates gives a set of class descriptors that are encoded into a joint embedding space together with the respective image. Standard CLIP averages encoded class descriptors (c()) into class queries Чe, and classifies to the class that has maximal cosine similarity with the encoded image. However, this ignores that some prompt templates describe the image of interest better than others (their embeddings have higher average similarity): for instance, when the image drawing, the template "A drawing of a <class name>" results in stronger class descriptors than other templates and should thus be weighted higher when computing class queries. AUTOCIIP determines such weights directly from class descriptor-image similarities in the embedding space. Here, the car image is taken from Atkinson (2015).
2

---

Published in Transactions on Machine Learning Research (08/2024)
We propose to not tune the prompts but instead use large set of predefined and fixed prompt templates and to adapt the weights of those prompt templates for each image at test-time. This approach has the major advantage that adaptation takes place entirely in the embedding space without requiring additional forward or backward passes through the VLM's encoders, which significantly lowers the test-time computation and memory overhead compared to prior TPT methods. Our work similar to Allingham et al. (2023). but comes with the major advantages that our approach can adapt weights for single samples and does not require access to the pre-training feature distribution.
We briefly summarize the standard way of constructing zero-shot classifiers from VLMs (see Figure | left). At first, collection of prompt templates is instantiated for each class to form a set of class descriptors (e.g,, "A photo of a car", and "A drawing of a car" are sample class descriptors of class car). These descriptors are processed by the text encoder and the resulting encoded descriptors are averaged to obtain the image-independent class queries (e.g. Hear). Besides, the image encoder processes the input image to be classified to get the image encoding, which lies in the same embedding space as class queries. The cosine similarity of the encoded image to every (averaged) class query computed, and the output prediction is assigned to the class with maximum similarity.
This work follows a similar zero-shot classification sctup, except that we change how class queries are computed. Instead of a simple average of the encoded class descriptors, we propose to take a weighted average, wherein weights of the encoded class descriptors are automatically tuned for each image separately. The weights are determined in a manner that prompt templates whose resulting class descriptors are closer to the respective image embedding get higher weightage than those being less similar (see Figure | right). Our approach is motivated by the intuition that prompt templates with high similarity describe relevant properties of the image better than ones with lower similarity (see Figure 6 for evidence supporting this intuition). We denote our method that automatically adapts the weights of the encoded class descriptors for each image as AUTOCLIP. We provide basic implementation of AUTOCLIP at
We empirically show that AUTOCLIP improves the performance of zero-shot classifiers across many datasets, VLMs, and prompt strategies with little inference-time overhead. Note that AUTOCLIP is fully zero-shot as it does not require any supervision from the target task. Furthermore, AUTOCLIP makes no assumptions on the underlying VLM and can thus be broadly applicd, potentially also to multi-modal models beyond VLMs such ImageBind (Girdhar et al. 2023b).
Overall, our main contributions are as follows: we introduce AUTOCLIP (Section 3.2 for constructing zero-shot classifiers from vision-language models. AUTOCLIP leverages statistics of class descriptor-image similarities to automatically determine weights of the prompt templates. We further discuss a method for automatically tuning AUToCLIP's step size such that the entropy of the prompt template's weights is controlled (Section 3.4). We propose a default entropy reduction factor, which is shared across all the experiments. By this, AUTOCLIP comes essentially without free hyperparameters, which is important as hyperparameters cannot be tuned in zero-shot settings. We evaluate AUTOCLIP on large number of datascts, vision-languagg models, and prompt templates (Section 4 as well as in a controlled setting (Section 5. We find that it improves performance on the vast majority (85%) of settings, by 0.45 percent point accuracy on average, and by up to 3 percent point in some settings. These gains come essentially for free with the only cost being a very small inference time overhcad (sce Scction A.1 in the appendix). as our approach operates entirely in the embedding space. Considering these benefits, we believe that the proposed AUTOCLIP can serve as default zero-shot inference strategy for VL.Ms.
novel procedure
2 Related Work
Vision-Language Pretraining. Deep learning with vision-langual pretraining has enabled zero-shot trans- fer capabilities, i.c., the resulting vision-language models (VLMs) are able to perform zero-shot classification on vastly diverse unseen target datasets given only text prompts of individual target classes. CLIP is one of the state-of-the-ar VLMs pretrained on the well-curated WeblmageText dataset containing 400 million imago-text pairs using a contrastive loss (Radford et al. 2021). In terms of datasots usod, ALIGN requires less dataset preprocessing enabling training on a dataset of over a billion image-text pairs (Jia et al., 2021]) p. Florence Yuua et al. 2021 expands models to other common modalities (e.g., videos). Tn terms of the
3

---

Published in Transactions on Machine Learning Research (08/2024)
training loss, CoCa Yu et al. 2022 leverages additional captioning loss allowing models to be used in generative applications. In our work, we study how to optimally use text prompts of the target classes with these VLMs.
Prompt Construction. Conventionally, one or several mamually designed text prompts per target class are employed for zero-shot classification (Radford et al. 2021 Jia al., 2021). Recent research demonstrates that introducing additional prompts can improve overall performance. DCLIP (Menon & Vondrick 2022) generates additional prompts based on querying the large-language model GPT-3 (Brown et al. 2020). WaffteCLIP Roth et al. 2023 has shown that classification performance can be further boosted by appending random words or characters to predefined prompt templates. To derive zero-shot classifier, these works weight all text prompts uniformly. In contrast, we propose an approach to adjust weights of individual prompts per input sample dynamically at test time.
Test- Time Adaptation. Our work can be considered as a test-time adaption approach for VLMs. TENT (aan et al. 2020) demonstrates that adapting models to minimize prediction entropy can improve model optimizes prompts of target classes demonstrates that minimizing the entropy objective can lead to overfitting under distribution shift and proposes adaptation based on average CLIP scores. In contrast to these previous works, we do not perform any adaptation of prompts or model parameters, but refine weights of individual (encoded) prompts, which considerably cheaper in terms of computation and memory consumption. Most similar to our work is Zero-shot Prompt Ensembling (ZPE) [lliinhham al., 2023). which also determines prompt weights in embedding space. However, ZPE requires entire batch of target domain samples and the availability of image features representing the feature distribution in pre-training ("source domain"). In contrast, our work operates on single images in source-free setting.
performance at test time. In the context of VLMs, TPT (Shu et al. based the entropy minimization objective. RLCF (Zhao et al.
2022
2023
3 AutoCLIP
We outline the common approach for building zero-shot classifiers for VLMs like CLIP in Section 3.1 Thereupon, we detail our proposed AUTOCLIP as an auto-tuned alternative in Section 3.2 describing how the required gradient can be calculated in closed-form in Section 3.3 and finally explain how AUTOCLIP's step size can be automatically determined in Section 3.4
followed by
3.1 Background: Zero-Shot Classifiers for Vision-Language Models
Let us consider a classification task X ++ C, where X' corresponds to the input domain and C fci.... .......... .ch is a set of C classes. We assumee that there exists a pretrained VLM such CLIP that provides joint embedding space & and corresponding embedding functions Fx X & that maps input data x X into embedding space È and ET T + & that maps text into the same embedding space E. Let there be K prompt templates ti....tx: : C D that map class name c E C to (textual) class descriptors d E T. These prompt templates can be either manually designed (Radford et al., 2021 ; generated by a large language model (Menon & Vondrick 2022), or randomly generated (Roth et al.] 2023). Algorithm summarizes the standard zero-shot classifier for VLMs: average the class descriptor encodings e(d) into class queries Oj, then compute cosine similarities sy between class query and encoded image e(*), and classify to the class that maximizes similarity.
3.2 Auto-Tuning Zero-Shot Classfiers
AUTOCLIP modifies Line 7 in Algorithm 1 Instead of computing class querics as simple average of class descriptor encodings 9, 11/K Σ 6ig AUTOCLIP uses weighted average: q; = Σ wxet with tunable w, satisfying w; 2 0, Σx Wi = 1, which we enforce by reparameterizing w = softmax(p) and R* AUTOCLIP's guiding intuition (see Figure [] is to assign higher weights w; to prompt templates t2 that (d)
(d)
(d)
result. in class descriptor encodings eg that are more similar to the encoded image e(a), that is: 4 with (zd) (d)
large (i E/j _e(x) (j 1,..., ,). This inspired by the observation that class descriptors having higher
4

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm 1 Zero-Shot Classifier for a single sample
D Generate K C' class descriptors
d (tz(cy) | E {1...., ",....,K,,, E {l....,CT)
3: & Encode image of interest æ with VLM
4: exx) + (x
5: P Encode all class descriptors with VLM
(d)
6: Eij <-
7: w; +1/K
D. Uniform prompt template weights
8: for jEl...., do
9:
> Class queries as average class descriptor encodings
9j + - Wi (d) (d)
10:
11:
> Cosine similarity between el*) and class query Lj
12: 8j e)x) 4j
13: end for
14: D Assign x to class cj. with maximum similarity
15: j t arg max, 83
similarity in the embedding space describe the image better (according to contrastive pretraining objectives in typical VLMs). In practice, AUTOCLIP tuncs w on a per-sample basis by one step of gradient ascent on a logsumexxp-asee objective function, which we detail below.
When determining the template's weights we have C descriptor-image similarities (ad) for each template 4. AutoCLIP needs to aggregate those C' similarities across classes when assigning larger weights to more relevant prompt templates. Intuitively, simply averaging all C similarities ("mean" aggregation) ignores that, in the classification objective, we ultimatcly only care about classes that result the descriptors closest e(*); however, laking only the class with highest similarity per template into account ("max" aggregation) ignores inherent ambiguity in the image and was found to be suboptimal (Roth et al. 2023 We propose middle ground of aggregating via a smooth approximation to the maximum function via logsumexp,(e_) (((d))
log E exp (zd) This logsumexp aggregation takes all classes into account but assigns higher importance to more relevant classes (ones resulting higher similarities the image æ)). AUTOCLIP then determines weights such that logsumexp, = 1ogsumexp;(E1: weei (ad) = (sollme) eng (gdd)) gets increased by one step of gradient ascent in the direction of Vp logsumeex (softmaxx))))) eng that logsumexp has been interpreted as the energy function of a data point (for appropriately trained classifiers) (Grathwohl et al. 2020); in this view, AUTOCLIP can be interpreted as minimizing the energy and maximizing the probability density p(z) of æ under the zero-shot classifier (see Section A.4 for more details).
(ad), ). We note
We summarize AUTOCLIP in Algorithm 2 We initialize p 0, which corresponds to unweighted average of the class descriptor encodings (Line 8). Similar to Algorithm 1 compute the pairwise cosine similarities 8, between encoded image e(*) and class queries qj (Line 9-14). Instead of directly classifying to the class with maximum similarity to the image, AUTOCLIP updates the class descriptor weights first. For this, the gradient is computed (Line 16), an appropriate step size a is selected (Line 18, see Section 3.4), and p= a- and w = (pftmaxx()) are updated (Line 20). Based on the new w, AUTOCLIP computes updated class queries lj and class-image similaritics (Line 21-26) and finally selcets the class with maximum similarity for the image (Line 28). It worth emphasizing that AUTOCLIP is permulatioon-nnvarianl in the prompt templates t,
= Vp logsummmxp
We note that Line 9-20 could be repeated for several itcrations with smaller step sizes; however preliminary experiments indicate no advantage of doing more than one ileration. We call AUTOCLIP "auto-tuned" because its weights w are automatically adapted for every input independently. Moreover, we note that in practice, models like CLIP scale e(ad) by a learned temperature (exponential logit scale) 7 obtain well calibrated classifiers; we use the same temperature for scaling (add) in the logsumexp aggregation (as there is labelled data in zero-shot setting on which temperature could be tuned).
5

---

Published in Transactions on Machine Learning Research (08/2024)
Algorithm AUTOCLIP: Auto-Tuned Zero-Shot Classifier for single sample
D Generate K C' class descriptors
d (tz(cy) | E {1,...,K), E [1,...,C]
& Encode image of interest æ with VLM
4: e(x) + Exx
5: P Encode all class descriptors with VLM
bi < -rr (r 222
(d)
7: & Uniform weights w; =
8: p=0;; 0; w; + softmax())
9: for j E L.....,C do
10: > Class queries as average class descriptor encodings (d)
11:
+ E Wi
12: Cosine similarity between e(#) and class query di
13: 8j 5 4j
14: end for
15: & Compute gradient (Section 3.3
16: g + Vplog Ei= exp(s;)
17: & Determine stepsize (Section 3.4
18: + BISECT -ISEECT(m__ctrropr/ cattiimeettopyare 9) Blog2 K, 00
19: & Update with one gradient ascent step and step size a
20: p at-g. g; f (x-ssoftmx(p))
21: for vEl....,C
22: > Class queries as average class descriptor encodings (d)
23: 9j + - WiCif
24: Cosine similarity between el*) and class query 9j
25:
e(x) .g
26: end for
27: Assign to class cj. with maximum similarity
28: j arg max, 83
3.3 Closed-form Computation of Gradient
While Vp logsummcxp((s can be casily computed using automatic diffcrentiation, we note that there can be runtime environments for inference such as edge devices where rumning automatic differentiation is undesirable. For such cases, the gradient V logsumexp, can also be computed in closed-form: (V, (ogsumexp,,s;)), E Softssofaaxx(x big function with 0,; and b.j = for
((gd)) wr(bi wx), with &3 being the Kronecker delta
3.4 Auto- Tuning the Step Size
The only free hyperparameter of AUTOCLIP the step size a. We note that in a zero-shot setting, there is by definition no labeled data on which such free hyperparameters can be tuned. Because of this, free hyperparameters need to be selected globally in dataset-indeeendeet manner. However, global choice for the step size is problematic since the scale of the gradient g V, logsumexp(s) depends on the dataset, and the step size would have to be adapted accordingly. We address this by proposing different parameterization in which the free hyperparameter is casily interpreted and the step size is a derived quantity. Specifically, we control the entropy of the query weights entropy(w)) = Σí w; log, wi- The standard, uniform weights have maximum entropy log, K and we set the target entropy to 8 log, K, where the entropy reduction factor I E [0, 1] the new free hyperparameter that we set globally 3 0.85. Intuitively, / 1 corresponds more equally weighted prompt templates while 3 to selecting the prompt template with maximum similarity. We present ablation of the effect of B's choice AUTOCLIP in Figure

---

Published in Transactions on Machine Learning Research (08/2024)
CLIP
CLIP
CLIP
CLIP
DataComp
CoCa
RN50
VIT-B-32
ViT-B-16
ViT-L-14
ViT-L-14
ViT-L-14
CUB200
47.75 (+0.5)
52.84
(+0.7)
57.12
(+1.3)
64.43
(+0.7)
84.79
(+0.8)
73.90
(+0.6)
Au
34.95 (-1.2)
46.16
(-0.7)
55.93 (+1.4)
55.09
(+0.6)
65.09
(+1.8)
54.77
(-0.4)
Food101
80.26 (+1.4)
84.13
(+1.3)
88.85 (+0.9)
93.71 (+0.4)
94.52
(+0.3)
90.46 (+0.4)
Oxford Pets
83.09 (+2.6)
85.63 (+2.9)
85.89 (+1.9)
91.64 (+0.9)
92.82
(+0.9)
92.03 (+1.2)
ImageNet
60.42
(+0.6)
63.80 (+0.6)
68.70 (+0.5)
75.89 (+0.3)
79.07 (+0.0)
75.63 (+0.2)
ImageNet V2
53.44
(+0.4)
56.49
(+0.8)
02.54(40.)
70.17 (+0.4)
72.21 (+0.2)
68.08 (+0.1)
ImageNetR
29.32
(+0.9)
51.04 (+1.0)
59.13 (+1.0)
73.98 (+0.4)
78.85 (+0.6)
75.59 (+0.8)
Table 1: Accuracy of AUTOCLIP (and A Accuracy to bascline zero-shot classifier in parenthesis) for K 100 WaffleCLIP prompt templates across models and datasets, averaged over runs.
With sm (aneropppy(aa g) denoting the entropy of the weights w = softmax( g), selecting the step size a is now equivalent to solving for f(a) = for f(a) = sm entropy - log, K. As sm entropy(a g) monotonically decreases with a, we use bisection on a E [0. 10 for finding with f(a) 0. We note that sm entropy(O g) = log, K and thus f(0) for all < 1; similarly, sm entropy(a g) for a 1010 in all settings considered and thus f(10'9) < 0 for all 3 > 0, which together satásfies the prerequisites for running bisection. The additional bisection has little overhead compared to the cost of encoding the image z with Eg (see Section A.1 in the appendix for details).
4 Experiments
Experimental Setting In this section, we compare AUTOCLIP to standard zero-shot classifiers wide range of zero-shot image classification benchmarks and a variety of settings. We conduct experiments
on the datasets CUB200 (Welinder et al. 2010), EuroSAT (Helber et al.) 2019 Food101 [Bossard et al.) 2014), Oxford Pets (Parkhi et al. 2012), ImageNet (Russakovsky et al., 2015). ImageNet (Kornblith et al., 2019), ImageNet-R (Hendrycks et al. 2021), and ImageNet-C (Hendrycks & Dietterich, 2019). We study six different vision-language models: from CLIP (Radford al.) (021). we use ResNet-50 (RN50) (He al. 2015) and vision transformer (VIT-B/32, ViT-B/16, and ViT-L/14) model variants (Dosovitskiy al.) [202]] Moreover, use the VIT-L/14 model variant from DataComp Gadre et al. 2023 and the one trained with
CoCa (Yu et al. 2022).
Additionally, we study three ways of generating prompt. templates: 1) using the 80 manually designed templates from Radford et al. (0221 (CLIP), 2) templates based on querying a large-language model (DCLIP) (Menon& Vondrick 2022), and 3) templates that append random words or characters to predefined prompt templates (WafteCLIP) (Roth al. 2023). We vary the number of templates from K to K 500; if there is a fixed number of templates available such as in CLIP/DCLII, templates sampled with replacement. To account for randomness in the template construction/sampplin
report results averaged over 7 runs. from Roth et al. 22023
We base our implementation on htttt::////tt htttps://githul htwp///spllaainabeeeetaaaaaaaaaa
and highly appreciate their code release under a permissible license. We report the difference of accuracy of AUTOCLIP compared to the bascline zero-shot classificr with uniform prompt template weights ('A Accuracy"). Absolute performance across different datasets and VLMs is shown in Table (and in Table and Table in the appendix).
Results We present the main results in Figure 2 Overall, the figure contains 990 different combinations comparing AUTOCLIP with the bascline; AUTOCLIP is better in 840 cases (~2 85%) and on average it is better by 0.45 percent point accuracy. We also observe a trend that for larger number of prompl. templates K, the advantage of AUTOCLIP (A Accuracy averaged across datasets, models and CLIP CLIP/DCLIC /DCLIP/ WWafteCLIPP increases: from A = 0.06 for K over A 0.33 for 10 and A 0.49 for K = 50 to A = 0.57 for K 200. When aggregating over models, datasets and number of prompt templates, AUTOCLIP achieves the largest average improvement for CaIfleCL.IP (A = 0.61), but still improves for CLIP (A 0.40) and
7

---

Published in Transactions on Machine Learning Research (08/2024)
CLIP RN50
CLIP VIT-B-32
CLIP ViT-B-16
CLIP VIT-L-14
CLIP DataComp VIT-L-14
CoCa ViT-L-14
DCUP
CUB200
1.0
WaffleCLIP
0.5
EUroSAT
Food101
1.0
0.0
A Accuracy
Oxford Pets
0.75
FErLi
ImageNet
0.50
0.00
-0.25
ImageNetV2
0.5
0.0
2.0
ImageNetR
1.5
1.0
0.0
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
10a
10'
10a
10a
#Prompts
#Prompts
#Prompts
#Prompts
#Prompts
#Prompts
Figure 2: Accuracy improvement (A Accuracy) of AUTOCLIP over baseline zero-shot classifier across models, datasets, and prompt ensembles. Shown are mean and standard error over runs.
8

---

Oxford Pets Published in Transactions on Machine Learning Research (08/2024)
CLIP RN50
CLIP ViT-B-32
CLIP VIT-B-16
CLIP VIT-L-14
CLIP DataComp ViT-L-14
0.50
Accuracy
0.25
DCLIP
-0.25
WaffleCLIP
-0.50
2
3
5
1
2
2
5
2
Severity
Severity
Severity
Severity
Severity
Figure 3: ImageNet-C C accuracy improvement (A Accuracy) of AUTOCLIP over baseline zero-shot classifier
for K 100 across models, corruption severity and prompt ensembles, averaged over corruptions and 7 runs.
2.5
CUB200
EuroSAT
2.0
Food101
A Accuracy
ImageNet
1.5
ImageNetR
ImageNetv2
1.0
Oxford Pets
Average
0.5
0.0
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Target Entropy Rate
Figure 4: Ablation on target entropy rate B. Shown the accuracy improvement (A Accuracy) of AUTOCLIP over baseline zero-shot classifier for CLIP ViT-B-16, and 100 WaffleCLIP prompt templates, averaged over runs.
DCLIP (A= 0.29). Taken together, the findings indicate that AUTOCLIP benefits from larger (increased K) and more diverse (WaffleCTIIP) sets of prompt templates.
When comparing different vision-language models, AUTOCLIP brings the biggest benefit for CLIP VIT-B-16 (A 0.68) and the smallest one for CoCa ViT-L-14 (A 0.19), with all other models having average A between 0.36 and 0.52. Comparing different datasets, AUTOCLIP performs strongest
(A 1.15) and worst on EuroSAT (A -0.24); we hypothesize that this is because EuroSAT is in general a challenging dataset for CLIP on which the image encoder produces embeddings that are not very informative about image properties, which deteriorates the prompt weight selection as it becomes harder to decide which prompt describes an image of interest well. We note that EuroSAT the only setting on which AUTOCLIP hurts performance on average; on other datasets, AUTOCLIP improves performance: (CCUB200) 0.5, A (Food101) 0.52, A A(TmageNet) 0.17, A(ImageNetV2) 0.2, and A(ImageNetRR- 0.71. While these

---

the same (unknown) class but vary across Published in Transactions on Machine Learning Research (08/2024)
2.0
Objective function logsumexp entropy
1.5
A Accuracy
1.0
mean
max
0.5
0.0
-05
-1.0
Oxford Pets
EuroSAT
CUB200
Food101 Dataset
ImageNetV2
ImageNetR
ImageNet
Figure 5: Comparison of different objective functions for auto-tuning. Shown is the accuracy improvement (A Accuracy) of AUTOCLIP over baseline zero-shot classifier for a ViT-B-16, and 100 WaffleCLIP prompt templates, averaged over runs.
improvements are of modest magnitude, they essentially come for free by a mere change of the inference procedure.
In Figure 3 we present results on ImageNNtt-C for WaffeCLIP with K 100 for different severities and averaged across corruptions. AUTOCLIP consistently improves performance for the smaller vision- languaee models (RN50, ViT-B-32, ViT-B-16) and sees minor drop of performance for the two ViT-I--14 variants. Averaged across all models, corruptions, and severities, AUTOCLIP improves performance by A 0.11. We provide plots for each corruption separately for WaflleLIIP prompt templates in the appendix in Figure g The biggest average benefit of AUTOCLIP is obtained for the wow-freuenccy corruptions "saturate" (A = 0.22),
"brightncss" (A = 0.22), and "contrast" " (A = 0.23); the smallest average benefit for "shot-noise" (A = 0.05)
and "snow" " AA = 0.06).
Ablations We ablate AUTOCLIP's choice of the target entropy rate B (which defaults to 0.85) and the objective function (defaults to logsumexp). In Figure | we observe that AUTOCLIP's performance for most datasots does not depend strongly on the specific choice the target entropy rate 3 as A Accuracy stays relatively constant the range 3 E [0.7,0.9]. This desirable property as zero-shot setting without labeled data, tuning 3 per dataset would be infeasible. For two datasets (Oxfort Pets and EuroSAT), our default value of 3 = 0.85 was suboptimal and a considerably smaller choice of 0.7 would have obtained considerably better results. Also average, 8 = 0.7 performs favorably and recommend this choice for future work on other datasets and tasks. We provide results for similar experiment in which we directly control the step size a in Section A.5) Directly controlling a reduces computation overhead further, but optimal choices of step size a vary more strongly across datasets than choices for the target entropy rate B.
We motivated the choice of logsumexp as AUroCLIP's striking a good compromisc between max and mcan aggregation. In Figure 5 we empirically confirm that the logsumexp aggregation performs favorably compared to max/ mmaa aggregation on all datasets. Moreover, it also outperforms entropy aggregation, which is popular choice for test-time adaptation (Wang al. 2020
cnarreaationooojccti function in Section 3.2 as
Shu et al. 2022)
In Figure 6 we show the prompt template weights (K = 30) obtained by AUTOCLIP on 500 Food101 samples. Samples are structured 10 blocks 50 samples each, where each block corresponds to one class. Prompt template weights are relatively similar for instances belonging
classes. Some templates like the ones starting with " "A tattoo of.." or "A drawing of..." " get consistently low weights the images of the Food101 dataset do not look like tattoos origami, while templates starting with
10

---

Published in Transactions on Machine Learning Research (08/2024)
0.25
tattoo the
photo
dark phoco plastie
drawing the
0.20
The cartoon
A photo the rice
platbie joog corruptes shoto
photo of walrd
A rendition of A photo of the ciean
low resolution photo of
close-up photo the
rer dering the
A (pog comuuttd phota af
crapped photo
bright photo
0.05
photo A bas photo the Graftti of
A photo of the hard to see
A sccupture of
0.00
chums
not and your soup
lamosit
deshimi
park chop
mrong relp
nonna cotaa
best Lerraee
graek Salad
tale gras
Figure 6: Illustration of prompt template weights w on 500 samples from the Food101 dataset, with blocks of 50 samples belonging to the same (unknown) class. CLIP backbone is ViT-B-16 and 30 CLIP prompt templates are used.
"A photo of... tend to get higher weights, Food101 contains mostly actual photos. Note that the weight distribution looks different on other datasots like ImageNet-R, RR, with higher weights for "artistic" prompts (see Figure in the appendix). Overall, this confirms that AUTOCLIP can adapt the zero-shot classifier on the fly to properties of the respective image. Moreover, AUroCLIP provides accuracy improvements with only minor additional inference overhead as discussed in Section A.l in the appendix.
5 Analysis in a Controlled Setting
We study AUTOCLIP in controlled setting, in which we directly sample embedding vectors in embedding space corresponding to encoded image and class descriptors, without actually encoding images text prompts. By this, we can control key propertics of the embeddings and study how they influcnce AUTOCLIP's performance. While the setting is strongly simplified, we will nevertheless gain some insights that provide possible explanations for some of the key findings from Section Code for reproducing the results of this section is available attps://githur cwt. hoom//bbschheeearc/aaattooo autocipp
We set the mumber of classes to C 5, the number of embedding dimensions to d 128, the mumber of prompt templates to K = 10, and the number of instance to 200. We sample class descriptor embeddings (dgg) as follows: let C ~ (o 1, 1, E Rd be C' d-dimensional standard normal-distributed class means, let Pa (O, 1, d) E Rd be K d-dimensional standard normal-distributed prompt embedding means, and V. ~ (O, 1,
E Rd be KC d-dimensional standard nornal-distributed prompt-class coupling terms. We then sct Er,d)) (1- p) + pv where p E [o, 1] controls the "entanglement" between class and prompt template embeddings. Intuitively, p = 0 simulates a setting in which the VLM's text encoder perfectly separates class and prompt template related information such that their combination is additive. Increasing results in stronger entanglement such that class and prompt template-related information interact more strongly and are no longer additive. We then set the image embeddings corresponding to Er(di) to Ex (x) = (d4y) + & with ~ (0, d). Here & controls the "instance noise", that is: how much the image embeddings are spread around the corresponding class descriptor embedding.
Figure compares the accuracy of mean-, max- and AutoCLIP-aggregate (5 = 0.85) for different values of entanglement, and instance noise In general, lower entanglement. favors mean aggregation (the
11

---

Published in Transactions on Machine Learning Research (08/2024)
Instance Noise: 2
Instance Noise: 5
Instance Noise: 10
0.8
0.50
0.9
0.7
Accuracy
Accuracy
Accuracy
0.45
0.8
0.5
0.35
max
AutoCLiP
0.4
0.30
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Entanglement
Entanglement
Entanglement
Figure 7: Comparison of AUTOCLIP to "mean" aggregation (Line 10 in Algorithm and "max" aggregation (sy +- max; cij 93) and "softmax" aggregation (which determines weights based the respective image- prompt template similarity, see Section A.3 in a controlled and simplified setting. "Instance Noise" controls how strongly instance encodings vary from their respective mean. "Fntanglement" controls how strongly the text encoding of prompt template and class name are entangled. Shown is mean over 100 random seeds.
(d)
standard in CLIP), while higher entanglement favors max-aggregation. Usually, VLM text encoders are good terms of disentangling concepts, which explains why prior work (Roth el al. 2023) has found max-aggregation to perform inferior compared to mean-aggregation. On the other hand, we observe that AUTOCLIP nearly always outperforms max-aggregation (except for the small instance noise and strong entanglement setting), also outperforms mean-aggregatiin for moderate entanglement (p 0.4), and performs very close to mean-aggregation for smaller entanglement.
This provides a possible explanation for the findings from Figure 2 for smaller (and weaker) VLMs, the text embeddings are more entangled and thus AUTOCLIP provides stronger benefits compared to standard mean-aggregation. For larger VLMSs like ViT-I-14 based ones, the entanglement decreases and thus the benefit of AUTOCLIP smaller. Morcover, for small entanglement and large instance noise, AUTOCLIP can also be slightly worse than mean-aggregation in Figure 7 This settings likely corresponds to the ViT-I-14 (low class-prte prompt entanglement) performs slightly worse than mean-aggregation. However, in general Figure 7 indicates that AUTOCLIP provides a favorable choice compared to mean- and max-aggregation for many practical settings.
ImageNet-C (large instance noise) in Figure 3 where AUTOCLIP also
6 Conclusion
We have proposed AUTOCLIP, a method for improving zero-shot classifiers on vision-language models. It automatically tunes per-image weights of prompt templates before aggregating them into class queries. AUTOCLIP improves performance over standard zero-shot classifiers on the vast majority of settings, with only minimal inference-time overhead. We believe that due its simplicity and low cost, AUTOCLIP has the potential be broadly applied in conjunction with vision-language models. For future work, it is exciting explore if AUTOCLIP also benefits other zero-shot tasks built top of multi-modal modals such as object detection with OWL-VIT Miiderer al. 2022 multi-modal prompting with ImageBind GGirdaar et al. 2023a). Moreover, exploring the potential of AUTOCLIP for few-shot classification is an interesting direction; we present initial promising results in Section A.6
12

---

Published in Transactions on Machine Learning Research (08/2024)
References
James Urquhart Allingham, Jie Ren, Michael W. Dusenberry, Xiuye Gu, Yin Cui, Dustin Tran, Jeremiah Zhe Liu, and Balaji Lakshninarayanan. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. In Proceedings of the 40th International Conference on Machine Learning,
TCML,23,, 2023.
Jeremy Atkinson.
Car beurre eta.
https://openverss wrg.://eeeereeeeeeoeeeeo /
5d960316-32000 300000000000000--544458888010000
2015. Accessed: 2023-09-27.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446 461. Springer, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL http::///ppparroeeeeeeeeettttrrs
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets, 2023.
Rohit Girdhar, Alaacldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVI /CFF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15180 15180-15190, June 2023a.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 15180 15190, 2023b.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2020. URL htttp:://operreviee net/ffrrummidd-Hkozx00tbb
Kaiming Hc, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. 2016 IEEE Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 770 778, 2015. URL https://api. semanticscholar. wwwpsschhoaar.rro//oooooo
wttp:/o//oo1:10005505000/. 206594692
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. TEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217 2226, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021.
Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hieu Pham, Quoe Lc, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
13

---

Published in Transactions on Machine Learning Research (08/2024)
Simon Korublith, Jonathon Shlens, and Quoc Le. Do better imagenet models transfer better? In Proceedings of the IERF/CVF conference on computer vision and paltern recognition, pp. 2661-2671, 2019.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint ar Xrx:22210.0000 070118 2022.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Amurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulayy object detection with vision transformers. arXiv preprint ar UrXiv:2205.06230 2022.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498 3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Karsten Roth, Jac Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. ar preprinl arXiv:2306.07288 2023.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211 252, 2015.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Newral Information Processing Systems, 12444-44288) 2022.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weekesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Îmaa Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10.1038/ 00.108/s411990 00/////4112-99-90660
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint 2rX 2020.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXim:2205.0190 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprinl Xrx:::2111 11. 1tf::211..1142,, 2021.
Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free free adaption of clip for few-shot classification. In ECCV'22, pp. 493 510, 2022.
Shuai Zhao, Xiaohan Wang, Linchao Zhu, and Yezhou Yang. Test-time adaptation with clip reward for zero-shot generalization in lanion-l models. ArXiv, abs/2305.18010, 2023. URL https://api
semanticscholar. .o../////oooo /CCCCoosI:::89559000
14

---

Published in Transactions on Machine Learning Research (08/2024)
A Appendix
A.1 Inference time overhead of AutoCLIP
In this paragraph, provide some measurements on inference time overhead by AUTOCLIP. We provide mumbers for the case of ViT-L-14 on the Oxford Pets dataset. Here, encoding an image takes 12.64ms on a V100 (minimum over 100 images). The baseline "averaging" zero-shot classifiers takes additional 0.08ms (average over 640 samples) on top to classify a sample. AUTOCLIP takes additional 1.54ms (average over 640 samples) for classification when rumning bisection for autotuning the step size. For a fixed step size, the overhead of AUTOCLIP 0.45ms. Thus, AUTOCLIP with autotuning raises inference time from 12.64ms 14.18ms. In contrast, TPT (Shu et al., 2022) and RLCF (Zhao et al., 2023), which did not report compute memory requirements, require encoding multiple image augmentations. TPT states "We augment single test image 63 times using random resized crops and construct a batch of 64 images, including the original one." ' which mcans that the image encoding time (for instance the 12.64ms from above) is increased by a factor of 64x, plus additional overhead for backpropagating through the text encoder, which likely brings the inference time per sample close to (or more if multiple test-time adaptation steps are conducted). We note that for bisection, we use an independent call scipy. ccipy.optin optimize. bisect (Virtanen al. 2020) (maxiter=100, xtol=1e-2, rtol=1c-2). A batched variant of biscction could speed-up many workloads.
A.2 Additional Experimental results
We present additional experimental results. Table and T show the absolute performance of AUTOCLIP on different datasets and VLMs for DCLIP and CLIP prompt templates, respectively, similar to Table in the main paper for WaffleCLIP templates. Figure iillustrates prompt weights on the ImageNetR dataset. Figure contains results of AUTOCLIP in terms of A Accuracy on ImageNetC for every corruption seperately.
In Figure 10 we show an additional comparison of AUTOCLIP to a stronger baseline which is based on TopR aggregation. In this TopR aggregation, for each image R prompt templates are selected whose resulting encoded class descriptors have maximum average cosine similarity to the encoded image. We note that choosing R non-trivial in a zero-shot setting due to the lack of labelled validation data. In the figure, we compare AUTOCLIP against this TopR-CLIP for K 100 DCLIP prompt template, across the same VLMs and datasets in Figure 2 We provide results for different choices of R: overall, for the best choice of R 20, AUTOCLIP is better on 86% of the cases and by 0.40 percent point accuracy average.
A.3 Softmax aggregation
Natural basclines for AUTOCLIP are "mean" aggregation (s, + 1/K 6ij 9j as in Algorithm and "max" aggregation (s3 + max, %ij 9j). The former corresponds to w; = 1/K and the latter to Wi- = 1 and = 0 Vi # i' with i* = arg max, Cij -qj. Naturally, it makes sense to also consider alternative "in-between" choices for w that are (in contrast to AUTOCLIP) not based on the gradient of some objective functions.
(d)
CLIP
CLIP
CLIP
CLIP
DataComp ViT-L-14
CoCa
RN50
VIT-B-32
ViT-B-16
ViT-L-14
ViT-L-14
CUB200
47.75
(+0.1)
53.00
(+0.4)
57.82
(+0.3)
64.57
(+0.3)
85.38
(+0.4)
73.69 (+0.1)
EuroSAT
36.39 (-1.2)
45.88 (-2.2)
59.22
(+2.5)
57.89
(+0.8)
60.08 (-0.7)
57.15 (-1.2)
Food101
79.12 (+0.9)
83.43
(+0.7)
88.53 (+0.5)
93.14
(+0.4)
93.89
(+0.2)
89.77 (+0.3)
Oxford Pets
85.92 (+1.3)
87.11
(+1.0)
88.53 (+0.9)
94.08 (+0.6)
94.00 (+0.4)
93.54 (+0.4)
ImageNet
60.62
(+0.3)
63.89 (+0.2)
69.10 (+0.3)
75.92 (+0.1)
79.02 (+0.0)
75.41 (+0.0)
ImageNet V
53.60
(+0.3)
56.73 (+0.5)
62.22 (+0.2)
70.01 (+0.1)
71.95 (-0.0)
67.91 (-0.0)
IeageNet 28.14
(+1.3)
49.51 (+1.6)
58.37 (+1.7)
73.12 (+0.6)
78.06 (+0.7)
73.73 (+1.1)
Table 2: Accuracy of AUTOCLIP (and A Accuracy to baseline zero-shot classifier in parenthesis) for K 100
DCLIP prompt templates across models and datasets, averaged over 7 runs.
15

---

Published in Transactions on Machine Learning Research (08/2024)
photo the
0.30
dark plushie drawing
3.25
plushie
joeg comupted
0.20
rendition
photo
0.15
origemi
close-up photo
0.10
loeg corrupted
cropped
bright photo the
pointing
3.05
photo my
photo
hard
sculpture
3.00
mushrosm
drawberr
arad
Vombooo
costa
vulturr
sabmarie
Chow Chow
Figure 8: Illustration of prompt template weights wo on 500 samples from the ImageNetR dataset, with blocks of 50 samples belonging to the same (unknown) class. CLIP backbone is a ViT-B-16 and 30 CLIP prompt templates used.
One such choice is the "softmax" weighting w softmax ( (II//C, ci 4)), which assigns higher weight to prompt templates 4 whose resulting class descriptors more similar to the image on average. Here the temperature controls the sharpness of the distribution. Instead of tuning directly, determine per example via bisection such that specific target entropy of is obtained (analogously to AUTOCLIP with f = 0.85).
We compare the different aggregation methods to AUTOCLIP in Table "softmax" aggregation often (but not always) outperforms "mean" and "max" aggregation. However, AutoCLIP clearly outperforms "softmax" aggregation, being superior softmax on all datasets except for ImageNetR, where both are nearly on par.
We hypothesize that while average image-prompt template alignment. contains useful information, directly determining weights based on it (such as in "softmax" aggregation) is problematic because it does not handle enlanglement between class and prompt lemplates (as defined in Section 5 in our paper) well. Indeed, when evaluating "softmax" aggregation in the controlled setting from Figure we find that it nearly always
CLIP
CLIP VIT-B-32
CLIP ViT-B-16
CLIP ViT-L-14
DataComp ViT-L-14
CoCa
RN50
ViT-L-14
CUB200
47.00
(+0.3)
52.36
(+0.7)
56.99
(+1.2)
63.94
(+0.5)
85.52
(+1.1)
73.99 (+0.1)
EuroSAT
32.28
(-3.7)
44.78 (-1.3)
56.76
(+0.4)
52.96
(+1.8)
61.94
(+1.4)
51.58 (-1.7)
Food101
79.69
(+1.1)
83.64 (+0.9)
88.83 (+0.6)
93.33
(+0.2)
94.55
(+0.3)
90.36 (+0.3)
Oxford Pets
84.30
(+1.7)
85.20
(+2.0)
88.42 (+0.9)
93.24
(+1.2)
93.79
(+1.3)
92.67 (+1.3)
ImageNet
59.90
(+0.2)
63.31
(+0.3)
68.43 (+0.2)
75.38
(+0.1)
79.29
(+0.1)
75.79 (+0.2)
ImageNet V
52.98
(+0.5)
56.00
(+0.4)
62.12 (+0.2)
69.56 (-0.1)
72.09
(+0.0)
67.90 (-0.0)
IeageNet 27.11
(+0.9)
47.74 (+0.9)
56.28 (+1.1)
71.30 (+0.4)
78.26 (+0.5)
74.51 (+0.9)
Table 3: Accuracy of AUTOCLIP (and A Accuracy to baseline zero-shot classifier in parenthesis) for K 100
CLIP prompt templates across models and datasets, averaged over 7 runs.
16

---

Published in Transactions on Machine Learning Research (08/2024)
brightness
contrast
defocus_blu
1.0
Accuracy
CLIP RN50
0.5
0.0
V-T-B-32
CLP
-0.5
CUP
V/T-8-16
1.0
CUP
VIT-L-14
2
4
5
2
3
DataComp
VIT-L114
elastic_transfon
fog
frost
1.0
4 Accuracy
0.5
0.0
0.5
-1.0
3
3
gaussian_blus blur
boassian_noisse
glass_blc blur
1.0
A Accuracy
0.5
0.0
-0.5
-1.0
3
impulse_noise
jpeg_compression
motion_blus
1.0
Accuracy
-0.5
-10
saturate
shot noise
snow
1.0
Accuracy
0.5
3
spatter
speckle_noise
foom_blus
1.0
Accuracy
0.5
0.0
0.5
2
4
3
Severity
Severity
Severity
Figure 9: ImageNetC Accuracy improvement (A Accuracy) of AUTOCLIP over bascline zero-shot classifier for WaffleCLIP across models, corruptions, averaged over runs.
17

---

Published in Transactions on Machine Learning Research (08/2024)
CLIP RN50
CLIP ViT-B-32
CUB200 200
Foodl 01
Accuracy
Ima geNet
Ima R
ImageNetV2
4
Oxford Pets
R
R
CLIP
CLIP
ViT-B-16
ViT-L-14
Accuracy
4
1 5 10
20
50
100
1.5
10
20
50
100
R
CLIP DataComp ViT-L-14
CoCa ViT-L-14
6
Accuracy
3
1
1 10
20
50
100
1.5 10
20
50
100
Figure 10: Accuracy improvement (A Accuracy) of AUTOCLIP with K = 100 DCLIP prompt templates over TopR zero-shot classifier with different values of R across models, averaged over datasets and 7 runs.
18

---

Published in Transactions on Machine Learning Research (08/2024)
AutoCLIP
max
mean
softmax
CUB200
57.12
56.19
55.78
55.67
EuroSAT
55.93
55.73
54.53
55.76
Food101
88.85
88.50
88.00
88.65
ImageNet
68.70
67.06
68.20
68.40
ImageNotR
59.13
56.92
58.15
59.17
ImaggNNtt
62.54
60.72
61.95
62.19
Oxford Pets
85.89
87.65
83.99
82.83
Average
68.31
67.54
67.23
67.52
Table 4: Comparison of AUTOCLIP to different aggregation methods for CLIP ViT-B-16 and K=100 WaffleCLIP prompt templates. Shown is accuracy, averaged over 7 runs.
performs neewwennmeane and "max" aggregation (as interpolates between the two) and thus typically worse than AUTOCLIP.
A.4 Motivation of Exg-Sum-Exp in AutoCLIP
3.2 AUTOCLIP determines weights wa such that lsgsumexp,(s,) (ad)) gets increased by one step of gradient ascent in the direction of V, logsumexp, (softmaxx()))) -eiy ((rd). Empirically, we find in Figure 5 that this logsumexp objective function outperfoms alternative loss functions such entropy, mean, max. On the one hand, we attribute this to logsumexp being "in-between" mean and max as as "smooth maximum". On the other hand, we hypothesize that logsumexp is a reasonable choice because its relationship to the energy function of a data point in that view AuroCT.IP can be interpreted as minimizing the energy and maximizing the probability density p(x) of x under the zero-shot classifier. We claborate this relationship in more detail the following paragraph.
As discussed in Section
logsumexp, (E wee, (y)e (essumeep,,ssolmmaa
According to (Grathwohl et al. 2020 for softmax-based classifier that assigns logits fx(x) to class- probablitics via p(y/x)= softmax (f())) one can slightly re-interpret the logits obtained from fo to define p(x, y) and p(x). For this, one re-uses the logits to define an energy-based model of the joint distribution of data point x and labels y via px = exp(Jo(x)ia) where Z(0) is the unknown normalizing constant and
Z(0)
Eg (xx v) (x) is the energy. Moreover, px()) Ey po(x, v) E, exp(fo 2(0) and accordingly the
energy corresponds to Ex(x) = - log E, exp(fo(x)x)))
In the case of AUTOCLIP, the logits fx( correspond to the cosine similarities 8, (with optional temperature scaling). Accordingly, - logsumexp, (sj) can be interpreted as the energy, and AUTOCLIP, which maximizes logsumexp, (sy), can be interpreted a method for minimizing the energy of the zero-shot classifier.
We note that this interpretation works best for classifiers that were trained with an energy-based loss term that encourages low energy on the data manifold and high energy elsewhere discussed by (Grathwohl et al. 2020 However, empirically we find it to also work well on based zero-shot classifiers.
A.5 Step Size Selection in AutoCLIP
AUTOCLIP controls the target entropy rate 3 as free hyperparameter as an alternative parametcrization to directly controlling the step size a. The reason for this is that optimal choices of step size vary more strongly across datasets than choices for the target entropy rate 3. We show this in Figure II (compared to Figure 4 while the reparametrization from a to B does not change the peak performance a dataset much, it affects the alignments of curves across datasets. We observe that the curves are more aligned when varying 3 than when varying a. Quantitatively, the mean pairwise Pearson product-moment
AUTOCLIP on
19

---

Published in Transactions on Machine Learning Research (08/2024)
2.5
CUB200
2.0
EuroSAT
Food101
Â Accuracy
ImageNet
1.5
ImageNetR
ImageNetV2
1.0
Oxford Pets
Average
0.5
0.0
10
102
103
Step Size
Figure 11: Control experiment when directly choosing a fixed step size (and not target entropy rate B)- Shown is the accuracy improvement (A Accuracy) of AUTOCLIP over a bascline zero-shot classifier for a CLIP ViT-B-16, and 100 WafleCLIP prompt templates, averaged over runs.
correlation coefficient between the curves for 33 in Figure ! is 0.66, while is only 0.45 for the curves for a in Figure 11 Thus, zero-shot setting which hyperparameters cannot be tuned on per-dataset basis, the more aligned behavior across datasets for target entropy rate 3 is preferable the the step size
A.6 AutoCLIP for Few-Shot Learning
While this work generally focuses on zero-shot learning settings in which no data from the task is available, it is often possible obtain a small of labeled data points for task. More specifically, we consider a few-shot classification setting in which we have K labeled datapoints per class. We study AUTOCLIP in combination with simple CLIP-based approach for few-shot classification: in this approach, cach image is considered one exemplar of its respective class. Similar to the prompt. template based approaches for zero-shot classification, this results in encoded class descriptors, namely éij for the i-th image of the j-th class. These encoded descriptors be plugged into line 6 of Algorithm 2 as replacements for the prompt template based descriptors. This allows applying AUTOCLIP or any other aggregation method in few-shot. setting.
(d)
Figure 12 shows the results for a CLIP ViT-B-16 on Oxford Pets for different aggregation methods. AUTOCLIP consistently outperforms other methods by a substantial margin or more percent points accuracy, This indicates that AUTOCLIP is a generally beneficial method for aggregating class-lovel information from several queries (lextual or other modalities), which can be used beyond zero-shot classification. We note that while these results show promise for AUTOCLIP in feo- settings, future work will be needed to integrate and evaluate AUTOCLIP in more powerful CLIP-based few-shot, learning approaches such with the training-free Tip-Adapter (Zhang et al. 2022
20

---

Published in Transactions on Machine Learning Research (08/2024)
80 aggregate
Softmax
75
AutoCLiP
mean
70
65
Accuracy
60
55
45
40
30
50
k-shot
Figure 12: Comparison of AUTOCLIP to other aggregation methods on few-shot CLIP-based classification
Oxfords Pets. Average over runs for CLIP ViT-B-16.
21


Reviews:
[{"review_id": "MZYwThnwbm", "reviewer": "Reviewer_M68R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. Good writing\n2. Thorough experiments,  including ablations studies of various parameters and design decisions\n3. Non-trivial performance improvements, with minor computational overhead\n\nCons:\n1. I was a bit confused about the method itself. Logsumexp makes sense, but then the way gradient ascent is applied and that it's just a single step was confusing. There's a note about \"logsumexp has been interpreted as the energy function of a data point (for appropriately trained classifiers) (Grathwohl et al., 2020)\", but it's not possible to understand from the paper what exactly this means. Would be helpful to add a self-contained explanation of where the method intuitively comes from and why it makes sense.\n2. Related to that, a natural alternative that comes to mind is to not do gradient ascent, but just have a temperature on the softmax used to compute the weights for aggreagation. and maybe another temperature in logsumexp. The paper notes that tuning the temperatures is not doable in zero-shot setting, but still those would be interesting baselines and maybe a constant temperature would do.\n3. Performance improvement on average is not large, 0.45%. \n4. (minor) In Figure 6, are the classes correct? It says Food101 and the prompts look food-related, but then the classes are animals. Vulture /quail/iguana with butter/apples/crust?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes AutoCLIP, an approach for improving performance of vision-language models on zero-shot image classification tasks. Specifically, AutoCLIP operates in the (fairly standard) setting where multiple prompts are being evaluated and suggests a strategy for aggregating the image-to-prompt similarity scores across prompts that is better than usual averaging. There is no need for multiple passes through the image encoder, so the method comes with a small compute overhead (~10%). The method is thoroughly evaluated and shown to perform better than simpler alternatives in most settings, by 0.45% on average", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the cons:\n1 (critical) - clarify better where the method comes from and why it makes sense\n2 (very desirable) - add comparisons with baselines with softmax/logsumexp, but without gradient descent\n4 (critical) - fix or explain", "broader_impact_concerns": "No specific concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZHAtO6flkT", "reviewer": "Reviewer_M2vc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths : \n\n[1] The proposed approach is conceptually simple yet effective in improving performance across a diverse range of settings.\n\n[2] The extensive evaluations across multiple datasets, vision-language models, and prompt generation strategies with consistent improvements over the  zero-shot classifier baseline in majority of cases. \n\n[3] The method operates entirely in the embedding space without requiring additional forward or backward passes through the model encoders, resulting in a minimal computational overhead during inference.\n\n[4] The paper provides a comprehensive analysis of AutoCLIP's performance, including ablation studies on the choice of the entropy reduction factor and the objective function, which further enhances the trust in the approach and its wider applicability. \n\nWeakness :\n\n[1] The paper lacks a theoretical justification of why the proposed approach should work better than the baseline. \n\n[2] Although AutoCLIP consistently improves performance in most cases, the gains are relatively modest in some settings, particularly for larger vision-language models like ViT-L/14. It would be useful to understand the factors that limit its effectiveness in these cases and explore ways to achieve more significant improvements. Having a deeper ablation around it will be useful for the community.\n\n[3] Quality and diversity of the prompt used holds the key to the overall performance of this approach. While the approach can adaptively weight the prompts, the authors do not address the issue of generating or selecting better prompt templates, which could further enhance the overall performance.\n\n[4] The evaluation is limited to majorly image classification tasks, it would be valuable to explore its applicability and effectiveness on other vision-language tasks, such as object detection, or segmentation, to demonstrate the broader applicability of the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "AutoCLIP introduces a novel approach to enhancing zero-shot image classification using vision-language models like CLIP. AutoCLIP dynamically adjusts the weights assigned to various text prompt templates for each input image during inference. This adjustment is based on the similarity between the encoded prompts and the encoded image within a joint embedding space.\n\nThe process of determining prompt weights is entirely unsupervised and does not depend on labeled data, making it well-suited for zero-shot settings. It operates efficiently within the embedding space without the need for additional forward or backward passes through model encoders, thus maintaining low computational overhead, which differentiates it from other approaches in this domain.\n\nComprehensive evaluations across multiple datasets, vision-language models, and prompt generation strategies demonstrate that AutoCLIP consistently outperforms baseline methods in the vast majority of settings.\n\nAdditional analysis and ablation studies examine factors such as the impact of the number of prompts, the choice of entropy reduction factor, and comparisons with other aggregation objectives like max, mean, and entropy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "[1] The authors could try to provide a more formal analysis or theoretical justification for why the proposed weighted averaging of prompt embeddings based on their similarities to the input image should lead to improved performance.\n\n[2] The authors should investigate the factors that limit AutoCLIP's effectiveness in certain settings where it fails. Having some insights on these cases will further strengthen the submission.\n\n[3] The authors could explore ways to generate or select better prompt templates that may further enhance the overall performance. This could involve incorporating domain knowledge, leveraging language models, or using other techniques to create more diverse and relevant prompts.\n\n[4] Having some evaluation on other vision-language tasks beyond image classification.", "broader_impact_concerns": "The work does not appear to raise any major ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xx7C2oy3QY", "reviewer": "Reviewer_CaZg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The idea is simple and easy to implement, yet effective. \n- The paper is easy to read.\n- Impressive experimental results on AutoCLIP.\n\nWeaknesses:\n\n- Discrepancy between theoretical understanding and empirical implementation: logsumexp(logit) is interpreted as an energy function. In AutoCLIP, logit = softmax(\\rho) * e. To minimize the energy is to simply update the weights accordingly, while the authors update \\rho with proportion to \\delta-logsumexp. In that sense, I was wondering about the motivation for this empirical implementation.\n\n- Validity of introducing entropy reduction factor \\beta: It sounds valid to introduce entropy reduction factor to free up the hyper-parameter of step size \\alpha. However, when observing Figure 4, I have doubts about this since the entropy for the best accuracy on each dataset differs a lot (e.g. ~0.5 for Pets and EuroSAT while ~0.8 for CUB and ImageNet). In that sense, it would be better to show how robust the results are when fixing (or slightly varying) the \\beta compared to \\alpha.\n\n- Why does AutoCLIP not outperform in out-of-distribution settings on large models, in Figure 3? What could be the hypothesis on this observation?\n\n- How does AutoCLIP work on DCLIP? DCLIP uses different descriptors per class, while AutoCLIP needs to use the same descriptors for every class. I assume, after looking at Figure 6, that the authors collected all the descriptors across the classes and used them in every class, but then this is not what DCLIP proposed. In that sense, I was wondering how AutoCLIP + DCLIP is compared to the DCLIP in Figure 2.\n\n(Minor suggestion)\n- Figure 1 could be improved to better deliver the high-level concepts of AutoCLIP. \n- Figure 2 could also be improved for better readability. For instance, remove small xticks beyond the power of 10, show standard deviation, etc.\n- Reporting exact numbers is important, but Table 1 conveys redundant information considering Figure 2 (if I understood correctly). I would recommend moving Table 1 to the Appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper aims to improve the zero-shot classification performance on vision-language models. To achieve this, the authors propose AutoCLIP, a method for auto-tuning zero-shot classifiers. Given multiple textual prompts, AutoCLIP determines the weights of the prompt features when averaging them based on the prompt-image similarity. To further free up the hyper-parameters when updating the weights, AutoCLIP uses fixed hyper-parameters for the weights to have a certain entropy. The authors evaluate AutoCLIP in 6 different CLIP models on 8 datasets, and show the effectiveness of the proposed method compared to the baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The proposals are mainly described in the Weaknesses. I request the authors to change/add the explanations/experiments/plots mentioned above.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary: this paper proposes a simple approach towards improving zero-shot performance in VLMs like CLIP. The basic idea for models like CLIP is to perform open classification by using a set of prompt templates. Each of these templates provides different scores that can be assembled into a prediction. The authors’ approach swaps out uniform weights on these different scores into weights set by an essentially self-supervised technique. The intuition is to set weights higher for closer distances. 

The reviewers generally liked this paper and were favorable. They provided several useful suggestions that made their way into the paper. I agree with the reviewers on most points. 

One reviewer encouraged the authors to provide a theoretical analysis of the proposed approach. This is likely difficult to do without first establishing some reasonable theoretical model for CLIP, which might require quite a bit of work in an orthogonal direction and should be left for future work. 

I recommend that the paper be accepted. I voted for a minor revision:
- I agree with one of the reviewers that the clarity of presentation in the core Section 3.2 could use a bit of revising. The step of gradient ascent being taken is performed towards maximizing an objective, though it is not initially presented that way. The authors get at explaining this later, but generally the whole section could be better ordered and more clear.
- This is not very critical, but it would be great to at least comment (or perhaps include a very simple experiment) on the few-shot setting. It is not unusual to be able to obtain a handful of labeled datapoints. Do the weights obtained from the zero-shot setting with the fixed \beta parameter work reasonably well compared to a few show setting where \beta (or the weights directly) are learned? If so (and the optimization problem is fairly simple so this is believable) that provides even more value to the authors' technique. On the other hand, if there’s much larger gains to be had from few-shot learning here, it would be nice to add that conclusion.
Decision: Accept (poster)