LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections
Authors: Muhammad Jehanzeb Mirza
Leonid Karlinsky
Wei Lin
Horst Possegger
Mateusz Kozinski
Rogerio Feris
Horst Bischof
Venue: NeurIPS 2023 poster (2023)
Keywords: VL Models
TL;DR: Training a visual classifier on text-only data and later employing it for unsupervised finetuning of vision language models.

Abstract: Recently, large-scale pre-trained Vision and Language (VL) models have set a new state-of-the-art (SOTA) in zero-shot visual classification enabling open-vocabulary recognition of potentially unlimited set of categories defined as simple language prompts. However, despite these great advances, the performance of these zero-shot classifiers still falls short of the results of dedicated (closed category set) classifiers trained with supervised fine-tuning. In this paper we show, for the first time, how to reduce this gap without any labels and without any paired VL data, using an unlabeled image collection and a set of texts auto-generated using a Large Language Model (LLM) describing the categories of interest and effectively substituting labeled visual instances of those categories. Using our label-free approach, we are able to attain significant performance improvements over the zero-shot performance of the base VL model and other contemporary methods and baselines on a wide variety of datasets, demonstrating absolute improvement of up to $11.7\%$ ($3.8\%$ on average) in the label-free setting. Moreover, despite our approach being label-free, we observe $1.3\%$ average gains over leading few-shot prompting baselines that do use 5-shot supervision.

LaFree: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections
M. Jehanzeb Mirza 11,2
Leonid Karlinsky Rogerio Feris
Wei Lin
Mateusz Kozinski!
Horst Possegger'
Horst Bischof
Institute of Computer Graphics and Vision, TU Graz, Austria. "Christian Doppler Laboratory for Embedded Machine Learning. MITTIBB Watson Al Lab, USA.
Project Page: https///mieniira.giimbbbooll/
Abstract
Recently, large-scale pre-trained Vision and Language (VL) models have set a new state-of-the-art (SOTA) in zero-shot visual classification enabling open-vocabulary recognition of potentially unlimited set of categories defined as simple language prompts. However, despite these great advances, the performance of these zero- shot classifiers still falls short of the results of dedicated (closed category set) classifiers trained with supervised fine-tuning. In this paper show, for the first time, how to reduce this gap without any labels and without any paired VL data, using unlabeled image collection and set of texts auto-generated using Large Language Model (LLM) describing the categories of interest and effectively substituting labeled visual instances of those categories. Using our label-free approach, we are able to attain significant performance improvements over the zero-shot performance of the base VL model and other contemporary methods and baselines on a wide variety of datasets, demonstrating absolute improvement of up 11.7% (3.8% on average) in the label-free setting. Moreover, despite our approach being label-free, we observe 1.3% average gains over leading few-shot prompting baselines that do use 5-shot supervision.
1 Introduction
Vision and Language (VL.) models [1-8] recently became the de-facto standard for generalized zero- shot learning enabling recognition of arbitrary (open) set of categories provided with just their text descriptions and without requiring any additional data or training. However, this incredible flexibility comes performance cost and all the VL models, including the most widely used CLIP [1], still require additional supervised training (e.g. tuning its vision encoder) to compete with the closed set (of target categories) supervised training methods. Naturally, this incurs undesirable adaptation costs for those, otherwise very versatile and flexible, foundation models. Image annotations are often expensive to collect. especially for legacy vision systems, such as traffic surveillance, quality control, or security.
In this paper we propose LaFTer, approach for completely label-free parameter efficient finetuning of VI. models to a set of target classes. Our goal is to substitute the need for expensive-to-obtain image annotations with unsupervised finetuning of VL models. We show that since the VL models share a common text-image embedding space (due to their contrastive learning objective), there is a possibility train using embeddings samples from one modality (e.g., auto-labeled text) and then successfully apply what we trained to classify embeddings of samples from the other modality
Cooreepoondecc: muhammad. mirzaeicg. tugraz.at.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Learning Visual Classifer Text
Text Dataset
111.7
Ramingo: A flamingo bas pink feathers and a long neck.
LIM
KK.8
famingo: A photo of
bonghhin
Aamingo.
Text Encoder
Text
&
Aamingo: flamingo flying sky.
Classifier
CRAR
16.6
Imagitin-
famingo: A blurry photo uf Namingo.
Templates
Ucy
-4.3
Finetuning Visual Data
Classification after Fineluning
bogestin
Unsupervised Finetuning
Cannh
Vakan
+2,7
with Visual Inputs
Figure 1: LaFTer proposes to first train a classifier on a natural language text dataset mined in controlled manner from of target classes by generating descriptions for each class label using an LLM and mixing them with handcrafted templates. The training objective is to classify each description the correct (source) class name (top-left). In the second stage, LaFTer employs the text- only classifier to generate pseudo-labels on the unlabeled data to further finetune the vision encoder manner (bottom-left) Finally, the finetuned visual encoder and text classifier is used for eventual classification poottommmiddle Combining our text-only pre-trale pre-trained classifier together with the proposed pseudo-labeling pipeline lets consistently improve the previous SOTA results for label-free finetuning, UPL. [12] (right).
parameter-efficien
(c.g., unlabeled images). More specifically, show that it is possible to train neural network (e.g., a classifier) to classify text instances that can successfully used to classify images, showing successful cross-modal transfer. Instead of collecting labeled visual instances, in label-free LaFTer approach, we are mining text descriptions of the target categories by prompting an LLM (e.g.. GPT-3 [9]) and combining them with handcrafted prompts, shown in Figure 1. After creating such a text dataset, we train a neural network to classify each text instance (sentence) in it to the source be readily used classify
class label that used produce the instance. This text classifier images when used on top of CLIP visual encoder. Furthermore, we take advantage of this text-only pre-trained classifier by employing further finetune the CLIP vision encoder on an unlabeled image collection. To reduce overfitting and keep the finetuning parameter efficient, we make use of Visual Prompt Tuning [1 combined with adapting the affine transformations (scale and shift) of the normalization layers in the otherwise frozen network.
pseudo-labeling pipeline (inspired by FixMatch [10]). to
To summarize, our contributions are as follows:
We offer completely label-free, and parameter-efficient (0.4% learned parameters), cross-modall transfer approach (LaFTer) to finetuning a VL model for improved recognition performance set of target classes using set of automatically generated texts for those classes and an unlabeled image collection.
We show that text-side-only training automatically collected language knowledge (e.g. through LLM prompting) can be effective in bootstrapping visual classifier built on top of aligned VL model to consistently improve its performance variety of downstream classification benchmarks.
In an extensive evaluation our approach, spanning over 12 popular benchmarks, we show that it: (i) attains up to 28% absolute improvement over the base VL model (CLIP) performance (6. 7% on average): (ii) improves over previous SOTA under same label-free setting by up to 11.7% (3.8% on average); and (iii) performs favorably while comparing with SOTA parameter efficient few-shot methods, matching from above average performance of 5-shot supervised setting despite using no labeled samples.

---

2 Related work
Large-scale & Language (VL) Models: Remarkable performance in many zero-shot down- stream tasks has been attained with VL models pre-trained using contrastive losses large-scale noisy image-text data (e.g., CLIP [1] and ALIGN [13]). Different ideas have been tried to improve the image-text representation alignment, such leveraging off-the-shed object detectors [14-16], or using cross-attention and additional objective functions such as image-text matching and masked language modeling [4, 17-19], filtering noisy captions (e.g., BLIP [4]). Some additional properties of language structure are utilized in [2, 20-23]. DeCLIP [2] finds additional positives for the con- trastive loss by seeking textual nearest-neighhors. Geometrically consistent representations (within and across the paired image-text modalities) are employed in CyClip [20]. Recently, few methods have attempted improving VL performance using additional supervision [3, 24), finer finer-grained interac- tions [25], modern Hopfield networks [26], optimal transport distillation [27], cycle consistency [28], and hierarchical feature alignment [29]. However, VL models performance still falls short of classi- fiers trained with supervision on the downstream target data. Most of the approaches to fine-tuning VL models [30, 31J leverage annotated data for this finetuning. In contrast, in our work we propose completely label-free approach for adapting VL model to the target task. Methods which are most related our work, UPL. [12] and CLIP-PR [32] also finetune VI. models in an unsupervised manner. UPL finetunes learnable text prompts (similar to [30]) by relying on confidence sampled pscudo-labeling, whereas CLIP-PR relies both on offline pseudo-labeling and label distribution prior from the training data. In contrast, in our work, we first leverage textual knowledge from LLMs and design a euf-suuereised text-only pre-training task showing that it can be employed in an unsupervised parameter- -ffficiieeeefeeenntteeeeee finetuning of the VL model on set of unlabeled images. Extensive empirical evaluations show the benefits of our approach w.r.t. UPL and CLIP-PR.
Prompt Tuning. Prompt tuning belongs to wider family of pararamter-efficien finetuning methods that can also be applied to VI. models [33, 34). Originating in NI.P [35, 36], visual prompt tuning was recently shown to be effective also for vision backbones [11, 33, 37J. CoOp [33] learns prompt
vectors by minimizing the prediction error using the cross-entropy loss. ProDA [37] learns diverse
prompts from data to cope with variance of the visual representations. UPL [34] proposes an unsupervised prompt learning framework. TPT [38] proposes a test-time prompt tuning framework. CLIP-Adapter [39] and Tip-Adapter [40] employ alternative parameter efficient finetuning strategy using additional adapter modules. Recently, CoCoOp [41] proposed a Meta-Net for generating image-adaptive prompts for improved generalization distribution shifts. In this work, we employ parameter-efffcieet Visual Prompt Tuning ( (VPT) [11] as the means for adapting the visual encoder of the VI. model, but in contrast with other works do not use any supervised data. Instead, use a classifier, trained in the text domain using texts automatically generated by an LLM from the set of target classes, to generate pseudo-labels for unlabeled image collection self-supervision. Furthermore, we propose to combine VPT with tuning the scale and shift parameters of normalization layers, previously proposed for handling domain shifts [42-44], but to the best of our knowledge never before used to tune VL models.
form of
Pseudo-labeling. Pseudo labeling, also known self-training. is commonly used semi- supervised learning technique. Popularized in the seminal works of [45, 46). pseudo-labeling was extended to utilize consistency regularization [47] and augmentation [48]. The pseudo-labeling pipeline of our method is inspired by FixMatch [10] that combined consistency regularization with confidence-basee filtering, surpassing SOTA semi-supervised techniques the time. It was later extended with a non-parametric classifier in PAWS [49]. These techniques are proposed for semi- supervised learning and require some amount of labeled instances. In contrast, we propose a label-free method for improving VL models performance on set of target classes. To achieve this, our method finetunes VL models in parameter-efficien manner by generating pseudo labels through text-only classifier trained on a corpus of text data generated by prompting language models.
3 LaFTer
CLIP [1] consists of a vision encoder and a text encoder, which project images and texts to a common embedding space. It has been trained on a very large volume (400M) of image-text pairs to align the embedding of each image to the embedding of its corresponding text, at the same time pushing it away from the embeddings of unrelated texts corresponding to other images. In [1], was demonstrated that the text and vision encoders enable effective zero-shot image classification. Given the set of class
3

---

Text Dataset
n ~ ()
Classes
Classes
our hos four legs and bil.
Cat Dog Bird
LLM
A dog las furry tall. Aghano of bind. hind Bying in the sky. bluuny phano of'a dog.
Text
Text Classitier
Cross Entropy Loss
Tomplene
Uplate Weights
Star Grd
Vision Eneodkr
Text Classifier
Pscudo-Label
Share
Cross Entropy Loss
Vision Enceder
Text Classifier
Class Logits
Updai: Classifier, LN Affine Picenpts
Figure 2: Overview of our LaFTer. (top) Given of class labels, generate data set of short texts by prompting Large Language Model (LLM) multiple times with each class name. We compute embeddings of these Text Classifier, to infer the class used to prompt the LLM from the embedding of the text it generated. Even though the Text Classifier has been trained exclusively image embeddings generated by CLIP vision encoder. (bottom) We further take advantage of the Text Classifier by leveraging
using CLIP text encoder. This lets train a neural network, the
text, performs well in classifying
pseudo-labeling setup to finetune the VL model.
names C, the text encoder used generate the embedding u. of prompt for each class e C, typically of the form 'A photo of...' completed with the class name. Each test image is encoded by the vision encoder and classified using its cosine similarity cos to the text embedding of each class. The likelihood of a predicted class is computed with softmax,
(c0((ue,00=))
le(x) Ecso (com(M).)()))
where denotes the temperature constant.
The zero-shot classifier (1) does not require any training data, but typically outperformed by networks trained on data from the target domain. Existing approaches to reducing this performance gap [30, 31] rely on fine-tuning the zero-shot classifier enhancing accuracy, they incur additional costs of curating and annotating the training data. In con- trast, we propose a two-step approach fine-tuning the network without requiring any annotations. In the next section 3.1, we describe technique to train a visual classifier purely textual data, conveniently generated by Large Language Models. Section. 3.2, we propose an unsupervised training setup that lets us benefit from unlabelled data from the target domain, when such data is available. Combining these techniques significantly reduces the performance gap to the supervised classifier and is competitive with few-shot approaches despite using any supervision. These results are provided in Section 4.2.
few-shot regime. While effectively
3.1 Learning an Image Classifier using Text
Aligning the text and image embedding spaces is the key idea underlying Vision Language models. gives CLIP and similar methods, their main advantages: the capacity to be trained on extremely large volume of data available on the Internet and the resulting effectiveness as zero-shot classifiers. Recently, Zhang et al. [50] explore yet another advantage of the alignment between text and image embedding spaces
allows diagnosing and rectifying vision models spawn from the VL model vision encoder by using the other modality. In particular, it hints that it might be possible to finetune a zero-shot image classifier on textual data.
While acquiring and annotating images requires manual labor, a training set of annotated texts can be constructed automatically, using Large Language Model, like the GPT-3 [9]. Generating text using LLM constitutes convenient alternative to text mining, LLMs represent extremely

---

large text corpora on which they were trained. Moreover, prompting LLMs is much more efficient than searching a large body of text for the words of interest. To construct our training set, we take inspiration from [51] and for each class e, we prompted GPT-3 with queries of the following kind: 'Describe what a [e] looks like." We repeated the prompting for each class c, in the dataset and complemented the resulting set of synthetic texts with text generated procedurally, using the same hand-crafted templates as for constructing prompts for the zero-shot classifier, for example, 'A photo of [c].'. We defer the full list of prompts (to LLMs) and templates to the supplementary material.
The capacity to finetune the classifier in a supervised text classification setting lifts the architectural constraints imposed by the zero-shot setup. More precisely. the class likelihoods no longer need to be produced according to (1), but can instead be computed by trainable classifier f. of arbitrary architecture. We train f with the Smoothed Cross Entropy loss LSCr [52]. To regularize training, we add Gaussian Noise n ~ (0, T) to the Iz-normalized feature vector from the clip text encoder. Formally, given a text training set T consisted of pairs of text fragments & and class labels c, the training objective is:
CSCE(fol uu() |e((t)|
min
Σ (L.c)ET (=-1)
+ m).c),
where 0 is the parameter of the classifier. Training of the text-only classifier is very efficient. For example, 3000 epochs of training the classifier on the data set of 130000 text sentences, representing the 1000 classes of the ImageNet [53] dataset is completed in 120 seconds on an NVIDIA 3090 graphics card.
3.2 Unsupervised Finetuning on Target Domain Images
Text-only training does not require any image data. However, in many applications, unlabeled images from the target domain are readily available, or can be acquired at low cost. Given a set of unlabeled images we propose to take advantage of the text-only pre-traue traitrained classifier and use it in pseudo-labeling pipeline on top of the vision encoder as demonstrated at the bottom part of Figure 2.
Inspired by Fixmatch [10], for each training image x, we generate two views: the weakly-augmenttd view (2) and the strongly-augmentt function. Contrary to Fixmatch, in our unsupervised finetuning we set aw as an identity transfor- mation. For on, we use the augmentations proposed in [54]. The weakly-augmented view serves to generate a pseudo-label. To that end. it is passed through the vision encoder and the text classifier f. yielding vector of class probabilities p. Class probabilities give rise to pseudo labels, that we denote by ((r). Formally,
view (r), where denotes stochastic augmentation
where p (=((aa((x))
a(x) = arg max Pe.
(3)
cEC'
d(z) is not differentiable, and we do not backpropagate the error signal through it during training. The pseudo labels pa, are used as ground truth for training the network on the meavily-augmented views. Since the vision encoders in the two branches share the weights, the pseudo-labels are generated in an online manner and are constantly refined as adaptation is progressing. This is in contrast to UPL [12] and CLIP-PR [32] which rely on offline pseudo-labeling by only generating them once from frozen CLIP encoders. We freeze the weights of the text-only pre-trained classifier in the pseudo-labeling branch since it leads to more stable adaptation.
When processing the heavily augmented views ay (r). we augment the vision encoder by visual prompt tuning [11]. That is, we append randomly initialized, learnable parameters (a matrix of size Np x d, where Np is the number of prompts and d, is the channel dimension of the vision encoder) to the input of the vision transformer after the initial embedding layer. This helps the network account for the heavy augmentation of the input and, as we show in Section 4.3, boosts the performance of the finetuned classifier. We denote the vision encoder with prompting by vº, The prediction for the heavily- augmente image is obtained as (an(()))))
The network is trained with the smoothed cross entropy loss LSCE [52]. We denote the set of unlabelled target domain images by D. and formalize the training objective as
min Σ 0.7
LSCr (fo (1) (a)((a)))) ((6)).
(4)
ZED
5

---

ImageNet
CIEAR-10
CIFAR-100
EuroSat
DTD
CALTECH-101
CLIP
61.9
88.8
64.2
45.1
42.9
90.5
CLIP-PR
60.4
89.3
63.2
44.2
40.1
84.8
UPL
61.2
89.2
65.8
62.2
48.0
90.6
LaFTer
64.2
95.8
74.6
73.9
46.1
93.3
UCF-101
Flowers-102
SUN-397
ImageNet-A
ImageNet-S
ImageNet-R
CLIP
61.0
66.6
60.8
29.6
40.6
65.8
CLIP-PR
57.9
57.7
54.7
11.6
38.6
54.1
UPL
63.9
71.5
66.0
26.9
42.4
65.6
LaFTer
68.2
71.0
64.5
31.5
42.7
72.6
Table I: Top-1 Classification Accuracy (%) while using the CLIP pre-trained VIT-B/32 backbone for 12 image classification benchmarks. LaFTer represents results obtained by first pre-training the visual classifier on text-only data and then performing unsupervised finetuning on the unlabeled image data. Highest accuracy is shown in bold, while second best is underlined.
where and r) denote the finetuned parameters of the classifier and of the vision encoder, respectively, The parameters are the visual prompts and the scale and shift (affine) parameters of the normalization layers of the vision encoder. The selection of r) is motivated by keeping the adaptation parameter- efficient. For LaFTer, the number of trainable parameters are less than 0.4% of the entire model parameters, making adaptation extremely lightweight.
Experimental Evaluation
Here we first provide description of the datasets and baselines we use in our evaluations, then explain our implementation details and later discuss our experimental results in detail.
4.1 Evaluation Setting
Datasets: We extensively evaluate approach 12 different datasets belonging to widely different domains. More specifically, we use four datasets containing common natural categories: ImageNet [53], CIFAR-10/100 [55] and Caltech-100 101 [56]. EuroSat [57] contains satellite images of 10 different locations. UCF-101 [58] is an action recognition dataset. SUN-397 [59] contains images from 397 naturally occuring scenes. Flowers- 102 [60] is fine-grained classification dataset for classifying different categories of flowers commonly occuring in the United Kingdom. Whereas, ImageNet-A (Adversarial) [61], ImageNet-S (Sketch) [62] and IeageNet-R (Rendition) [63] are different versions of the original ImageNet validation set. In our setting, we divide the ImageNet-A, ImageNet-S and ImageNet- provided by [30].
80% train and 20% test set. For all other datasets we use the splits
Baselines: We compare LaFTer with baselines which are label-free (not requiring any additional labeled images):
CLIP [1] denotes zero-shot classification scores by computing the cosine similarity between embeddings from frozen CLIP encoders.
UPL [12] adds learnable text prompts to the CLIP text encoder and finetunes them in an unsupervised manner by employing mafidennce-smmlle offline pseudo-labeling.
CLIP-PR [32] optimizes an adapter top of the CLIP vision encoder by using label distribution priors from the training set of the downstream datasets and generating offline pseudo-labels.
For completeness, apart from these baselines, we also provide a comparison with few-shot fine- tuning method CoOp [30], which learns soft text prompts using & labeled images per class (k-shot).
Implementation Details: For all our experiments, unless otherwise stated, we use a ViT/B-32 CLIP pre-trained model from OpenAl [1]. The Text-Only classifier (Section 3.1) is implemented as

---

ImageNet
CIEAR-10
CIFAR-100
EuroSat
DTD
CALTECH-I01
LaFTer (no-shot)
64.2
95.8
74.6
73.9
46.1
93.3
CoOp (1-shot)
60.6
83.0
55.6
58.4
40.1
91.7
CoOp (5-shot)
61.3
86.6
63.2
71.8
41.1
93.2
CoOp (10-shot)
62.3
88.5
66.6
81.6
65.8
94.6
PEFT (1-shot)
50.7
62.7
50.2
37.5
42.6
90.6
PEFT (5-shot)
59.3
80.0
67.3
55.3
59.9
94.5
PEFT (10-shot)
62.8
87.9
74.1
67.9
67.3
96.1
UCF-101
Flowers-102
SUN-397
ImageNet-A
ImageNet-S
ImageNet-R
LaFTer (no-shot)
68.2
71.0
64.5
31.5
42.7
72.6
CoOp (1-shot)
63.8
71.2
64.1
24.5
39.9
60.0
CoOp (5-shot)
74.3
85.8
67.3
30.0
46.5
61.6
CoOp (10-shot)
77.2
92.1
69.0
35.0
63.6
PEFT (1-shot)
60.5
66.9
58.3
20.9
38.5
PEFT (5-shot)
72.6
91.1
68.7
33.3
55.3
66.4
PEFT (10-shot)
79.8
95.2
72.3
40.2
61.1
71.0
Table 2: Top-1 Accuracy (%) for our LaFTer (no-shot) compared few-shot methods. We compare CoOp [30] in 1-, 5- and 10-shot supervised finetuning regimes. Parameter Efficient Finetuning (PEFT) represents tuning the same parameters in LaFTeer (prompts, classifier, affine) but in few-shot manner. For each dataset/compared method, blue highlights the highest number of shots outperformed by no-shot LaFTeer Notably, LalTeer improves 10-shot and all compared methods in 4 datasets, including ImageNet, where 10-shot 10K labeled samples.
single linear layer, with the output units equal the number of classes in the dataset. For training this classifier, we load the complete text dataset single batch and optimize the network using AdamW as optimizer, with a learning rate of 0.001. For unsupervised fine-tuning using visual data (Section 3.2), we again use the AdamW optimizer with learning rate of 0.0001, batch size of 50 and optimize the learnable parameters for total of 50 epochs. Thanks to our Text-Only classifier pre-training, through empirical evaluations we find that 10 epochs are sufficient for fine-tuning the large-scale ImageNet dataset. To produce an augmented view of the image, we employ the augmentations used in SimSiam [54]: Gaussian blur, random resized crop, random horizontal flip, color jitter and random gray scaling. For generating class descriptions we use different LLM's M's e.g., GPT-3 [9] and Alpaca [64]. In total generate 50 descriptions for each category in the dataset. We ablate the LLM choice in section 4.3. To construct the text dataset we combine the class descriptions from LLM's and dataset-speciffc prompt templates provided by [1].
4.2 Results
We test our LaFTer extensively on 12 image classification datasets. These results are provided in Table 1. Our LaFTer consistently improves the zero-shot CLIP model on all the 12 datasets. On some datasets, for example EuroSat, our LaFTer shows an absolute improvement over 28% on the zero-shot CLIP classification results. Even the large scale ImageNet dataset show considerable improvement of 2.3% over the zero-shot CLIP classification results.
We also see that we outperform CLIP-PR on all datasets. Since CLIP-PR relies on offline pseudo- labels, which are generated only once for the entire datasct and only proposes to finetune an adapter on top of frozen CLIP visual encoder. We conjecture that their approach might be less expressive. Moreover, it requires label distribution prior from the dataset on which it is being finetuned on. Our LaFTer is free from these requirements and proposes more general solution for unsupervised finetuning on downstream datasets.
We also compare to the other unsupervised adaptation baseline UPL, which relies on finetuning learnable text prompts attached to the CLIP text encoder through knowledge distillation. Table 1, we see that our method out performs UPL on most of the datasets (9 out of 12). On some datasets such as FuroSat, shows huge gain of 11.7 percentage-poine over UPL. On datasets such as Describable Textures Dataset (DTD), Flowers-100 and SUN-397 our method is marginally behind

---

IN
CIFAR-10
CIFAR-100
IN-A
IN-S
IN-R
Mean
Class Name
58.4
87.1
59.0
30.7
37.7
65.5
56.4
61.1
88.1
63.1
30.4
40.3
65.9
58.1
Pataset-Tempatte
60.1
89.3
64.7
31.1
41.2
67.5
59.0
Alpaca Descriptions
54.8
88.4
59.4
25.7
36.3
58.7
53.9
GPT Descriptions
60.5
87.8
63.0
30.2
39.6
63.6
57.4
GPT + Templates
61.9
89.3
65.4
31.5
40.9
67.8
59.5
Table 3:
Top-1 Classification Accuracy (%) for a ViT-B/32 model while ablating different text generation strategies for training the text-only classifier in the first stage of LaFTer To obtain these results, we evaluate the test set of the respective datasets by using the text-only pre-trained classifier on top of the frozen vision encoder from CLIP. IN ImageNet.
Clip w/o Aug w/o Prompts w/o A Affine w/o Cls w/o Stop Grad w/o Text Pre-train LaFTer
CIFAR-10
88.8
93.5
92.5
94.6
94.1
94.7
95.1
95.8
CIFAR-100
64.2
72.6
71.7
72.9
67. 4
73.2
73.1
74.6
UCF-101
61.0
65.3
66.1
63.6
63.4
67.5
67.7
68.2
EuroSat
45.1
63.2
61.2
64.2
60.9
69.2
69.7
73.9
ImageNet-A
29.9
29.8
31.1
28.9
31.5
ImageNet-S S
40.6
40.3
40.7
41.1
40.9
42.1
41.3
42.7
ImageNet- R
65.8
68.0
67.7
66.8
66.1
71.8
70.2
72.6
Average
56.4
61.8
61.4
62.0
60.4
64.2
63.7
65.6
Table 4: Top-1 Accuracy (%) for LaFTer while ablating the various critical design choices in methodology. For each of these experiments, we disable one component from our framework and test the resulting method. Aug: Augmentations, Cls: Classifier.
augmentations in one of our streams during adaptation phase, might result noisy gradients during the learning process. Specially since datasets like DTD and Flowers- 102 can depend on color cues for distinguishing the classes. For the large scale ImageNet dataset, we see that the performance of UPL is below the CLIP zero-shot performance. This can be because UPL generates offline pseudo-labels and in ImageNet, due fine-grained classes the pseudo-labels might not be very confident from the zero-shot CLIP classifier. On the other hand, LaFTeer benefits first from classifier which has learned discriminative visual features through text-only training and later makes use of parameter efficient finetuning (PEFT). Furthermore, since our pscudo labels are generated in an online manner for each iteration during the optimization, they are also constantly refined as the training is progressing.
UPL. We conjecture that since
In Table 2 provide comparison of LaFTer with the few-shot learning method CoOp [30] and also test Parameter Efficient Fine- Fine-Tuning (PEFT), which tunes the same learnable parameters LaFTer (prompts, classifier, and affine parameters of the normalization layers). but in few-shot manner. Interestingly, we see that our unsupervised representation learning method conveniently outperforms CoOp for 1- and 5-shots. For example, LaFTer (no-shots) is average 7.1% better than CoOp (1-shot) and even 1.3% better in the 5-shot learning regime, while remaining competitive for 10-shots. It is also worth noting that for the large-scale ImageNet our LaFTer (requiring no labels) performs better than CoOp (10-shots), requiring 10000 labeled instances from the dataset. Results also follow similar trend when compared with PEFT.
4.3 Ablation Studies
To understand the significance of all the components in our LaFTer we minutely study each design aspect. First we discuss the performance of our Text-Only pre-trained classifier, then ablate each component in our unsupervised adaptation phase using unlabeled images and finally provide results by adapting other pre-trained backbones from CLIP [1]. Due limited evaluation resources and space constraints, we perform these ablations on subset of datasets with different complexity.
Text-Only Pre-trained Classifier. A main component of our LaFTer is the text-only pre-trained visual classifier, later used in our pseudo-labeling pipeline. The motivation behind it being, that since

---

Performancc Gains (%%
+4.2
C-10 C-100 UCF
EuroSat IN-R
+2.4224
11.8
CLIP (B/16)
89.2
68.1
64.7
48.4
73.8
11.514
LaFTer
96.5
76.3
67.2
72.1
81.4
+1.0
10.8
+0.7
CLIP (L/14)
95.3
75.8
72.0
60.3
84.9
EursSar
IN-A
DTD
Flawore
SUN
e rer
Calledh
C-0
ECF
LaFTer
99.0
87.2
77.2
77.2
91.5
Figure 3: Performance gains of using the text- only pre-trained visual classifier vs not using it in LaFTer pseudo-labeling pipeline scheme.
Table 5: Top-1 Accuracy (%) for CLIP pre-trained ViT-B/16 and ViT-L/14 backbones. Results are pro- vided for Base VL model (CLIP) and our LaFTer.
63.0
87.5
Accuracy (%)
86.5
62.0
59
86.0
61.5
58
85.5
61.0
CIPAR 10
CIPAR 100
ImageNet
85.0
10
20
30
400
50
10
20
30
40
50
10
20
30
40
50
Descripsioms Per Class
Descriptions Per Class
Descriptions
Per Class
Figure 4: Effect of diversity of descriptions on Top-1 Accuracy (%). We train the text classifier by randomly choosing (with increment of 5) certain number of descriptions per class for each evaluation step. For all the main evaluations, we use a maximum of 50 descriptions per class.
CLIP trained order have shared text and vision embedding space so a classifier trained to classify the embeddings from any one of the modalities should also be able to classify embeddings from the other modality. We show that it possible train a classifier classify images by only training it to classify natural language. To design this supe-suue objective of classifying text (described detail in 3.1) we test different ways of generating the text dataset. For example, simple- template such as A photo of a.... dataset-speciffi templates from CL.IP [1] and descriptions from I.LMs. These results are presented in Table Note that for these results, we first train the classifier on the generated text dataset and then evaluate the (images) test set of the respective dataset by using the trained classifier to classify the visual embeddings from the CLIP visual encoder. The simplest text dataset generation strategy: classifying the classname is also able to show reasonably strong visual classification performance. Furthermore, different strategies have slight difference in performance. We also find that descriptions from GPT-3 work better than descriptions from Alpaca. Our choice of generating class descriptions by prompting GPT-3 and complementing them by adding handcrafted templates from [1] works best. While averaging over 6 datasets, we gain performance improvement of 3.1 1% in comparison to the simplest text dataset generation strategy of classifying the classnames themselves.
Text Classifier Contribution to LaFTeer Pre-training a visual classifier on text-only data and then employing it in our pseudo-labeling pipeline helps LaFTeer gain improvements all the 12 datasets which we evaluate in our paper. In Figure 3 we analyze the performance gains for our LaFTer conjunction with our pseudo-labeling pipeline. On some
when using the text-only pre-training datasets, for example EuroSat [57], our text-only pre-training helps our LaFTer to gain an absolute performance improvement of 4.2%. On other datasets, the performance gains are also consistent.
Design Choices for Unsupervised Finetuning. In order to study the effect on performance of each individual component we ablate our LaFTer in Table 4. We see that removing the classifier and instead using the CLIP Cosine similarity scores in both branches results in the highest degradation of results. Similarly, all other design choices also have a certain effect. For example, removing the leamable visual prompts results in a decrease of 4.2 percentage-poine as compared to our LaFTer.

---

Visual Embedding- Text Embeddings
Component 2
Component
Component
Component
(a) Base CLIP (ViT-B/32).
(b) LaFTer Adapted CLIP (ViT-B/32).
Figure 5: TSNE projections for visual (circles) and text (triangles) embeddings for 10 classes of the EuroSAT dataset from (a) Base (un-adapted) CLIP ViT-B/32 model and (b) after adaptation with LaFTer. For the Base CLIP model, the text embeddings the output feature vector from the CLIP text encoder and for LaFTer we use the weights of the adapted text-classifier. Visual embeddings are always the output features from the vision encoder.
Different CLIP Backbones. For our main experimental results described in Tables 1 and 2 and ablations in Tables 3 and 4, the ViT-B/32 CLIP backhone. In Table also provide results with different backbones for our LaFTer We observe consistent improvements by LaFTer over CLIP also for larger backbones. For example, for ViT-B/16 our method shows 23.7% absolute improvement for EuroSat, while for ViT-L/14, our method improves CLIP zero-shot by 16.9% on the same dataset.
Diversity of Deseriptions. To test the effect of diversity of generated descriptions classification results for the text-only pre-training of the text-classifier, we ablate the number of descriptions chosen for each class (for ImageNet and CIFAR- 10/10))) and provide the results in Figure 4. We find that increasing the diversity of descriptions to generate the text dataset has a positive effect for all the datasets ablated.
the visual
Visualization of Embeddings. To gain further insights into the LaFTer adaptation we visualize the latent embeddings from the CLIP encoders in Figure 5, before and after adaptation for all 10 classes in the EuroSAT dataset. Analyzing the TSNE projections, we observe that the adaptation with our LaFTer results in pronounced category clusters with larger inter-class distances, and also that after applying LaFTer the class-label text embeddings are better aligned with these category clusters.
Conclusion & Limitations
We propose a completely label-free finetuning method Vision-Language models by first showing cross-modality transfer and learning classifier natural language inputs which can successfully classify visual data. Later, we leverage this text-only pre-trained classifier in our pseudo-labeling pipeline to further finetune the VL models in parameter efficient manner. We extensively evaluate our LaFTer and achieve state-of-the-ant results when comparing to other methods in an unsupervised finetuning paradigm, while also performing favorably in comparison to methods relying on few-shot supervised learning routines.
Limitations. One contribution of LaFTer is to show cross-modality transfer by training classifier to classify natural language sentences and showing that it can readily applied classify visual data as well. In our work, we experimented with single linear layer classifier. Due to the risk of over-fitting because of the sparse nature of natural language, we did not experiment with more complex structures. Experimentation with expanding the text dataset coupled with designing more expressive classifier head is left as an exciting future direction.
Acknowledgements
We gratefully acknowledge the financial support by the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development and Christian Doppler Research Association, This work was also partially funded by the FWF Austrian Science
10

---

Fund Lise Meitner grant (M3374) and the Austrian Research Promotion Agency (FFG) under the SAFER (894164) project.
References
[1] A. Radford, W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell,
P. Mishkin, J. Clark, et al., "Learning Transferable Visual Models from Natural Language Supervision," Proc. ICML, 2021.
[2] Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and Yan, "Supervision
Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm,"
arXiv:2 10.05208, 2021.
[3] N. Mu, A. Kirillov, D. Wagner, and S. Xie, "Slip: Self-supervision Meets Language-image Pre-training." arXiv:2112, 12750, 2021.
[4] J. Li, D. Li, C. Xiong. and S. Hoi, "BLIP: Bootstrapping Language-Image Pre- Pre-training for
Unified Vision-Language Understanding and Generation," D:Xii:2201./2080 2022.
[5] L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, "FILIP: Pre- "eeinirin arXiv:2111.0775 2021.
Fine-grained Interactive Language-Iman
[6] A. Singh, R. Hu, Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela, "FLAVA:
A Foundational Language And Vision Alignment Model," arXív:21 1nt:2//22.44483 2021.
[7] J. Yu, Z. Wang. V. Vasudevan, Yeung, M. Seyedhosseini, and Y. Wu, "CoCa: Contrastive Captioners are Image-Text Foundation Models," arXiv:2205.01911 2022.
[8] S. Reed et al., "A Generalist Agent," arXiv:2205.0611
2022.
[9] T. B. Brown et al., "Language Models are Few-Shot Learners," arXiv:2005. 2020.
[10] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Ku- rakin, and C.-L. Li, "Fixmatch: Simplifying cupi-supervised Learning with Consistency and Confidence," in NeurlPS, 2020.
[11] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie. B. Hariharan, and S.-N. Lim, "Visual Prompt Tuning." " Proc. ECCV, 2022.
[12] T. Huang, J. Chu, and F. Wei, "Unsupervised Prompt Learning for Vision-Language Models,"
arXiv:2204. TIt::2204.03649, 2022.
[13] C. Jia, Y. Yang. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung. Z. Li, and Duerig, "Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision," in Proc. ICML, 2021.
[14] H. Tan and M. Bansal, "Lxmert: Learning Cross-modality Encoder Representations from Transformers," arXiv: /908. 07490, 2019.
[15] Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, "Uniter: Universal Image-text Representation Learning," in Proc. ECCV, 2020.
[16] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang. H. Hu, L. Dong, F. Wei, et al., "Oscar: Object-semantics Aligned Pre-training for Vision-language Tasks," in Proc. ECCV, 2020.
[17] W. Kim, B. Son, and I. Kim, "Vilt: Vision-and--angguage Transformer without Convolution or Region Supervision," in Proc. /CML, 2021.
[18) J. Li R. Selvaraju, A. D. Gotmare, S. Joty, C. Xiong, and S. Hoi, "Align before Fuse: Vision and Language Representation Learning with Momentum Distillation," arXiv:2
2021.
[19] J. Yang, Duan, S. Tran, Y. Xu, Chanda, L. Chen, B. Zeng, T. Chilimbi, and J. Huang, "Vision-Language Pre-Training with Triple Contrastive Learning"" in Proc. CVPR, 2022. [20] S. Goel, H. Bansal, S. Bhatia, R. A. Rossi, V. Vinay, and A. Grover, "CyCLIP: Cyclic Contrastive Language-Image Pretraining," arXiv:2205. /4459, 2022.
[21] L. Yao, R. Huang. L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiing, and C. "Filip: Fine-grained Interactive Language-image Pre-training." arXiv:211 11.0783, 2021.
[22] A. Fürst, Rumetshofer, V. Tran, Ramsauer, Tang. Lehner, Kreil, M. Kopp,
G. Klambauer, A. Sitto-Nemling et al., "Cloob: Modern Hopfield Networks with Infoloob Outperform Clip," arXiv:2 10.113161616 2021.
[23] Gao, J. Liu, Z. Xu, J. Zhang, K. Li, and C. Shen, "PyramidCLIP: Hierarchical Feature Alignment for Aision-language Model Pretraining," arXiv:2204.1400 14095, 2022.

---

[24] Y. Li, F. Liang. L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, "Supervision Exists Everywhere: A data Efficient Contrastive Language-Iman Pre-training Paradigm." "
arXiv:21100052000 2021.
[25] L. Yao, R. Huang. L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, "Filip: 2021. [26] A. Fürst, E. Rumetshofer, V. Tran, H. Ramsauer, F. Tang. J. Lehner, D. Kreil, M. Kopp, G. Klambauer, A. Bitto-Nemling, et al., "Cloob: Modern Hopfield Networks with InfoLOOB
Fine-graine -eeeegraindd Interactive Language- Language-image Pre-training." arXiv:21
Outperform Clip," (00.i::21101111166 2021.
[27] B. Wu, R. Cheng, P. Zhang. P. Vajda, and J. E. Gonzalez, "Data Efficient Languae--suprrvieed Zero-shot Recognition with Optimal Transport Distillation," urXiv:21/2.094/4 2021.
[28] S. Goel, H. Bansal, S. Bhatia, R. A. Rossi, V. Vinay, and A. Grover, "CyCLIP: Cyclic Contrastive Language-Image Pretraining," arXiv:2205. 14459, 2022.
[29] Y. Gao, J. Liu, Z. Xu. J. Zhang, K. Li, and C. Shen, "PyramidCLIP. Hierarchical Feature Alignment for Vision-language Model Pretraining," arXiv:2204. 2022.
[30] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, "Learning to Prompt for Vision-Language Models," IJCV, 2022.
[31] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, "Conditional Prompt Learning for Vision-Language Models," in Proc. CVPR, 2022.
[32] J. Kahana, N. Cohen, and Y. Hoshen, "Improving Zero-Shot Models with Label Distribution 2022.
Priors," arXiv:2212.0077
[33] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, "Learning to Prompt for Vision-language Models," IJCV, 2022.
[34] T. Huang, J. Chu, and F. Wei, "Unsupervised Prompt Learning for Vision-L.anguage Models," arXivv2204. 03649, 2022.
[35] Z. Zhong. D. Friedman, and D. Chen, "Factual Probing is [MASK]: Learning vs. Learning to Recall," Stti:200040024-00 2021.
[36] B. Lester, R. Al-Rfou, and N. Constant, "The Power of Scale for Parameter-efficien Prompt Tuning," " Stfiv2/04.00663 2021.
[37] Y. Lu, J. Liu, Y. Zhang, Y. Liu, and X. Tian, "Prompt Distribution Learning," in Proc. CVPR,
2022.
[38] M. Shu, W. Nie, D.-A. Huang. Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao, "Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models," in NeurIPS, 2022.
[39] P. Gao, S. Geng, R. Zhang. T. Ma, R. Fang. Y. Zhang, H. Li. and Y. Qiao, "Clip-adapter: Better Vision-language Models with Feature Adapters," arXiv:2 10.44444, 2021.
[40] R. Zhang, R. Fang, P. Gao, W. Zhang, K. Li, J. Dai, Y. Qiao, and H. Li, "Tip-adapter: Training- Modeling," arXiv:2111.0030 2021. [41] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, "Conditional Prompt Learning for Vision-language Models," in Proc. CVPR, 2022.
free Clip-adapter for Better Vision-Languras
[42] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell, "Tent: Fully Test-time Adaptation by Entropy Minimization," in Proc. ICLR, 2020.
[43] M. J. Mirza, P. J. Soneira, W. Lin, M. Kozinski, H. Possegger, and H. Bischof, "ActMAD: Activation Matching to Align Distributions for Test-Time Training," " in Proc. CVPR, 2023.
[44] S. Niu, J. Wu, Y. Zhang, Y. Chen, S. Zheng, P. Zhao, and M. Tan, "Efficient Test-Tine Time Model Adaptation without Forgetting," in Proc. ICML, 2022.
[45] D.-H. Lee. "Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks," in Proc. ICMLW, 2013.
[46] A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, "Semi-supervised Learning with Ladder Networks," in NeurlPS, 2015.
[47] A. Tarvainen and H. Valpola, "Mean Teachers are Better Role Models: Weight-averaged Consistency Targets Improve Semi-supervised Deep Learning Results," in NeurlPS, 2017.
[48] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, "Unsupervised Data Augmentation for Consis- tency Training," in NeurIPS, 2020.
[49] M. Assran, M. Caron, I. Misra, P. Bojanowski, A. Joulin, N. Ballas, and M. Rabbat, "Semi- supervised Learning of Visual Features by non- Parammetttrillly Predicting View Assignments with Support Samples," in Proc. CVPR, 2021.
12

---

[50] Y. Zhang. J. Z. HaoChen, S.-C. Huang. K.-C. Wang. J. Zou, and S. Yeung, "DrML: Diagnosing and Rectifying Vision Models using Language," " in Proc. ICLR, 2023. [Online]. Available: https://openreet
net/forrumPic--losuSTTaaPP
[51] S. Pratt, R. Liu, and A. Farhadi, "What does Platypus Look Like? Generating Customized Prompts for Zero-shot Image Classificattinn" arXiv:2209,03333 03320, 2022.
[52] C. Szegedy, V. Vanhoucke, S. loffe. J. Shlens, and Z. Wojna, "Rethinking the Inception Architecture for Computer Vision," in Proc. CVPR, 2016.
[53] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: Large-scale Hierarchical Image Database," in Proc. CVPR, 2009.
[54] X. Chen and K. He, "Exploring Simple Siamese Representation Learning," in Proc. CVPR, 2021.
[55] A. Krizhevsky and G. Hinton, "Learning Multiple Layers of Features from Tiny Images," Department Computer Science, University of Toronto, Tech. Rep., 2009.
[56] L. Fei-Fei, R. Fergus, and P. Perona, "Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories," in Proc. CVPR, 2004.
[57] P. Helber, B. Bischke, A. Dengel, and D. Borth, "Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification," in Proc. IGARSS, 2018.
[58) K. Soomro, A. R. Zamir, and M. Shah, "UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild," arXiv:121..0402, 2012.
[59] J. Xiao, Hays, K. A. Ehinger, A. Oliva, and A. Torralba, "SUN Database: Large-scale Scene Recognition from Abbey to Zoo," in Proc. CVPR, 2010.
[60] M.-E. Nilsback and A. Zisserman," " "Automated Flower Classification Over a Large Number of Classes," in Proc. ICVGIP, 2008.
[61] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, "Natural Adversarial Examples,"
in Proc. CVPR, 2021.
[62] H. Wang, S. Ge, Z. Lipton, and E. P. Xing, "Learning Robust Global Representations by Penalizing Local Predictive Power," in NeurIPS, 2019.
[63] Hendrycks el al., "The Many Faces of Robustness: A Critical Analysis of Out-of- Distribution Generalization,," in Proc. ICCV, 2021.
[64] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, Stanford Alpaca: An followinn LLaMA model, 2023.
13


Reviews:
[{"review_id": "1dLASIGYrS", "reviewer": "Reviewer_YPJ1", "summary": "This paper proposes a new method to improve the zero-shot classification accuracy for a pre-trained vision-language (VL) model. By leveraging the shared embedding space between the vision and language modalities, and relatively accessible text data generated by large language models (LLMs), the proposed method trains a classifier on the embedding of generated texts for the zero-shot classes, which can also work on the image features. The proposed method also includes a pseudo-label strategy using the trained classifier to augment unlabeled image data. The paper shows compelling performance improvement over the original VL model and other non-label fine-tuning methods.", "questions": "LLM can just generate random text descriptions of the class. For example, it might generate the history of an object which is not very useful in visual classification. Did you consider imposing any constraints to make the LLMs generate only visually meaningful texts?", "limitations": "To me, the biggest limitations are \n1. The method might rely on a super good LLM which might not be accessible in many real-world scenarios.\n2. The generated text for visual classification might not be visually meaningful.\nNo negative societal impact was observed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Overall I think it is an interesting idea. By using the shared embedding from the two modalities, it generates relatively \"cheap\" data by using an LLM to train a classifier directly for the target classes. This classifier further gives a better \"starting point\" for the pseudo-labeling in the next stage.\n2. The evaluations on the benchmark datasets show considerable improvements over other label-free methods. It also outperforms 1-shot fine-tuning methods on certain datasets.", "weaknesses": "1. What bothers me the most is that the performance of only using Llama descriptions is quite underwhelming (Table 3). It has a big gap to only using class names. Even the pure GPT description is only 1% better than class names on average. I think the failure of Llama needs more explanations. Also, these results might imply: 1) the generated texts are not that effective; 2) you need a really good LLM (GPT3 or better) to generate decent text descriptions to make this method work. IMO, both points will limit the contribution and impact of the proposed method. \n2. I feel the diversity and quality of the generated texts will be very important in the proposed approach, as it will affect the generalization ability of the trained classifier. It is better to have some ablations on these aspects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort spent in reviewing our paper. In the following, we provide a response to the questions raised in the review.\n\n**Failure of Llama needs more explanation.** While comparing the descriptions of CIFAR-10 classes generated by Llama and GPT respectively, we found the following differences which could have an effect on the eventual classification performance.  \n\n- In most cases, GPT mentions the class name in its descriptions, while Llama does not. Our experiments (in Table 1 of global response), highlight that this property influences the results. \n    \n- The Llama descriptions often contain technical terms (e.g., in $\\sim53$% of the reviewed descriptions), while GPT descriptions use more common terms (e.g., in $\\sim94$% of the reviewed descriptions). To describe the airplane category, Llama mentioned terms like empennage, livery, composite material, whereas GPT descriptions contain terms, such as tires, metal plates etc. As (intuitively) technical terms were rarely encountered during the CLIP text encoder pre-training (e.g., the airplane class contained only $\\sim8$% technical terms in the reviewed LAOIN-400m [1] captions, which can be a representative of CLIP alt-text), it might be harder for it to leverage those terms for classification. \n    \n- The Llama descriptions had a larger variety of sentence structures than GPT, combining technical information with descriptive elements, whereas the GPT used simpler sentence structure with less variety, mentioning important details in a more succinct and structured manner. As the CLIP text encoder is trained on web collection of images and their alt-text, it is logical that it responds better to tuning with simpler sentence structure.  \n\nTo test the benefits of including the class names, we prepend the class names to each of the Llama descriptions and provide results in Table 1 of the global response.\nWe see that by simply prepending the class name the performance increased by $3.8$% for CIFAR-100 and the average result is also $0.8$% better than the results obtained by using the GPT descriptions (while the results obtained by using descriptions from both the LLMs are on average $2.8$% better than using class names alone).\n\n**Generated descriptions might not be so effective.** In Table $3$ of the main manuscript, we see that the GPT descriptions perform well on datasets composed of naturally occurring images (CIFAR-10/100 and ImageNet), showing $4.0$% increase on CIFAR-100 and $2.1$% increase on the large-scale ImageNet, as compared to using only the class names. \n\nOn the out-of-distribution datasets like ImageNet-Adversarial (A) and ImageNet-Rendition (R), GPT descriptions show a degradation. \nFor ImageNet-A it is expected, as learning a visual classifier with detailed LLM descriptions of classes enhances attention to those class details, while ImageNet-A contains adversarial images collected such that details of wrong classes appear in the images to confuse the classifier. On ImageNet-R, composed of renditions of ImageNet classes, we can easily improve the performance by slightly changing the LLM prompts to generate class descriptions for text-only pre-training in our LaFTer. For example, we prompt the LLMs (Llama and GPT) to provide us with descriptions of different types of renditions of objects present in the ImageNet-R dataset (e.g., art, graffiti, embroidery). An example prompt to the LLM is: \n\n- Describe what an _embroidery rendition_ of a {category} looks like. \n\nBy using these _targeted descriptions_ for the datasets, we gain substantial improvements. For GPT, we obtain an improvement of $4.9$% ($63.6$% $\\to$ $68.5$%), and for Llama descriptions prepended with class names we gain $2.6$% improvement in accuracy ($63.5$% $\\to$ $66.1$%). Generating dataset-specific descriptions opens interesting future work directions for cross-domain text-only tuning (as already indicated by our preliminary results on ImageNet-R). Since, the targeted responses help to mitigate distribution shifts from the text side, it is naturally reflected on the vision side due to the shared embedding space.\n\n**Method requires a really good LLM (GPT or better).** In our paper, we perform experiments with the open-source Llama and in the rebuttal discuss the reasons why the results obtained by using Llama descriptions lag behind GPT, and how they can be improved (Table 1 of global response). \nIn an effort to improve the accessibility of LaFTer, we also perform experiments with recently open-sourced Llama-2 by META. \nThe results (Table 1 of global response) show that the descriptions generated by Llama-2 provide competitive results to the GPT descriptions and even outperform them on average (by $1.2$%) when prepended with the class names. These results show that the recently open-sourced LLMs can be readily used as an alternative to GPT.  \n\n**Diversity and quality of descriptions.** In Figure 1 of the global response, we plot the resulting accuracy after randomly sampling a certain amount of GPT descriptions per class. We observe that as we increase the amount of descriptions per class, the accuracy also increases, highlighting that the diversity of generated descriptions is indeed important. Furthermore, our experiments with generating _targeted descriptions_ for ImageNet-R (described above), demonstrate that the quality of responses from the LLM has a strong influence on the eventual classification performance. \n\n**Imposing constraints on LLMs for visually meaningful texts.**\nWe expect that our prompts to the LLMs would impose conditions on the responses to produce visually meaningful texts. Two example prompts to the LLM are: \n\n- Describe what a {category} looks like?\n- How can you identify a {category}?\n\nThese prompts instruct the LLM to produce _category descriptions_ in a visually meaningful way. Please refer to the Supplementary Section 3.1, for a complete list of prompts.\n\n[1] Schuhmann et al. LAION-400m, NeurIPS 2021."}, {"review_id": "HRkY7j6OfM", "reviewer": "Reviewer_bBjN", "summary": "This paper proposed a new approach to improve zero-shot vision recognition capability which leveraged the common embedding space for image and text. Specifically, with a pre-trained VLM (vision-language model), the authors leveraged LLM to automatically generate multiple language prompts for training a text-based classifier, which can be adapted for vision classification thanks to the shared embedding space. The classifier is further applied to augmented images to provide pseudo labels for finetuning the visual encoder in a label-free manner (pseudo labels generated from the text classifier). The authors perform extensive experiments to validate the effectiveness of the proposed training paradigm and on multiple benchmarks the model achieved significant performance gain.", "questions": "1. Some other recent work used similar ideas for finetuning VLMs so maybe it's worth mentioning and provide a comparison. For example, [1] used a similar idea to generate multiple text and image prompts for finetuning VLMs domain adaptation.\n\n[1] https://arxiv.org/abs/2306.16658", "limitations": "The paper has a dedicated section discussing about the limitations with the single linear layer of text classifier. Some more important limitations may need emphasizing as I suggested in the weaknesses section that the general assumption of the effectiveness of the proposed approach is the high quality of the underlying VLM model. No obvious potential negative societal impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is intuitive and theoretically sound. The shared image-text embedding space naturally serves as a bridge for applying text-trained classifiers on image domains. The simple idea turns out to be very effective and brings significant performance boost across multiple datasets.\n2. The conclusion is supported by extensive experiments with various ablation studies, showing many interesting patterns.\n3. The paper is well-written and in good shape, it's easy to read.", "weaknesses": "1. Section 3.2 is named as \"unsupervised finetuning\" but technically since the model is trained with smoothed cross entropy loss, it's not truly \"unsupervised\", just that the labels are from text-based classifier (thus \"pseudo labels\"). The name is somewhat misleading as audience may get confused with unsupervised approaches.\n2. CLIP is pre-trained with massive data thus demonstrating good zero-shot capability. Although this paper experimented on multiple benchmark datasets, the images of these datasets are all natural thus similar in style. The proposed approach achieves significant performance gain without real labels but the assumption is that the pseudo labels are relatively accurate due to image consistency. Therefore it is better to emphasize the point at least as a potential limitations. It might be better to be more careful with some claims e.g. L317 regarding the cross-modality transfer capability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort spent in reviewing our paper. In the following, we provide a response to the questions raised in the review.\n\n**Unsupervised finetuning naming convention.** Thank you for pointing it out. In our manuscript, we refer to the second \nstage of LaFTer (Section 3.2 of the originally submitted manuscript) as 'Unsupervised Finetuning on Target Domain Images' \nbecause we do not use any _ground truth_ labels from the training samples of the downstream datasets.\nThe supervision in the Cross-Entropy Loss is generated by the output predictions of the model itself. \nHowever, we do acknowledge the source of confusion, which might be caused \nbecause 'Unsupervised Finetuning' has been used in a variety of contexts, more commonly for \n'Unsupervised Representation Learning'. Following the suggestion, we will consider changing the name of Section 3.2 to \n'Label-Free Finetuning on Target Domain Images'. \nWe would also be happy to incorporate any further suggestions which may be provided during the discussion period. \n\n**Distribution of images of the downstream datasets.** In addition to experimenting with our LaFTer on datasets \ncontaining images collected in the real-world (e.g., CIFAR, ImageNet, SUN-397, UCF-101), we also tested our method \non out-of-distribution variant of the original ImageNet dataset (ImageNet-Adversarial), which contains adversarial images for the original categories in the ImageNet dataset, as well as on satellite data (EuroSat), which has images from space that are \nlikely very rare in CLIP web train data. Please note that our method shows considerable performance gains by adapting \nto the out-of-distribution images as well. For example, for EuroSat we obtain a performance gain of $28.8$% over the \noriginal CLIP model. However, we do agree that since CLIP is trained on a large corpus of data scraped from the \ninternet, it is extremely challenging to know the _true_ data distribution of the training set and hence, it might \nbe the case that CLIP has seen similar images to the out-of-distribution variants during its large-scale pre-training. \nWe will emphasize this point and also rephrase $\\text{L}317$ in the limitation section, of the revised manuscript.  \n\n**Recent work for finetuning VLMs.** Thank you for highlighting this relevant work. \nPlease note that it was released on ArXiv on $29^{th}$ of June, 2023 (long after the NeurIPS deadline). \nHowever, we will discuss (and compare, provided the code is open-sourced) this work in the related work section \nof our revised manuscript. "}, {"review_id": "nszKEBofia", "reviewer": "Reviewer_zXQC", "summary": "This paper proposes a finetuning approach for Vision-Language models that does not require any labels. It begins by demonstrating the transfer of information between different modalities and training a classifier using natural language inputs. This classifier achieves the successful classification of visual data. Furthermore, it utilizes this pre-trained classifier that relies solely on text inputs in our pseudo-labeling pipeline. This enables it to effectively and efficiently fine-tune the Vision-Language models with fewer parameters.", "questions": "Please answer all my concerns raised in the weaknesses section. ", "limitations": "In this paper, the authors acknowledge a limitation of their proposed model. They state that in their current work, they employed a straightforward neural network-based classifier and consider exploring more complex neural networks as part of future work. However, it is worth noting that from my observation of Table-1, the proposed model also faces challenges when applied to fine-grained datasets.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1- The proposal introduces a label-free approach to zero-shot learning, capitalizing on the benefits of pretrained LLM models. Additionally, it enhances the model's performance by finetuning it in an unsupervised manner. This novel approach presents a new direction for zero-shot learning research, eliminating the need for labeled datasets even for seen classes.\n2- The proposed method extends its experiments to include a few-shot framework and demonstrates a noteworthy improvement compared to the baseline approaches.\n3- The experiments are conducted on twelve (12) diverse datasets, encompassing coarse-grained, fine-grained, and natural scene datasets. A comprehensive ablation study is conducted to demonstrate the effectiveness of the proposed model and the contribution of its individual components.", "weaknesses": "1- Upon analyzing Table-1, it is evident that the proposed model exhibits lower performance for the Flower and SUN datasets, both commonly utilized in zero-shot learning and are fine-grained datasets. This observation raises surprise as the proposed model struggles specifically with fine-grained datasets.\n2- The visualization of embeddings obtained from both the text encoder and image encoder would provide a clearer understanding of the effectiveness of the pretrained CLIP (LLM) model. Additionally, generating images based on textual descriptions of classes would offer valuable insights into how the proposed model operates within the zero-shot framework.\n3- The distinction between the weakly-augmented view and the strongly-augmented view is unclear when it comes to unsupervised fine-tuning. What are the benefits of utilizing the weakly-augmented view for pseudo-labels instead of the strongly-augmented view?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort spent in reviewing our paper. In the following, we provide a response to the questions raised in the review.\n\n**Performance on fine-grained datasets.** On datasets like Flower-102 and SUN-397 our method improves the base \nCLIP model by $4.4$% and $3.7$% respectively.\nHowever, as compared to the other unsupervised finetuning baseline UPL, our LaFTer slightly lags behind \non Flowers-102 and on SUN-397 (while significantly improving over UPL on average across all tested datasets).\nOne potential reason could be the descriptions generated from the LLM for the fine-grained categories of these (fine-grained) datasets.\nTo probe into this, we consider the example of Flowers-102 dataset. \nIt contains categories such as,`love in the mist, mexican aster, alpine sea holly, ruby-lipped cattleya`, \nwhich are names of flowers. \nIt could be challenging for an LLM to provide descriptions for these categories without having any prior knowledge \nabout the _parent category_ of these classes.\nIn order to generate more exact descriptions about these finegrained classes, we slightly modify the prompts to the \nLLM (GPT) and add the parent category (flower) in the prompts. \nA couple of examples for the modified prompts are as follows (please refer to the supplementary Section 3.1, \nfor the original list of prompts): \n\n\n- Describe what the _flower_ type {category} looks like.\n- How can you identify the _flower_ type {category}?\n\n\nBy generating descriptions of the finegrained categories by prompting the LLM with the parent category _flower_ we get \nan improvement of $1.4$% ($71.0$% $\\to$ $72.4$%) for our LaFTeR final classification accuracy for this dataset.\nThe improved results show a gain of $0.9$% over the UPL baseline (similarly, we gain $0.3$% over UPL for SUN-397), and once again demonstrate the ease with which \nthe text-only pre-training proposed in LaFTer can facilitate further customization to downstream tasks and domains \nvia small (essentially cost-free) changes in the prompts. \nBy adapting the LLM instructions to include the _parent category_, we can easily generate better-targeted \ndescriptions for any fine-grained dataset. \nNotice that we also experiment with _targeted prompts_ for the out-of-distribution dataset ImageNet-Rendition \nand show a considerable performance improvement. \nWe discussed those results in our response to `Reviewer YPJ1`.\nAt this point, we leave the exploration of _targeted prompts_ towards downstream datasets as an interesting and \nseemingly very promising future work direction.\n\n**Visualization of embeddings.** In Figure 2 of the global response, we provide TSNE projections of the \nvisual and text embeddings, before and after adaptation for the $10$ classes in the EuroSAT dataset. \nAnalyzing the TSNE projections, we observe that the adaptation with our LaFTer \nresults in more pronounced category clusters with larger inter-class distances, and also that after applying LaFTer the class-label text \nembeddings are better aligned with these category clusters. \n\n\n**Benefits of weakly-augmented view for pseudo-labels.** In the context of self-supervised learning with pseudo-labels, \nseveral approaches have introduced the concept of consistency regularization by deriving pseudo-labels from \nweakly-augmented data samples, as demonstrated in works like FixMatch [1].\nThe rationale behind utilizing weakly-augmented views for generating pseudo-labels stems from the expectation that the \nmodel's predictions on these mildly augmented versions will exhibit greater confidence compared to \npredictions from strongly augmented counterparts.\nThe substantial alterations introduced by strong augmentations may distort the inherent data structure of the \nimage, potentially introducing noise to the predictions.\nConversely, when pseudo-labels are generated from weakly-augmented views, they tend to be less noisy, \nfacilitating smoother and more effective guidance of the learning process.\n\n[1] FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence, Sohn et al., NeurIPS 2020.\n\n"}, {"review_id": "R9sKEFpm4D", "reviewer": "Reviewer_o7C2", "summary": "The paper proposes a novel method for improving zero-shot classification of VL models, in an unsupervised manner. This is done without additional visual labeled examples, yet with additional unlabeled examples. They rely on LLM to generate a dataset of text describing the desired classes. Then, they train a text classifier on top of the VL textual encoder using the generated dataset. As a second stage, they use a set of relevant unsupervised images as well as a set of designed augmentations, to finetune the classifier and visual prompts, adjusting it to the VL models image representations. In an extensive evaluation, the authors show a significant performance enhancement, and ablate their method methodically. ", "questions": "-\tCould the authors please verify if the experiments denoted by LaFTer* are just the second stage of LaFTer using CLIP as the initial text classifier? \nIf so, I think these results are more suitable to appear in the ablations section (Table 4).  \n", "limitations": "-\tThe method requires access to a set of unlabeled images from the same distribution.\n-\tThe method is limited by the quality of the visual representation of the VL model.\n", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "-\tThe paper is clearly written and easy to follow. \n-\tThe performance enhancement shown by LaFTer is very significant and consistent across most test cases. \n-\tAlthough inspired by previous works, the idea of tunning the visual encoder of VL models using text supervision is highly interesting. \n\nOverall, I believe the idea of the paper is very interesting, and the results are a significant improvement over CLIP.\n", "weaknesses": "My main concern is the performance of CLIPPR baseline. In most cases, it has been found to be less successful than the original CLIP, which is not intuitive with the method. Could the authors explain this gap please?\n\nOne other minor concern is the statement in the abstract about LaFTer being the first to reduce the gap from the supervised baseline. Table 1 shows LaFTer is not the first. Table 2 shows that it does not close the gap for about 50% of the cases, using only few-shot supervision. I believe this is misleading statement. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort spent in reviewing our paper. In the following, we provide a response to the questions raised in the review. \n\n**Performance of CLIP-PR baseline.** The CLIP-PR reported image classification results only on CIFAR-10 and ImageNet. \nWe used the official codebase provided by the CLIP-PR authors to evaluate their method on all the additional \ndatasets reported in our paper. We used the values for all hyperparameters as recommended by the CLIP-PR authors in \ntheir official release.\n\nWe would also like to note that the zero-shot classification accuracy of the original (non-adapted) CLIP baseline \nreported in CLIP-PR is lower than we observed in our experiments and reported in our paper.\nFor example, in Table $3$ of the CLIP-PR paper, they report $57.59$% CLIP zero-shot Top-1 Accuracy (%) for \nImageNet, whereas, we measured CLIP to obtain $61.9$% (Table $1$ of our submitted manuscript) while using the same ViT-B/32 backbone from \nOpenAI as reported to be used by CLIP-PR and observed from their official code. \nSimilarly, for CIFAR-10 we obtain zero-shot accuracy of $88.8$% for base CLIP, while CLIP-PR reports $85.17$%.\n\n**Confusion about LaFTer being the first to reduce the gap from the supervised baselines.** Thank you for \npointing it out, and we agree that the referred statement (in its current form) might lead to confusion.\nHowever, the purpose was to highlight that LaFTer is the first method to jointly combine two concepts in order to \neffectively finetune Vision-Language models in an unsupervised manner:\n\n- Substituting labeled visual instances of the categories by generating text through Large Language Models (Learning an Image Classifier using Text, Section 3.1). \n- Finetuning the visual encoder in an unsupervised manner by using unlabeled image data (Unsupervised Finetuning on Target Domain Images, Section 3.2).\n\nIt is true that CLIP-PR and UPL use unlabeled image data for unsupervised finetuning of Vision-Language models, however, \nLaFTer is the first method to also take advantage of auto-generated text from the Large Language Models which \nsubstitutes visual instances. \nWe will make the referred statement in the abstract concise and more clear in the updated version of the manuscript. \n\n**Move LaFTer$^\\*$ results to the ablation section.** Thank you for the suggestion. \nAnalyzing the tables again, we agree that it will be more suitable to move LaFTer$^*$ results to the ablations section \n(Table  $4$). \nWe will rearrange the results in the updated manuscript."}]

Author rebuttal:
We thank all the reviewers for their efforts to review our paper and for 
providing insightful feedback. 

We are happy to see that they found our work: **novel** `(YPJ1, zXQC, bBjN, o7C2)`, 
**interesting** `(YPJ1, o7C2)` and **theoretically sound** `(bBjN)`.
Furthermore, we also thank them for highlighting that our work is **extensively evaluated** 
on different benchmarks and shows **strong empirical results** `(YPJ1, bBjN, zXQC, o7C2)`, 
contains **appropriate ablation studies** showing many interesting patterns and contributions of 
individual components of the approach `(bBjN, zXQC)`, and also for finding our work 
**easy to read and understand** `(bBjN, zXQC)`.

In the attached PDF, we present more insights to gain a further understanding of our LaFTer. We refer to this PDF throughout the response (to each reviewer)  as **`global response`**.  

Meta-review: The paper proposes a technique to train a classifier zero-shot (with no labelled data) using a pre-trained CLIP model and a language model to generate pseudo-labels. The reviews appreciate that the method was interesting and intuitive, as well as the breadth of the experiments and strength of the results. During rebuttal, the addition of adjustments to one of the baselines and discussion of the generated texts were useful. At the end of the discussion, the decision is a unanimous accept.
Decision: Accept (poster)