Title: LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections
Venue: NeurIPS 2023
Accepted: yes
Keywords: VL Models
Abstract: Recently, large-scale pre-trained Vision and Language (VL) models have set a new state-of-the-art (SOTA) in zero-shot visual classification enabling open-vocabulary recognition of potentially unlimited set of categories defined as simple language prompts. However, despite these great advances, the performance of these zero-shot classifiers still falls short of the results of dedicated (closed category set) classifiers trained with supervised fine-tuning. In this paper we show, for the first time, how to reduce this gap without any labels and without any paired VL data, using an unlabeled image collection and a set of texts auto-generated using a Large Language Model (LLM) describing the categories of interest and effectively substituting labeled visual instances of those categories. Using our label-free approach, we are able to attain significant performance improvements over the zero-shot performance of the base VL model and other contemporary methods and baselines on a wide variety of datasets, demonstrating absolute improvement of up to $11.7\%$ ($3.8\%$ on average) in the label-free setting. Moreover, despite our approach being label-free, we observe $1.3\%$ average gains over leading few-shot prompting baselines that do use 5-shot supervision.

LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections

Abstract

Recently, large-scale pre-trained Vision and Language (VL) models have set a new state-of-the-art (SOTA) in zero-shot visual classification enabling open-vocabulary recognition of potentially unlimited set of categories defined as simple language prompts. However, despite these great advances, the performance of these zeroshot classifiers still falls short of the results of dedicated (closed category set) classifiers trained with supervised fine-tuning. In this paper we show, for the first time, how to reduce this gap without any labels and without any paired VL data, using an unlabeled image collection and a set of texts auto-generated using a Large Language Model (LLM) describing the categories of interest and effectively substituting labeled visual instances of those categories. Using our label-free approach, we are able to attain significant performance improvements over the zero-shot performance of the base VL model and other contemporary methods and baselines on a wide variety of datasets, demonstrating absolute improvement of up to 11 . 7% ( 3 . 8% on average) in the label-free setting. Moreover, despite our approach being label-free, we observe 1 . 3% average gains over leading few-shot prompting baselines that do use 5 -shot supervision.

1 Introduction

Vision and Language (VL) models [1-8] recently became the de-facto standard for generalized zeroshot learning enabling recognition of arbitrary (open) set of categories provided with just their text descriptions and without requiring any additional data or training. However, this incredible flexibility comes at a performance cost and all the VL models, including the most widely used CLIP [1], still require additional supervised training (e.g. tuning its vision encoder) to compete with the closed set (of target categories) supervised training methods. Naturally, this incurs undesirable adaptation costs for those, otherwise very versatile and flexible, foundation models. Image annotations are often expensive to collect, especially for legacy vision systems, such as traffic surveillance, quality control, or security.

In this paper we propose LaFTer, an approach for completely label-free parameter efficient finetuning of VL models to a set of target classes. Our goal is to substitute the need for expensive-to-obtain image annotations with unsupervised finetuning of VL models. We show that since the VL models share a common text-image embedding space (due to their contrastive learning objective), there is a possibility to train using embeddings of samples from one modality (e.g., auto-labeled text) and then successfully apply what we trained to classify embeddings of samples from the other modality

3

flamingo:

A flamingo has

pink feathers and a long neck.

flamingo:

A photo of a

flamingo.

flamingo:

A flamingo flying

in the sky.

flamingo:

A blurry photo of a

flamingo.

Text

Classifier

Text

Classifier

Text

Encoder

Vision

Encoder

LLM

Unsupervised

Finetuning

with Visual Inputs

Learning Visual Classifer on Text

Finetuning on Visual Data

Classification after Finetuning

0

3

6

9

12

EuroSat

CIFAR-100

CIFAR-10

ImageNet-A

UCF

ImageNet

Caltech

ImageNet-R

Flamingo

Trilobite

Trantule

Brambling

+2.7

+3.0

+4.3

+4.6

+6.6

+7.0

+8.8

+11.7

(e.g., unlabeled images). More specifically, we show that it is possible to train a neural network (e.g., a classifier) to classify text instances that can be successfully used to classify images, showing successful cross-modal transfer. Instead of collecting labeled visual instances, in our label-free LaFTer approach, we are mining text descriptions of the target categories by prompting an LLM (e.g., GPT-3 [9]) and combining them with handcrafted prompts, as shown in Figure 1. After creating such a text dataset, we train a neural network to classify each text instance (sentence) in it to the source class label that was used to produce the instance. This text classifier can be readily used to classify images when used on top of a CLIP visual encoder. Furthermore, we take advantage of this text-only pre-trained classifier by employing it in a pseudo-labeling pipeline (inspired by FixMatch [10]), to further finetune the CLIP vision encoder on an unlabeled image collection. To reduce overfitting and keep the finetuning parameter efficient, we make use of Visual Prompt Tuning [11] combined with adapting the affine transformations (scale and shift) of the normalization layers in the otherwise frozen network.

To summarize, our contributions are as follows:

We offer a new, completely label-free, and parameter-efficient ( 0 . 4% learned parameters), cross-modal transfer approach (LaFTer) to finetuning a VL model for improved recognition performance on a set of target classes using a set of automatically generated texts for those classes and an unlabeled image collection.

We show that text-side-only training on an automatically collected language knowledge (e.g. through LLM prompting) can be effective in bootstrapping a visual classifier built on top of an aligned VL model to consistently improve its performance in a variety of downstream classification benchmarks.

In an extensive evaluation of our approach, spanning over 12 popular benchmarks, we show that it: (i) attains up to 28% absolute improvement over the base VL model (CLIP) performance ( 6 . 7 %on average); (ii) improves over previous SOTA under the same label-free setting by up to 11 . 7% ( 3 . 8 %on average); and (iii) performs favorably while comparing with SOTA parameter efficient few-shot methods, matching from above average performance of 5 -shot supervised setting despite using no labeled samples.

2 Related work

Large-scale Vision&Language (VL) Models: Remarkable performance in many zero-shot downstream tasks has been attained with VL models pre-trained using contrastive losses on large-scale noisy image-text data (e.g., CLIP [1] and ALIGN [13]). Different ideas have been tried to improve the image-text representation alignment, such as leveraging off-the-shelf object detectors [14-16], or using cross-attention and additional objective functions such as image-text matching and masked language modeling [4, 17-19], or filtering noisy captions (e.g., BLIP [4]). Some additional properties of language structure are utilized in [2, 20-23]. DeCLIP [2] finds additional positives for the contrastive loss by seeking textual nearest-neighbors. Geometrically consistent representations (within and across the paired image-text modalities) are employed in CyClip [20]. Recently, few methods have attempted improving VL performance using additional supervision [3, 24], finer-grained interactions [25], modern Hopfield networks [26], optimal transport distillation [27], cycle consistency [28], and hierarchical feature alignment [29]. However, VL models performance still falls short of classifiers trained with supervision on the downstream target data. Most of the approaches to fine-tuning VL models [30, 31] leverage annotated data for this finetuning. In contrast, in our work we propose a completely label-free approach for adapting a VL model to the target task. Methods which are most related to our work, UPL [12] and CLIP-PR [32] also finetune VL models in an unsupervised manner. UPL finetunes learnable text prompts (similar to [30]) by relying on confidence sampled pseudo-labeling, whereas CLIP-PR relies both on offline pseudo-labeling and label distribution prior from the training data. In contrast, in our work, we first leverage textual knowledge from LLMs and design a self-supervised text-only pre-training task showing that it can be employed in an unsupervised parameter-efficient finetuning of the VL model on a set of unlabeled images. Extensive empirical evaluations show the benefits of our approach w.r.t. UPL and CLIP-PR.

Prompt Tuning. Prompt tuning belongs to a wider family of pararamter-efficient finetuning methods that can also be applied to VL models [33, 34]. Originating in NLP [35, 36], visual prompt tuning was recently shown to be effective also for vision backbones [11, 33, 37]. CoOp [33] learns prompt vectors by minimizing the prediction error using the cross-entropy loss. ProDA [37] learns diverse prompts from data to cope with variance of the visual representations. UPL [34] proposes an unsupervised prompt learning framework. TPT [38] proposes a test-time prompt tuning framework. CLIP-Adapter [39] and Tip-Adapter [40] employ an alternative parameter efficient finetuning strategy using additional adapter modules. Recently, CoCoOp [41] proposed a Meta-Net for generating image-adaptive prompts for improved generalization to distribution shifts. In this work, we employ parameter-efficient Visual Prompt Tuning (VPT) [11] as the means for adapting the visual encoder of the VL model, but in contrast with other works we do not use any supervised data. Instead, we use a classifier, trained in the text domain using texts automatically generated by an LLM from the set of target classes, to generate pseudo-labels for an unlabeled image collection as a form of self-supervision. Furthermore, we propose to combine VPT with tuning the scale and shift parameters of normalization layers, previously proposed for handling domain shifts [42-44], but to the best of our knowledge never before used to tune VL models.

Pseudo-labeling. Pseudo labeling, also known as self-training, is commonly used as a semisupervised learning technique. Popularized in the seminal works of [45, 46], pseudo-labeling was extended to utilize consistency regularization [47] and augmentation [48]. The pseudo-labeling pipeline of our method is inspired by FixMatch [10] that combined consistency regularization with confidence-based filtering, surpassing SOTA semi-supervised techniques at the time. It was later extended with a non-parametric classifier in PAWS [49]. These techniques are proposed for semisupervised learning and require some amount of labeled instances. In contrast, we propose a label-free method for improving VL models performance on a set of target classes. To achieve this, our method finetunes VL models in a parameter-efficient manner by generating pseudo labels through a text-only classifier trained on a corpus of text data generated by prompting language models.

3 LaFTer

CLIP [1] consists of a vision encoder and a text encoder, which project images and texts to a common embedding space. It has been trained on a very large volume (400M) of image-text pairs to align the embedding of each image to the embedding of its corresponding text, at the same time pushing it away from the embeddings of unrelated texts corresponding to other images. In [1], it was demonstrated that the text and vision encoders enable effective zero-shot image classification. Given the set of class

- Cat

- Dog

- Bird

.....

Classes

LLM

Text Dataset

Text

Encoder

Text

Classifier

Text

Classifier

Classes

Cross Entropy

Loss

Update Classifier Weights

Update Classifier, LN Affine, Prompts

Cross

Entropy

Loss

Vision

Encoder

Vision

Encoder

Pseudo-Label

Class Logits

Prompts

Stop Grad

+

A

cat

has four legs and a

tail.

A

dog

has furry tail.

A photo of a

bird

.

A

bird

flying in the sky.

A blurry photo of a

dog

.

Text Dataset

A

cat

has four legs and a

tail.

A

dog

has furry tail.

A photo of a

bird

.

A

bird

flying in the sky.

A blurry photo of a

dog

.

Text

Classifier

Shared

names C , the text encoder u is used to generate the embedding u c of a prompt for each class c ∈ C , typically of the form ' A photo of ... ', completed with the class name. Each test image x is encoded by the vision encoder v and classified using its cosine similarity cos to the text embedding of each class. The likelihood of a predicted class ˆ c is computed with a softmax,

where τ denotes the temperature constant.

The zero-shot classifier (1) does not require any training data, but is typically outperformed by networks trained on data from the target domain. Existing approaches to reducing this performance gap [30, 31] rely on fine-tuning the zero-shot classifier in a few-shot regime. While effectively enhancing accuracy, they incur additional costs of curating and annotating the training data. In contrast, we propose a two-step approach for fine-tuning the network without requiring any annotations. In the next section 3.1, we describe a technique to train a visual classifier on purely textual data, conveniently generated by Large Language Models. In Section. 3.2, we propose an unsupervised training setup that lets us benefit from unlabelled data from the target domain, when such data is available. Combining these techniques significantly reduces the performance gap to the supervised classifier and is even competitive with few-shot approaches despite not using any supervision. These results are provided in Section 4.2.

3.1 Learning an Image Classifier using Text

Aligning the text and image embedding spaces is the key idea underlying Vision Language models. It gives CLIP and similar methods, their main advantages: the capacity to be trained on an extremely large volume of data available on the Internet and the resulting effectiveness as zero-shot classifiers. Recently, Zhang et al. [50] explore yet another advantage of the alignment between text and image embedding spaces - it allows diagnosing and rectifying vision models spawn from the VL model vision encoder by using the other modality. In particular, it hints that it might be possible to finetune a zero-shot image classifier on textual data.

While acquiring and annotating images requires manual labor, a training set of annotated texts can be constructed automatically, using a Large Language Model, like the GPT-3 [9]. Generating text using an LLM constitutes a convenient alternative to text mining, as LLMs represent extremely

large text corpora on which they were trained. Moreover, prompting LLMs is much more efficient than searching a large body of text for the words of interest. To construct our training set, we take inspiration from [51] and for each class c , we prompted GPT-3 with queries of the following kind: ' Describe what a [ c ] looks like. ' We repeated the prompting for each class c , in the dataset and complemented the resulting set of synthetic texts with text generated procedurally, using the same hand-crafted templates as for constructing prompts for the zero-shot classifier, for example, ' A photo of a [ c ]. '. We defer the full list of prompts (to LLMs) and templates to the supplementary material.

The capacity to finetune the classifier in a supervised text classification setting lifts the architectural constraints imposed by the zero-shot setup. More precisely, the class likelihoods no longer need to be produced according to (1), but can instead be computed by a trainable classifier f , of arbitrary architecture. We train f with the Smoothed Cross Entropy loss L SCE [52]. To regularize training, we add Gaussian Noise n ∼ N (0 , I ) to the l 2 -normalized feature vector from the clip text encoder. Formally, given a text training set T consisted of pairs of text fragments t and class labels c , the training objective is:

where θ is the parameter of the classifier. Training of the text-only classifier is very efficient. For example, 3000 epochs of training the classifier on the data set of 130000 text sentences, representing the 1000 classes of the ImageNet [53] dataset is completed in ∼ 120 seconds on an NVIDIA 3090 graphics card.

3.2 Unsupervised Finetuning on Target Domain Images

Text-only training does not require any image data. However, in many applications, unlabeled images from the target domain are readily available, or can be acquired at a low cost. Given a set of unlabeled images we propose to take advantage of the text-only pre-trained classifier and use it in a pseudo-labeling pipeline on top of the vision encoder as demonstrated at the bottom part of Figure 2.

Inspired by Fixmatch [10], for each training image x , we generate two views: the weakly-augmented view α w ( x ) and the strongly-augmented view α h ( x ) , where α denotes a stochastic augmentation function. Contrary to Fixmatch, in our unsupervised finetuning we set α w as an identity transformation. For α h we use the augmentations proposed in [54]. The weakly-augmented view serves to generate a pseudo-label. To that end, it is passed through the vision encoder v and the text classifier f , yielding a vector of class probabilities p . Class probabilities give rise to pseudo labels, that we denote by ˆ c ( x ) . Formally,

ˆ c ( x ) is not differentiable, and we do not backpropagate the error signal through it during training. The pseudo labels p ˆ c , are used as ground truth for training the network on the heavily-augmented views. Since the vision encoders in the two branches share the weights, the pseudo-labels are generated in an online manner and are constantly refined as adaptation is progressing. This is in contrast to UPL [12] and CLIP-PR [32] which rely on offline pseudo-labeling by only generating them once from frozen CLIP encoders. We freeze the weights of the text-only pre-trained classifier in the pseudo-labeling branch since it leads to more stable adaptation.

When processing the heavily augmented views α h ( x ) , we augment the vision encoder by visual prompt tuning [11]. That is, we append randomly initialized, learnable parameters (a matrix of size N P × d v where N P is the number of prompts and d v is the channel dimension of the vision encoder) to the input of the vision transformer after the initial embedding layer. This helps the network account for the heavy augmentation of the input and, as we show in Section 4.3, boosts the performance of the finetuned classifier. We denote the vision encoder with prompting by v p . The prediction for the heavily-augmented image is obtained as f ( v p ( α h ( x ))) .

The network is trained with the smoothed cross entropy loss L SCE [52]. We denote the set of unlabelled target domain images by D , and formalize the training objective as

where θ and η denote the finetuned parameters of the classifier and of the vision encoder, respectively. The parameters η are the visual prompts and the scale and shift (affine) parameters of the normalization layers of the vision encoder. The selection of η is motivated by keeping the adaptation parameterefficient . For LaFTer, the number of trainable parameters are less than 0 . 4% of the entire model parameters, making adaptation extremely lightweight.

4 Experimental Evaluation

Here we first provide a description of the datasets and baselines we use in our evaluations, then explain our implementation details and later discuss our experimental results in detail.

4.1 Evaluation Setting

Datasets: We extensively evaluate our approach on 12 different datasets belonging to widely different domains. More specifically, we use four datasets containing common natural categories: ImageNet [53], CIFAR-10/100 [55] and Caltech-101 [56]. EuroSat [57] contains satellite images of 10 different locations. UCF-101 [58] is an action recognition dataset. SUN-397 [59] contains images from 397 naturally occuring scenes. Flowers-102 [60] is a fine-grained classification dataset for classifying different categories of flowers commonly occuring in the United Kingdom. Whereas, ImageNet-A (Adversarial) [61], ImageNet-S (Sketch) [62] and ImageNet-R (Rendition) [63] are different versions of the original ImageNet validation set. In our setting, we divide the ImageNet-A, ImageNet-S and ImageNet-R in to 80% train and 20% test set. For all other datasets we use the splits provided by [30].

Baselines: We compare LaFTer with baselines which are label-free (not requiring any additional labeled images):

CLIP [1] denotes zero-shot classification scores by computing the cosine similarity between embeddings from frozen CLIP encoders.

UPL [12] adds learnable text prompts to the CLIP text encoder and finetunes them in an unsupervised manner by employing confidence-sampled offline pseudo-labeling.

CLIP-PR [32] optimizes an adapter on top of the CLIP vision encoder by using label distribution priors from the training set of the downstream datasets and generating offline pseudo-labels.

For completeness, apart from these 3 baselines, we also provide a comparison with few-shot finetuning method CoOp [30], which learns soft text prompts using k labeled images per class ( k -shot).

Implementation Details: For all our experiments, unless otherwise stated, we use a ViT/B-32 CLIP pre-trained model from OpenAI [1]. The Text-Only classifier (Section 3.1) is implemented as a

single linear layer, with the output units equal to the number of classes in the dataset. For training this classifier, we load the complete text dataset as a single batch and optimize the network using AdamW as optimizer, with a learning rate of 0 . 001 . For unsupervised fine-tuning using visual data (Section 3.2), we again use the AdamW optimizer with a learning rate of 0 . 0001 , batch size of 50 and optimize the learnable parameters for a total of 50 epochs. Thanks to our Text-Only classifier pre-training, through empirical evaluations we find that 10 epochs are sufficient for fine-tuning on the large-scale ImageNet dataset. To produce an augmented view of the image, we employ the augmentations used in SimSiam [54]: Gaussian blur, random resized crop, random horizontal flip, color jitter and random gray scaling. For generating class descriptions we use different LLM's, e.g., GPT-3 [9] and Alpaca [64]. In total we generate 50 descriptions for each category in the dataset. We ablate the LLM choice in section 4.3. To construct the text dataset we combine the class descriptions from LLM's and dataset-specific prompt templates provided by [1].

4.2 Results

We test our LaFTer extensively on 12 image classification datasets. These results are provided in Table 1. Our LaFTer consistently improves the zero-shot CLIP model on all the 12 datasets. On some datasets, for example EuroSat, our LaFTer shows an absolute improvement of over 28% on the zero-shot CLIP classification results. Even on the large scale ImageNet dataset we show a considerable improvement of 2 . 3% over the zero-shot CLIP classification results.

We also see that we outperform CLIP-PR on all datasets. Since CLIP-PR relies on offline pseudolabels, which are generated only once for the entire dataset and only proposes to finetune an adapter on top of frozen CLIP visual encoder. We conjecture that their approach might be less expressive. Moreover, it requires label distribution prior from the dataset on which it is being finetuned on. Our LaFTer is free from these requirements and proposes a more general solution for unsupervised finetuning on downstream datasets.

We also compare to the other unsupervised adaptation baseline UPL, which relies on finetuning learnable text prompts attached to the CLIP text encoder through knowledge distillation. In Table 1, we see that our method out performs UPL on most of the datasets ( 9 out of 12 ). On some datasets such as EuroSat, it shows a huge gain of 11 . 7 percentage-points over UPL. On datasets such as Describable Textures Dataset (DTD), Flowers-102 and SUN-397 our method is marginally behind

UPL. We conjecture that since we use augmentations in one of our streams during our adaptation phase, it might result in noisy gradients during the learning process. Specially since datasets like DTD and Flowers-102 can depend on color cues for distinguishing the classes. For the large scale ImageNet dataset, we see that the performance of UPL is below the CLIP zero-shot performance. This can be because UPL generates offline pseudo-labels and in ImageNet, due to fine-grained classes the pseudo-labels might not be very confident from the zero-shot CLIP classifier. On the other hand, LaFTer benefits first from a classifier which has learned discriminative visual features through text-only training and later makes use of parameter efficient finetuning (PEFT). Furthermore, since our pseudo labels are generated in an online manner for each iteration during the optimization, they are also constantly refined as the training is progressing.

In Table 2 we provide a comparison of our LaFTer with the few-shot learning method CoOp [30] and also test Parameter Efficient Fine-Tuning (PEFT), which tunes the same learnable parameters as LaFTer (prompts, classifier, and affine parameters of the normalization layers), but in a few-shot manner. Interestingly, we see that our unsupervised representation learning method conveniently outperforms CoOp for 1 - and 5 -shots. For example, LaFTer ( no-shots) is on average 7 . 1% better than CoOp (1-shot) and even 1 . 3% better in the 5 -shot learning regime, while remaining competitive for 10 -shots. It is also worth noting that for the large-scale ImageNet our LaFTer (requiring no labels) performs better than CoOp (10-shots), requiring 10000 labeled instances from the dataset. Results also follow a similar trend when compared with PEFT.

4.3 Ablation Studies

To understand the significance of all the components in our LaFTer we minutely study each design aspect. First we discuss the performance of our Text-Only pre-trained classifier, then we ablate each component in our unsupervised adaptation phase using unlabeled images and finally provide results by adapting other pre-trained backbones from CLIP [1]. Due to limited evaluation resources and space constraints, we perform these ablations on a subset of datasets with different complexity.

Text-Only Pre-trained Classifier. A main component of our LaFTer is the text-only pre-trained visual classifier, later used in our pseudo-labeling pipeline. The motivation behind it being, that since

+0.5

1

2

3

4

+0.8 +0.7

+1.0

+1.6 +1.5 +1.4

+1.9 +1.8

+2.4

+2.6

+4.2

EuroSat

IN-A

DTD

Flowers

SUN

C-100

C-10

UCF

IN-S

Caltech

IN

IN-R

10

20

30

40

50

Descriptions Per Class

85

.

0

85

.

5

86

.

0

86

.

5

87

.

0

87

.

5

Accuracy (%)

CIFAR-10

10

20

30

40

50

Descriptions Per Class

61

.

0

61

.

5

62

.

0

62

.

5

63

.

0

Accuracy (%)

CIFAR-100

10

20

30

40

50

Descriptions Per Class

57

58

59

60

Accuracy (%)

ImageNet

CLIP is trained in order to have a shared text and vision embedding space so a classifier trained to classify the embeddings from any one of the modalities should also be able to classify embeddings from the other modality. We show that it is possible to train a classifier to classify images by only training it to classify natural language. To design this self-supervised objective of classifying text (described in detail in 3.1) we test different ways of generating the text dataset. For example, simpletemplate such as A photo of a . . . , dataset-specific templates from CLIP [1] and descriptions from LLMs. These results are presented in Table 3. Note that for these results, we first train the classifier on the generated text dataset and then evaluate on the (images) test set of the respective dataset by using the trained classifier to classify the visual embeddings from the CLIP visual encoder. The simplest text dataset generation strategy: classifying the classname is also able to show reasonably strong visual classification performance. Furthermore, different strategies have slight difference in performance. We also find that descriptions from GPT-3 work better than descriptions from Alpaca. Our choice of generating class descriptions by prompting GPT-3 and complementing them by adding handcrafted templates from [1] works best. While averaging over 6 datasets, we gain a performance improvement of 3 . 1% in comparison to the simplest text dataset generation strategy of classifying the classnames themselves.

Text Classifier Contribution to LaFTer. Pre-training a visual classifier on text-only data and then employing it in our pseudo-labeling pipeline helps LaFTer gain improvements on all the 12 datasets which we evaluate in our paper. In Figure 3 we analyze the performance gains for our LaFTer when using the text-only pre-training in conjunction with our pseudo-labeling pipeline. On some datasets, for example EuroSat [57], our text-only pre-training helps our LaFTer to gain an absolute performance improvement of 4 . 2% . On other datasets, the performance gains are also consistent.

Design Choices for Unsupervised Finetuning. In order to study the effect on performance of each individual component we ablate our LaFTer in Table 4. We see that removing the classifier and instead using the CLIP Cosine similarity scores in both branches results in the highest degradation of results. Similarly, all other design choices also have a certain effect. For example, removing the learnable visual prompts results in a decrease of 4 . 2 percentage-points as compared to our LaFTer.

Component 1

Component 2

(a) Base CLIP (ViT-B/32).

Component 1

Component 2

Visual Embeddings

Text Embeddings

(b) LaFTer Adapted CLIP (ViT-B/32).

Different CLIP Backbones. For our main experimental results described in Tables 1 and 2 and ablations in Tables 3 and 4, we use the ViT-B/32 CLIP backbone. In Table 5 we also provide results with different backbones for our LaFTer . We observe consistent improvements by LaFTer over CLIP also for larger backbones. For example, for ViT-B/16 our method shows 23.7% absolute improvement for EuroSat, while for ViT-L/14, our method improves CLIP zero-shot by 16 . 9 %on the same dataset.

Diversity of Descriptions. To test the effect of diversity of generated descriptions on the visual classification results for the text-only pre-training of the text-classifier, we ablate the number of descriptions chosen for each class (for ImageNet and CIFAR-10/100) and provide the results in Figure 4. We find that increasing the diversity of descriptions to generate the text dataset has a positive effect for all the 3 datasets ablated.

Visualization of Embeddings. To gain further insights into the LaFTer adaptation we visualize the latent embeddings from the CLIP encoders in Figure 5, before and after adaptation for all 10 classes in the EuroSAT dataset. Analyzing the TSNE projections, we observe that the adaptation with our LaFTer results in more pronounced category clusters with larger inter-class distances, and also that after applying LaFTer the class-label text embeddings are better aligned with these category clusters.

5 Conclusion & Limitations

We propose a completely label-free finetuning method for Vision-Language models by first showing cross-modality transfer and learning a classifier on natural language inputs which can successfully classify visual data. Later, we leverage this text-only pre-trained classifier in our pseudo-labeling pipeline to further finetune the VL models in a parameter efficient manner. We extensively evaluate our LaFTer and achieve state-of-the-art results when comparing to other methods in an unsupervised finetuning paradigm, while also performing favorably in comparison to methods relying on few-shot supervised learning routines.

Limitations. One contribution of LaFTer is to show cross-modality transfer by training a classifier to classify natural language sentences and showing that it can be readily applied to classify visual data as well. In our work, we experimented with a single linear layer as a classifier. Due to the risk of over-fitting because of the sparse nature of natural language, we did not experiment with more complex structures. Experimentation with expanding the text dataset coupled with designing a more expressive classifier head is left as an exciting future direction.

References

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. , 'Learning Transferable Visual Models from Natural Language Supervision,' in Proc. ICML , 2021.

Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, 'Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm,' arXiv:2110.05208 , 2021.

N. Mu, A. Kirillov, D. Wagner, and S. Xie, 'Slip: Self-supervision Meets Language-image Pre-training,' arXiv:2112.12750 , 2021.

J. Li, D. Li, C. Xiong, and S. Hoi, 'BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation,' arXiv:2201.12086 , 2022.

L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, 'FILIP: Fine-grained Interactive Language-Image Pre-Training,' arXiv:2111.07783 , 2021.

A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela, 'FLAVA: A Foundational Language And Vision Alignment Model,' arXiv:2112.04482 , 2021.

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu, 'CoCa: Contrastive Captioners are Image-Text Foundation Models,' arXiv:2205.01917 , 2022.

S. Reed et al. , 'A Generalist Agent,' arXiv:2205.06175 , 2022.

T. B. Brown et al. , 'Language Models are Few-Shot Learners,' arXiv:2005.14165 , 2020.

K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, 'Fixmatch: Simplifying Semi-supervised Learning with Consistency and Confidence,' in NeurIPS , 2020.

M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, 'Visual Prompt Tuning,' in Proc. ECCV , 2022.

T. Huang, J. Chu, and F. Wei, 'Unsupervised Prompt Learning for Vision-Language Models,' arXiv:2204.03649 , 2022.

C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig, 'Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,' in Proc. ICML , 2021.

H. Tan and M. Bansal, 'Lxmert: Learning Cross-modality Encoder Representations from Transformers,' arXiv:1908.07490 , 2019.

Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, 'Uniter: Universal Image-text Representation Learning,' in Proc. ECCV , 2020.

X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al. , 'Oscar: Object-semantics Aligned Pre-training for Vision-language Tasks,' in Proc. ECCV , 2020.

W. Kim, B. Son, and I. Kim, 'Vilt: Vision-and-language Transformer without Convolution or Region Supervision,' in Proc. ICML , 2021.

J. Li, R. R. Selvaraju, A. D. Gotmare, S. Joty, C. Xiong, and S. Hoi, 'Align before Fuse: Vision and Language Representation Learning with Momentum Distillation,' arXiv:2107.07651 , 2021.

J. Yang, J. Duan, S. Tran, Y. Xu, S. Chanda, L. Chen, B. Zeng, T. Chilimbi, and J. Huang, 'Vision-Language Pre-Training with Triple Contrastive Learning,' in Proc. CVPR , 2022.

S. Goel, H. Bansal, S. Bhatia, R. A. Rossi, V. Vinay, and A. Grover, 'CyCLIP: Cyclic Contrastive Language-Image Pretraining,' arXiv:2205.14459 , 2022.

L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, 'Filip: Fine-grained Interactive Language-image Pre-training,' arXiv:2111.07783 , 2021.

A. Fürst, E. Rumetshofer, V. Tran, H. Ramsauer, F. Tang, J. Lehner, D. Kreil, M. Kopp, G. Klambauer, A. Bitto-Nemling, et al. , 'Cloob: Modern Hopfield Networks with Infoloob Outperform Clip,' arXiv:2110.11316 , 2021.

Y. Gao, J. Liu, Z. Xu, J. Zhang, K. Li, and C. Shen, 'PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining,' arXiv:2204.14095 , 2022.

Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, 'Supervision Exists Everywhere: A data Efficient Contrastive Language-Image Pre-training Paradigm,' arXiv:2110.05208 , 2021.

L. Yao, R. Huang, L. Hou, G. Lu, M. Niu, H. Xu, X. Liang, Z. Li, X. Jiang, and C. Xu, 'Filip: Fine-grained Interactive Language-image Pre-training,' arXiv:2111.07783 , 2021.

A. Fürst, E. Rumetshofer, V. Tran, H. Ramsauer, F. Tang, J. Lehner, D. Kreil, M. Kopp, G. Klambauer, A. Bitto-Nemling, et al. , 'Cloob: Modern Hopfield Networks with InfoLOOB Outperform Clip,' arXiv:2110.11316 , 2021.

B. Wu, R. Cheng, P. Zhang, P. Vajda, and J. E. Gonzalez, 'Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation,' arXiv:2112.09445 , 2021.

S. Goel, H. Bansal, S. Bhatia, R. A. Rossi, V. Vinay, and A. Grover, 'CyCLIP: Cyclic Contrastive Language-Image Pretraining,' arXiv:2205.14459 , 2022.

Y. Gao, J. Liu, Z. Xu, J. Zhang, K. Li, and C. Shen, 'PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining,' arXiv:2204.14095 , 2022.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu, 'Learning to Prompt for Vision-Language Models,' IJCV , 2022.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu, 'Conditional Prompt Learning for Vision-Language Models,' in Proc. CVPR , 2022.

J. Kahana, N. Cohen, and Y. Hoshen, 'Improving Zero-Shot Models with Label Distribution Priors,' arXiv:2212.00784 , 2022.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu, 'Learning to Prompt for Vision-language Models,' IJCV , 2022.

T. Huang, J. Chu, and F. Wei, 'Unsupervised Prompt Learning for Vision-Language Models,' arXiv:2204.03649 , 2022.

Z. Zhong, D. Friedman, and D. Chen, 'Factual Probing is [MASK]: Learning vs. Learning to Recall,' arXiv:2104.05240 , 2021.

B. Lester, R. Al-Rfou, and N. Constant, 'The Power of Scale for Parameter-efficient Prompt Tuning,' arXiv:2104.08691 , 2021.

Y. Lu, J. Liu, Y . Zhang, Y . Liu, and X. Tian, 'Prompt Distribution Learning,' in Proc. CVPR , 2022.

M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao, 'Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models,' in NeurIPS , 2022.

P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao, 'Clip-adapter: Better Vision-language Models with Feature Adapters,' arXiv:2110.04544 , 2021.

R. Zhang, R. Fang, P. Gao, W. Zhang, K. Li, J. Dai, Y. Qiao, and H. Li, 'Tip-adapter: Trainingfree Clip-adapter for Better Vision-Language Modeling,' arXiv:2111.03930 , 2021.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu, 'Conditional Prompt Learning for Vision-language Models,' in Proc. CVPR , 2022.

D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell, 'Tent: Fully Test-time Adaptation by Entropy Minimization,' in Proc. ICLR , 2020.

M. J. Mirza, P. J. Soneira, W. Lin, M. Kozinski, H. Possegger, and H. Bischof, 'ActMAD: Activation Matching to Align Distributions for Test-Time Training,' in Proc. CVPR , 2023.

S. Niu, J. Wu, Y. Zhang, Y. Chen, S. Zheng, P. Zhao, and M. Tan, 'Efficient Test-Time Model Adaptation without Forgetting,' in Proc. ICML , 2022.

D.-H. Lee, 'Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks,' in Proc. ICMLW , 2013.

A. Rasmus, M. Berglund, M. Honkala, H. Valpola, and T. Raiko, 'Semi-supervised Learning with Ladder Networks,' in NeurIPS , 2015.

A. Tarvainen and H. Valpola, 'Mean Teachers are Better Role Models: Weight-averaged Consistency Targets Improve Semi-supervised Deep Learning Results,' in NeurIPS , 2017.

Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, 'Unsupervised Data Augmentation for Consistency Training,' in NeurIPS , 2020.

M. Assran, M. Caron, I. Misra, P. Bojanowski, A. Joulin, N. Ballas, and M. Rabbat, 'Semisupervised Learning of Visual Features by non-Parametrically Predicting View Assignments with Support Samples,' in Proc. CVPR , 2021.

Y. Zhang, J. Z. HaoChen, S.-C. Huang, K.-C. Wang, J. Zou, and S. Yeung, 'DrML: Diagnosing and Rectifying Vision Models using Language,' in Proc. ICLR , 2023. [Online]. Available: https://openreview.net/forum?id=losu6IAaPeB .

S. Pratt, R. Liu, and A. Farhadi, 'What does a Platypus Look Like? Generating Customized Prompts for Zero-shot Image Classification,' arXiv:2209.03320 , 2022.

C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, 'Rethinking the Inception Architecture for Computer Vision,' in Proc. CVPR , 2016.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A Large-scale Hierarchical Image Database,' in Proc. CVPR , 2009.

X. Chen and K. He, 'Exploring Simple Siamese Representation Learning,' in Proc. CVPR , 2021.

A. Krizhevsky and G. Hinton, 'Learning Multiple Layers of Features from Tiny Images,' Department of Computer Science, University of Toronto, Tech. Rep., 2009.

L. Fei-Fei, R. Fergus, and P. Perona, 'Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories,' in Proc. CVPR , 2004.

P. Helber, B. Bischke, A. Dengel, and D. Borth, 'Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification,' in Proc. IGARSS , 2018.

K. Soomro, A. R. Zamir, and M. Shah, 'UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild,' arXiv:1212.0402 , 2012.

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, 'SUN Database: Large-scale Scene Recognition from Abbey to Zoo,' in Proc. CVPR , 2010.

M.-E. Nilsback and A. Zisserman, 'Automated Flower Classification Over a Large Number of Classes,' in Proc. ICVGIP , 2008.

D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, 'Natural Adversarial Examples,' in Proc. CVPR , 2021.

H. Wang, S. Ge, Z. Lipton, and E. P. Xing, 'Learning Robust Global Representations by Penalizing Local Predictive Power,' in NeurIPS , 2019.

D. Hendrycks et al. , 'The Many Faces of Robustness: A Critical Analysis of Out-ofDistribution Generalization,' in Proc. ICCV , 2021.

R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, Stanford Alpaca: An Instruction-following LLaMA model , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method for improving zero-shot classification of VL models, in an unsupervised manner. This is done without additional visual labeled examples, yet with additional unlabeled examples. They rely on LLM to generate a dataset of text describing the desired classes. Then, they train a text classifier on top of the VL textual encoder using the generated dataset. As a second stage, they use a set of relevant unsupervised images as well as a set of designed augmentations, to finetune the classifier and visual prompts, adjusting it to the VL models image representations. In an extensive evaluation, the authors show a significant performance enhancement, and ablate their method methodically.\n\nSTRENGTHS:\n-\tThe paper is clearly written and easy to follow. \n-\tThe performance enhancement shown by LaFTer is very significant and consistent across most test cases. \n-\tAlthough inspired by previous works, the idea of tunning the visual encoder of VL models using text supervision is highly interesting. \n\nOverall, I believe the idea of the paper is very interesting, and the results are a significant improvement over CLIP.\n\nWEAKNESSES:\nMy main concern is the performance of CLIPPR baseline. In most cases, it has been found to be less successful than the original CLIP, which is not intuitive with the method. Could the authors explain this gap please?\n\nOne other minor concern is the statement in the abstract about LaFTer being the first to reduce the gap from the supervised baseline. Table 1 shows LaFTer is not the first. Table 2 shows that it does not close the gap for about 50% of the cases, using only few-shot supervision. I believe this is misleading statement.\n\nQUESTIONS:\n-\tCould the authors please verify if the experiments denoted by LaFTer* are just the second stage of LaFTer using CLIP as the initial text classifier? \nIf so, I think these results are more suitable to appear in the ablations section (Table 4).\n\nLIMITATIONS:\n-\tThe method requires access to a set of unlabeled images from the same distribution.\n-\tThe method is limited by the quality of the visual representation of the VL model.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a finetuning approach for Vision-Language models that does not require any labels. It begins by demonstrating the transfer of information between different modalities and training a classifier using natural language inputs. This classifier achieves the successful classification of visual data. Furthermore, it utilizes this pre-trained classifier that relies solely on text inputs in our pseudo-labeling pipeline. This enables it to effectively and efficiently fine-tune the Vision-Language models with fewer parameters.\n\nSTRENGTHS:\n1- The proposal introduces a label-free approach to zero-shot learning, capitalizing on the benefits of pretrained LLM models. Additionally, it enhances the model's performance by finetuning it in an unsupervised manner. This novel approach presents a new direction for zero-shot learning research, eliminating the need for labeled datasets even for seen classes.\n2- The proposed method extends its experiments to include a few-shot framework and demonstrates a noteworthy improvement compared to the baseline approaches.\n3- The experiments are conducted on twelve (12) diverse datasets, encompassing coarse-grained, fine-grained, and natural scene datasets. A comprehensive ablation study is conducted to demonstrate the effectiveness of the proposed model and the contribution of its individual components.\n\nWEAKNESSES:\n1- Upon analyzing Table-1, it is evident that the proposed model exhibits lower performance for the Flower and SUN datasets, both commonly utilized in zero-shot learning and are fine-grained datasets. This observation raises surprise as the proposed model struggles specifically with fine-grained datasets.\n2- The visualization of embeddings obtained from both the text encoder and image encoder would provide a clearer understanding of the effectiveness of the pretrained CLIP (LLM) model. Additionally, generating images based on textual descriptions of classes would offer valuable insights into how the proposed model operates within the zero-shot framework.\n3- The distinction between the weakly-augmented view and the strongly-augmented view is unclear when it comes to unsupervised fine-tuning. What are the benefits of utilizing the weakly-augmented view for pseudo-labels instead of the strongly-augmented view?\n\nQUESTIONS:\nPlease answer all my concerns raised in the weaknesses section.\n\nLIMITATIONS:\nIn this paper, the authors acknowledge a limitation of their proposed model. They state that in their current work, they employed a straightforward neural network-based classifier and consider exploring more complex neural networks as part of future work. However, it is worth noting that from my observation of Table-1, the proposed model also faces challenges when applied to fine-grained datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new approach to improve zero-shot vision recognition capability which leveraged the common embedding space for image and text. Specifically, with a pre-trained VLM (vision-language model), the authors leveraged LLM to automatically generate multiple language prompts for training a text-based classifier, which can be adapted for vision classification thanks to the shared embedding space. The classifier is further applied to augmented images to provide pseudo labels for finetuning the visual encoder in a label-free manner (pseudo labels generated from the text classifier). The authors perform extensive experiments to validate the effectiveness of the proposed training paradigm and on multiple benchmarks the model achieved significant performance gain.\n\nSTRENGTHS:\n1. The proposed method is intuitive and theoretically sound. The shared image-text embedding space naturally serves as a bridge for applying text-trained classifiers on image domains. The simple idea turns out to be very effective and brings significant performance boost across multiple datasets.\n2. The conclusion is supported by extensive experiments with various ablation studies, showing many interesting patterns.\n3. The paper is well-written and in good shape, it's easy to read.\n\nWEAKNESSES:\n1. Section 3.2 is named as \"unsupervised finetuning\" but technically since the model is trained with smoothed cross entropy loss, it's not truly \"unsupervised\", just that the labels are from text-based classifier (thus \"pseudo labels\"). The name is somewhat misleading as audience may get confused with unsupervised approaches.\n2. CLIP is pre-trained with massive data thus demonstrating good zero-shot capability. Although this paper experimented on multiple benchmark datasets, the images of these datasets are all natural thus similar in style. The proposed approach achieves significant performance gain without real labels but the assumption is that the pseudo labels are relatively accurate due to image consistency. Therefore it is better to emphasize the point at least as a potential limitations. It might be better to be more careful with some claims e.g. L317 regarding the cross-modality transfer capability.\n\nQUESTIONS:\n1. Some other recent work used similar ideas for finetuning VLMs so maybe it's worth mentioning and provide a comparison. For example, [1] used a similar idea to generate multiple text and image prompts for finetuning VLMs domain adaptation.\n\n[1] https://arxiv.org/abs/2306.16658\n\nLIMITATIONS:\nThe paper has a dedicated section discussing about the limitations with the single linear layer of text classifier. Some more important limitations may need emphasizing as I suggested in the weaknesses section that the general assumption of the effectiveness of the proposed approach is the high quality of the underlying VLM model. No obvious potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method to improve the zero-shot classification accuracy for a pre-trained vision-language (VL) model. By leveraging the shared embedding space between the vision and language modalities, and relatively accessible text data generated by large language models (LLMs), the proposed method trains a classifier on the embedding of generated texts for the zero-shot classes, which can also work on the image features. The proposed method also includes a pseudo-label strategy using the trained classifier to augment unlabeled image data. The paper shows compelling performance improvement over the original VL model and other non-label fine-tuning methods.\n\nSTRENGTHS:\n1. Overall I think it is an interesting idea. By using the shared embedding from the two modalities, it generates relatively \"cheap\" data by using an LLM to train a classifier directly for the target classes. This classifier further gives a better \"starting point\" for the pseudo-labeling in the next stage.\n2. The evaluations on the benchmark datasets show considerable improvements over other label-free methods. It also outperforms 1-shot fine-tuning methods on certain datasets.\n\nWEAKNESSES:\n1. What bothers me the most is that the performance of only using Llama descriptions is quite underwhelming (Table 3). It has a big gap to only using class names. Even the pure GPT description is only 1% better than class names on average. I think the failure of Llama needs more explanations. Also, these results might imply: 1) the generated texts are not that effective; 2) you need a really good LLM (GPT3 or better) to generate decent text descriptions to make this method work. IMO, both points will limit the contribution and impact of the proposed method. \n2. I feel the diversity and quality of the generated texts will be very important in the proposed approach, as it will affect the generalization ability of the trained classifier. It is better to have some ablations on these aspects.\n\nQUESTIONS:\nLLM can just generate random text descriptions of the class. For example, it might generate the history of an object which is not very useful in visual classification. Did you consider imposing any constraints to make the LLMs generate only visually meaningful texts?\n\nLIMITATIONS:\nTo me, the biggest limitations are \n1. The method might rely on a super good LLM which might not be accessible in many real-world scenarios.\n2. The generated text for visual classification might not be visually meaningful.\nNo negative societal impact was observed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}