SlowGuess/NeurIPS_2023_OCR huggingface dataset:
M. Jehanzeb Mirza†,2
Horst Possegger<sup>1</sup>
Leonid Karlinsky3
Rogerio Feris<sup>3</sup>
Wei Lin<sup>1</sup>
Mateusz Kozinski
Horst Bischof1,2
$^{1}$ Institute of Computer Graphics and Vision, TU Graz, Austria.
2Christian Doppler Laboratory for Embedded Machine Learning.
$^{3}$ MIT-IBM Watson AI Lab, USA.
Project Page: https://jmiemirza.github.io/LaFTer/
Recently, large-scale pre-trained Vision and Language (VL) models have set a new state-of-the-art (SOTA) in zero-shot visual classification enabling open-vocabulary recognition of potentially unlimited set of categories defined as simple language prompts. However, despite these great advances, the performance of these zero-shot classifiers still falls short of the results of dedicated (closed category set) classifiers trained with supervised fine-tuning. In this paper we show, for the first time, how to reduce this gap without any labels and without any paired VL data, using an unlabeled image collection and a set of texts auto-generated using a Large Language Model (LLM) describing the categories of interest and effectively substituting labeled visual instances of those categories. Using our label-free approach, we are able to attain significant performance improvements over the zero-shot performance of the base VL model and other contemporary methods and baselines on a wide variety of datasets, demonstrating absolute improvement of up to  $11.7\%$  ( $3.8\%$  on average) in the label-free setting. Moreover, despite our approach being label-free, we observe  $1.3\%$  average gains over leading few-shot prompting baselines that do use 5-shot supervision.
Vision and Language (VL) models [1-8] recently became the de-facto standard for generalized zero-shot learning enabling recognition of arbitrary (open) set of categories provided with just their text descriptions and without requiring any additional data or training. However, this incredible flexibility comes at a performance cost and all the VL models, including the most widely used CLIP [1], still require additional supervised training (e.g. tuning its vision encoder) to compete with the closed set (of target categories) supervised training methods. Naturally, this incurs undesirable adaptation costs for those, otherwise very versatile and flexible, foundation models. Image annotations are often expensive to collect, especially for legacy vision systems, such as traffic surveillance, quality control, or security.
In this paper we propose LaFTer, an approach for completely label-free parameter efficient finetuning of VL models to a set of target classes. Our goal is to substitute the need for expensive-to-obtain image annotations with unsupervised finetuning of VL models. We show that since the VL models share a common text-image embedding space (due to their contrastive learning objective), there is a possibility to train using embeddings of samples from one modality (e.g., auto-labeled text) and then successfully apply what we trained to classify embeddings of samples from the other modality
†Correspondence: muhammad.mirza@icg.tugraz.at
Figure 1: LaFTer proposes to first train a classifier on a natural language text dataset mined in a controlled manner from a set of target classes by generating descriptions for each class label using an LLM and mixing them with handcrafted templates. The training objective is to classify each description to the correct (source) class name (top-left). In the second stage, LaFTer employs the text-only classifier to generate pseudo-labels on the unlabeled data to further finetune the vision encoder in a parameter-efficient manner (bottom-left). Finally, the finetuned visual encoder and text classifier is used for eventual classification (bottom-middle). Combining our text-only pre-trained classifier together with the proposed pseudo-labeling pipeline lets us consistently improve the previous SOTA results for label-free finetuning, UPL [12] (right).
(e.g., unlabeled images). More specifically, we show that it is possible to train a neural network (e.g., a classifier) to classify text instances that can be successfully used to classify images, showing successful cross-modal transfer. Instead of collecting labeled visual instances, in our label-free LaFTer approach, we are mining text descriptions of the target categories by prompting an LLM (e.g., GPT-3 [9]) and combining them with handcrafted prompts, as shown in Figure 1. After creating such a text dataset, we train a neural network to classify each text instance (sentence) in it to the source class label that was used to produce the instance. This text classifier can be readily used to classify images when used on top of a CLIP visual encoder. Furthermore, we take advantage of this text-only pre-trained classifier by employing it in a pseudo-labeling pipeline (inspired by FixMatch [10]), to further finetune the CLIP vision encoder on an unlabeled image collection. To reduce overfitting and keep the finetuning parameter efficient, we make use of Visual Prompt Tuning [11] combined with adapting the affine transformations (scale and shift) of the normalization layers in the otherwise frozen network.
- We offer a new, completely label-free, and parameter-efficient (0.4% learned parameters), cross-modal transfer approach (LaFTer) to finetuning a VL model for improved recognition performance on a set of target classes using a set of automatically generated texts for those classes and an unlabeled image collection.
- We show that text-side-only training on an automatically collected language knowledge (e.g. through LLM prompting) can be effective in bootstrapping a visual classifier built on top of an aligned VL model to consistently improve its performance in a variety of downstream classification benchmarks.
- In an extensive evaluation of our approach, spanning over 12 popular benchmarks, we show that it: (i) attains up to  $28\%$  absolute improvement over the base VL model (CLIP) performance  $(6.7\%$  on average); (ii) improves over previous SOTA under the same label-free setting by up to  $11.7\%$ $(3.8\%$  on average); and (iii) performs favorably while comparing with SOTA parameter efficient few-shot methods, matching from above average performance of 5-shot supervised setting despite using no labeled samples.
Large-scale Vision&Language (VL) Models: Remarkable performance in many zero-shot downstream tasks has been attained with VL models pre-trained using contrastive losses on large-scale noisy image-text data (e.g., CLIP [1] and ALIGN [13]). Different ideas have been tried to improve the image-text representation alignment, such as leveraging off-the-shelf object detectors [14-16], or using cross-attention and additional objective functions such as image-text matching and masked language modeling [4, 17-19], or filtering noisy captions (e.g., BLIP [4]). Some additional properties of language structure are utilized in [2, 20-23]. DeCLIP [2] finds additional positives for the contrastive loss by seeking textual nearest-neighbors. Geometrically consistent representations (within and across the paired image-text modalities) are employed in CyClip [20]. Recently, few methods have attempted improving VL performance using additional supervision [3, 24], finer-grained interactions [25], modern Hopfield networks [26], optimal transport distillation [27], cycle consistency [28], and hierarchical feature alignment [29]. However, VL models performance still falls short of classifiers trained with supervision on the downstream target data. Most of the approaches to fine-tuning VL models [30, 31] leverage annotated data for this finetuning. In contrast, in our work we propose a completely label-free approach for adapting a VL model to the target task. Methods which are most related to our work, UPL [12] and CLIP-PR [32] also finetune VL models in an unsupervised manner. UPL finetunes learnable text prompts (similar to [30]) by relying on confidence sampled pseudo-labeling, whereas CLIP-PR relies both on offline pseudo-labeling and label distribution prior from the training data. In contrast, in our work, we first leverage textual knowledge from LLMs and design a self-supervised text-only pre-training task showing that it can be employed in an unsupervised parameter-efficient finetuning of the VL model on a set of unlabeled images. Extensive empirical evaluations show the benefits of our approach w.r.t. UPL and CLIP-PR.
Prompt Tuning. Prompt tuning belongs to a wider family of pararameter-efficient finetuning methods that can also be applied to VL models [33, 34]. Originating in NLP [35, 36], visual prompt tuning was recently shown to be effective also for vision backbones [11, 33, 37]. CoOp [33] learns prompt vectors by minimizing the prediction error using the cross-entropy loss. ProDA [37] learns diverse prompts from data to cope with variance of the visual representations. UPL [34] proposes an unsupervised prompt learning framework. TPT [38] proposes a test-time prompt tuning framework. CLIP-Adapter [39] and Tip-Adapter [40] employ an alternative parameter efficient finetuning strategy using additional adapter modules. Recently, CoCoOp [41] proposed a Meta-Net for generating image-adaptive prompts for improved generalization to distribution shifts. In this work, we employ parameter-efficient Visual Prompt Tuning (VPT) [11] as the means for adapting the visual encoder of the VL model, but in contrast with other works we do not use any supervised data. Instead, we use a classifier, trained in the text domain using texts automatically generated by an LLM from the set of target classes, to generate pseudo-labels for an unlabeled image collection as a form of self-supervision. Furthermore, we propose to combine VPT with tuning the scale and shift parameters of normalization layers, previously proposed for handling domain shifts [42-44], but to the best of our knowledge never before used to tune VL models.
Pseudo-labeling. Pseudo labeling, also known as self-training, is commonly used as a semi-supervised learning technique. Popularized in the seminal works of [45, 46], pseudo-labeling was extended to utilize consistency regularization [47] and augmentation [48]. The pseudo-labeling pipeline of our method is inspired by FixMatch [10] that combined consistency regularization with confidence-based filtering, surpassing SOTA semi-supervised techniques at the time. It was later extended with a non-parametric classifier in PAWS [49]. These techniques are proposed for semi-supervised learning and require some amount of labeled instances. In contrast, we propose a label-free method for improving VL models performance on a set of target classes. To achieve this, our method finetunes VL models in a parameter-efficient manner by generating pseudo labels through a text-only classifier trained on a corpus of text data generated by prompting language models.
3 LaFTer
CLIP [1] consists of a vision encoder and a text encoder, which project images and texts to a common embedding space. It has been trained on a very large volume (400M) of image-text pairs to align the embedding of each image to the embedding of its corresponding text, at the same time pushing it away from the embeddings of unrelated texts corresponding to other images. In [1], it was demonstrated that the text and vision encoders enable effective zero-shot image classification. Given the set of class
Figure 2: Overview of our LaFTer. (top) Given a set of class labels, we generate a data set of short texts by prompting a Large Language Model (LLM) multiple times with each class name. We compute embeddings of these texts using CLIP text encoder. This lets us train a neural network, the Text Classifier, to infer the class used to prompt the LLM from the embedding of the text it generated. Even though the Text Classifier has been trained exclusively on text, it performs well in classifying image embeddings generated by CLIP vision encoder. (bottom) We further take advantage of the Text Classifier by leveraging it in a pseudo-labeling setup to finetune the VL model.
names  $C$ , the text encoder  $u$  is used to generate the embedding  $u_{c}$  of a prompt for each class  $c \in C$ , typically of the form 'A photo of ...', completed with the class name. Each test image  $x$  is encoded by the vision encoder  $v$  and classified using its cosine similarity  $\cos$  to the text embedding of each class. The likelihood of a predicted class  $\hat{c}$  is computed with a softmax,
$$
l _ {\hat {c}} (x) = \frac {e ^ {\cos \left(u _ {\hat {c}} , v (x)\right) / \tau}}{\sum_ {c \in C} e ^ {\cos \left(u _ {c} , v (x)\right) / \tau}}, \tag {1}
$$
where  $\tau$  denotes the temperature constant.
The zero-shot classifier (1) does not require any training data, but is typically outperformed by networks trained on data from the target domain. Existing approaches to reducing this performance gap [30, 31] rely on fine-tuning the zero-shot classifier in a few-shot regime. While effectively enhancing accuracy, they incur additional costs of curating and annotating the training data. In contrast, we propose a two-step approach for fine-tuning the network without requiring any annotations. In the next section 3.1, we describe a technique to train a visual classifier on purely textual data, conveniently generated by Large Language Models. In Section 3.2, we propose an unsupervised training setup that lets us benefit from unlabelled data from the target domain, when such data is available. Combining these techniques significantly reduces the performance gap to the supervised classifier and is even competitive with few-shot approaches despite not using any supervision. These results are provided in Section 4.2.
3.1 Learning an Image Classifier using Text
Aligning the text and image embedding spaces is the key idea underlying Vision Language models. It gives CLIP and similar methods, their main advantages: the capacity to be trained on an extremely large volume of data available on the Internet and the resulting effectiveness as zero-shot classifiers. Recently, Zhang et al. [50] explore yet another advantage of the alignment between text and image embedding spaces - it allows diagnosing and rectifying vision models spawn from the VL model vision encoder by using the other modality. In particular, it hints that it might be possible to finetune a zero-shot image classifier on textual data.
While acquiring and annotating images requires manual labor, a training set of annotated texts can be constructed automatically, using a Large Language Model, like the GPT-3 [9]. Generating text using an LLM constitutes a convenient alternative to text mining, as LLMs represent extremely
large text corpora on which they were trained. Moreover, prompting LLMs is much more efficient than searching a large body of text for the words of interest. To construct our training set, we take inspiration from [51] and for each class  $c$ , we prompted GPT-3 with queries of the following kind: 'Describe what a  $[c]$  looks like.' We repeated the prompting for each class  $c$ , in the dataset and complemented the resulting set of synthetic texts with text generated procedurally, using the same hand-crafted templates as for constructing prompts for the zero-shot classifier, for example, 'A photo of a  $[c]$ .'. We defer the full list of prompts (to LLMs) and templates to the supplementary material.
The capacity to finetune the classifier in a supervised text classification setting lifts the architectural constraints imposed by the zero-shot setup. More precisely, the class likelihoods no longer need to be produced according to (1), but can instead be computed by a trainable classifier  $f$ , of arbitrary architecture. We train  $f$  with the Smoothed Cross Entropy loss  $\mathcal{L}_{\mathrm{SCE}}$  [52]. To regularize training, we add Gaussian Noise  $n \sim \mathcal{N}(0, I)$  to the  $l_{2}$ -normalized feature vector from the clip text encoder. Formally, given a text training set  $T$  consisted of pairs of text fragments  $t$  and class labels  $c$ , the training objective is:
$$
\min  _ {\theta} \sum_ {\substack {(t, c) \in T \\ n \sim \mathcal {N} (0, I)}} \mathcal {L} _ {\mathrm {S C E}} \left(f _ {\theta} \left(\frac {u (t)}{\| u (t) \|} + n\right), c\right), \tag{2}
$$
where  $\theta$  is the parameter of the classifier. Training of the text-only classifier is very efficient. For example, 3000 epochs of training the classifier on the data set of 130000 text sentences, representing the 1000 classes of the ImageNet [53] dataset is completed in  $\sim 120$  seconds on an NVIDIA 3090 graphics card.
3.2 Unsupervised Finetuning on Target Domain Images
Text-only training does not require any image data. However, in many applications, unlabeled images from the target domain are readily available, or can be acquired at a low cost. Given a set of unlabeled images we propose to take advantage of the text-only pre-trained classifier and use it in a pseudo-labeling pipeline on top of the vision encoder as demonstrated at the bottom part of Figure 2.
Inspired by Fixmatch [10], for each training image  $x$ , we generate two views: the weakly-augmented view  $\alpha_w(x)$  and the strongly-augmented view  $\alpha_h(x)$ , where  $\alpha$  denotes a stochastic augmentation function. Contrary to Fixmatch, in our unsupervised finetuning we set  $\alpha_w$  as an identity transformation. For  $\alpha_h$  we use the augmentations proposed in [54]. The weakly-augmented view serves to generate a pseudo-label. To that end, it is passed through the vision encoder  $v$  and the text classifier  $f$ , yielding a vector of class probabilities  $p$ . Class probabilities give rise to pseudo labels, that we denote by  $\hat{c}(x)$ . Formally,
$$
\hat {c} (x) = \underset {c \in C} {\arg \max } p _ {c}, \quad \text {w h e r e} \quad p = f \left(v \left(\alpha_ {w} (x)\right)\right). \tag {3}
$$
$\hat{c}(x)$  is not differentiable, and we do not backpropagate the error signal through it during training. The pseudo labels  $p_{\hat{c}}$  are used as ground truth for training the network on the heavily-augmented views. Since the vision encoders in the two branches share the weights, the pseudo-labels are generated in an online manner and are constantly refined as adaptation is progressing. This is in contrast to UPL [12] and CLIP-PR [32] which rely on offline pseudo-labeling by only generating them once from frozen CLIP encoders. We freeze the weights of the text-only pre-trained classifier in the pseudo-labeling branch since it leads to more stable adaptation.
When processing the heavily augmented views  $\alpha_{h}(x)$ , we augment the vision encoder by visual prompt tuning [11]. That is, we append randomly initialized, learnable parameters (a matrix of size  $N_P \times d_v$  where  $N_P$  is the number of prompts and  $d_v$  is the channel dimension of the vision encoder) to the input of the vision transformer after the initial embedding layer. This helps the network account for the heavy augmentation of the input and, as we show in Section 4.3, boosts the performance of the finetuned classifier. We denote the vision encoder with prompting by  $v^p$ . The prediction for the heavily-augmented image is obtained as  $f(v^p(\alpha_h(x)))$ .
The network is trained with the smoothed cross entropy loss  $\mathcal{L}_{\mathrm{SCE}}$  [52]. We denote the set of unlabelled target domain images by  $D$ , and formalize the training objective as
$$
\min  _ {\theta , \eta} \sum_ {x \in D} \mathcal {L} _ {\mathrm {S C E}} \left(f _ {\theta} \left(v _ {\eta} ^ {p} \left(\alpha_ {h} (x)\right)\right), \hat {c} (x)\right), \tag {4}
$$
ImageNet	CIFAR-10	CIFAR-100	EuroSat	DTD	CALTECH-101
CLIP	61.9	88.8	64.2	45.1	42.9	90.5
CLIP-PR	60.4	89.3	63.2	44.2	40.1	84.8
UPL	61.2	89.2	65.8	62.2	48.0	90.6
LaFTer	64.2	95.8	74.6	73.9	46.1	93.3
UCF-101	Flowers-102	SUN-397	ImageNet-A	ImageNet-S	ImageNet-R
CLIP	61.0	66.6	60.8	29.6	40.6	65.8
CLIP-PR	57.9	57.7	54.7	11.6	38.6	54.1
UPL	63.9	71.5	66.0	26.9	42.4	65.6
LaFTer	68.2	71.0	64.5	31.5	42.7	72.6
Table 1: Top-1 Classification Accuracy (%) while using the CLIP pre-trained ViT-B/32 backbone for 12 image classification benchmarks. LaFTer represents results obtained by first pre-training the visual classifier on text-only data and then performing unsupervised finetuning on the unlabeled image data. Highest accuracy is shown in bold, while second best is underlined.
where  $\theta$  and  $\eta$  denote the finetuned parameters of the classifier and of the vision encoder, respectively. The parameters  $\eta$  are the visual prompts and the scale and shift (affine) parameters of the normalization layers of the vision encoder. The selection of  $\eta$  is motivated by keeping the adaptation parameter-efficient. For LaFTer, the number of trainable parameters are less than  $0.4\%$  of the entire model parameters, making adaptation extremely lightweight.
Here we first provide a description of the datasets and baselines we use in our evaluations, then explain our implementation details and later discuss our experimental results in detail.
4.1 Evaluation Setting
Datasets: We extensively evaluate our approach on 12 different datasets belonging to widely different domains. More specifically, we use four datasets containing common natural categories: ImageNet [53], CIFAR-10/100 [55] and Caltech-101 [56]. EuroSat [57] contains satellite images of 10 different locations. UCF-101 [58] is an action recognition dataset. SUN-397 [59] contains images from 397 naturally occurring scenes. Flowers-102 [60] is a fine-grained classification dataset for classifying different categories of flowers commonly occurring in the United Kingdom. Whereas, ImageNet-A (Adversarial) [61], ImageNet-S (Sketch) [62] and ImageNet-R (Rendition) [63] are different versions of the original ImageNet validation set. In our setting, we divide the ImageNet-A, ImageNet-S and ImageNet-R in to  $80\%$  train and  $20\%$  test set. For all other datasets we use the splits provided by [30].
Baselines: We compare LaFTer with baselines which are label-free (not requiring any additional labeled images):
- CLIP [1] denotes zero-shot classification scores by computing the cosine similarity between embeddings from frozen CLIP encoders.
- UPL [12] adds learnable text prompts to the CLIP text encoder and finetunes them in an unsupervised manner by employing confidence-sampled offline pseudo-labeling.
- CLIP-PR [32] optimizes an adapter on top of the CLIP vision encoder by using label distribution priors from the training set of the downstream datasets and generating offline pseudo-labels.
For completeness, apart from these 3 baselines, we also provide a comparison with few-shot fine-tuning method CoOp [30], which learns soft text prompts using  $k$  labeled images per class ( $k$ -shot).
Implementation Details: For all our experiments, unless otherwise stated, we use a ViT/B-32 CLIP pre-trained model from OpenAI [1]. The Text-Only classifier (Section 3.1) is implemented as a
ImageNet	CIFAR-10	CIFAR-100	EuroSat	DTD	CALTECH-101
LaFTer (no-shot)	64.2	95.8	74.6	73.9	46.1	93.3
CoOp (1-shot)	60.6	83.0	55.6	58.4	40.1	91.7
CoOp (5-shot)	61.3	86.6	63.2	71.8	41.1	93.2
CoOp (10-shot)	62.3	88.5	66.6	81.6	65.8	94.6
PEFT (1-shot)	50.7	62.7	50.2	37.5	42.6	90.6
PEFT (5-shot)	59.3	80.0	67.3	55.3	59.9	94.5
PEFT (10-shot)	62.8	87.9	74.1	67.9	67.3	96.1
UCF-101	Flowers-102	SUN-397	ImageNet-A	ImageNet-S	ImageNet-R
LaFTer (no-shot)	68.2	71.0	64.5	31.5	42.7	72.6
CoOp (1-shot)	63.8	71.2	64.1	24.5	39.9	60.0
CoOp (5-shot)	74.3	85.8	67.3	30.0	46.5	61.6
CoOp (10-shot)	77.2	92.1	69.0	35.0	49.1	63.6
PEFT (1-shot)	60.5	66.9	58.3	20.9	38.5	57.2
PEFT (5-shot)	72.6	91.1	68.7	33.3	55.3	66.4
PEFT (10-shot)	79.8	95.2	72.3	40.2	61.1	71.0
Table 2: Top-1 Accuracy (\%) for our LaFTer (no-shot) compared to few-shot methods. We compare to CoOp [30] in 1-, 5- and 10-shot supervised finetuning regimes. Parameter Efficient Finetuning (PEFT) represents tuning the same parameters as in LaFTer (prompts, classifier, affine) but in a few-shot manner. For each dataset/compared method, blue highlights the highest number of shots outperformed by no-shot LaFTer. Notably, LaFTer improves over 10-shot and all compared methods in 4 datasets, including ImageNet, where 10-shot = 10K labeled samples.