SlowGuess/NeurIPS_2023_OCR huggingface dataset:
single linear layer, with the output units equal to the number of classes in the dataset. For training this classifier, we load the complete text dataset as a single batch and optimize the network using AdamW as optimizer, with a learning rate of 0.001. For unsupervised fine-tuning using visual data (Section 3.2), we again use the AdamW optimizer with a learning rate of 0.0001, batch size of 50 and optimize the learnable parameters for a total of 50 epochs. Thanks to our Text-Only classifier pre-training, through empirical evaluations we find that 10 epochs are sufficient for fine-tuning on the large-scale ImageNet dataset. To produce an augmented view of the image, we employ the augmentations used in SimSiam [54]: Gaussian blur, random resized crop, random horizontal flip, color jitter and random gray scaling. For generating class descriptions we use different LLM's, e.g., GPT-3 [9] and Alpaca [64]. In total we generate 50 descriptions for each category in the dataset. We ablate the LLM choice in section 4.3. To construct the text dataset we combine the class descriptions from LLM's and dataset-specific prompt templates provided by [1].
We test our LaFTer extensively on 12 image classification datasets. These results are provided in Table 1. Our LaFTer consistently improves the zero-shot CLIP model on all the 12 datasets. On some datasets, for example EuroSat, our LaFTer shows an absolute improvement of over  $28\%$  on the zero-shot CLIP classification results. Even on the large scale ImageNet dataset we show a considerable improvement of  $2.3\%$  over the zero-shot CLIP classification results.
We also see that we outperform CLIP-PR on all datasets. Since CLIP-PR relies on offline pseudolabels, which are generated only once for the entire dataset and only proposes to finetune an adapter on top of frozen CLIP visual encoder. We conjecture that their approach might be less expressive. Moreover, it requires label distribution prior from the dataset on which it is being finetuned on. Our LaFTer is free from these requirements and proposes a more general solution for unsupervised finetuning on downstream datasets.
We also compare to the other unsupervised adaptation baseline UPL, which relies on finetuning learnable text prompts attached to the CLIP text encoder through knowledge distillation. In Table 1, we see that our method out performs UPL on most of the datasets (9 out of 12). On some datasets such as EuroSat, it shows a huge gain of 11.7 percentage-points over UPL. On datasets such as Describable Textures Dataset (DTD), Flowers-102 and SUN-397 our method is marginally behind
IN	CIFAR-10	CIFAR-100	IN-A	IN-S	IN-R	Mean
Class Name	58.4	87.1	59.0	30.7	37.7	65.5	56.4
Simple-Template	61.1	88.1	63.1	30.4	40.3	65.9	58.1
Dataset-Templates	60.1	89.3	64.7	31.1	41.2	67.5	59.0
Alpaca Descriptions	54.8	88.4	59.4	25.7	36.3	58.7	53.9
GPT Descriptions	60.5	87.8	63.0	30.2	39.6	63.6	57.4
GPT + Templates	61.9	89.3	65.4	31.5	40.9	67.8	59.5
Table 3: Top-1 Classification Accuracy (%) for a ViT-B/32 model while ablating different text generation strategies for training the text-only classifier in the first stage of LaFTer. To obtain these results, we evaluate the test set of the respective datasets by using the text-only pre-trained classifier on top of the frozen vision encoder from CLIP. IN = ImageNet.
Clip	w/o Aug	w/o Prompts	w/o Affine	w/o Cls	w/o Stop Grad	w/o Text Pre-train	LaFTer
CIFAR-10	88.8	93.5	92.5	94.6	94.1	94.7	95.1	95.8
CIFAR-100	64.2	72.6	71.7	72.9	67.4	73.2	73.1	74.6
UCF-101	61.0	65.3	66.1	63.6	63.4	67.5	67.7	68.2
EuroSat	45.1	63.2	61.2	64.2	60.9	69.2	69.7	73.9
ImageNet-A	29.6	29.9	29.8	30.8	30.1	31.1	28.9	31.5
ImageNet-S	40.6	40.3	40.7	41.1	40.9	42.1	41.3	42.7
ImageNet-R	65.8	68.0	67.7	66.8	66.1	71.8	70.2	72.6
Average	56.4	61.8	61.4	62.0	60.4	64.2	63.7	65.6
Table 4: Top-1 Accuracy (%) for our LaFTer while ablating the various critical design choices in our methodology. For each of these experiments, we disable one component from our framework and test the resulting method. Aug: Augmentations, Cls: Classifier.
UPL. We conjecture that since we use augmentations in one of our streams during our adaptation phase, it might result in noisy gradients during the learning process. Specially since datasets like DTD and Flowers-102 can depend on color cues for distinguishing the classes. For the large scale ImageNet dataset, we see that the performance of UPL is below the CLIP zero-shot performance. This can be because UPL generates offline pseudo-labels and in ImageNet, due to fine-grained classes the pseudo-labels might not be very confident from the zero-shot CLIP classifier. On the other hand, LaFTer benefits first from a classifier which has learned discriminative visual features through text-only training and later makes use of parameter efficient finetuning (PEFT). Furthermore, since our pseudo labels are generated in an online manner for each iteration during the optimization, they are also constantly refined as the training is progressing.
In Table 2 we provide a comparison of our LaFTer with the few-shot learning method CoOp [30] and also test Parameter Efficient Fine-Tuning (PEFT), which tunes the same learnable parameters as LaFTer (prompts, classifier, and affine parameters of the normalization layers), but in a few-shot manner. Interestingly, we see that our unsupervised representation learning method conveniently outperforms CoOp for 1- and 5-shots. For example, LaFTer (no-shots) is on average  $7.1\%$  better than CoOp (1-shot) and even  $1.3\%$  better in the 5-shot learning regime, while remaining competitive for 10-shots. It is also worth noting that for the large-scale ImageNet our LaFTer (requiring no labels) performs better than CoOp (10-shots), requiring 10000 labeled instances from the dataset. Results also follow a similar trend when compared with PEFT.
To understand the significance of all the components in our LaFTer we minutely study each design aspect. First we discuss the performance of our Text-Only pre-trained classifier, then we ablate each component in our unsupervised adaptation phase using unlabeled images and finally provide results by adapting other pre-trained backbones from CLIP [1]. Due to limited evaluation resources and space constraints, we perform these ablations on a subset of datasets with different complexity.
Text-Only Pre-trained Classifier. A main component of our LaFTer is the text-only pre-trained visual classifier, later used in our pseudo-labeling pipeline. The motivation behind it being, that since
Figure 3: Performance gains of using the text-only pre-trained visual classifier vs not using it in LaFTer pseudo-labeling pipeline scheme.
C-10	C-100	UCF	EuroSat	IN-R
CLIP (B/16)	89.2	68.1	64.7	48.4	73.8
LaFTer	96.5	76.3	67.2	72.1	81.4
CLIP (L/14)	95.3	75.8	72.0	60.3	84.9
LaFTer	99.0	87.2	77.2	77.2	91.5
Table 5: Top-1 Accuracy (%) for CLIP pre-trained ViT-B/16 and ViT-L/14 backbones. Results are provided for Base VL model (CLIP) and our LaFTer.
Figure 4: Effect of diversity of descriptions on the Top-1 Accuracy (\%). We train the text classifier by randomly choosing (with an increment of 5) a certain number of descriptions per class for each evaluation step. For all the main evaluations, we use a maximum of 50 descriptions per class.
CLIP is trained in order to have a shared text and vision embedding space so a classifier trained to classify the embeddings from any one of the modalities should also be able to classify embeddings from the other modality. We show that it is possible to train a classifier to classify images by only training it to classify natural language. To design this self-supervised objective of classifying text (described in detail in 3.1) we test different ways of generating the text dataset. For example, simple-template such as A photo of a ..., dataset-specific templates from CLIP [1] and descriptions from LLMs. These results are presented in Table 3. Note that for these results, we first train the classifier on the generated text dataset and then evaluate on the (images) test set of the respective dataset by using the trained classifier to classify the visual embeddings from the CLIP visual encoder. The simplest text dataset generation strategy: classifying the classname is also able to show reasonably strong visual classification performance. Furthermore, different strategies have slight difference in performance. We also find that descriptions from GPT-3 work better than descriptions from Alpaca. Our choice of generating class descriptions by prompting GPT-3 and complementing them by adding handcrafted templates from [1] works best. While averaging over 6 datasets, we gain a performance improvement of  $3.1\%$  in comparison to the simplest text dataset generation strategy of classifying the classmates themselves.
Text Classifier Contribution to LaFTer. Pre-training a visual classifier on text-only data and then employing it in our pseudo-labeling pipeline helps LaFTer gain improvements on all the 12 datasets which we evaluate in our paper. In Figure 3 we analyze the performance gains for our LaFTer when using the text-only pre-training in conjunction with our pseudo-labeling pipeline. On some datasets, for example EuroSat [57], our text-only pre-training helps our LaFTer to gain an absolute performance improvement of  $4.2\%$ . On other datasets, the performance gains are also consistent.
Design Choices for Unsupervised Finetuning. In order to study the effect on performance of each individual component we ablate our LaFTer in Table 4. We see that removing the classifier and instead using the CLIP Cosine similarity scores in both branches results in the highest degradation of results. Similarly, all other design choices also have a certain effect. For example, removing the learnable visual prompts results in a decrease of 4.2 percentage-points as compared to our LaFTer.
(a) Base CLIP (ViT-B/32).
(b) LaFTer Adapted CLIP (ViT-B/32).
Figure 5: TSNE projections for visual (circles) and text (triangles) embeddings for 10 classes of the EuroSAT dataset from (a) Base (un-adapted) CLIP ViT-B/32 model and (b) after adaptation with LaFTer. For the Base CLIP model, the text embeddings are the output feature vector from the CLIP text encoder and for LaFTer we use the weights of the adapted text-classifier. Visual embeddings are always the output features from the vision encoder.
Different CLIP Backbones. For our main experimental results described in Tables 1 and 2 and ablations in Tables 3 and 4, we use the ViT-B/32 CLIP backbone. In Table 5 we also provide results with different backbones for our LaFTer. We observe consistent improvements by LaFTer over CLIP also for larger backbones. For example, for ViT-B/16 our method shows  $23.7\%$  absolute improvement for EuroSat, while for ViT-L/14, our method improves CLIP zero-shot by  $16.9\%$  on the same dataset.
Diversity of Descriptions. To test the effect of diversity of generated descriptions on the visual classification results for the text-only pre-training of the text-classifier, we ablate the number of descriptions chosen for each class (for ImageNet and CIFAR-10/100) and provide the results in Figure 4. We find that increasing the diversity of descriptions to generate the text dataset has a positive effect for all the 3 datasets ablated.
Visualization of Embeddings. To gain further insights into the LaFTer adaptation we visualize the latent embeddings from the CLIP encoders in Figure 5, before and after adaptation for all 10 classes in the EuroSAT dataset. Analyzing the TSNE projections, we observe that the adaptation with our LaFTer results in more pronounced category clusters with larger inter-class distances, and also that after applying LaFTer the class-label text embeddings are better aligned with these category clusters.
5 Conclusion & Limitations
We propose a completely label-free finetuning method for Vision-Language models by first showing cross-modality transfer and learning a classifier on natural language inputs which can successfully classify visual data. Later, we leverage this text-only pre-trained classifier in our pseudo-labeling pipeline to further finetune the VL models in a parameter efficient manner. We extensively evaluate our LaFTer and achieve state-of-the-art results when comparing to other methods in an unsupervised finetuning paradigm, while also performing favorably in comparison to methods relying on few-shot supervised learning routines.
Limitations. One contribution of LaFTer is to show cross-modality transfer by training a classifier to classify natural language sentences and showing that it can be readily applied to classify visual data as well. In our work, we experimented with a single linear layer as a classifier. Due to the risk of over-fitting because of the sparse nature of natural language, we did not experiment with more complex structures. Experimentation with expanding the text dataset coupled with designing a more expressive classifier head is left as an exciting future direction.
We gratefully acknowledge the financial support by the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development and Christian Doppler Research Association. This work was also partially funded by the FWF Austrian Science
Fund Lise Meitner grant (M3374) and the Austrian Research Promotion Agency (FFG) under the SAFER (894164) project.
[50] Y. Zhang, J. Z. HaoChen, S.-C. Huang, K.-C. Wang, J. Zou, and S. Yeung, "DrML: Diagnosing and Rectifying Vision Models using Language," in Proc. ICLR, 2023. [Online]. Available: https://openreview.net/forum?id=losu6IAaPeB.
[51] S. Pratt, R. Liu, and A. Farhadi, "What does a Platypus Look Like? Generating Customized Prompts for Zero-shot Image Classification," arXiv:2209.03320, 2022.
[52] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Schlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” in Proc. CVPR, 2016.
[53] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A Large-scale Hierarchical Image Database," in Proc. CVPR, 2009.
[54] X. Chen and K. He, “Exploring Simple Siamese Representation Learning,” in Proc. CVPR, 2021.
[55] A. Krizhevsky and G. Hinton, "Learning Multiple Layers of Features from Tiny Images," Department of Computer Science, University of Toronto, Tech. Rep., 2009.
[56] L. Fei-Fei, R. Fergus, and P. Perona, “Learning Generative Visual Models from Few Training Examples: An Incremental Bayesian Approach Tested on 101 Object Categories,” in Proc. CVPR, 2004.
[57] P. Helber, B. Bischke, A. Dengel, and D. Borth, "Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification," in Proc. IGARSS, 2018.
[58] K. Soomro, A. R. Zamir, and M. Shah, "UCF101: A Dataset of 101 Human Actions Classes from Videos in the Wild," arXiv:1212.0402, 2012.
[59] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “SUN Database: Large-scale Scene Recognition from Abbey to Zoo,” in Proc. CVPR, 2010.
[60] M.-E. Nilsback and A. Zisserman, "Automated Flower Classification Over a Large Number of Classes," in Proc. ICVGIP, 2008.
[61] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, "Natural Adversarial Examples," in Proc. CVPR, 2021.
[62] H. Wang, S. Ge, Z. Lipton, and E. P. Xing, “Learning Robust Global Representations by Penalizing Local Predictive Power,” in NeurIPS, 2019.
[63] D. Hendrycks et al., "The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization," in Proc. ICCV, 2021.
[64] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, Stanford Alpaca: An Instruction-following LLaMA model, 2023.