SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training Parameters. For most of our experiments on CC3M, CC12M, and RedCaps, we utilized the ViT-B/16 architecture [17] and trained the models with a batch size of 8,192 and the AdamW optimizer [26]. Additionally, we explored the ViT-L/16 and ViT-S/16 architectures in ablation studies. For LAION-400M, we used both the ViT-B/32 and ViT-B/16 architecture with a batch size of 32,768, and followed the exact training setup outlined in [44], training the model for 32 epochs. Appendix B contains a detailed breakdown of our training hyperparameters.
Evaluation Setup. We consider three evaluation metrics for the trained models: Zero-Shot (ZS) classification accuracy, Few-Shot (FS) classification accuracy and Linear Probing (LP) accuracy. For zero-shot classification, we adopt the same prompt templates as described in the CLIP paper [44]. The class text embeddings are used to compute distance with the image feature, and images are classified to class with the shortest distance. For few-shot classification, we follow the set up in [44] and perform 5-way 5-shot classification with a weighted kNN classifier on top of the frozen features. For linear probing, following [44, 18], we freeze the pre-trained image encoder and extract features for every image in the downstream dataset. We then train a linear classifier using L-BFGS optimizer on top of the extracted features. ZS and LP are evaluated on both ImageNet (IN) and 15 Downstream (DS) datasets. FS are evaluated on the same downstream datasets. In the ablation studies we report the performance on IN and the mean on DS.
4.1 Zero-shot Evaluation
We provide a comprehensive analysis of the zero-shot transfer performance on ImageNet and downstream datasets in Table 1. Remarkably, across all pretrained datasets, our LaCLIP approach achieves a significant performance improvement over the baseline CLIP model on both ImageNet and down-
Table 2: Few-shot transfer evaluation of different models. We report 5-way, 5-shot classification accuracy for all downstream datasets. We highlight the best performance of each setting in **bold**. Similar to zero-shot, in nearly all cases, pre-training using language rewrites outperforms vanilla pre-training.
Data	Model	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average
Model Architecture: ViT-B/32
LAION-400M	CLIP	92.5	87.2	89.0	98.0	98.5	78.9	87.4	94.5	99.2	99.0	96.1	82.8	94.3	79.8	49.7	88.5
LaCLIP	93.5	91.0	90.7	98.2	99.1	82.2	87.5	95.7	99.4	99.2	97.2	80.1	94.2	80.4	52.2	89.4
Model Architecture: ViT-B/16
CC3M	CLIP	67.6	64.2	73.6	94.1	54.4	46.1	74.4	76.7	93.3	94.3	84.6	81.4	87.1	66.9	37.3	73.1
LaCLIP	70.0	69.1	76.8	95.2	57.6	49.2	75.8	77.4	95.2	95.0	89.5	81.1	85.5	71.0	37.3	75.0
CC12M	CLIP	87.0	77.5	82.1	97.2	90.9	62.0	83.3	91.1	98.2	97.6	92.6	83.4	91.2	70.6	44.3	83.3
LaCLIP	89.9	81.3	85.0	98.0	95.3	68.1	84.9	93.4	98.9	98.4	95.9	83.0	92.4	76.4	46.7	85.8
SLIP	87.6	79.2	83.0	97.5	85.6	56.4	85.8	88.1	97.7	97.1	92.5	84.9	91.0	62.4	43.0	82.1
LaSLIP	90.5	84.9	86.6	98.1	91.6	61.0	86.7	89.8	98.7	97.8	94.2	84.0	92.8	65.8	45.4	84.5
RedCaps	CLIP	94.4	80.6	85.3	95.9	88.5	54.5	82.6	94.5	97.8	99.0	94.8	84.9	91.3	75.3	40.6	84.0
LaCLIP	95.8	81.4	85.4	96.2	90.9	58.8	82.4	94.1	98.0	99.2	95.6	86.2	92.1	76.5	42.6	85.0
LAION-400M	CLIP	95.0	90.1	90.7	98.2	99.2	80.8	88.7	96.2	99.5	99.4	97.1	84.5	95.0	77.7	55.1	89.8
LaCLIP	95.8	92.7	91.9	98.4	99.5	86.1	89.0	97.1	99.6	99.5	98.1	82.9	95.0	80.9	57.9	91.0
stream datasets. For instance, when training models on the CC12M dataset, our LaCLIP method achieves over  $8\%$  improvement in absolute top-1 accuracy on ImageNet and  $7\%$  improvement on average over the other downstream datasets. LaCLIP and CLIP share the exact same amount of parameters and computation cost during training.
Adaptability to other methods. It is noteworthy that LaCLIP is compatible with other techniques intended to enhance CLIP's performance. Once the augmented texts are generated, integrating LaCLIP into any CLIP-based framework can be achieved seamlessly without incurring additional computational or memory overhead. As demonstrated in Table 1, we applied language augmentation to the SLIP framework and yield LaSLIP, resulting in significant performance improvements across all evaluation metrics. Notably, even though SLIP already incorporates additional self-supervision to enhance CLIP's performance, our proposed language augmentation further boosts its effectiveness. This showcases the generalization capability of our proposed text augmentation strategy.
Generalization to Larger Datasets. Consistently, the results highlight the substantial margin by which LaCLIP outperforms CLIP across various datasets. Noteworthy is the scalability of our method with dataset size, as it demonstrates improvement even when trained on the massive LAION-400M dataset, which contains hundreds of millions of data points<sup>1</sup>. These findings suggest that our LaCLIP approach can be seamlessly integrated as a plug-and-play component for training vision-language foundation models.
4.2 Few-Shot & Linear-Probing
We present the 5-way 5-shot classification performance in Table 2 and the linear-probing performance in Table 3. Our approach consistently outperforms vanilla CLIP or SLIP in the vast majority of cases. Interestingly, SLIP performs worse than vanilla CLIP in the few-shot setting, despite introducing additional self-supervision from the image side. However, by incorporating our proposed language augmentation strategy, SLIP's few-shot performance improves, surpassing vanilla CLIP. This result highlights the effectiveness of text augmentations in the few-shot setting.
Furthermore, it is important to highlight that the improvements observed in the few-shot and linear-probing results are solely achieved through the utilization of image encoders. This demonstrates the efficacy of our proposed text augmentation approach in enhancing not only the joint image-text embedding space, which aligns image and text features more effectively, but also the quality of the
<sup>1</sup>The version used in our experiment contains  $\sim 340\mathrm{M}$  samples, slightly less than original due to link rot. We use OpenCLIP implementation (https://github.com/mlfoundations/openclip) and achieves  $62.0\%$  ImageNet zero-shot accuracy for CLIP, comparable to their model with  $62.9\%$  trained on the full dataset.
Table 3: Linear Probing comparison of different models. Performances on ImageNet and 15 common downstream datasets are reported. We highlight the best performance of each setting in **bold**. Similar to ZS and FS, linear probe performance for our approach is almost always better than that of vanilla CLIP or SLIP.
Data	Model	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Model Architecture: ViT-B/32
LAION-400M	CLIP	85.8	95.8	83.6	75.1	89.2	54.3	79.7	86.9	94.5	96.8	97.9	96.3	93.5	88.6	23.1	82.7	74.6
LaCLIP	85.1	96.2	84.2	75.6	90.1	56.1	79.6	89.1	94.8	97.7	98.4	95.8	93.6	88.6	22.9	83.2	75.3
Model Architecture: ViT-B/16
CC3M	CLIP	62.6	86.8	68.1	58.5	32.8	40.9	63.4	69.6	82.0	89.4	91.7	95.9	89.0	71.9	13.3	67.7	54.5
LaCLIP	63.8	87.7	69.5	60.2	32.4	42.7	64.0	71.1	83.3	90.2	93.4	95.8	89.7	74.6	13.2	68.8	56.5
CC12M	CLIP	81.6	93.8	79.3	72.0	75.1	52.6	75.6	86.2	92.2	95.3	97.3	96.7	93.1	80.6	19.7	79.4	70.3
LaCLIP	82.9	94.7	79.7	73.8	79.9	54.5	75.7	87.7	93.0	96.4	98.0	96.4	93.0	81.9	19.7	80.5	72.3
SLIP	84.4	94.2	79.1	73.5	74.2	54.6	76.5	86.1	92.7	95.7	97.6	96.8	93.7	74.0	20.6	79.6	73.2
LaSLIP	85.2	94.6	80.8	75.1	77.0	53.8	78.5	85.6	93.7	96.5	97.9	96.8	93.5	76.1	21.1	80.4	74.4
RedCaps	CLIP	89.1	94.1	78.8	65.6	74.0	52.5	73.2	91.5	91.4	97.7	98.0	96.3	93.5	80.8	17.0	79.6	71.8
LaCLIP	90.1	94.3	78.5	66.6	77.6	53.6	73.9	90.8	91.5	97.9	97.6	96.6	92.7	80.8	17.2	80.0	71.9
LAION-400M	CLIP	90.5	96.9	85.0	78.1	92.1	57.2	80.0	90.9	95.7	98.0	98.7	96.7	94.7	90.3	27.0	84.8	78.6
LaCLIP	90.7	96.7	85.5	78.7	92.8	63.1	81.3	92.8	96.2	98.8	99.1	96.4	94.6	89.5	27.5	85.6	79.9
image representations themselves. This improvement is realized by pairing images with a wider range of diverse texts, providing richer and more varied supervision to the vision encoder. As a result, the vision encoder is able to learn more generalizable and robust image representations that can be effectively utilized across a range of downstream tasks. To further emphasize the effectiveness of our approach, we provide detailed t-SNE visualizations of the learned features with different approaches in Appendix F. These visualizations qualitatively demonstrate that features learned with LaCLIP exhibit clearer class boundaries across different classes, reinforcing the enhanced discriminative power of our approach.
Varying Augmentation Strategies. Table 4 presents a comparison of our proposed LLM-based text augmentation strategy with existing language augmentation methods, namely EDA [58], which involves word-level operations such as synonym replacements, and back translation [50], which translates the text to another language and then back to the original language. Although these simpler augmentation techniques demonstrate some improvements, the enhancements achieved are relatively marginal, especially when compared to our proposed LLM-based approach. This disparity can be attributed to the significantly higher diversity in sentence structures that our method achieves. In Appendix D, we provide further details and qualitative comparisons of the various text augmentation techniques using different methods.
Scaling with Number of Augmentations. Figure 3 demonstrates the impact of the number of augmented texts per sample in our LaCLIP approach on ImageNet zero-shot accuracy. A value of 0 augments corresponds to vanilla CLIP without any text augmentation. The results clearly demonstrate that simpler augmentation strategies exhibit poor scalability as the number of augments increases. This is attributed to their limited diversity. Conversely, our LLM-based text augmentation consistently improves performance as more augmentations are added.
Different Meta-Input-Output Pair for ICL. In order to assess the impact of different meta-input-output caption pairs used for LLaMA ICL, we conducted an experiment by disentangling LaCLIP and training four models. Each model was trained using only the ground truth caption combined with a specific type of LLM augmented version, using a particular meta-input-output pair (ChatGPT, Bard, COCO, or human). The performance comparison between these models is presented in Table 5. It can be observed that different strategy yield similar performance, with the model trained with augmentations using the Human pair slightly outperforming the others. We conjecture the reason
Table 4: Performance comparison of CLIP training with different text augmentations.
Augment	ZS	FS	LP
DS	IN	DS	IN
N/A (CLIP)	38.8	40.2	83.3	79.4	70.3
EDA [58]	40.6	41.2	83.4	79.4	70.5
Back Trans [50]	40.4	41.6	83.9	79.8	70.7
LLM (Ours)	46.2	48.4	85.8	80.5	72.3
Table 5: Ablation of CLIP trained with augmented texts prompted by different meta-input-output pairs.
Source	ZS	FS	LP
DS	IN	DS	IN
ChatGPT	42.3	44.5	84.8	79.8	71.2
Bard	41.7	44.8	85.0	79.6	71.2
MSCOCO	42.1	44.6	84.8	79.8	71.3
Human	43.0	45.1	84.8	79.9	71.3
Table 6: Ablation on different network architectures.
Backbone	Method	ZS	FS	LP
DS	IN	DS	IN
ViT-S/16	CLIP	36.3	36.9	82.2	77.0	67.1
LaCLIP	44.1	46.3	84.5	78.0	69.1
ViT-B/16	CLIP	38.8	40.2	83.3	79.4	70.3
LaCLIP	46.2	48.4	85.8	80.5	72.3
ViT-L/16	CLIP	42.6	44.0	85.1	81.3	72.9
LaCLIP	46.6	49.1	86.8	81.9	73.7
Figure 3: ImageNet zero-shot accuracy with different num of text augments.
Table 7: Performance comparison of CLIP, LaCLIP and LaCLIP-MT on different datasets.
Dataset	Method	ZS	FS	LP
DS	IN	DS	IN
CC12M	CLIP	38.8	40.2	83.3	79.4	70.3
LaCLIP	46.2	48.4	85.8	80.5	72.3
LaCLIP-MT	45.2	49.0	85.8	80.6	72.4
RedCaps	CLIP	42.6	42.9	84.0	79.6	71.8
LaCLIP	45.0	46.2	85.0	80.0	71.9
LaCLIP-MT	46.1	48.1	85.3	80.3	72.4
is that during the meta-input-output generation process, humans have the advantage of viewing the corresponding image, which allows them to generate more accurate and diverse rewrites.
Different Backbone Architecture. We further investigate the performance of LaCLIP using different backbone architectures. Table 6 summarizes the results obtained with ViT-S/16, ViT-B/16, and ViT-L/16. We observe that LaCLIP scales with model size and consistently outperforms vanilla CLIP across all network architectures. These findings highlight the effectiveness of LaCLIP in improving the performance of CLIP models, irrespective of the underlying backbone architecture.
Comparison with Pre-trained Text Encoder. To deepen the comprehension on whether LaCLIP outperforms vanilla CLIP simply due to a better text encoder, we conducted experiments comparing LaCLIP with CLIP models trained with a pre-trained text encoder. Here we employed the BERT model as the pre-trained text encoder. The experiment result in Table 8 demonstrates that fine-tuning based on the pretrained text encoder exhibits some improvements, whereas freezing the pre-trained text en
Table 8: Performance comparison of LaCLIP and CLIP models trained with different pre-trained text encoder configurations.
Method	Text Encoder	ZS	FS	LP
Pre-train	Freeze	DS	IN	DS	IN
CLIP	X	X	38.8	40.2	83.3	79.4	70.3
✓	X	42.1	42.9	83.6	79.5	70.4
✓	✓	24.5	23.2	80.3	74.9	66.0
LaCLIP	X	X	46.2	48.4	85.8	80.5	72.3
coder weights substantially degrades performance. This observation aligns with the findings in LiT [63]. In contrast, LaCLIP consistently outperforms all configurations with a pre-trained text encoder, underscoring the benefit and necessity for explicit sentence augmentation strategies.
5 Multi-Text Training Loss with LaCLIP
It is important to highlight that once we have generated multiple text augmentations, and with a slight tolerance for computational cost, we can create multi-positive training pairs for each training iteration. These pairs are formed by pairing each image with not only the original text but also with all the rewritten versions of the text. By adopting this approach, we introduce a Multi-Text version
of LaCLIP, referred to as LaCLIP-MT, which incorporates a multi-positive contrastive training loss. The training process iterates through all the images in the following manner:
$$
L _ {I *} = - \frac {1}{M} \sum_ {i = 1} ^ {N} \sum_ {j = 0} ^ {M} \log \frac {\exp \left(\operatorname {s i m} \left(f _ {I} \left(\operatorname {a u g} _ {I} \left(x _ {I} ^ {i}\right)\right) , f _ {T} \left(x _ {T j} ^ {i}\right)\right) / \tau\right)}{\sum_ {k = 1} ^ {N} \exp \left(\operatorname {s i m} \left(f _ {I} \left(\operatorname {a u g} _ {I} \left(x _ {I} ^ {i}\right)\right) , f _ {T} \left(x _ {T j} ^ {k}\right)\right) / \tau\right)}, \tag {4}
$$
Since each text can still be paired with a single image, the training loss that iterates through all the texts remains unchanged, with the only difference being that it now iterates over all of the texts instead of just the augmented ones. Consequently, the final training loss is given by the average of the image loss  $(L_{I*})$  and the text loss  $(L_T)$ , resulting in  $L = (L_{I*} + L_T) / 2$ .
In order to showcase the efficacy of the multi-positive contrastive training loss in boosting the performance of LaCLIP, we conducted additional experiments on the CC12M and RedCaps datasets. The results of these experiments are summarized in Table 7, which compares the performance of LaCLIP-MT with both LaCLIP and vanilla CLIP. The results clearly indicate that LaCLIP-MT could further improve upon LaCLIP across most metrics. By allowing each image to be paired with all of the diverse texts describing its content, LaCLIP-MT leverages the additional and richer supervision from the language modality to enhance the formation of image-text embeddings. This improvement highlights the benefits of the multi-positive contrastive training loss in facilitating better alignment between images and diverse text descriptions.
6 Conclusion, Limitations and Broader Impact
Conclusion. We have introduced LaCLIP, a straightforward yet highly effective CLIP training strategy that incorporates text augmentations through text rewriting, leveraging the in-context learning capabilities of LLMs. Through this simple and versatile approach, we have demonstrated significant improvements in the performance of CLIP embeddings across various pre-training scales and datasets. Additionally, we have proposed a novel multi-text training loss to further enhance the training process. As LLMs continue to improve in performance and in-context learning capabilities, our approach stands to directly benefit from these advancements.
Limitations. While the training process itself does not entail any additional memory or computation overhead compared to vanilla CLIP, the process of generating text rewrites using LLMs can be computationally expensive, requiring significant GPU resources and taking hours for large datasets. Additionally, the quality of the rewritten text generated by LLaMA is not filtered, which may result in some irrelevant details that do not align well with the corresponding images. This misalignment could impact the transferability of the learned embeddings to downstream tasks. To address these limitations, future work could focus on developing more efficient methods for generating text rewrites using LLMs, reducing the computational burden without sacrificing performance. Furthermore, techniques for filtering the rewritten texts could be explored, aiming to retain only the most relevant and accurate versions while discarding those with misleading details. This would enable the model to learn a better embedding space that is robust and transferable across different downstream datasets, improving overall performance and alignment between vision and text encoders.
Broader Impact. We propose a general text augmentation strategy that can generate diverse rewrites for any given text. This strategy not only improves the performance of vision-language models but also has the potential to enhance models in pure natural language processing tasks, such as language understanding and reasoning. On the other hand, we acknowledge that LLMs are trained on large-scale web data, which may contain factual errors and hallucinations. Consequently, the rewritten versions of texts may also inherit these limitations. Therefore, we encourage researchers to implement additional data filtering methods before deploying these models in real-world scenarios. Additionally, the current LLM-based rewriting strategy requires significant GPU/TPU computation, which can contribute to a higher carbon footprint. However, it is also possible that such rewriting strategy can significantly reduce the number of training iterations for larger models to reach similar performances as vanilla CLIP.
We would like to thank Mathilde Caron for the insightful early manuscript review, and the anonymous reviewers for their helpful comments and suggestions. Additionally, we appreciate the support and discussions with the VisCAM team at Google Research.
[56] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2, 3, 4, 5
[57] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 27
[58] Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196, 2019. 2, 3, 8, 9, 22, 24, 26
[59] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010. 6, 16
[60] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783, 2021. 2
[61] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 2
[62] Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6995-7004, 2021. 2
[63] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133, 2022. 2, 9
We provide additional experiment details, results and analysis in the supplementary material.
A. Dataset Details
A.1 Pre-training Datasets
We utilized four image-text pre-training datasets of varying scales, namely CC3M [51], CC12M [7], RedCaps [15], and LAION-400M [49], to train both CLIP and LaCLIP models. Additionally, we trained SLIP and LaSLIP models on the CC12M dataset. It is important to note that due to image link rot within the datasets, the versions we obtained may have slightly fewer images compared to the original versions. As a result, there may be slight performance differences when compared to models trained on the full image versions. Below are detailed descriptions of the four pre-training datasets:
CC3M [51]: This dataset comprises 3.3 million image-text pairs extracted from 5 billion webpages. The image descriptions are derived from the HTML alt-text attribute. The version we used consists of 2.8 million unique samples.
CC12M [7]: With a similar procedure as CC3M, CC12M consists of 12.4 million image-text pairs. The filters used in this dataset are more relaxed, resulting in a wider range of topics and visual concepts, making it more reflective of real-world scenarios. The version we acquired contains 10.0 million samples.
RedCaps [14]: RedCaps encompasses 12.0 million image-caption pairs gathered exclusively from Reddit across 350 subreddits. The captions are sourced from Reddit instead of HTML alt-text. The version we acquired includes 11.7 million unique samples.
LAION-400M [49]: This dataset is constructed by processing and filtering the Common Crawl dataset. The original version contains 413 million unique samples, while the version we obtained consists of 340 million samples.
For all datasets, we resized the images such that the shorter side measured 256 pixels.
A.2 Downstream Datasets
We conducted evaluations on our pre-trained model using both ImageNet [13] and 15 widely-used downstream datasets. To prepare the downstream datasets, we utilized torchvision and VISSL [21]. The detailed information about the downstream datasets can be found in Table A1.
B. Implementation Details
Encoders We employed the standard ViT-S/16, ViT-B/16, ViT-B/32 and ViT-L/16 architectures from [17, 55] as our vision encoders. Specifically, ViT-B/16 is used on all CC3M, CC12M and RedCaps datasets. ViT-B/32 is used on LAION-400M. ViT-S/16 and ViT-L/16 are used on CC12M. Following the approach in SLIP [41], we utilized the smallest text encoder from CLIP [44]. Our tokenizer was consistent with CLIP, having a vocabulary size of 49, 408 and a maximum context length of 77. Further details about the encoders can be found in Table A2.
Hyper-Parameters Table A3 provides an overview of the pre-training hyperparameters used for CLIP on all datasets. Following [44, 41], we perform RandomResizedCrop augmentation for the images. For SLIP training, the learning rate was set to  $3 \times 10^{-3}$ , weight decay was set to 0.1, and all other parameters remained the same. Further details can be found in Table A4. The pre-training process was conducted on four machines with eight A100 GPUs each.