SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lijie Fan $^{1,2,*}$
Dilip Krishnan<sup>1</sup>
Phillip Isola2
Dina Katabi²
Yonglong Tian $^{1,*}$
<sup>1</sup>Google Research,
$^{2}$ MIT CSAIL,
Contrastive Language-Image Pre-training (CLIP) stands as one of the most effective and scalable methods for training transferable vision models using paired image and text data. CLIP models are trained using contrastive loss, which typically relies on data augmentations to prevent overfitting and shortcuts. However, in the CLIP training paradigm, data augmentations are exclusively applied to image inputs, while language inputs remain unchanged throughout the entire training process, limiting the exposure of diverse texts to the same image. In this paper, we introduce Language augmented CLIP (LaCLIP), a simple yet highly effective approach to enhance CLIP training through language rewrites. Leveraging the in-context learning capability of large language models, we rewrite the text descriptions associated with each image. These rewritten texts exhibit diversity in sentence structure and vocabulary while preserving the original key concepts and meanings. During training, LaCLIP randomly selects either the original texts or the rewritten versions as text augmentations for each image. Extensive experiments on CC3M, CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with language rewrites significantly improves the transfer performance without computation or memory overhead during training. Specifically for ImageNet zero-shot accuracy, LaCLIP outperforms CLIP by  $8.2\%$  on CC12M and  $2.4\%$  on LAION-400M. Code is available at https://github.com/LijieFan/LaCLIP.
Pre-trained vision-language multi-modal encoders, exemplified by CLIP [44], have proven to be extremely useful in learning transferable features from paired image and text data. CLIP's training process can leverage two scalable paradigms: data and compute. Firstly, the availability of large-scale vision-language paired data [49, 48] enables effective training at a substantial scale. Secondly, CLIP's utilization of language and image co-embeddings grants it favorable scaling properties with respect to compute resources [27]. Consequently, CLIP embeddings consistently outperform other visual pre-training approaches such as SimCLR [8] or MAE [22] across various downstream tasks [27]. Follow-up methods that build upon language-image pre-training, such as SLIP [41] and FLIP [35], exhibit similar advantageous scaling performance.
The CLIP framework is built upon contrastive learning, which typically relies on data augmentations to prevent overfitting and the learning of ineffective shortcuts [8, 47]. However, in the CLIP training process, this beneficial feature is applied exclusively to image inputs, which undergo augmentations in every epoch. In contrast, text inputs are neglected and remain unchanged throughout training, lacking any form of augmentation. This input asymmetry leads to scenarios where the same text is consistently paired with slightly augmented images, while the augmented version of the same image is always paired with the exact same words. Such asymmetry presents two issues. Firstly, the image encoders receive limited supervision from the language side since the same image is consistently paired with the same words. Consequently, the language aspect provides less guidance to the image
encoders. Secondly, the text encoders repeatedly encounter the exact same texts in each epoch, which increases the risk of text overfitting and significantly impacts zero-shot transferability.
Hence, it becomes crucial to incorporate a suitable augmentation strategy for the text inputs. However, existing language augmentation methods are not sufficiently effective. Most previous approaches [58] focus on word-level treatments like replacement or masking, which have limited impact on enriching text structures and are comparatively weaker than image augmentations. Currently, there is a lack of language rewriting strategies that can effectively augment sentences while preserving key concepts and meanings. Such strategies are urgently needed for CLIP training to ensure optimal performance.
Meanwhile, alongside the development of CLIP, the field has witnessed significant advancements in large language models (LLMs) like GPT models [5, 44, 45] and LaMDA [3]. These LLMs have seen tremendous growth in terms of data, computational resources, and performance. Instruction fine-tuned models such as ChatGPT [2] and Bard [1] have also emerged, incorporating fine-tuning through supervised and reinforcement learning. These models have exhibited exceptional performance surpassing human capabilities across a wide range of natural language tasks. Motivated by these advancements, we naturally explored the potential of leveraging LLMs to effectively generate diverse rewritten versions of a given text. While a straightforward approach would involve directly employing instruction fine-tuned models like ChatGPT or Bard, their closed-source nature renders it infeasible to use them for rewriting the hundreds of millions of image descriptions in the datasets. Fortunately, open-sourced LLMs such as LLaMA [56], despite lacking fine-tuning with instructions, possess excellent In-Context Learning (ICL) capabilities, enabling predictions with a limited context. By thoughtfully designing contextual examples, LLaMA can generate diverse and rich text rewrites for the entire dataset.
Building upon this foundation, we propose Language augmented CLIP (LaCLIP), a straightforward yet highly effective approach for enhancing CLIP model performance by harnessing the power of LLMs. Our method leverages ICL using LLaMA to generate diverse variants of each caption within the text-image pairs of a given pre-training dataset. To facilitate ICL prompts, we have devised multiple strategies to generate a small set of meta-input-output caption pairs. These strategies involve utilizing ChatBots, human rewriters, or existing image captioning datasets. Once we have acquired the meta-input-output pairs, we employ them as examples to prompt LLaMA, enabling the rewriting of millions of texts within the entire dataset. Unlike existing strategies for text rewriting [58, 50], which tend to preserve the sentence structure, LLMs exhibit the remarkable ability to generate language rewrites with greater richness and diversity. This is attributed to their emergent properties and extensive training data. Following the caption rewriting process conducted by LLaMA ICL, each image is now accompanied by a collection of diverse captions resulting from the rewriting process. Utilizing these rewritten texts, we proceed to train CLIP models with augmentation also on the text side. The text augmentation could be performed by randomly selecting one out of the many captions associated with each image.
Extensive experiments on various pretraining datasets at different scales demonstrate our proposed LaCLIP could significantly improve the transferability of CLIP. For instance, on the LAION-400M dataset [49], we observe a notable improvement over CLIP in the zero-shot performance on ImageNet, increasing from  $62.0\%$  to  $64.4\%$ . We firmly believe that this strategy presents a simple, scalable approach that contributes to the array of training strategies available for training image embeddings.
Vision-Language models. There are a number of earlier works demonstrating the effectiveness of learning visual representations from the supervision of corresponding text [25, 31, 14, 62]. Contrastive Language-Image Pretraining (CLIP) [44] has attracted significant attention due to its superior representation learning and zero-shot transfer ability. This performance is achieved through contrastive learning on top of image and text features. Another related approach is ALIGN [24], which achieves similar performance with larger and noisier datasets. There have been numerous follow-up works that attempt to improve the efficacy and data efficiency of CLIP training. SLIP [41] and DeCLIP [34] proposes to improve the performance by incorporating self-supervised training techniques. FILIP [60] proposes to leverage cross-modal fine-grained alignment between image patches and text words. CoCa [61] introduces an additional decoder and captioning loss. LIT [63] proposes to boost zero-shot transferring performance by fine-tuning the text encoders. BLIP series [33, 32] include
additional captioners and incorporates iterative image captioning within the training pipeline, which intricately link the generated captions with the image content. However, most of these follow-up works introduce additional training inputs and losses, which can have a negative impact on training efficiency and memory consumption.
Text augmentation and Large Language Models. The success of natural language processing (NLP) tasks is strongly dependent on the quality and quantity of available data. Deep neural networks benefit from more data [40, 58]. Common practices in data augmentation include synonym replacement [58], random masking [30], and back translation [50]. The advent of self-supervised large language models like BERT [16] and GPT series [45, 46] has been a game-changer, as they do not require labeled data and can scale up to web-level data to achieve superior transfer ability. Recently, even larger foundation models with billion-level parameters have emerged, revolutionizing the NLP community. Models like the 175B GPT-3 [5] and 540B PaLM [10] achieve superior performance on various NLP tasks. GPT-3 also demonstrates the few-shot in-context learning ability of large language models [5]. Open-sourced LLaMA [56] also achieve comparable performances on various benchmarks. In addition to them, ChatGPT and Bard are chatbots trained with reinforcement learning human feedback (RLHF), and have achieved human-comparable performances on various language understanding tasks [6].
3 Improving CLIP with Language Rewrites
This section outlines the core design of our LaCLIP framework, highlighting the key components and strategies involved. We provide a comprehensive description of our approach, including the generation of a small set of meta-input-output text pairs from diverse sources, the process of rewriting image descriptions across the entire dataset using LLM ICL, and the enhanced CLIP training strategies incorporating these rewritten descriptions.
CLIP. The Contrastive Language-Image Pretraining (CLIP) method has proven to be highly effective to train vision models using language supervision. In this framework, a large batch of  $N$  paired images and text  $\{x_{I}, x_{T}\}$  are sampled from the training dataset during each training step. The images are pre-processed using data augmentations, and image and text features are extracted using dedicated encoders and normalization functions  $f_{I}$  and  $f_{T}$ . The image text features are used to compute the InfoNCE loss, where the paired images and text form the positive pairs and the unpaired ones are treated as negative samples. The training loss can be formulated as follows:
$$
L _ {I} = - \sum_ {i = 1} ^ {N} \log \frac {\exp \left(\operatorname {s i m} \left(f _ {I} \left(\operatorname {a u g} _ {I} \left(x _ {I} ^ {i}\right)\right) , f _ {T} \left(x _ {T} ^ {i}\right)\right) / \tau\right)}{\sum_ {k = 1} ^ {N} \exp \left(\operatorname {s i m} \left(f _ {I} \left(\operatorname {a u g} _ {I} \left(x _ {I} ^ {i}\right)\right) , f _ {T} \left(x _ {T} ^ {k}\right)\right) / \tau\right)}, \tag {1}
$$
In this scenario,  $(x_I^i,x_T^i)$  is the  $i^{th}$  image-text pair, and  $\mathrm{aug}_I()$  denotes the image augmentation functions. The  $\mathrm{sim}(\cdot ,\cdot)$  function measures distance using the dot product, while the temperature  $\tau$  is a learnable parameter that scales the logits. To simplify the presentation, we only show the training loss iterating over images. A symmetrical loss  $L_{T}$  that iterates over texts is also computed during the training process. The total training loss is  $L = (L_{I} + L_{T}) / 2$
Language Rewrites as Text Augmentation. In Equation 1, the standard CLIP loss applies augmentation exclusively to images, leaving the text inputs unchanged throughout the whole training process. Recognizing this gap, we propose to generate text augmentations, denoted as  $\mathrm{aug}_T$ , where  $\mathrm{aug}_T(x_T)$  is utilized as the input for  $f_{T}$  instead of the original  $x_{T}$ . In the following subsections, we introduce the methodology employed to generate these text augmentations using LLMs, as well as the integration process during CLIP training. By addressing this gap, we aim to enhance the training process and expand the benefits of augmentation to the text inputs, leading to improved performance and a more comprehensive learning framework.
3.2 Meta-Input-Output Text Pair Generation
A recently uncovered property of autoregressive LLMs is In-Context Learning (ICL) [5, 39]. This property allows LLMs to learn a new task by conditioning on a few examples and then make predictions for a test input. To harness ICL for text rewriting, we first need to generate several
Figure 1: Illustration of using ChatGPT to generate meta-input-output pairs: we first sample source captions randomly from a few datasets. We then use prompts such as "Rewrite this image caption" to guide the ChatGPT model to generate rewritten target captions. The resulting target captions have different sentence structure than the source texts but, crucially, keep the major objects and subjects intact (in bold). These meta-input-output pairs are later used as contexts for ICL.
rewriting examples that can be used as examples in the prompt context. We name these examples meta-input-output text pairs. To this end, we explored different strategies for generating these pairs:
- Rewriting with Chatbots. We randomly sample texts from image-text datasets, and prompt ChatGPT[2] and Bard[1] web portals to generate target texts using a prompt such as "Rewrite this caption of an image vividly, and keep it less than thirty words:" . Illustrations of this process can be found in Figure 1. Here we leverage the extremely powerful rewriting abilities of these models to provide modified captions that keep the essence of the original caption but change the style and details. This ensures that the semantics associated with the corresponding image do not change, which is important for representation learning purposes.
- MSCOCO Sampling. Multiple text descriptions for the same image are available in many existing image captioning datasets. To leverage this characteristic, we utilize the widely used MS-COCO dataset [36]. Within this dataset, each image is associated with five distinct text descriptions, which have been meticulously annotated by human workers. From this dataset, we randomly select a subset of images. For each selected image, we choose one description as the meta-input text and another as the meta-output text.
- Human Rewriting. We randomly sample several image-text pairs from various image-text datasets. To ensure diverse and varied textual variations, we engage human annotators and task them with rewriting the captions based on the content depicted in the corresponding observed images. This process results in the creation of meta-input-output pairs, consisting of the original text and the rewritten version by human annotators.
Through the utilization of diverse generation strategies, we acquire four distinct types (ChatGPT, Bard, COCO, and Human) of meta-input-output text pairs, which then serve as valuable examples within the input context for the In-Context Learning (ICL) framework. For each specific strategy, we randomly select 16 original captions from image-text datasets and generate target captions using that strategy, resulting in a total of 16 meta-input-output pairs. These pairs encompass a range of sources and variations, facilitating a comprehensive and diverse training experience for our framework.
3.3 Large scale Language Rewriting
Generating rewrites for hundreds of millions of texts using closed-source models like ChatGPT or Bard is impractical due to the significant financial and time costs associated with API usage. Therefore, to facilitate the rewriting of text samples within any given image-text dataset, we employ LLaMA [56]—an open-source state-of-the-art large language model known for its robust performance in text completion tasks. Despite not being fine-tuned with instructions, LLaMA exhibits exceptional ICL capabilities. Leveraging the meta-input-output text pairs generated as described in Section 3.2, we employ LLaMA's ICL ability to rewrite every text entry within the image-text dataset.
Given a text sample to be rewritten, we formulate a context input as the following three parts: Firstly, we include a sentence that informs the LLM about the task of rewriting image descriptions. This serves as an initial contextual clue for the LLM to understand the objective at hand. The second part of the context encompasses three examples sampled from the meta-input-output pairs described in Section 3.2. We randomly three distinct meta-input-output caption pairs from a specific strategy (e.g.,
Rewrite the following image descriptions:
white red cheerful combination in the bedroom for girl  $= >$  A bright and lively white-and-red color scheme in a girl's bedroom, creating a cheerful ambiance.
A tourist taking a photograph of river looking towards suspension bridge and office  $\Rightarrow$  Tourist snaps photo of suspension bridge and office building across the river.
tree hollow and green leaves of a tree top in summer  $=>$  Amidst lush green leaves on the top of a tree, a hollow creates a natural shelter, typical of summer foliage.
man driving a car through the mountains  $= >$
Augmented Caption
LLaMa
A man confidently navigating a winding mountain road with breathtaking views.
Figure 2: Illustration of our proposed in-context learning based strategy for language rewriting: The left box depicts the input context for LLaMA, responsible for text completion. The blue and green texts represent the meta-input-output text pairs, with blue indicating the input and green indicating the output. These pairs are defined in Section 3.2 and visualized in Fig. 1. The final blue line represents the text from the image-text datasets intended for rewriting. On the right box, we showcase the completion result generated by the open-source 7B parameter LLaMA model [56], which represents the rewritten version of the text of interest.
ChatGPT). Each pair is clearly separated by a "  $= >$  " symbol. These pairs provide explicit instances that showcase the desired rewriting behavior for the LLM to learn from. The additional random sampling process further enable the LLaMA model to generate more diverse text rewrites. Finally, the last part of the context includes the text sample that requires rewriting, followed by the separation symbol. This ensures that the LLM receives the specific text to be rewritten as part of its context input. By incorporating these three parts, we create a comprehensive context that guides the LLM in effectively generating diverse and contextually appropriate text rewrites.
Utilizing the constructed context input as a prompt, LLaMA exhibits its ability to perform text completion and generate rewritten versions of the corresponding text samples. This process is conducted for each text sample present in the pre-training image-text dataset. Specifically, we employ the LLaMA-7B model to generate four distinct rewrites for every text sample in the dataset, with each rewrite corresponding to one of the four different meta-input-output sources (ChatGPT, Bard, COCO, or Human). It takes 7 hours to generate one rewrite for the entire CC3M dataset on a 8 A100 GPU machine. By incorporating multiple sources and leveraging the capabilities of LLaMA, we ensure the generation of diverse and contextually relevant text rewrites for each text sample within the dataset.
3.4 LaCLIP: Training CLIP with Language Augmentations
Having generated  $M$  different rewrites for each caption (in our case,  $M = 4$ ), we are now able to bridge the augmentation gap between the image and text inputs, thereby presenting the training strategy for our LaCLIP framework. The key addition to the CLIP framework is the augmentation function for the text inputs, which can be easily implemented through a straightforward random sampling process, where we randomly select a text sample from either the original text or one of the generated rewrites:
$$
\operatorname {a u g} _ {T} \left(x _ {T}\right) \sim \text {U n i f o r m} \left(\left[ x _ {T 0}, x _ {T 1} \dots , x _ {T M} \right]\right) \tag {2}
$$
Here  $x_{Tk}$  is the  $k^{th}$  rewritten sample for  $x_{T}$ . For the sake of simplicity, we denote the original text as  $x_{T0}$ . The training loss over the images becomes:
$$
L _ {I} = - \sum_ {i = 1} ^ {N} \log \frac {\exp \left(\operatorname {s i m} \left(f _ {I} \left(\operatorname {a u g} _ {I} \left(x _ {I} ^ {i}\right)\right) , f _ {T} \left(\operatorname {a u g} _ {T} \left(x _ {T} ^ {i}\right)\right)\right) / \tau\right)}{\sum_ {k = 1} ^ {N} \exp \left(\operatorname {s i m} \left(f _ {I} \left(\operatorname {a u g} _ {I} \left(x _ {I} ^ {i}\right)\right) , f _ {T} \left(\operatorname {a u g} _ {T} \left(x _ {T} ^ {k}\right)\right)\right) / \tau\right)}, \tag {3}
$$
The only difference with the original CLIP training here is the additional text augmentation  $\mathrm{aug}_T$ , and all other parts remain the same, which does not bring any additional computation or parameter overheads compared to original CLIP during training. By incorporating text augmentations into CLIP, we introduce variability and diversity into the training data, enabling the model to learn from both the original text and the augmented versions. This simple yet effective strategy enhances the training process and contributes to the overall performance and adaptability of the LaCLIP framework.
Table 1: Zero-shot transfer evaluation of different models. Performance on ImageNet and 15 common downstream datasets are reported. We highlight the best performance of each setting in **bold**. We see that regardless of the scale of the pre-training dataset, LaCLIP outperforms CLIP [44] and LaSLIP outperforms SLIP [41], by up to  $8.2\%$  absolute accuracy.
Data	Model	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Model Architecture: ViT-B/32
LAION-400M	CLIP	79.9	91.8	72.0	64.6	77.0	15.8	49.9	84.8	89.3	64.4	95.3	43.2	60.6	36.9	14.5	62.7	62.0
LaCLIP	79.7	92.4	73.0	64.9	81.9	20.8	55.4	87.2	91.8	70.3	97.3	50.6	61.5	49.4	16.0	66.1	64.4
Model Architecture: ViT-B/16
CC3M	CLIP	10.3	54.9	21.8	25.0	0.8	1.4	10.5	12.8	43.3	10.2	77.6	14.1	19.1	6.9	0.6	20.6	15.8
LaCLIP	14.2	57.1	27.5	35.1	1.6	1.6	16.6	15.6	52.7	14.7	86.2	15.0	24.3	6.4	1.0	24.6	21.5
CC12M	CLIP	50.8	64.9	38.5	44.7	24.1	2.4	19.4	64.1	77.4	33.2	91.0	20.1	38.9	7.3	5.1	38.8	40.2
LaCLIP	60.7	75.1	43.9	57.0	36.3	5.6	31.0	72.4	83.3	39.9	95.1	27.3	44.3	12.7	8.9	46.2	48.4
SLIP	52.5	80.7	46.3	48.8	24.9	2.3	25.1	58.6	77.6	29.2	89.1	25.8	36.6	6.0	5.7	40.6	42.1
LaSLIP	62.9	82.0	50.2	59.6	32.2	4.4	30.1	70.6	82.4	37.4	95.0	20.4	45.6	10.1	9.2	46.1	49.7
RedCaps	CLIP	81.5	70.4	39.9	33.2	19.2	1.9	19.7	82.7	72.8	53.9	92.8	23.3	33.6	8.3	6.2	42.6	42.9
LaCLIP	85.0	74.8	40.7	40.3	21.3	2.2	23.9	78.2	76.4	59.0	91.4	27.1	41.3	5.6	7.6	45.0	46.2
LAION-400M	CLIP	85.5	93.0	71.7	66.8	83.5	16.7	52.8	90.1	91.2	63.9	97.3	42.4	63.3	46.2	17.8	65.5	67.0
LaCLIP	86.5	93.5	73.9	67.9	87.1	24.2	58.9	90.9	92.4	73.1	98.4	48.3	65.8	46.1	19.6	68.4	69.3
Datasets. Our experiments were conducted on four different image-text datasets at different scale: Conceptual Captions 3M (CC3M) [51], Conceptual Captions 12M (CC12M) [7], RedCaps [15], and LAION-400M[49]. RedCaps is a 12M-instance dataset collected exclusively from Reddit, potentially exhibiting distinct distributions compared to other datasets. The majority of our ablation studies were performed on the CC12M dataset. We evaluate all the models on ImageNet and 15 common downstream datasets like Food101 [4], SUN397 [59] and FGVCAircraft [38]. Appendix A contains the details for all datasets.