SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The pre-training was performed on CC12M and RedCaps datasets. The results highlight the potential of the multi-text version of the CLIP loss to enhance the performance of LaCLIP even further. By pairing each image with all corresponding texts, the vision encoder receives more diverse supervision during training iterations. he improvements are particularly significant for the RedCaps dataset, where LaCLIP-MT achieves an additional  $1.9\%$  increase in zero-shot classification accuracy on ImageNet.
H. Detailed Experiment Results for Different Backbone
In Table A10, we present the detailed experiment results on CC12M using different backbone architectures, including ViT-S/16, ViT-B/16, and ViT-L/16 encoders. The results consistently demonstrate that our proposed LaCLIP outperforms the vanilla CLIP baseline across all backbone architectures. This highlights the scalability of LaCLIP, as it consistently improves performance on various downstream tasks while leveraging encoders of different sizes.
I. Ablation on LLaMA model
We performed two ablation studies on the LLaMA model to assess the impact of modifying key components on the performance of LaCLIP. The studies focused on two factors: model size and temperature. By systematically investigating these factors, we aimed to shed light on their influence and provide valuable insights into the effectiveness and adaptability of the LLM-based augmentation approach. All experiments were conducted on LaCLIP using a single text augmentation strategy
Table A11: Ablation study on LaCLIP trained with text rewrites generated with different LLaMA model size on CC12M.
(a) Zero-shot and Linear-probing Experiment Results
Model Size	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Zero-shot
N/A (CLIP)	50.8	64.9	38.5	44.7	24.1	2.4	19.4	64.1	77.4	33.2	91.0	20.1	38.9	7.3	5.1	38.8	40.2
7B	57.0	71.1	38.9	51.2	31.6	3.9	25.5	63.0	80.8	36.9	92.9	24.5	39.6	10.1	6.9	42.3	44.5
13B	55.4	71.5	39.3	51.3	29.6	4.0	26.4	65.7	80.7	36.0	93.8	17.0	38.7	9.0	7.6	41.7	44.8
33B	56.7	76.0	37.7	52.0	31.2	4.5	24.3	60.7	80.9	35.4	94.4	26.7	40.4	11.6	7.0	42.6	44.4
65B	57.5	69.2	38.9	51.6	31.1	4.1	25.3	65.2	79.0	36.8	93.1	31.7	40.2	15.0	7.4	43.1	44.4
Linear-Probing
N/A (CLIP)	81.6	93.8	79.3	72.0	75.1	52.6	75.6	86.2	92.2	95.3	97.3	96.7	93.1	80.6	19.7	79.4	70.3
7B	81.8	94.3	79.7	73.3	77.5	55.0	75.4	87.4	92.5	96.3	97.6	96.9	92.6	81.3	20.2	80.1	71.2
13B	82.1	93.7	78.2	73.0	77.6	55.6	74.6	87.4	92.7	96.0	97.4	96.3	93.2	82.5	20.0	80.0	71.2
33B	81.8	94.1	79.4	73.3	78.6	54.1	75.0	86.4	92.4	96.1	97.3	96.6	93.1	81.5	19.8	80.0	71.4
65B	82.2	94.2	79.3	73.0	78.7	54.0	75.4	87.3	91.9	95.4	97.5	96.7	92.7	82.5	20.0	80.1	71.3
(b) Few-shot Experiment Results
Model Size	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211
N/A (CLIP)	87.0±0.5	77.5±0.6	82.1±0.7	97.2±0.2	90.9±0.5	62.0±1.0	83.3±0.6	91.1±0.5	98.2±0.2	97.6±0.2	92.6±0.4	83.4±0.5	91.2±0.4	70.6±0.8	44.3±0.7
7B	88.8±0.5	78.4±0.6	83.3±0.6	97.7±0.2	93.4±0.4	66.5±1.0	84.4±0.6	92.5±0.4	98.6±0.2	98.0±0.2	94.3±0.3	84.0±0.5	92.3±0.4	73.7±0.8	45.6±0.7
13B	89.1±0.5	79.2±0.6	82.8±0.7	97.9±0.2	94.0±0.4	66.3±1.0	84.1±0.6	92.9±0.4	98.5±0.2	98.2±0.2	94.4±0.3	83.2±0.5	91.6±0.4	73.6±0.8	45.7±0.7
33B	88.6±0.5	80.3±0.6	83.6±0.6	97.8±0.2	94.3±0.4	65.4±1.0	84.7±0.6	92.8±0.4	98.6±0.2	98.2±0.2	94.5±0.3	84.2±0.5	92.1±0.4	72.0±0.8	45.8±0.7
65B	88.8±0.5	79.2±0.6	82.9±0.6	97.8±0.2	94.1±0.4	66.6±1.0	84.3±0.6	93.1±0.4	98.6±0.2	98.1±0.2	94.5±0.3	85.6±0.5	91.9±0.4	72.5±0.8	45.6±0.7
with the ChatGPT meta-input-output prompting pairs. The models were pre-trained on the CC12M dataset.
Model Size. Given that LLaMA offers multiple models with varying numbers of parameters, including 7B, 13B, 33B, and 65B, it is widely acknowledged that larger models tend to excel in NLP tasks involving reasoning and comprehension. Building upon this observation, we sought to explore the potential benefits of incorporating larger LLaMA models into our framework, with the aim of enhancing the performance of LaCLIP on downstream tasks.
To investigate whether the use of larger LLaMA models would yield improved results, we conducted a series of experiments where LaCLIP was trained using text augmented by LLaMA models of different sizes. We compared the performance of LaCLIP across these different configurations and summarized the results in Table A11.
Through our analysis, we have observed that even the smallest and relatively lightweight LLaMA model (7B) is sufficient to significantly boost the performance of LaCLIP on CLIP. Although larger LLaMA models showed some improvement on certain downstream datasets, the overall impact was relatively modest in our experimental setups focused on training vision-language models. It is worth mentioning that different model sizes may benefit from different temperature settings during the sampling process, and we leave this as a topic for future research. In the following sections, we specifically examine the effect of temperature on the 7B model.
Temperature. The temperature parameter plays a crucial role in the LLaMA token sampling process as it controls the balance between diversity and precision in the generated text. Higher values of temperature increase text diversity, but excessively high values can introduce random words or non-English tokens, negatively impacting the results.
Table A12: Ablation study on LaCLIP trained with text rewrites generated with different LLaMA temperature on CC12M.
(a) Zero-shot and Linear-probing Experiment Results
temperature	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Zero-shot
0.3	52.1	66.6	40.1	44.0	30.0	4.0	22.3	62.2	79.7	34.8	90.7	21.3	37.3	10.9	6.3	40.2	43.6
0.5	54.0	69.5	36.4	46.1	31.8	3.4	22.9	62.3	80.2	35.8	93.0	22.4	38.1	10.9	6.1	40.9	44.0
0.7	53.6	67.2	37.5	48.3	31.5	3.9	24.0	63.5	78.6	34.6	91.9	24.2	42.9	8.1	6.7	41.1	43.8
0.9	57.0	71.1	38.9	51.2	31.6	3.9	25.5	63.0	80.8	36.9	92.9	24.5	39.6	10.1	6.9	42.3	44.5
1.1	55.8	72.8	39.2	53.1	28.6	4.2	23.6	64.7	80.6	34.2	93.1	21.8	37.4	15.2	7.6	42.1	44.0
Linear-Probing
0.3	82.1	94.0	79.0	72.9	77.9	54.9	75.3	87.6	92.7	96.2	97.5	96.7	92.8	81.9	19.6	80.1	71.1
0.5	82.1	94.0	79.2	72.6	78.3	53.7	75.7	86.8	92.0	95.9	97.5	96.6	93.2	81.5	19.7	79.9	71.0
0.7	81.9	94.3	78.9	73.2	78.7	54.7	75.6	86.8	92.4	96.0	97.5	96.5	92.8	80.6	19.9	80.0	71.2
0.9	81.8	94.3	79.7	73.3	77.5	55.0	75.4	87.4	92.5	96.3	97.6	96.9	92.6	81.3	20.2	80.1	71.2
1.1	81.7	94.0	78.8	73.4	77.2	54.0	74.3	87.0	92.2	95.7	97.6	96.1	93.1	80.4	20.1	79.7	71.3
(b) Few-shot Experiment Results
temperature	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211
0.3	89.1±0.5	77.8±0.6	82.3±0.7	97.6±0.2	93.3±0.4	66.1±1.0	84.3±0.6	93.0±0.4	98.5±0.2	98.1±0.2	93.6±0.3	83.8±0.5	91.8±0.4	71.9±0.8	45.3±0.7
0.5	88.6±0.5	77.8±0.6	82.4±0.7	97.6±0.2	93.2±0.4	65.5±1.0	84.1±0.6	92.9±0.4	98.5±0.2	98.1±0.2	93.8±0.3	85.0±0.5	92.0±0.4	71.9±0.8	45.4±0.7
0.7	88.8±0.5	78.0±0.6	82.2±0.7	97.8±0.2	93.3±0.4	65.5±1.0	84.1±0.6	92.5±0.5	98.6±0.2	98.1±0.2	93.9±0.3	84.0±0.5	91.8±0.4	72.5±0.8	45.7±0.7
0.9	88.8±0.5	78.4±0.6	83.3±0.6	97.7±0.2	93.4±0.4	66.5±1.0	84.4±0.6	92.5±0.4	98.6±0.2	98.0±0.2	94.3±0.3	84.0±0.5	92.3±0.4	73.7±0.8	45.6±0.7
1.1	88.7±0.5	80.1±0.5	83.6±0.7	97.8±0.2	93.4±0.4	64.8±1.0	83.8±0.6	92.5±0.4	98.7±0.2	98.1±0.2	95.2±0.3	82.3±0.5	91.4±0.4	70.7±0.8	45.7±0.7
We conducted experiments with temperature values ranging from 0.3 to 1.1, and the detailed results of employing different temperatures for LLaMA generation are provided in Table A12. The results show that overall the performance is quite robust across temperatures. Generally as the temperature increases, the performance initially improves, reaching a peak around  $\tau = 0.9$ , and then begins to decline. Therefore,  $\tau = 0.9$  appears to be the optimal temperature for text rewriting in the context of text augmentation, and we consistently use this value in all of our experiments.
J. Ablation on Non-contrastive Training
Language rewrites techniques used in LaCLIP holds potential for broader applications. In order to evaluate the impact of the language augmentation strategy on noncontrastive vision-language pre-training methods, we integrated this strategy into Virtex's training pipeline [14], leading to the formation of Language-augmented Virtex (La-Virtex). We replicated the identical setup in their official implementation and trained the two models on the CC12M dataset. Table A13 shows the linear classification performance on the PASCAL VOC07 dataset of the two
pre-trained models. The result demonstrates that the incorporation of language rewrites in La-Virtex outperforms the standard Virtex. This indicates that language augmentation could potentially be more generic and beneficial to non-contrastive vision-language model training methods as well.
Table A13: Comparison of Virtex training with and without Language Augmentation on VOC07 classification.
Model	Language Aug	VOC07
Virtex	×	78.40
La-Virtex	✓	80.92
Figure A2: Training loss and validation accuracy for CLIP and LaCLIP trained on various datasets including CC3M, CC12M, RedCaps and LAION-400M. Top row is the training loss curve, bottom row is the validation accuracy, measured by zero-shot accuracy on ImageNet. Each column corresponds to a specific training dataset. For each figure, X-axis is the training epoch.
CLIP LaCLIP (Ours)
Figure A3: Visualization of image examples corrected by LaCLIP on the ImageNet validation set.
K. Training and Validation Curves
We show the training and validation curve on CC3M, CC12M, RedCaps and LAION-400M datasets in Figure A2. The results demonstrate LaCLIP consistently achieves higher validation accuracy and higher training loss across different datasets. This indicates language augmentation is improving the model's generalization ability rather than its optimization process. This is because Language augmentations used in LaCLIP makes the pre-training task more challenging and therefore improves the generalization ability of the model.
L. Visualization of Examples being Corrected
To provide a qualitative understanding of the categories most impacted by language augmentation, we present examples from the three categories with the most significant accuracy improvements by LaCLIP on the ImageNet validation set in Figure A3. This demonstrate LaCLIP has the ability to distinguish between some fine-grained categories where vanilla CLIP faces challenges.
M. Detailed Experiment Results for Pre-trained Text-encoder
In Table A14, we present the detailed experiment results on CC12M between LaCLIP and CLIP models trained with different text encoder setups. For the CLIP models where pre-trained text encoders are used, we replaced the text encoder and tokenizer with the pre-trained BERT-Base model while keeping all other parameters to be the same. The experiments with pre-trained BERT encoder are conducted in two distinct setups: (a) fine-tuning the entire model, and (b) freezing the weights of the text encoder. The results show using the pre-trained text encoder as initialization and fine-tune
Table A14: Ablation study between LaCLIP and CLIP models trained with different pre-trained Text Encoder (Text-Enc) setups on CC12M. For the models with a pre-trained text encoder, we use the pre-trained BERT-base model.
(a) Zero-shot and Linear-probing Experiment Results
Method	Text-EnC
Pre-trained	Text-EnC
Frozen	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Zero-shot
CLIP	X	X	50.8	64.9	38.5	44.7	24.1	2.4	19.4	64.1	77.4	33.2	91.0	20.1	38.9	7.3	5.1	38.8	40.2
CLIP	✓	X	55.3	66.8	38.1	50.3	33.1	4.2	25.9	64.8	78.6	34.5	92.5	27.9	39.2	13.5	7.0	42.1	42.9
CLIP	✓	✓	15.0	57.0	23.9	30.8	5.1	2.6	17.6	16.4	54.8	5.2	85.4	19.5	27.4	3.0	4.0	24.5	23.2
LaCLIP	X	X	60.7	75.1	43.9	57.0	36.3	5.6	31.0	72.4	83.3	39.9	95.1	27.3	44.3	12.7	8.9	46.2	48.4
Linear-Probing
CLIP	X	X	81.6	93.8	79.3	72.0	74.9	52.7	75.6	86.2	92.2	95.3	97.3	96.7	93.1	80.5	19.7	79.4	70.3
CLIP	✓	X	81.5	93.6	79.1	72.5	76.7	54.2	74.3	86.4	92.1	95.8	97.0	96.6	93.2	79.3	20.0	79.5	70.4
CLIP	✓	✓	74.9	91.9	74.4	68.9	60.0	48.0	71.0	79.6	87.5	90.7	96.2	95.6	91.1	75.7	18.1	74.9	66.0
LaCLIP	X	X	82.9	94.7	79.7	73.8	79.9	54.5	75.7	87.7	93.0	96.4	98.0	96.4	93.0	81.9	19.7	80.5	72.3
(b) Few-shot Experiment Results
Method	Text-Enc
Pre-trained	Text-Enc
Fusion	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average
CLIP	X	X	87.0	77.5	82.1	97.2	90.9	62.0	83.3	91.1	98.2	97.6	92.6	83.4	91.2	70.6	44.3	83.3
CLIP	✓	X	87.8	75.5	81.6	97.5	91.7	64.9	83.5	91.9	98.3	97.6	93.3	83.0	91.6	71.5	44.6	83.6
CLIP	✓	✓	80.2	75.0	78.6	96.8	82.9	60.7	80.4	85.7	96.7	93.9	94.5	79.5	87.9	67.1	44.8	80.3
LaCLIP	X	X	89.9	81.3	85.0	98.0	95.3	68.1	84.9	93.4	98.9	98.4	95.9	83.0	92.4	76.4	46.7	85.8
the entire model can bring some benefit to vanilla CLIP training, while freezing the text encoder weights will result in performance drop. In the meantime, LaCLIP shows superior performance and outperforms all pre-trained text encoder counterparts, demonstrating the effectiveness of language augmentation strategy.