SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Original: Handmade mirror in a wooden toned frame texture cracked paint with reflection green apple on the table. Handmade mirror in a wooden toned frame texture cracked stock photos
EDA Aug 1: handmade mirror in a wooden toned frame texture cracked on with reflection green apple paint the table handmade mirror in a wooden toned frame texture cracked stock photos
EDA Aug 2: handmade mirror in a wooden toned frame texture crock up cracked paint with reflection green apple angstrom unit on the table handmade mirror in a wooden toned frame texture cracked stock photos
Back Translation Spanish: Hand-made mirror in the toned wood frame cracked texture paint with green apple reflection on the table. Hand-made mirror in the toned wood frame cracked texture stock photos
Back Translation French: Hand-made mirror in a wood toned frame texture cracked paint with green apple reflection on the table. Hand-made mirror in a wood toned frame texture cracked stock photos
Back Translation German: Handmade mirror in a wooden toned frame texture cracked color with reflection green apple on the table. Handmade mirror in a wooden toned frame texture cracked stock photos
LLM ChatGPT: Mirror on wooden painted frame, over cracked stone, with green apple on the table. Handmade mirror in a wooden toned frame texture cracked stock photo
LLM Bard: Green apple, still life, studio photography, abstract background
LLM MSCOCO: a mirror on a table
LLM Human: A closeup on a cracked mirror that reflects an apple on the table. The frame is covered in a golden toned wood grain texture.
- Original: Traffic jam on the road, a lot of cars which go towards each other and to the different directions
EDA Aug 1: traffic jam on the road a lot of cars which go towards each other and to the different focus
EDA Aug 2: traffic jam on the road a lot of cars which go towards each other and dissimilar to the different directions
Back Translation Spanish: Traffic jam on the road, a lot of cars going towards the other and to the different directions
Back Translation French: Traffic jams on the road, lots of cars going towards each other and towards different directions
Back Translation German: Traffic jam on the road, a lot of cars that go to each other and to the different directions
LLM ChatGPT: Traffic jam on the road, with a lot of cars moving in different directions, as well as towards each other.
LLM Bard: A lot of cars line the street, all heading toward the same intersection.
LLM MSCOCO: Traffic jam on the road with cars going in different directions, some cars are on the same lane, others are on different lanes.
LLM Human: A traffic jam on the road. There are a lot of cars and many of them are going in different directions.
- Original: Green young small newborn tree growing on concrete road among dry fallen leaves given concept of the life beginning or strong surviving
EDA Aug 1: green young small newborn tree growing on concrete road among dry fallen leaves given concept of the life beginning or strong surviving
EDA Aug 2: green young small new sprung tree growing on concrete road among dry fallen leaves given conception of the life beginning or strong surviving
Back Translation Spanish: Young green small newborn tree growing on the concrete road between fallen dry leaves given the concept of the beginning of life or strong survival
Back Translation French: Young young-born green tree growing on the concrete road among the dried leaves fallen according to the concept of the beginning of life or strong survivor
Back Translation German: Green young little newborn tree growing on concrete road under dry fallen leaves given concept of life beginning or strong survival
LLM ChatGPT: Bold forest green, from endless tangle of vines, create a strong vertical accent on the concrete road. On the right side, two brown fallen leaves, littering the road, sit quietly and gracefully, almost as though they are praying.
LLM Bard: Green young small newborn tree growing on concrete road among dry fallen leaves giving concept of the life beginning or strong surviving
LLM MSCOCO: A bare tree trunk surrounded by fallen leaves.
LLM Human: a young and tall green tree on a concrete road among dry fallen leaves, its leaves on the floor. Concept is a strong surviving in harsh environment
- Original: Caused by the european solid wood floor lamp lamp creative decorative floor lamp floor lamp minimalist home japanese wood living room bedroom lights
EDA Aug 1: room by the european solid wood floor lamp lamp creative decorative floor lamp floor lamp minimalist home japanese wood bedroom caused living lights
EDA Aug 2: caused house by the european solid wood floor lamp lamp creative decorative floor lamp floor lamp minimalist home japanese wood living have room bedroom lights
Back Translation Spanish: Caused by European solid wood floor lamp European decorative floor lamp creative minimalist floor lamp Japanese home wood living room bedroom lights
Back Translation French: Cause of the European floor lamp in solid wood decorative floor lamp creative floor lamp minimalist floor lamp Japanese house living room light room bedroom
Back Translation German: Conditioned by the European solid wood floor lamp lamp creative decorative floor lamp floor lamp minimalist house Japanese wood living room bedroom lights
LLM ChatGPT: Due to the european style lighting system, the decorative floor lamp will enhance your living room decor.
LLM Bard: Because of the european solid wood floor lamp, creative floor lamp, decorative floor lamp, floor lamp, minimalist home, japanese wood living room, bedroom lights.
LLM MSCOCO: A furniture store with lamps on the shelves. One lamp is lit.
LLM Human: The european solid wood floor lamp creative decorative floor lamp floor lamp minimalist home japanese wood living room bedroom lights
- Original: A man, likely a railroad employee, stands in the forefront of the station by the tracks. In the background, groups of people lean against the building, perhaps waiting for a train. In the foreground is the switch tower.
EDA Aug 1: a man likely a railroad employee stands in the forefront of the station by the tracks in the background groups of inch people lean against the building perhaps waiting for a hulk train in the foreground is the transposition switch tower
EDA Aug 2: a military personnel likely a railroad employee stands in the forefront of the station by the tracks in the background groups of people lean against the building perhaps ready and waiting for a train in the foreground is the throw tower
Back Translation Spanish: A man, probably a railroad employee, is at the forefront of the station by the tracks. Deep down, groups of people lean on the building, perhaps waiting for a train. In the foreground is the switch tower.
Back Translation French: A man, probably a railway employee, stands at the vanguard of the station by the tracks. In the background, groups of people lean against the building, perhaps waiting for a train.
Back Translation German: A man, probably a railway worker, is standing at the top of the station
Table A7: Performance comparison of LaCLIP trained with different text augmentation strategies on CC12M. (a) Zero-shot and Linear-probing Experiment Results
Augmentation	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Zero-shot
N/A (CLIP)	50.8	64.9	38.5	44.7	24.1	2.4	19.4	64.1	77.4	33.2	91.0	20.1	38.9	7.3	5.1	38.8	40.2
EDA [58]	51.9	67.6	36.5	48.2	27.7	2.8	25.4	64.7	78.2	33.3	92.8	21.9	40.0	10.8	6.6	40.6	41.2
Back Translation [50]	49.3	71.0	36.7	47.9	27.8	3.7	25.7	63.9	77.4	32.0	90.6	22.0	41.3	10.7	6.1	40.4	41.6
LLM (Ours)	60.7	75.1	43.9	57.0	36.3	5.6	31.0	72.4	83.3	39.9	95.1	27.3	44.3	12.7	8.9	46.2	48.4
Linear-Probing
N/A (CLIP)	81.6	93.8	79.3	72.0	75.1	52.6	75.6	86.2	92.2	95.3	97.3	96.7	93.1	80.6	19.7	79.4	70.3
EDA [58]	81.6	94.0	78.2	72.9	76.2	53.7	74.8	85.6	92.2	95.5	97.2	96.8	92.9	79.9	20.1	79.4	70.5
Back Translation [50]	81.8	94.2	78.2	73.0	77.5	54.6	75.5	87.1	91.6	96.0	97.5	97.1	93.1	80.0	20.0	79.8	70.7
LLM (Ours)	82.9	94.7	79.7	73.8	79.9	54.5	75.7	87.7	93.0	96.4	98.0	96.4	93.0	81.9	19.7	80.5	72.3
(b) Few-shot Experiment Results
Augmentation	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211
N/A (CLIP)	87.0±0.5	77.5±0.6	82.1±0.7	97.2±0.2	90.9±0.5	62.0±1.0	83.3±0.6	91.1±0.5	98.2±0.2	97.6±0.2	92.6±0.4	83.4±0.5	91.2±0.4	70.6±0.8	44.3±0.7
EDA [58]	88.1±0.5	76.1±0.6	81.3±0.7	97.6±0.2	91.7±0.5	62.9±1.0	83.4±0.6	91.9±0.5	98.4±0.2	97.8±0.2	93.5±0.3	84.3±0.5	91.6±0.4	68.4±0.8	44.6±0.7
Back Trans [50]	88.1±0.5	76.9±0.6	82.5±0.7	97.5±0.2	91.8±0.4	65.1±1.0	83.7±0.6	92.5±0.4	98.3±0.2	97.9±0.2	94.2±0.3	83.3±0.5	91.1±0.4	70.8±0.8	45.1±0.7
LLM (Ours)	89.9±0.5	81.3±0.5	85.0±0.6	98.0±0.2	95.3±0.3	68.1±1.0	84.9±0.6	93.4±0.4	98.9±0.2	98.4±0.2	83.0±0.5	92.4±0.4	76.4±0.8	46.7±0.7
on the tracks. In the background, groups of people are leaning against the building, perhaps waiting for a train.
LLM ChatGPT: A man, likely a railroad employee, stands by the tracks in front of the station. In the background, groups of people lean against the building. In the foreground is the switch tower.
LLM Bard: man leaning against the rail station and switch tower in a dark night with a fog
LLM MSCOCO: A portrait of the man in the front of the station is seen in the upper left. In the lower right is a man leaning on a post with his arms crossed.
LLM Human: An image of a man, likely a railroad employee, standing in the foreground of a train station by the tracks. In the background are groups of people, some leaning against the building, which could be waiting for a train. In the foreground are the tracks with a switch tower in the distance.
D.2 Detailed Experiment Results on Augmentation Strategy
We conducted a quantitative comparison of different augmentation strategies while ensuring a fair evaluation by generating a consistent number of augmented texts per original sentence (i.e., 4).
For the EDA strategy, we created 4 distinct versions of each sentence by randomly applying their predefined augmentation operations. As for the back translation approach, we translated the original texts into four different languages (Spanish, French, German, and Italic languages) and then back to English, resulting in 4 rewritten versions of the original texts. In our LLM-based augmentation, we used LLaMA ICL to generate 4 augmentations prompted by the 4 predefined meta-input-output pairs (ChatGPT, Bard, Human, and MSCOCO).
A comprehensive comparison of these strategies is presented in Table A7. The results demonstrate that while the baseline augmentation strategies improve the performance of the vanilla CLIP baseline, our proposed LLM-based augmentation strategy consistently achieves superior results across various datasets and evaluation metrics, outperforming the other augmentation methods significantly.
Table A8: Performance comparison of CLIP and LaCLIP trained with different text augmentation strategies with different number of augmentations per original text on CC12M.
(a) Zero-shot and Linear-probing Experiment Results
Augment / Num	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Zero-shot
N/A (CLIP) / 0	50.8	64.9	38.5	44.7	24.1	2.4	19.4	64.1	77.4	33.2	91.0	20.1	38.9	7.3	5.1	38.8	40.2
EDA / 1	52.2	66.2	34.3	46.6	25.6	3.6	22.2	64.5	79.3	33.5	90.9	24.1	37.6	13.4	5.6	40.0	40.2
EDA / 2	49.8	62.4	32.1	47.1	28.1	2.2	25.3	64.6	79.1	31.4	92.3	12.6	38.0	13.1	5.7	38.9	41.1
EDA / 3	50.4	62.8	35.4	49.7	26.8	2.5	24.5	69.5	77.4	33.1	92.8	24.9	37.3	15.1	6.7	40.6	41.7
EDA / 4	51.9	67.6	36.5	48.2	27.7	2.8	25.4	64.7	78.2	33.3	92.8	21.9	40.0	10.8	6.6	40.6	41.2
Back Trans / 1	49.7	61.5	34.6	45.5	26.7	4.0	20.7	59.2	77.2	32.1	88.2	27.1	40.0	12.6	5.8	39.0	40.1
Back Trans / 2	50.0	55.4	35.5	44.3	29.0	5.2	21.0	67.4	78.5	32.6	89.4	19.6	38.4	7.6	6.2	38.7	41.0
Back Trans / 3	49.9	67.3	37.6	46.9	26.7	4.1	22.8	65.7	76.8	34.3	91.7	20.0	34.3	12.5	6.3	39.8	41.5
Back Trans / 4	49.3	71.0	36.7	47.9	27.8	3.7	25.7	63.9	77.4	32.0	90.6	22.0	41.3	10.7	6.1	40.4	41.6
LLM (Ours) / 1	57.0	71.1	38.9	51.2	31.6	3.9	25.5	63.0	80.8	36.9	92.9	24.5	39.6	10.1	6.9	42.3	44.5
LLM (Ours) / 2	57.0	70.3	41.3	54.2	34.2	5.8	29.0	64.0	79.5	38.5	94.4	33.0	38.6	9.1	8.2	43.8	46.5
LLM (Ours) / 3	59.7	75.0	42.6	56.5	34.0	5.1	29.4	65.8	81.3	38.2	94.7	18.7	42.4	13.4	8.7	44.4	47.7
LLM (Ours) / 4	60.7	75.1	43.9	57.0	36.3	5.6	31.0	72.4	83.3	39.9	95.1	27.3	44.3	12.7	8.9	46.2	48.4
Linear-Probing
N/A (CLIP) / 0	81.6	93.8	79.3	72.0	75.1	52.6	75.6	86.2	92.2	95.3	97.3	96.7	93.1	80.6	19.7	79.4	70.3
EDA / 1	81.5	93.3	78.0	72.1	75.6	53.1	76.5	85.9	91.5	95.8	97.3	96.4	92.6	80.0	19.9	79.3	70.4
EDA / 2	81.4	94.1	80.2	72.5	76.7	52.9	75.7	85.8	92.1	95.7	97.2	96.7	92.7	81.6	19.9	79.7	70.6
EDA / 3	81.3	93.6	78.8	72.3	74.5	53.3	75.1	86.0	91.1	95.6	97.3	96.7	93.0	79.1	19.7	79.2	70.6
EDA / 4	81.6	94.0	78.2	72.9	76.2	53.7	74.8	85.6	92.2	95.5	97.2	96.8	92.9	79.9	20.1	79.4	70.5
Back Trans / 1	81.5	93.4	78.3	72.4	76.9	52.5	74.8	85.7	92.0	95.5	97.4	96.9	93.2	81.6	19.8	79.5	70.5
Back Trans / 2	81.5	93.9	78.5	72.4	76.3	52.8	74.5	86.2	91.7	95.5	97.5	96.8	92.4	80.5	19.4	79.3	70.5
Back Trans / 3	81.6	93.5	78.0	72.4	75.9	52.1	73.8	86.2	92.1	95.1	97.3	96.5	92.3	79.4	19.9	79.1	70.5
Back Trans / 4	81.8	94.2	78.2	73.0	77.5	54.6	75.5	87.1	91.6	96.0	97.5	97.1	93.1	80.0	20.0	79.8	70.7
LLM (Ours) / 1	81.8	94.3	79.7	73.3	77.5	55.0	75.4	87.4	92.5	96.3	97.6	96.9	92.6	81.3	20.2	80.1	71.2
LLM (Ours) / 2	82.3	94.0	79.1	73.3	77.6	52.7	76.0	86.8	91.8	96.1	97.7	96.6	93.1	83.3	20.1	80.0	71.7
LLM (Ours) / 3	82.3	94.7	80.0	73.7	79.2	56.0	75.7	87.0	92.9	96.2	98.0	96.6	92.9	83.1	20.0	80.6	71.9
LLM (Ours) / 4	82.9	94.7	79.7	73.8	79.9	54.5	75.7	87.7	93.0	96.4	98.0	96.4	93.0	81.9	19.7	80.5	72.3
(b) Few-shot Experiment Results
Augment / Num	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211
N/A (CLIP) / 0	87.0±0.5	77.5±0.6	82.1±0.7	97.2±0.2	90.9±0.5	62.0±1.0	83.3±0.6	91.1±0.5	98.2±0.2	97.6±0.2	92.6±0.4	83.4±0.5	91.2±0.4	70.6±0.8	44.3±0.7
EDA / 1	87.6±0.5	75.4±0.6	81.3±0.7	97.4±0.2	91.3±0.5	62.6±1.0	83.5±0.6	91.5±0.5	98.2±0.2	97.8±0.2	93.0±0.3	83.2±0.5	91.4±0.4	68.9±0.8	44.4±0.7
EDA / 2	87.9±0.5	77.3±0.6	82.0±0.6	97.4±0.2	91.9±0.4	62.8±1.0	83.5±0.6	92.1±0.4	98.4±0.2	97.8±0.2	93.6±0.3	82.8±0.6	91.6±0.4	70.0±0.8	44.8±0.7
EDA / 3	87.5±0.5	76.5±0.6	82.0±0.7	97.6±0.2	91.2±0.5	62.7±1.0	83.8±0.6	91.3±0.5	98.2±0.2	97.7±0.2	94.2±0.3	84.0±0.5	91.4±0.4	72.0±0.8	44.3±0.7
EDA / 4	88.1±0.5	76.1±0.6	81.3±0.7	97.6±0.2	91.7±0.5	62.9±1.0	83.4±0.6	91.9±0.5	98.4±0.2	97.8±0.2	93.5±0.3	84.3±0.5	91.6±0.4	68.4±0.8	44.6±0.7
Back Trans / 1	87.8±0.5	76.4±0.6	81.8±0.7	97.4±0.2	91.7±0.5	63.4±1.0	83.8±0.6	91.7±0.5	98.3±0.2	97.7±0.2	93.1±0.3	83.9±0.5	91.6±0.4	70.1±0.8	44.7±0.8
Back Trans / 2	87.8±0.5	75.6±0.6	81.6±0.7	97.5±0.2	92.3±0.4	62.8±1.0	83.7±0.6	92.5±0.4	98.3±0.2	97.8±0.2	93.6±0.3	83.8±0.5	91.1±0.4	68.8±0.8	44.7±0.7
Back Trans / 3	88.2±0.5	77.0±0.6	82.8±0.6	97.4±0.2	91.7±0.4	62.6±1.0	83.8±0.6	91.6±0.5	98.3±0.2	97.7±0.2	93.3±0.3	83.1±0.5	91.8±0.4	71.0±0.8	45.0±0.7
Back Trans / 4	88.1±0.5	76.9±0.6	82.5±0.7	97.5±0.2	91.8±0.4	65.1±1.0	83.7±0.6	92.5±0.4	98.3±0.2	97.9±0.2	94.2±0.3	83.3±0.5	91.1±0.4	70.8±0.8	45.1±0.7
LLM (Ours) / 1	88.8±0.5	78.4±0.6	83.3±0.6	97.7±0.2	93.4±0.4	66.5±1.0	84.4±0.6	92.5±0.4	98.6±0.2	98.0±0.2	94.3±0.3	84.0±0.5	92.3±0.4	73.7±0.8	45.6±0.7
LLM (Ours) / 2	89.2±0.5	79.1±0.6	83.6±0.6	97.9±0.2	94.2±0.4	65.6±1.0	84.2±0.6	93.2±0.4	98.8±0.2	98.2±0.2	93.5±0.3	83.6±0.5	91.7±0.4	75.6±0.8	46.1±0.7
LLM (Ours) / 3	89.8±0.5	82.5±0.5	84.2±0.6	98.0±0.2	94.4±0.4	68.5±1.0	85.0±0.6	93.4±0.4	98.7±0.2	98.4±0.2	95.9±0.2	83.9±0.5	91.6±0.4	75.1±0.8	46.9±0.7
LLM (Ours) / 4	89.9±0.5	81.3±0.5	85.0±0.6	98.0±0.2	95.3±0.3	68.1±1.0	84.9±0.6	93.4±0.4	98.9±0.2	98.4±0.2	95.9±0.2	83.0±0.5	92.4±0.4	76.4±0.8	46.7±0.7
Vanilla CLIP
(a) CIFAR-10.
EDA
Back Translation
LaCLIP
(b) The first 10 classes on Food101.
EDA
(c) STL-10.
EDA
Vanilla CLIP
Figure A1: t-SNE visualization of image features learned from Vanilla CLIP, two baseline text augmentations strategies (EDA and back translation), and our proposed LaCLIP on CIFAR-10, Food101, STL-10, and EuroSAT datasets. Image features learned from our proposed LaCLIP have a clearer class boundaries and cluster centroids.
(d) EuroSAT.
EDA
E. Number of Augmentations per Original Text
We conducted experiments to investigate how the performance varies with the number of augmentations generated for each text and the differences between augmentation strategies as the number of augmentations per original text increases. We examined the performance of each strategy with 0 to 4 augmentations per original text, where 0 corresponds to vanilla CLIP without any text augmentation. Specifically, for each specific number of augmentations  $k$ : For EDA, we selected  $k$  versions out of the 4 generated versions. In the case of back translation, we used Spanish, Spanish+French, Spanish+French+German, and Spanish+French+German+Italic languages for  $k = 1,2,3,4$ , respectively. Regarding our LLM-based augmentation, we used ChatGPT, ChatGPT+Bard, ChatGPT+Bard+MSCOCO, and ChatGPT+Bard+MSCOCO+Human as augmentations corresponding to  $k = 1,2,3,4$ , respectively.
The detailed comparison can be found in Table A8. From the results, we observe that the performance of the baseline augmentation strategies does not scale well with the number of augmentations per sentence, indicating limited diversity in the rewritten texts. This aligns with the findings in [58], where the best results are obtained with four different augmentations. In contrast, LaCLIP trained
Table A9: Performance comparison of CLIP, LaCLIP and LaCLIP-MT trained on CC12M and RedCaps.
(a) Zero-shot and Linear-probing Experiment Results
Data	Model	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Zero-shot
CC12M	CLIP	50.8	64.9	38.5	44.7	24.1	2.4	19.4	64.1	77.4	33.2	91.0	20.1	38.9	7.3	5.1	38.8	40.2
LaCLIP	60.7	75.1	43.9	57.0	36.3	5.6	31.0	72.4	83.3	39.9	95.1	27.3	44.3	12.7	8.9	46.2	48.4
LaCLIP-MT	59.2	69.5	39.0	56.8	34.4	5.5	30.7	72.8	83.1	42.5	95.2	24.8	43.4	13.1	8.3	45.2	49.0
RedCaps	CLIP	81.5	70.4	39.9	33.2	19.2	1.9	19.7	82.7	72.8	53.9	92.8	23.3	33.6	8.3	6.2	42.6	42.9
LaCLIP	85.0	74.8	40.7	40.3	21.3	2.2	23.9	78.2	76.4	59.0	91.4	27.1	41.3	5.6	7.6	45.0	46.2
LaCLIP-MT	84.2	74.9	43.1	40.5	23.0	1.9	24.0	84.7	77.1	60.9	91.0	31.9	40.3	6.1	7.9	46.1	48.1
Linear-Probing
CC12M	CLIP	81.6	93.8	79.3	72.0	75.1	52.6	75.6	86.2	92.2	95.3	97.3	96.7	93.1	80.6	19.7	79.4	70.3
LaCLIP	82.9	94.7	79.7	73.8	79.9	54.5	75.7	87.7	93.0	96.4	98.0	96.4	93.0	81.9	19.7	80.5	72.3
LaCLIP-MT	82.9	94.5	79.7	73.7	79.4	55.0	76.0	87.9	93.0	96.4	97.6	96.2	93.1	82.7	20.2	80.6	72.4
RedCaps	CLIP	89.1	94.1	78.8	65.6	74.0	52.5	73.2	91.5	91.4	97.7	98.0	96.3	93.5	80.8	17.0	79.6	71.8
LaCLIP	90.1	94.3	78.5	66.6	77.6	53.6	73.9	90.8	91.5	97.9	97.6	96.6	92.7	80.8	17.2	80.0	71.9
LaCLIP-MT	90.2	94.0	79.0	67.3	79.2	53.2	75.3	91.7	91.0	98.3	98.1	96.9	93.0	80.6	17.2	80.3	72.4
(b) Few-shot Experiment Results
Model	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211
Pre-trained on CC12M
CLIP	87.0±0.5	77.5±0.6	82.1±0.7	97.2±0.2	90.9±0.5	62.0±1.0	83.3±0.6	91.1±0.5	98.2±0.2	97.6±0.2	92.6±0.4	83.4±0.5	91.2±0.4	70.6±0.8	44.3±0.7
LaCLIP	89.9±0.5	81.3±0.5	85.0±0.6	98.0±0.2	95.3±0.3	68.1±1.0	84.9±0.6	93.4±0.4	98.9±0.2	98.4±0.2	95.9±0.2	83.0±0.5	92.4±0.4	76.4±0.8	46.7±0.7
LaCLIP-MT	89.5±0.5	80.1±0.5	84.4±0.6	98.0±0.2	94.8±0.4	69.6±1.0	84.6±0.6	93.7±0.4	98.8±0.2	98.4±0.2	96.0±0.2	83.8±0.5	92.0±0.4	76.8±0.7	46.4±0.7
Pre-trained on RedCaps
CLIP	94.4±0.3	80.6±0.5	85.3±0.6	95.9±0.3	88.5±0.6	54.5±0.9	82.6±0.6	94.5±0.4	97.8±0.2	99.0±0.1	94.8±0.3	84.9±0.5	91.3±0.4	75.3±0.8	40.6±0.7
LaCLIP	95.8±0.3	81.4±0.5	85.4±0.6	96.2±0.3	90.9±0.5	58.8±1.0	82.4±0.6	94.1±0.4	98.0±0.2	99.2±0.1	95.6±0.2	86.2±0.5	92.1±0.4	76.5±0.8	42.6±0.7
LaCLIP-MT	95.9±0.3	81.8±0.5	86.0±0.6	96.5±0.3	91.4±0.5	58.1±1.0	82.7±0.6	94.8±0.4	98.2±0.2	99.3±0.1	95.4±0.2	87.5±0.4	92.2±0.4	76.5±0.8	42.5±0.7
with our LLM-based augmentation demonstrates good scalability with the number of augmentations. This can be attributed to the rich and diverse nature of LLaMA ICL in the rewriting process, allowing for continued performance improvement with more augmentations.
F. t-SNE Visualizations
To gain a deeper understanding of the distinctions between the features learned from LaCLIP and vanilla CLIP, as well as the impact of different augmentation strategies used in LaCLIP training, we visualize the vision encoder features on different downstream datasets using t-SNE [57] in Figure A1. We generate feature visualizations for CIFAR-10, Food101, STL-10, and EuroSAT datasets, as they provide sufficient samples per class for meaningful visualizations. Other datasets have a limited number of samples per class in the test set, making it difficult to generate reliable visualizations. For Food101 we visualize the features from the first 10 classes.
The visualization reveals that LaCLIP trained with our proposed LLM-based rewriting strategy exhibits clearer class boundaries and more distinct clusters compared to other approaches. This observation suggests that language augmentations not only enhance the performance of text encoders, but also improve the ability of vision encoders to learn a more effective image embedding space that is well-suited for downstream tasks.
G. Detailed Experiment Results for LaCLIP-MT
In Table A9, we present a detailed performance comparison among CLIP, LaCLIP, and the Multi-Text version LaCLIP-MT, as introduced in Section 5.
Table A10: Performance comparison of CLIP and LaCLIP trained with different backbone architectures, ViT-S/16, ViT-B/16 and ViT-L/16, on CC12M.
(a) Zero-shot and Linear-probing Experiment Results
Backbone	Model	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Zero-shot
ViT-S/16	CLIP	44.0	54.7	32.6	41.9	20.2	2.5	20.1	56.9	74.0	29.3	88.0	29.1	36.0	11.2	4.4	36.3	36.9
LaCLIP	57.6	70.6	37.1	55.6	29.1	6.6	29.7	71.2	81.1	39.5	93.6	26.7	40.0	14.7	8.4	44.1	46.3
ViT-B/16	CLIP	50.8	64.9	38.5	44.7	24.1	2.4	19.4	64.1	77.4	33.2	91.0	20.1	38.9	7.3	5.1	38.8	40.2
LaCLIP	60.7	75.1	43.9	57.0	36.3	5.6	31.0	72.4	83.3	39.9	95.1	27.3	44.3	12.7	8.9	46.2	48.4
ViT-L/16	CLIP	54.1	76.0	44.3	49.7	31.2	3.4	20.9	65.8	79.9	34.7	92.6	30.6	41.1	9.0	6.1	42.6	44.0
LaCLIP	60.5	80.4	47.3	58.1	38.8	5.7	31.0	71.5	82.0	39.6	95.8	18.6	46.8	13.0	9.2	46.6	49.1
Linear-Probing
ViT-S/16	CLIP	78.9	91.7	75.3	70.5	69.1	46.5	74.4	84.3	90.8	94.8	96.3	95.9	91.7	76.5	17.9	77.0	67.1
LaCLIP	80.3	93.0	76.6	71.8	73.0	49.0	74.3	85.3	91.8	95.1	97.0	95.4	90.7	78.4	18.2	78.0	69.1
ViT-B/16	CLIP	81.6	93.8	79.3	72.0	75.1	52.6	75.6	86.2	92.2	95.3	97.3	96.7	93.1	80.6	19.7	79.4	70.3
LaCLIP	82.9	94.7	79.7	73.8	79.9	54.5	75.7	87.7	93.0	96.4	98.0	96.4	93.0	81.9	19.7	80.5	72.3
ViT-L/16	CLIP	83.5	95.3	81.4	73.4	80.1	57.8	76.8	88.4	93.3	96.5	97.9	97.0	94.0	82.9	20.8	81.3	72.9
LaCLIP	83.8	95.8	82.8	74.4	81.4	58.1	77.2	88.6	93.9	97.2	98.2	97.0	93.7	85.2	20.5	81.9	73.7
(b) Few-shot Experiment Results
Backbone	Model	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211
ViT-S/16	CLIP	85.4±0.6	75.1±0.6	81.4±0.7	97.1±0.3	89.2±0.5	58.5±1.0	83.2±0.6	91.0±0.5	97.6±0.3	97.5±0.2	91.9±0.4	82.9±0.5	90.7±0.5	67.9±0.8	43.5±0.7
LaCLIP	88.3±0.5	79.4±0.6	81.7±0.7	97.7±0.2	94.0±0.4	65.2±1.0	84.5±0.6	92.4±0.5	98.4±0.2	98.0±0.2	95.5±0.3	81.7±0.5	91.3±0.4	72.2±0.8	46.5±0.7
ViT-B/16	CLIP	87.0±0.5	77.5±0.6	82.1±0.7	97.2±0.2	90.9±0.5	62.0±1.0	83.3±0.6	91.1±0.5	98.2±0.2	97.6±0.2	92.6±0.4	83.4±0.5	91.2±0.4	70.6±0.8	44.3±0.7
LaCLIP	89.9±0.5	81.3±0.5	85.0±0.6	98.0±0.2	95.3±0.3	68.1±1.0	84.9±0.6	93.4±0.4	98.9±0.2	98.4±0.2	95.9±0.2	83.0±0.5	92.4±0.4	76.4±0.8	46.7±0.7
ViT-L/16	CLIP	89.1±0.5	81.1±0.5	84.8±0.6	97.8±0.2	93.0±0.5	66.4±1.0	84.3±0.6	93.2±0.4	98.7±0.2	98.2±0.2	93.4±0.3	84.6±0.5	92.2±0.4	74.1±0.8	45.2±0.7
LaCLIP	90.3±0.4	84.5±0.5	86.4±0.6	98.0±0.2	95.6±0.3	70.5±1.0	84.6±0.6	94.6±0.4	99.1±0.1	98.8±0.2	96.0±0.2	85.0±0.5	92.8±0.4	78.9±0.8	47.2±0.7