SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zero-shot Classification We follow a similar prompt ensemble strategy as described in [44] and employ the same set of prompting templates. For each class name, we compute the average text embedding across all templates. These averaged embeddings are then used to calculate the similarity between each test image and the class embeddings. Specifically, for zero-shot evaluation on ImageNet, models trained on the LAION-400M dataset use the exact 80 prompts provided by [44] to ensure a
Table A1: Details of the downstream classification datasets.
Dataset	Metric	Categories	Train Size	Test Size
Food-101 [4]	Accuracy	101	75,750	25,250
CIFAR-10 [29]	Accuracy	10	50,000	10,000
CIFAR-100 [29]	Accuracy	100	50,000	10,000
SUN397 [59]	Accuracy	397	19,850	19,850
Stanford Cars [28]	Accuracy	196	8,144	8,041
FGVC Aircraft [38]	Mean per class	100	6,667	3,333
DTD [11]	Accuracy	47	3,760	1,880
Oxford Pets [43]	Mean per class	37	3,680	3,669
Caltech-101 [19]	Mean per class	102	3,060	6,085
Oxford Flowers [42]	Mean per class	102	2,040	6,149
STL-10 [12]	Accuracy	10	1,000	8,000
EuroSAT [23]	Accuracy	10	10,000	5,000
RESISC45 [9]	Accuracy	45	25,200	6,300
GTSRB [53]	Accuracy	43	26,640	12,630
Country211 [44, 54]	Accuracy	211	42,200	21,100
Table A2: Encoder details.
Model	Patch size	Input resolution	Embedding dimension	Vision Transformer	Text Transformer	Vocab size	Text length
Layers	Width	Heads	Layers	Width	Heads
ViT-S/16	16	224	512	12	384	12	12	512	8
ViT-B/16	16	224	512	12	768	12	12	512	8	49,408	77
ViT-B/32	32	224	512	12	768	12	12	512	8
ViT-L/16	16	224	512	24	1024	16	12	512	8
fair comparison. For models trained on other datasets, we use a subset of 7 templates recommended by [44] to expedite the evaluation process.
Few-shot Classification Following the settings in [18], we evaluate the 5-way 5-shot performance across 15 downstream datasets. We use Prototypical Networks [52] as classifier on top of the features extracted from vision encoders without data augmentation. Only Resize followed by CenterCrop is applied here for all images. We evaluate each model for 600 randomly sampled episodes, and for each episode, images are sampled from the combination of training, validation and testing sets. We always sample 15 images for each class as query set. The mean accuracy across all episodes are reported in the main paper, and we also report the  $95\%$  confidence interval in the appendix.
Linear-Probing For linear probing on ImageNet, we keep the image encoder frozen and train a Linear Classifier on the extracted features. The only augmentation applied is RandomHorizontalFlip. We sweep the base learning rate across the range of [0.002, 0.005, 0.01, 0.015, 0.02, 0.03, 0.05] and report the best performance achieved. The learning rate is scaled linearly based on the actual batch size, following the approach outlined in [20]. Details of all other hyperparameters can be found in Table A5. For linear probing on all other downstream datasets, we train a logistic regression layer on top of the frozen features extracted from the vision encoders, without applying any data augmentation. The model is optimized using L-BFGS with Scikit-learn, and the maximum number of iterations is set to 500. To determine the optimal  $\ell_2$  regularization term for each model and dataset, we perform a sweep across 45 steps that are logarithmically spaced ranging from  $10^{-6}$  to  $10^{5}$  on the validation set. For the final results, we fit the model on the combined training and validation sets and report the performance on the separate test set.
C. Meta-input-output Details
C.1 Meta-input-output Pairs
Here we provide the exact 16 meta-input-output pairs we used as templates for all four set ups: ChatGPT, Bard, Human and MSCOCO, described in Section 3.2. We use 'Source' to represent the meta-input text we sampled from the image-text datasets, and use 'Target' to represent the meta-output text generated by each of the strategies. Note the meta-input-output pairs showed in Figure 1 and Figure 2 in the main text are for illustration only, please refer to this section for the real pairs used in the experiments.
Table A3: Detailed pre-training hyper-parameters for CLIP training on all four image-text datasets.
(a) Pre-training hyper-parameter on CC3M.
Config	Value
Batch size	8,192
Optimizer	AdamW [37]
Learning rate	1 × 10-3
Weight decay	0.5
Adam β	β1,β2=(0.9,0.98)
Adam ε	1 × 10-8
Total epochs	40
Warm up epochs	1
Learning rate schedule	cosine decay
(b) Pre-training hyper-parameter on CC12M.
Config	Value
Batch size	8,192
Optimizer	AdamW [37]
Learning rate	1 × 10-3
Weight decay	0.5
Adam β	β1,β2=(0.9,0.98)
Adam ε	1 × 10-8
Total epochs	35
Warm up epochs	1
Learning rate schedule	cosine decay
(c) Pre-training hyper-parameter on RedCaps.
Config	Value
Batch size	8,192
Optimizer	AdamW [37]
Learning rate	1 × 10-3
Weight decay	0.5
Adam β	β1,β2=(0.9,0.98)
Adam ε	1 × 10-8
Total epochs	30
Warm up epochs	1
Learning rate schedule	cosine decay
(d) Pre-training hyper-parameter on LAION-400M.
Config	Value
Batch size	32,768
Optimizer	AdamW [37]
Learning rate	5 × 10-4
Weight decay	0.2
Adam β	β1,β2=(0.9,0.98)
Adam ε	1 × 10-6
Total epochs	32
Warm up iterations	2,000
Learning rate schedule	cosine decay
Table A4: SLIP hyper-parameters.
Config	Value
Batch size	8,192
Optimizer	AdamW [37]
Learning rate	3 × 10-3
Weight decay	0.1
Adam β	β1,β2=(0.9,0.98)
Adam ε	1 × 10-8
Total epochs	35
Warm up epochs	1
Learning rate schedule	cosine decay
Table A5: Detailed hyper-parameters on LinearProbing on ImageNet.
Config	Value
Batch size	1,024
Optimizer	SGD
Base learning rate	sweep
Weight decay	0
Momentum	0.9
Training epochs	90
Learning rate schedule	cosine decay
ChatGPT:
1. Source: white and red cheerful combination in the bedroom for girl
Target: A bright and lively white-and-red color scheme in a girl's bedroom, creating a cheerful ambiance.
2. Source: vintage photograph of a young boy feeding pigeons.
Target: A charming vintage photograph capturing a young boy feeding a flock of pigeons in a bustling city square.
3. Source: businessman with smartphone sitting on ledge by the sea
Target: Serene coastal view as a businessman sits on a ledge by the sea, using his smartphone.
4. Source: a tourist taking a photograph of river looking west towards suspension bridge and office
Target: Tourist snaps photo of suspension bridge and office building across the river.
5. Source: glass of foods and food product on a sunny day
Target: An assortment of food items and products displayed in a glass container, illuminated by bright sunshine.
6. Source: turtles and large fish in the pond
Target: A tranquil pond where large fish and turtles coexist peacefully, creating a harmonious natural habitat.
7. Source: the frescoes inside the dome
Target: The elaborate and intricate paintings or artworks adorning the inner surface of the dome, typically found in religious buildings.
8. Source: fight over a loose ball
Target: Intense competition as players struggle to gain control of a loose ball during the game.
9. Source: love this winter picture by person.
Target: Mesmerizing winter landscape by person: serene snowy scenery with gentle snowflakes, skillfully framed with perfect contrast and depth.
10. Source: one of several paths through the woods.
Target: A narrow forest path, one among many weaving through the lush trees, underbrush, and dappled sunlight.
11. Source: ripe strawberries falling through the water.
Target: Juicy ripe strawberries plummeting through a stream of water, splashing and creating ripples in the liquid.
12. Source: a city reflected on a red sunglasses.
Target: The cityscape reflected on a pair of red sunglasses, creating a distorted but fascinating view of the urban environment.
13. Source: man driving a car through the mountains
Target: A man confidently navigating a winding mountain road with breathtaking views.
14. Source: maritime museum from the historical dock.
Target: A museum dedicated to seafaring history, located on a historic dock where visitors can view a collection of artifacts and vessels.
15. Source: tree hollow and green leaves of a tree top in summer
Target: Amidst lush green leaves on the top of a tree, a hollow creates a natural shelter, typical of summer foliage.
16. Source: musician of musical group performs on stage on the first day of festival
Target: On the opening day of the festival, a musician from a musical group performs energetically on stage to a lively crowd.
Bard:
1. Source: man driving a car through the mountains
Target: A man drives his car through the mountains, the road winding its way through the towering peaks.
2. Source: a bicycle hanging above the entrance to a store
Target: A bicycle suspended from a chain above the entrance to a bike shop advertises their services.
3. Source: government agency released underwater footage of the unique movements of starfish
Target: New footage released by government agency shows starfish swimming in unique ways, shedding light on their underwater behavior.
4. Source: unique red chair among other white chairs at the stadium
Target: Red chair stands out among white seats at stadium, a beacon of color.
5. Source: person looks comfortable as he connects with a free - kick during the session and is in line to return against hull on saturday
Target: The player strikes a free-kick with confidence during training, looking sharp and ready for his return to the team against Hull on Saturday.
6. Source: animal in front of a white background
Target: An animal stands out against the stark white background, its feathers on full display.
7. Source: a mother and daughter lying on a lawn
Target: A mother and daughter embrace in the grass, enjoying the warmth of the sun and the sound of birdsong.
8. Source: sign is seen outside the home
Target: A for sale sign stands in the front yard, a reminder that change is always on the horizon.
9. Source: portrait of person against an abstract background stock photo
Target: person's face stands out against a swirling, abstract background, their expression one of mystery and intrigue.
10. Source: state flag waving on an isolated white background.
Target: The state flag waves proudly against a stark white background. The colors of the flag are vibrant and the design is intricate, representing the history and culture of the state
11. Source: actor wears a gorgeous blush pink coloured gown at festival.
Target: Actor stuns in blush pink gown at festival, her beauty a match for the flowers in bloom.
12. Source: person answering the phones again at the office.
Target: A friendly voice answers the phone, eager to help the caller with their needs.
13. Source: little boy sitting on the grass with drone and remote controller
Target: A young boy's eyes light up as he launches his drone into the sky, his imagination taking flight with it.
14. Source: golfer competes during day held
Target: Golfer focuses on the ball as she competes in a tournament on a beautiful day.
15. Source: golden fish in a bowl
Target: A golden fish swims lazily in a bowl, its scales shimmering in the sunlight. The fish is a beautiful shade of orange, with black spots on its fins and tail.
16. Source: businessman with smartphone sitting on ledge by the sea
Target: A businessman relaxes on a seaside ledge, checking his phone and enjoying the view.
Human:
1. Source: Honey buttermilk biscuits on a cooling rack being drizzled with honey
Target: A warm stack of freshly baked honey buttermilk biscuits, sit on a cooling rack as they are drizzled with golden honey
2. Source: happy corgi time
Target: Delighted corgi stands in the hallway, looking at its owner
3. Source: <PERSON> dog looking at dirt from the ground
Target: <Person>’s dog, lying on the ground, looks at the dirt
4. Source: navy vintage pants - lime green bag - ivory Maison Simons t-shirt - Zara clogs
Target: A young beautiful lady wearing navy vintage pants and ivory Maison Simons t-shirt, is holding a lime green bag.
5. Source: O Oak Barbie City Shine
Target: A custom-made Barbie doll with a city-inspired look shines brightly
6. Source: Real Wedding on a NYC Rooftop
Target: a couple is kissing each other during their rooftop wedding in NYC
7. Source: the proud of my beloved italian bracco after leg amputation due to a tumor.
Target: my italian bracco lied down proudly under the sunshile, despite of leg amputation due to a tumor.
8. Source: Pineapple Wearing Headphones Art Print by Philip Haynes
Target: An art from Philip Haynes depicts a pineapple that wears headphones
9. Source: Omnous thunderclouds behind the Capitol Building
Target: Thunderclouds loom over the Capitol Building, casting a dark shadow
10. Source: Steampunk woman with gun
Target: A fierce and stylish steampunk woman holds a toy revolver in her hands
11. Source: a new watch with some old friends
Target: The watch sits besides a cartoon picture, evoking memories of cherished times shared with long-time friends
12. Source: Particularly important to Africa is the East African Highland Banana (EAHB), a staple food for 80 million people. Uganda alone has about 120 varieties of this type of banana.
Target: An African man holds a bunch of bananas, which is particularly important to Africa
13. Source: Electric Blue Guitar There Goes My Hero, Rock The Vote, <PERSON>, <PERSON>, Music Photo, Red Eyes, Photo Quotes, Electric Blue, MusicLyrics
Target: <PERSON> is playing an electric blue guitar, eyes bloodshot from the stage lights
14. Source: Advanced Bicycle Skills Video - Valuable Video for Safe Cycl
Target: A Cyclist is demonstrating advanced bicycle skills in a video that will help people stay safe.
15. Source: grilled turkey pesto sandwich
Target: A grilled turkey pesto sandwich with melted cheese and fresh arugula is served on a plate.
16. Source: Actress <PERSON> during the launch of international fashion brand Forever 21 store at a mall in Mumbai on Saturday, October 12th, 2013.
Target: The young beautiful actress attended the launch of fashion brand Forever 21 at a mall.
MSCOCO:
For the meta-input-output sampling using the MSCOCO strategy, we utilize the fact that there are five different captions associated with each image. In our approach, we randomly select two texts from the available five, with one serving as the meta-input and the other as the meta-output. Below is a list of the captions we employ for this purpose.
1. Caption 1: A herd of goats walking down a road way.
Caption 2 : Three lambs stand next to each other and look different directions.
Caption 3 : The animals standing in the clearing are 3 varieties of sheep.
Caption 4 : Three small sheep are standing on a road.
Caption 5: Some animals are standing on a dirt path
2. Caption 1: A boy is preparing to toss a frisbie while another boy is sitting in the background in a park.
Caption 2 : Several people are out in the woods on a path playing a game.
Caption 3: A man in a park playing a throwing game.
Caption 4: A group of people that are hanging out together.
Caption 5: A boy gets ready to throw a frisbee
3. Caption 1: A pizza sitting on top of a metal pan.
Caption 2 : The large pepperoni pizza is covered with chives.
Caption 3 : A pizza that is sitting on a tray.
Caption 4 : A large pizza with toppings sitting on a tray.
Caption 5 : a pizza with fresh basil tomato sauce and cheese baked
4. Caption 1: A woman sits on top of a motorcycle in a parade.
Caption 2 : Woman wearing starts on helmet and shorts rides motorcycle
Caption 3 : A woman wearing attire that matches her motorcycle is driving on.
Caption 4: A person that is on top of a motorcycle.
Caption 5 : Woman on a motorcycle rides in a parade
5. Caption 1: the people are sampling wine at a wine tasting.
Caption 2 : Group of people tasting wine next to some barrels.
Caption 3 : People are gathered around a man tasting wine.
Caption 4: A man pouring wine from casks for patrons
Caption 5: People gather around a table while sampling wine.
6. Caption 1: A herd of sheep walking down a street in front of a bus.
Caption 2 : There are three animals walking down the road.
Caption 3 : a van is stuck behind a few traveling goats
Caption 4 : a van that has some kind of animal out front of it
Caption 5 : A herd of animals walking down the road behind a truck.
7. Caption 1: A sandwich with meat and cheese sits on a plate with a small salad.
Caption 2 : A sandwich with cheese and a bowl with a salad.
Caption 3 : Two plates with sandwiches on them next to a bowl of vegetables.
Caption 4: A long sandwich and a salad is on a plate.
Caption 5 : a sandwich and a bowl of vegetables on a plate
8. Caption 1: A NASA airplane carrying a space shuttle on its back.
Caption 2 : A large plan with a smaller plan on top of it.
Caption 3 : A NASA airplane carrying the old Space Shuttle
Caption 4: A NASA airplane glides through the sky while carrying a shuttle.
Caption 5 : This jet is carrying a space shuttle on it
9. Caption 1: A one way sign under a blue street sign.
Caption 2 : a view from below of a one way sign
Caption 3: A street sign stating that the road is one way beneath a blue sky.
Caption 4: A "One Way" street sign pointing to the right.
Caption 5 : A one way road sign mounted above a street sign.
10. Caption 1: A bowl of food containing broccoli and tomatoes.
Caption 2 : A large salad is displayed in a silver metal bowl.
Caption 3 : A bowl of food with tomatoes, sliced apples, and other greens
Caption 4: A silver bowl filled with various produce discards.
Caption 5: The salad in the bowl contains many fresh fruits and vegetables.
11. Caption 1: a cake made to look like it has candy decorations on it
Caption 2 : A photograph of a highly decorated cake on a table.
Caption 3: A cake decorated with lollipops and a piece of pie.
Caption 4: A piece of cake with lollypops, pie and caterpillar designs.
Caption 5 : A layered cake with sweet treats and a caterpillar as decorations.
12. Caption 1: A young man riding a skateboard on a cement walkway.
Caption 2 : a guy riding a skateboard by a car
Caption 3: A young man on a skateboard near a car
Caption 4 : an image of a boy on a skateboard doing tricks
Caption 5: A young man is riding on his skateboard.
Table A6: Performance comparison of LaCLIP trained with different meta-input-output strategies on CC12M.
(a) Zero-shot and Linear-probing Experiment Results
Source	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211	Average	ImageNet
Zero-shot
ChatGPT	57.0	71.1	38.9	51.2	31.6	3.9	25.5	63.0	80.8	36.9	92.9	24.5	39.6	10.1	6.9	42.3	44.5
Bard	55.2	70.1	39.4	51.7	31.5	4.6	25.2	63.3	80.6	34.5	92.5	20.7	39.6	10.1	7.2	41.7	44.8
MSCOCO	54.9	66.3	39.1	52.6	29.0	4.2	24.9	67.7	79.3	33.1	93.8	27.8	38.2	13.2	7.1	42.1	44.6
Human	56.4	69.1	39.1	51.7	31.4	3.8	22.9	68.1	80.6	38.4	94.3	26.9	43.0	11.7	7.5	43.0	45.1
Linear-Probing
ChatGPT	81.5	94.0	79.4	73.0	77.2	54.7	75.1	87.1	92.2	96.0	97.3	96.6	92.3	81.0	19.9	79.8	71.2
Bard	82.0	93.7	79.4	72.7	77.6	53.8	74.4	86.3	92.0	95.7	97.1	96.2	92.5	81.7	19.6	79.6	71.2
MSCOCO	81.9	94.1	79.2	73.3	76.0	53.4	75.4	86.8	92.8	95.9	97.6	96.5	92.7	82.5	19.4	79.8	71.3
Human	82.3	94.2	79.4	73.3	76.2	55.1	75.6	87.0	92.0	96.3	97.5	96.2	92.8	81.3	19.8	79.9	71.3
(b) Few-shot Experiment Results
Source	Food-101	CIFAR-10	CIFAR-100	SUN397	Cars	Aircraft	DTD	Pets	Caltech-101	Flowers	STL-10	EuroSAT	RESISC45	GTSRB	Country211
ChatGPT Bard	88.8±0.5	78.4±0.6	83.3±0.6	97.7±0.2	93.4±0.4	66.5±1.0	84.4±0.6	92.5±0.4	98.6±0.2	98.0±0.2	94.3±0.3	84.0±0.5	92.3±0.4	73.7±0.8	45.6±0.7
89.2±0.5	80.1±0.6	83.4±0.6	97.7±0.2	93.3±0.4	66.3±1.0	84.3±0.6	93.2±0.4	98.6±0.2	98.1±0.2	94.9±0.3	83.2±0.5	92.2±0.4	74.2±0.8	45.6±0.7
MSCOCO Human	88.6±0.5	79.5±0.6	82.7±0.6	97.8±0.2	93.7±0.4	65.1±1.0	84.4±0.6	92.5±0.4	98.7±0.2	98.1±0.2	95.0±0.3	84.9±0.5	91.6±0.4	74.3±0.8	44.9±0.7
88.6±0.5	78.4±0.6	83.2±0.6	97.7±0.2	93.7±0.4	66.1±1.0	84.7±0.6	93.0±0.4	98.6±0.2	98.2±0.2	94.4±0.3	83.5±0.5	92.2±0.4	74.1±0.8	45.7±0.7
13. Caption 1: A small brown dog sitting on display behind a window.
Caption 2 : A small fuzzy dog stares longingly out a window.
Caption 3 : The dog is brown shaggy with a red collar.
Caption 4: A dog sits alone and stares out of a window.
Caption 5: A furry and cute dog sitting in a window looking outside.
14. Caption 1: A herd of sheep standing on a lush green hillside.
Caption 2 : Several animals standing on the side of a hill.
Caption 3 : A number of sheep eat on a steep grassy hill.
Caption 4 : a couple of sheep are standing in some grass
Caption 5: The side of a small hill of grass with several sheep grazing in the grass and houses in the background on the upper hill.
15. Caption 1: The tennis player on the blue court has his racquet raised.
Caption 2 : A man swinging a tennis racket at a pro tennis match.
Caption 3 : A tennis player wearing a NIKE shirt swings his racket
Caption 4: Man posing in front of the camera holding up a tennis racket.
Caption 5: A man wearing a white shirt playing tennis.
16. Caption 1: A surfer riding a wave in a tempestuous ocean
Caption 2 : Man in body suit surfing on a large wave.
Caption 3 : A surfer is sideways on a wave of water on a surfboard.
Caption 4: The surfer is riding sideways along a wave.
Caption 5 : a surfer wearing a wet suit is surfing on a white board
C.2 Detailed Experiment Results on Meta-Input-Output
We present a detailed analysis of the experiment results comparing different meta-input-output strategies. Specifically, for each of the four meta-input-output strategy (ChatGPT, Bard, Human, MSCOCO), we use this specific strategy as example candidates for LLaMA ICL, and generate a rewrite for every text in CC12M. Then we train four LaCLIP models, each model trained with the original captions and the rewrite version of one specific meta-input-output strategy. The comprehensive results of these experiments are summarized in Table A6. The results indicate that different meta-input-output strategy achieves similar performance.
D. Augmentation Strategy Details
To help understand the effect of our proposed language rewriting strategy by LLaMA ICL, here we compare our proposed strategy with two widely used language augmentation baselines: EDA [58] and back translation [50].
- EDA contains four types of different randomly performed augmentation operations: Synonym Replacement, Random Insertion, Random Swap, and Random Deletion. We used the official implementation and kept all the default parameters as used in [58].
- Back Translation first translates the text to another language and then translate it back to English to generate slightly different versions of the text. We chose four different languages for our experiments: Spanish, French, German and Italic languages.
D.1 Augmentation Qualitative Comparison
We begin by presenting qualitative comparisons of different text augmentation strategies. It is observed that the EDA and back translation approaches primarily focus on word-level modifications, often preserving the sentence structures and leading to limited diversity in the rewritten texts. In contrast, our LLM-based augmentation strategy demonstrates the ability to substantially alter the sentence structure while maintaining the original key concepts and meaning intact. This results in more effective, diverse, and enriched rewritten texts, highlighting the superiority of our approach over word-based methods.