SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Downsample	CIFAR-10	CIFAR-100	Aircraft	Cars	DTD	Flowers	Pets	SUN397	Caltech-101	Food-101	VOC2007	Average
Real	SimCLR	88.3	70.3	47.1	45.5	76.2	92.5	70.1	65.4	83.8	75.0	81.2	72.3
SimCLR	✓	92.3	75.4	47.8	44.4	77.1	91.8	69.2	65.1	84.7	74.9	81.2	73.1 (+0.8)
CLIP	94.0	79.0	53.2	75.8	75.7	96.0	86.7	72.5	92.7	81.6	86.1	81.2
CLIP	✓	95.8	82.9	51.5	76.5	74.7	95.3	87.2	72.5	92.7	81.7	86.2	81.5 (+0.3)
Syn	SimCLR	84.8	65.2	51.0	53.2	74.5	93.3	74.2	65.0	81.7	74.8	81.8	72.7
SimCLR	✓	92.6	76.2	50.4	53.2	74.7	93.3	74.7	64.8	86.0	74.7	81.2	74.7 (+2.0)
CLIP	87.3	69.5	53.5	79.5	75.8	95.4	85.8	69.2	90.9	78.3	84.5	79.1
CLIP	✓	93.4	78.8	52.4	79.6	75.1	95.0	85.0	69.5	90.9	78.4	84.7	80.3 (+1.2)
StableRep	90.7	74.4	57.6	80.3	79.0	96.7	87.1	73.2	94.0	83.5	87.2	82.2
StableRep	✓	96.2	84.1	58.3	80.9	78.1	97.2	87.5	73.0	94.6	83.6	87.2	83.7 (+1.5)
A.2 Standard self-supervised learning
We follow the default settings for standard self-supervised learning algorithms, and present the training details in Table 9 and Table 10. We use the linear  $lr$  scaling rule:  $lr = \text{base\_lr} \times \text{bsz} / 256$ . For BYOL [25], we did not follow the hyperparameters  $(blr = 1.0e - 4, wd = 0.03)$  in [11], as we found our setting here yielded better accuracy. For DINO [6], we did not use the multi-crop strategy and only pre-trained the model with two  $224 \times 224$  crops.
Table 8: We ablate the effects of Random Downsample augmentation on linear probing benchmarks from various domains. This augmentation brings significant improvements on CIFAR-10 and CIFAR-100 datasets, while maintaining the performance on other datasets.
config	MAE	SimCLR
optimizer	AdamW	AdamW
base learning rate	1.5e-4	2.0e-4
weight decay	0.05	0.1
optimizer momentum	β1, β2=0.9, 0.95	β1, β2=0.9, 0.98
batch size	4096	4096
learning rate schedule	cosine decay	cosine decay
epochs	300 (cc3m) / 80 (cc12m)	100 (cc3m) / 35 (cc12m)
warmup epochs	10 (cc3m) / 4 (cc12m)	5 (cc3m) / 1 (cc12m)
augmentation	RandomResizedCrop, Flip	SimCLR Aug. [10]
Table 9: Self-supervised pre-training settings. MAE and SimCLR.
config	DINO	BYOL/MoCo-v3
optimizer	AdamW	AdamW
base learning rate	5.0e-4	1.5e-4
weight decay	0.04 to 0.4, cosine	0.1
optimizer momentum	β1, β2=0.9, 0.999	β1, β2=0.9, 0.95
batch size	4096	4096
learning rate schedule	cosine decay	cosine decay
epochs	100 (cc3m) / 35 (cc12m)	100 (cc3m) / 35 (cc12m)
warmup epochs	5 (cc3m) / 2 (cc12m)	5 (cc3m) / 2 (cc12m)
momentum update λ	0.996 to 1, cosine	0.996 to 1, cosine
augmentation	BYOL Aug. [25]	BYOL Aug. [25]
teacher temp. τt	0.04 to 0.07 in warmup
student temp. τs	0.1
A.3 StableRep pre-training
The hyperparameters for StableRep is presented in Table 11. Indeed, they are the same as that in SimCLR. The difference is that the base_lr in StableRep is for 512 images while in SimCLR it is for 256 images, because each image in StableRep only has one single crop. We ended up using a batch size of 8256 images, since we trained our model with 32 GPUs and 8192 is not divisible over  $32 \times 6$ . The computation for StableRep has been converted to SimCLR-equivalent epochs.
Table 10: Self-supervised pre-training settings. DINO, BYOL and MoCo v3.
config	StableRep
batch size	8256 (m = 6, n = 1376)
optimizer	AdamW
base learning rate	2.0e-4
peak learning rate	base_lr × bsz/512
weight decay	0.1
optimizer momentum	β1, β2=0.9, 0.98
learning rate schedule	cosine decay
epochs	35 / 70 / 105
warmup epochs	1.2 / 2.3 / 3.5
augmentation	Downsample Aug. + SimCLR Aug. [10]
A.4 CLIP training
Table 11: StableRep pre-training settings.
config	CLIP
batch size	8192
optimizer	AdamW
peak learning rate	1e-3
weight decay	0.5
optimizer momentum	β1, β2=0.9, 0.98
learning rate schedule	cosine decay
epochs	35
warmup epochs	1
augmentation	RandomResizedCrop(scale=(0.5, 1.0))
Table 12: CLIP training settings.
Model	Patch size	Input resolution	Embedding dimension	Vision Transformer	Text Transformer	Vocab size	Text length
Layers	Width	Heads	Layers	Width	Heads
ViT-B/16	16	224	512	12	768	12	12	512	8	49,408	77
We follow the hyperparameter setting used in [51] since it is better than that from the original CLIP [58] paper. Table 12 summarizes the training details, and Table 13 presents the architecture of CLIP encoders. With this training setup, we are able to produce  $40.2\%$  ImageNet zero-shot accuracy when training CLIP on CC12M dataset. As a comparison, [51] reports  $36.0\%$  using the same architecutre.
A.5 ImageNet linear probing
We follow prior work [11, 6] to train the linear classifier. It has been generally observed that regularization such as weight decay hurts the performance [78]. Following [78, 11], we set weight decay as 0, and only use RandomResizedCrop and RandomHorizontalFlip as data augmentation. We sweep the base_lr over  $\{0.1, 0.2, 0.5, 1, 2, 5, 10, 20, 50\} \times 10^{-2}$ .
Table 13: CLIP encoder details.
config	value
batch size	1024
optimizer	SGD
base learning rate	sweep
weight decay	0
optimizer momentum	0.9
learning rate schedule	cosine decay
epochs	90
augmentation	RandomResizedCrop, Flip
For StableRep trained with 35 epochs, we find that adding an extra BatchNorm layer without affine transformation improves and stabilizes the linear probing results. However, this additional BatchNorm does not help when StableRep is trained with a longer schedule, e.g., 105 epochs. We conjecture that BatchNorm is helpful when StableRep is not convergent, and present the comparison in Table 15.
Table 14: ImageNet linear probing settings.
w/ BN	w/o BN
StableRep, 35 epochs	73.5	71.4
StableRep, 105 epochs	75.2	75.4
Table 15: ImageNet linear probing results w/ or w/o extra BatchNorm layer for the linear classifier.
A.6 Fine-grained linear classification
Following [10, 25, 20], we fit a regularized multinomial logistic regression model on top of the frozen CLS token. In training and testing, we do not perform any data augmentation; images are resized to 224 pixels along the shorter side using bicubic resampling, followed by a center crop of  $224 \times 224$ . We minimize the cross-entropy objective using L-BFGS with  $\ell_2$ -regularization. We select this  $\ell_2$ -regularization constant on the validation set over 45 logarithmically spaced values between  $10^{-6}$  and  $10^5$ . The maximum number of L-BFGS iterations is set to 500.
The details about the fine-grained classification datasets are presented in Table 16.
A.7 Few-shot image classification
Following the settings in [20, 19], we evaluate the 5-way 5-shot performance on 10 different datasets. We do not use data augmentation; images are resized to 224 pixels along the shorter side using
Dataset	Metric	Categories	Train Size	Test Size
CIFAR-10 [42]	Accuracy	10	50,000	10,000
CIFAR-100 [42]	Accuracy	100	50,000	10,000
Aircraft [47]	Mean per class	100	6,667	3,333
Cars [41]	Accuracy	196	8,144	8,041
DTD [13]	Accuracy	47	3,760	1,880
Flowers [53]	Mean per class	102	2,040	6,149
Pets [55]	Mean per class	37	3,680	3,669
SUN397 [85]	Accuracy	397	19,850	19,850
Caltech-101 [22]	Mean per class	102	3,060	6,085
Food-101 [5]	Accuracy	101	75,750	25,250
VOC2007 [21]	Mean per class	20	5,011	4,952
bicubic resampling, followed by a center crop of  $224 \times 224$ . We report the mean accuracy of 600 randomly sampled tasks (also known as episodes). For each task, images are randomly sampled from the combination of training, validation and testing sets. We sample 15 query images for each class in every task for evaluation purpose.
B.1 Fine-grained classification
Table 16: Details of the fine-grained linear classification datasets.
CIFAR-10	CIFAR-100	Aircraft	Cars	DTD	Flowers	Pcs	SUN397	Caltech-101	Food-101	VOC2017	Average
Pre-training on Redcaps for 35 epochs
Real	SimCLR	90.2	72.0	46.8	42.8	77.9	94.6	83.0	61.2	82.7	81.3	80.9	73.9
CLIP	94.2	78.9	52.9	74.9	73.9	97.8	91.6	66.2	91.6	89.2	85.4	81.5
Syn	SimCLR	85.1	65.4	48.7	53.7	74.6	95.0	79.6	61.8	84.5	79.7	80.4	73.5
CLIP	88.7	71.4	53.7	77.3	76.0	96.9	88.2	67.3	90.3	83.7	84.5	79.8
StableRep	96.7	84.6	57.2	78.8	79.0	98.4	90.9	70.7	94.9	88.1	86.6	84.2
Longer training for StableRep
cc12m	35 epochs	96.2	84.1	58.3	80.9	78.1	97.2	87.5	73.0	94.6	83.6	87.2	83.7
105 epochs	96.7	84.7	59.2	83.5	80.1	97.3	88.3	74.3	94.7	85.1	87.9	84.7
redcaps	35 epochs	96.7	84.6	57.2	78.8	79.0	98.4	90.9	70.7	94.9	88.1	86.6	84.2
105 epochs	96.9	85.6	60.2	83.9	80.0	98.5	91.7	72.5	94.8	89.4	87.8	85.6
OpenAI's CLIP trained on WIT-400M (numbers copied from [58])
WIT-400M	96.2	83.1	59.5	86.7	79.2	98.1	93.1	78.4	94.7	92.8	89.2	86.5
Table 17: Linear transfer results on fine-grained datasets. All results are with ViT-B/16. Results of StableRep are marked. Upper: different methods pre-trained on RedCaps. Middle: StableRep with different training schedules. Lower: OpenAI's CLIP trained on WIT-400M dataset. Our StableRep trained with synthetic images only is approaching the performance of CLIP trained with 400 millions of real images.
In Table 17, we further present the fine-grained linear classification results by models from RedCaps or models that are trained longer (2x or 3x longer). When pre-training on RedCaps, StableRep achieves the best average accuracy. Longer training of StableRep further improves transferability. Notably, our StableRep trained with synthetic images only is approaching the performance of OpenAI's CLIP trained with 400 millions of real images.
B.2 Few-shot image classification
We further summarizes the few-shot image classification results in Table 18. The  $95\%$  confidence interval is provided. StableRep stands out on the majority of the evaluated datasets.
CFAR-10	CFAR-100	Aircraft	Cars	DTD	Flowers	Pets	SUN397	Caltech-101	Food-101	Average
Pre-training on cc12m
Real	SimCLR	64.0±0.7	70.4±0.8	40.7±0.9	50.9±0.8	82.2±0.6	92.1±0.5	74.4±0.8	94.0±0.4	90.4±0.5	70.4±0.7	73.0
CLIP	77.5±0.6	82.1±0.7	62.0±1.0	90.9±0.5	83.3±0.6	97.6±0.2	91.1±0.5	97.2±0.2	98.2±0.2	87.0±0.5	86.7
Syn	SimCLR	50.0±0.6	58.9±0.8	45.2±1.0	54.2±0.8	79.8±0.6	92.0±0.5	74.6±0.8	92.9±0.4	89.1±0.6	71.0±0.7	70.8
CLIP	63.1±0.6	73.5±0.7	61.3±1.0	92.5±0.4	81.7±0.6	96.9±0.3	91.5±0.5	96.7±0.2	96.8±0.3	82.5±0.6	83.7
StableRep	92.3±0.3	91.8±0.5	62.6±1.0	91.8±0.5	86.4±0.5	98.2±0.2	91.7±0.5	97.3±0.2	98.8±0.2	87.3±0.5	89.8
Pre-training on redcaps
Real	SimCLR	62.3±0.6	69.4±0.7	39.6±0.9	51.0±0.8	82.7±0.6	94.8±0.4	85.4±0.6	91.8±0.5	88.5±0.6	79.1±0.7	74.5
CLIP	80.6±0.5	85.3±0.6	54.5±0.9	88.5±0.6	82.6±0.6	99.0±0.1	94.5±0.4	95.9±0.3	97.8±0.2	94.4±0.3	87.3
Syn	SimCLR	52.9±0.6	60.8±0.8	40.9±0.9	53.2±0.8	79.5±0.6	94.3±0.4	78.3±0.7	92.0±0.4	88.9±0.5	75.9±0.7	71.7
CLIP	65.7±0.6	75.7±0.7	55.2±1.0	90.1±0.5	82.6±0.6	98.2±0.2	92.0±0.5	96.3±0.3	96.9±0.3	88.1±0.5	84.1
StableRep	92.7±0.3	92.9±0.4	57.3±1.0	89.4±0.6	86.2±0.5	99.2±0.1	94.5±0.4	96.8±0.3	98.9±0.2	91.8±0.4	90.0
Table 18: Few-shot image classification results with  $95\%$  confidence interval provided. All models here are trained for 35 epochs. Upper: pre-training on CC12M dataset. Upper: pre-training on RedCaps dataset.
C Image Generation
We use Stable Diffusion [61] v1.5. During sampling, we generate images by 50 DDIM [74] steps. To accelerate the generation process, we leverage xFormers library for efficient attention computation, which brings down the sampling time to  $\sim 0.8$ s per image on a single A100 GPU and  $\sim 2.3$ s per image on a V100 GPU.
Image resolution. The image resolution may affect the quality of representations learned by self-supervised learning algorithms. We try to make a relative fair comparison by storing all synthetic and real images in similar resolutions. The synthetic images generated by Stable Diffusion are  $512 \times 512$ ; we resized them to  $256 \times 256$  before storing them on the disk. The real images have various sizes, ranging from less than a hundred of pixels in shorter side to thousands of pixels; we resize the shorter side of all real images to 256.
C.2 Generation examples
Some examples of synthetic images are visualized in Figure 8.
Synthesis. The slowest part of the StableRep pipeline is the image generation. We use 512 V100 GPUs to synthesize images, which takes  $\sim 13$  hours for every ten million images.
Pre-training. Each of our StableRep models with ViT-B/16 is trained on 4 nodes, each of which has 8 A100 GPUs and 96 CPU cores. It takes  $\sim 20$  hours to complete 35 SimCLR-equivalent epochs of training on CC12M and  $\sim 23$  hours on RedCaps. For ViT-L/16, we use 64 A100 80GB GPUs spread over 8 nodes.
Figure 8: Examples of synthetic images. We show examples for 4 different text prompts. For each prompt, we provide examples synthesized with different guidance scale  $w$ , as well as the original real image.