SlowGuess/NeurIPS_2023_OCR huggingface dataset:
m	1	2	4	6	8	10
acc.	60.5	68.7	69.6	69.6	69.8	69.5
(b) Given a batch size of  $C$ , we form each batch by sampling  $C / m$  captions and  $m$  images per caption. We "abuse"  $m = 1$  to represent SimCLR.
Table 5: Ablation experiments on CC3M. ImageNet linear probing results with design choices relevant to data generation parameter  $l$ , and batch sampling parameter  $m$ .
Fixed generation budget. We first study the question: given a fixed value for the number of total synthetic images generated  $(T)$ , should we generate more images per caption  $(l)$ , and therefore use fewer captions  $(T / l)$  or the reverse. We assume an image budget of  $T = 2.7$  million. During training, we use the same total batch size (8192) for all  $l$ , and set the sampling parameter  $m$  as  $l$ . Table 5a presents the results. There is a clear benefit of generating more than 1 image per caption, e.g.,  $l = 8$  improves over  $l = 1$  by  $4.8\%$ . But this benefit saturates around  $l = 10$ . We thus generate 10 images per caption for our final experiments.
How to form the batch. Suppose we have generated 10 images for each of the 2.7 million captions. Now given a fixed batch size, i.e.,  $n * m = C$  (recall that  $n$  is the number of captions,  $m$  is the number of images per caption, inside each batch), a larger  $m$  encourages stronger invariance of images from the same caption, while larger  $n$  incorporates more negatives and thus encourages better separability
Figure 5: ImageNet zero-shot accuracy with different Stable Diffusion generation guidance scale  $w$ , using CLIP as pre-training.
Figure 6: ImageNet linear probing accuracy comparison between StableRep+ on synthetic images and CLIP on real images on LAION subsets. For this experiment, only 2 images are generated for each caption.
of representations. To study this trade-off, we vary the sampling parameter  $m$  from 2 to 10 while keeping  $n = C / m$ . As shown in Table 5b, The linear probing accuracy are similar between  $m = 4$  and  $m = 10$  (peak at  $m = 8$  with  $69.8\%$  accuracy), showing the robustness of StableRep w.r.t.  $m$ . We choose  $m = 6$  as our default setup. We "abuse"  $m = 1$  to represent SimCLR.
After the above study, we continue to ablate the following factors on CC12M and RedCaps.
Case	IN	avg.
small	72.8	82.2
large	70.8	80.4
mixed	71.9	81.5
(a) Guidance scale  $w$ . Smaller  $w$  yields better linear accuracy.
Size	IN	avg.
ViT-B/16	72.8	82.2
ViT-L/16	74.7	82.9
(b) Model size. Our approach scales up with model size.
Epochs	CC12M	RedCaps
35	72.8	73.7
70	75.0	76.3
105	75.7	76.7
(c) Training epochs. Longer training further improves accuracy.
Table 6: Ablation experiments by pre-training on CC12M or RedCaps. We report linear probing accuracy on ImageNet (IN) and/or average accuracy over the 11 fine-grained classification datasets (avg.). The colored cell indicates the default setup on each dataset: ViT-B/16 trained for 35 epochs with small guidance scale  $w$ .
Guidance score for training. We consider three configurations for the classifier free guidance scale  $w$ : (1) large scale  $-w \in \{8, 10\}$ ; (2) small scale  $-w \in \{2, 3\}$ ; (3) mixed scale  $-w \in \{2, 3, 4, 5, 6, 8, 10, 12\}$ . As shown in Table 6a, small scale gives the best linear transfer accuracy on ImageNet and fine-grained classification datasets. This is possibly because smaller  $w$  leads to larger intra-caption variation between generated images, which enforces StableRep to learn stronger invariance. This is different from SimCLR which requires larger  $w$  (recall Section 2.1), as SimCLR only models intra-image invariance and thus higher image quality (larger  $w$ ) helps more.
Model scale. We switch the backbone architecture to ViT-L/16. Table 6b presents the results. The accuracy improves by  $1.9\%$  on ImageNet linear probing and  $0.7\%$  on the average over fine-grained classification datasets. We found that pre-training with ViT-L was unstable. The loss kept exploding to NaN, and we resumed from the checkpoint before NaN. But this led to a higher convergent loss than ViT-B (ViT-L loss is lower before exploding). This may partly be due to the usage of BatchNorm.
Longer training. To investigate the scaling behavior of StableRep w.r.t. training compute, we further increase the pre-training computation budget to 2x and 3x epochs, and report the linear probing accuracy on ImageNet in Table 6c. The results indicate that StableRep scales well with longer training, e.g., improving by 2.2 for 2x and 2.9 for 3x on CC12M pre-training, and by 2.6 for 2x and 3.0 for 3x on RedCaps pre-training.
5 Adding Language Supervision
How would training CLIP using synthetic images work? We study this question by generating a copy (one image per caption) for each guidance scale  $w$  in  $\{1,2,3,4,6,8,10\}$  and training CLIP using
Figure 7: Examples of misalignment between input text and synthesized image, which can lead to suboptimal performance for CLIP trained on synthetic images. Upper: require headhammer shark but Stable Diffusion often generates sharks without headhammer; Lower: "Andrex Puppies" is a brand of toilet rolls.
each copy. Figure 5 plots the zero-shot ImageNet accuracy. Contrary to SSL methods, CLIP favors lower  $w$ . With the optimal  $w = 2$ , CLIP achieves  $34.9\%$  zero-shot accuracy. This is  $5.4\%$  lower than training on real images  $(40.2\%)$ . Such gap may be explained by misalignment between the generated images and the input text, shown in Figure 7. This is especially true for fine-grained classes.
We can add language supervision to StableRep by adding  $0.5 * (\mathcal{L}_{i2t} + \mathcal{L}_{t2i})$  to StableRep loss, where  $\mathcal{L}_{i2t}, \mathcal{L}_{t2i}$  are image-to-text and text-to-image contrastive losses described by Eq. 4. Adding supervision improves StableRep from  $72.8\%$  to  $74.4\%$  on CC12M and from  $73.7\%$  to  $75.4\%$  on RedCaps for ImageNet linear probing. We term it as StableRep+. We then further scale StableRep+ to a randomly selected 50M subset of LAION-400M [70]. For this experiment, we only generate 2 images per caption with  $w = 2$ , and train CLIP with real images and StableRep+ with synthetic images using different scales of random subsets of the 50M data. We plot the results in Figure 6. StableRep+ consistently achieves better accuracy than CLIP. Notably, StableRep+ with 10M captions outperforms CLIP with 50M captions, yielding a 5x time caption efficiency (2.5x image efficiency).
5.1 Fairness and compositionality
We further study the fairness and compositional understanding of the learned models on FairFace [37] and ARO [89] benchmarks, respectively. The results are presented in Table 7.
pre-train data	FairFace	ARO
mean acc.	best-class acc.	worst-class acc.	relation acc.
cc12m	CLIP	Real	28.2	60.2	0.3	46.4
Syn	30.4	64.0	3.1	50.0
StableRep+	Syn	37.2	74.9	10.0	47.3
redcaps	CLIP	Real	9.3	31.1	0.4	59.0
Syn	22.3	52.4	1.0	56.0
StableRep+	Syn	27.3	64.4	2.1	52.3
Table 7: Results of fairness and compositionality evaluation.
Fairness. We perform zero-shot classification on FairFace. We jointly classify both races and genders, e.g., treating Black male, Black female, Indian female, and so on as different classes at the same time. For cc12m models, CLIP with real data only achieved  $0.3\%$  accuracy with Southeast Asian male class, and CLIP with synthetic data improves this class to  $3.1\%$ , while our StableRep+ furthers it to  $27.2\%$ . For redcaps models, real CLIP only has  $0.4\%$  accuracy for East Asian Male, while StableRep+ improves this class to  $22.8\%$ . In summary, training with synthetic data is able to improve the worst class accuracy. However, a obvious geographic bias still exists in all models.
Compositionality. The results of compositionality evaluation are less clear. While training with synthetic data on cc12m slightly improves the relational understanding, an accuracy drop is observed in models trained with synthetic data on redcaps. An in-depth investigation may be further needed.
Text-to-Image generative models. Text-to-image models trained on large image and text pairs have recently enabled the creation of rich and diverse images encompassing many genres and themes [7, 61, 67, 88]. The resulting creations have become a sensation, with Stable Diffusion having
millions of downloads and many tools for image manipulation built on top [66, 38, 90]. Most of these models are built on denoising diffusion models [31, 73] with some notable exceptions [8, 7]. In this paper, we leverage this latest generation of diffusion-based pre-trained generative models for the task of representation learning.
Visual representation learning. Early approaches for visual representation learning often relied on pretext tasks such as inpainting [56] to train image encoders. More recent advancements have shown that mask image modeling, a form of self-supervised training, can be highly effective. In particular, Masked Autoencoder (MAE) [26] has demonstrated significant improvements in downstream finetuning performance. Another line of research focuses on contrastive learning, which aims to learn visual representations by maximizing agreement between two augmented views of the same image while distinguishing it from negative examples [10, 78, 54, 84, 27, 79]. Meanwhile CLIP [58] and its subsequent works [51] leverage contrastive learning to train image representations using language supervision, leading to impressive transferability across various tasks.
Learning from synthetic data. It has been common to train machine learning models with synthetic data in different domains [72, 81, 14, 63, 44, 64, 50, 43, 76, 87, 29, 49]. In computer vision, synthetic images have been used as a source for training models, such as optical flow [48, 23], autonomous driving [1], semantic segmentation [12, 62], object detection [65, 57], human pose estimation [82, 34] or classification [2, 69, 28]. The closest set of work are the ones that conduct representation learning on synthetic images [60, 45, 4, 35]. In [60], a model is trained to perform multi-task learning on synthetic images. The main method in [45, 4, 35] is to manipulate the latent variable of deep generative models [45, 4, 35] or image generation procedures [4], to form meaningful synthetic images for their representation learning methods. Our method falls into this category, but we use text-to-image diffusion models, which have also been explored by [2, 28, 69]. The key difference is that they conducted supervised learning while we use synthetic data for pre-training representations.
We have shown that solely synthetic data generated from state of the art text-to-image models can be used to train powerful visual representations. By harnessing the stochastic nature of Stable Diffusion in combination with a multi-positive contrastive loss, our approach yields a representation that surpasses the performance achieved through training on real data alone. Through a series of experiments, we establish that pre-training with synthetic datasets of varying scales yields impressive results across different downstream tasks, including linear probing and few-shot classification. Interestingly, we discover that even vanilla self-supervised methods trained on synthetic data can either outperform or achieve comparable results to those trained on real data.
Despite demonstrating the potential of training with synthetic data, this paper acknowledges its limitations. Firstly, we have yet to comprehend the reasons behind the effectiveness of training self-supervised methods on synthetic images compared to an equal amount of real images. It is possible that this observation is confined to our particular evaluation methodology. Furthermore, the current image generation process remains slow, with approximately 0.8s per image on a A100 GPU or 2.2s per image on a V100 GPU while xFormers is enabled. Consequently, we are not able to train StableRep models with non-repetitive images synthesized online. Additionally, we have not addressed the issue of semantic mismatch between the input prompts and the generated images, which may impact the quality and usefulness of the synthetic data. Moreover, synthetic data has the potential to exacerbate biases due to mode collapse and a predisposition to output "prototypical" images. Lastly, image attribution becomes a challenge when working with synthetic data.
Broader impacts. This paper focuses on the fundamentals of visual representation learning, and we believe it will be beneficial to the practice of this field. Our method presents an immediate application by reducing the reliance on collecting a vast amount of real images for learning representations. This approach brings potential benefits in terms of cost-effectiveness and minimizing biases introduced through human collection and curation processes. However, it is important to acknowledge that our method relies on text-to-image generative models trained on large-scale, uncurated web data. Such data may conceal social biases and errors that would have been exposed through human curation. Additionally, we must recognize that the text prompts we employed are not completely bias-free; the selection of prompts influences the synthesized images. Thus, the choice of prompts assumes a role similar to the selection of real images for self-supervised visual representation learning.
We would like to thank anonymous reviewers and Shumeet Baluja for reviewing our manuscript and providing many helpful comments and suggestions. We also appreciate the helpful discussions and the general supports from the VisCam teammates in Google Research.
[1] Hassan Abu Alhajja, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, and Carsten Rother. Augmented reality meets computer vision: Efficient data generation for urban driving scenes. IJCV, 2018. 10
[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Sahara, Mohammad Norouzi, and David J Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023. 10
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. edifi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022. 2
[4] Manel Baradad Jurjo, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. In NeurIPS, 2021. 10
[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In ECCV, 2014. 19
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021. 3, 4, 16, 18
[7] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023. 2, 9, 10
[8] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022. 10
[9] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. 2, 3, 5, 6
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 3, 6, 10, 16, 17, 18
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021. 3, 4, 16, 18
[12] Yuhua Chen, Wen Li, Xiaoran Chen, and Luc Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach. In CVPR, 2019. 10
[13] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014. 19
[14] Yabo Dan, Yong Zhao, Xiang Li, Shaobo Li, Ming Hu, and Jianjun Hu. Generative adversarial networks (gan) based efficient sampling of chemical composition space for inverse design of inorganic materials. NPJ Computational Materials, 2020. 10
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2, 3, 6
[16] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.11431, 2021. 2, 5, 6
[36] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. arXiv preprint arXiv:2303.05511, 2023.2
[37] Kimmo Kärkkäinen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv:1908.04913, 2019. 9
[38] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. arXiv preprint arXiv:2210.09276, 2022. 10
[39] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv:2004.11362, 2020.5
[40] Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In CVPR, 2019. 6
[41] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshop, 2013. 19
[42] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009. 19
[43] Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre-trained transformer models. arXiv preprint arXiv:2003.02245, 2020. 10
[44] Jason Li, Ravi Gadde, Boris Ginsburg, and Vitaly Lavrukhin. Training neural speech recognition systems with synthetic speech augmentation. arXiv preprint arXiv:1811.00707, 2018. 10
[45] Hao Liu, Tom Zahavy, Volodymyr Mnih, and Satinder Singh. Palm up: Playing in the latent manifold for unsupervised pretraining. arXiv preprint arXiv:2210.10913, 2022. 10
[46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6
[47] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013. 19
[48] Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR, 2016. 10
[49] Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. arXiv preprint arXiv:2202.04538, 2022.10
[50] Masato Mimura, Sei Ueno, Hirofumi Inaguma, Shinsuke Sakai, and Tatsuya Kawahara. Leveraging sequence-to-sequence speech synthesis for enhancing acoustic-to-word speech recognition. In SLT, 2018. 10
[51] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In ECCV, 2022. 6, 10, 18
[52] Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-20). Tech Report, 1996. 2
[53] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008. 19
[54] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 5, 10
[74] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 20
[75] Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 1991. 2
[76] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models., 2023. 10
[77] Yonglong Tian, Olivier J Henaff, and Aaron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. In ICCV, 2021. 3
[78] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv:1906.05849, 2019. 10, 18
[79] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NeurIPS, 2020. 10
[80] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In ECCV, 2020. 7
[81] Allan Tucker, Zhenchen Wang, Ylenia Rotalinti, and Puja Myles. Generating high-fidelity synthetic patient data for assessing machine learning healthcare software. NPJ digital medicine, 2020. 10
[82] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, 2017. 10
[83] Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning. arXiv preprint arXiv:1911.04623, 2019.7
[84] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. 10
[85] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 19
[86] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018. 7
[87] Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmentation for commonsense reasoning. arXiv preprint arXiv:2004.11546, 2020. 10
[88] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022. 2, 9
[89] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In ICLR, 2022. 9
[90] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023. 10
[91] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019. 7
A.1 Downsample augmentation
Synthetic images have constant high resolutions (e.g.,  $512 \times 512$  for Stable Diffusion). We find this leads to a domain gap when transferring to situations involving low resolution images, such as CIFAR-10 or CIFAR-100. To address this issue, we introduce Random Downsample augmentation, which randomlyizes images to a resolution of 64 or 128 (equally probable) and then resize them back to 224. During pre-training, we apply this augmentation with a probability of 0.05 and preprocess it to other augmentations.
In Table 8, we ablate the effects of applying this random downsample augmentation to different pretraining methods. This augmentation brings significant improvements on CIFAR-10 and CIFAR-100 datasets, while maintaining the performance on other datasets. On average this augmentation is more beneficial for pre-training with synthetic images than real ones.