SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huiwen Chang<sup>1</sup>
<sup>1</sup>Google Research, <sup>2</sup>MIT CSAIL, *equal contribution
Code: https://github.com/google-research/syn-rep-learn
We investigate the potential of learning visual representations using synthetic images generated by text-to-image models. This is a natural question in the light of the excellent performance of such models in generating high-quality images. We consider specifically the Stable Diffusion, one of the leading open source text-to-image models. We show that (1) when the generative model is configured with proper classifier-free guidance scale, training self-supervised methods on synthetic images can match or beat the real image counterpart; (2) by treating the multiple images generated from the same text prompt as positives for each other, we develop a multi-positive contrastive learning method, which we call StableRep. With solely synthetic images, the representations learned by StableRep surpass the performance of representations learned by SimCLR and CLIP using the same set of text prompts and corresponding real images, on large scale datasets. When we further add language supervision, StableRep trained with 20M synthetic images achieves better accuracy than CLIP trained with 50M real images.
(A) Traditional Representation Learning
(B) Representation Learning with Synthetic Data
Figure 1: Left: traditional visual representation learning relies on a dataset of real images to train an image embedding function. Right: we view generative models as datasets that allow us to sample images from the data distribution. In our study, we leverage text-to-image models (Stable Diffusion [61]) and treat multiple images synthesized from the same prompt as positives for contrastive representation learning.
Data has assumed a paramount role as the key component for the success of modern machine learning systems. Such systems, especially foundation models in various domains, heavily rely on vast and diverse datasets to acquire knowledge, make accurate predictions, and generate content. The quality, quantity, and diversity of the data significantly impacts the performance and effectiveness of these
models, as they learn from the collective information encapsulated within the data. In this data-centric era, a central question is: how can we collect such large amounts of varied data to train AI models?
As an example, suppose we are trying to solve a new computer vision problem, and need to collect data (images) for it. An ideal situation is to place a camera anywhere in the weld and capture whatever we need. But in reality, collecting data is historically not easy. In the 1990s, researchers needed to take photos by themselves to create datasets for objects [52] and faces [68, 24]. To collect data in the 2000s, people crawled the Internet [15]. Noisy, uncurated data collected in such a manner can exhibit domain gaps with the real world problem and reflect imbalances due to societal bias. Removing or reducing such imperfection in data of high volume by human labeling is costly and can be prohibitive.
However, what if data collection could be simplified to the utterance of a natural language command, specifying what you want? What if, for hardly any cost, you could take a photo every few milliseconds? This sounds fanciful, but modern text-to-image generative models are approaching this vision. It has long been a dream that someday we could use these as our data sources, rather than taking photos [75, 30, 35]. In this paper, we study if this is now a practical option in the context of large scale visual representation learning.
To achieve this, we choose to work with Stable Diffusion [61], one of the leading open source text-to-image models. We synthesize images by prompting Stable Diffusion with text from large scale image-text datasets, such as CC12M [9] and RedCaps [16]. Surprisingly, our investigation reveals that when the classifier-free guidance scale is properly configured for Stable Diffusion, it is able to synthesize images on which training self-supervised methods can perform at par with or better than training on real images of the same sample size. Inspired by the idea of contrastive self-supervised learning, which promotes intra-image invariance, we develop a representation learning approach that promotes intra-caption invariance. We achieve this by treating the multiple images generated from the same text prompt as positives for each other and use them in a multi-positive contrastive loss (see Figure 1). Despite training with solely synthetic images, this approach, called StableRep, even outperforms state-of-the-art methods such as CLIP [58] using the same text set, but with corresponding real images, on various representation evaluation benchmarks.
Intuitively, one reason that synthetic data can be better than real data is because we are able to achieve a greater degree of control in the sampling, such as via the guidance scale in Stable Diffusion, or via text prompts and latent noise variables. Furthermore, generative models have the potential to generalize beyond their training data and therefore provide a richer (synthetic) training set than the corresponding real data alone. Our key contributions are:
1. We discover that training modern self-supervised methods on synthetic images from Stable Diffusion can be surprisingly effective. The learned representations are often better than representations learned from real images of the same sample size.
2. We develop StableRep, a novel representation learning approach by capturing invariance between images generated from the same text prompt, and propose a multi-positive contrastive loss.
3. With StableRep, we are able to achieve  $76.7\%$  linear accuracy on ImageNet with ViT-B/16, using solely synthetic images.
4. When coupled with language supervision, our StableRep trained with 20M synthetic images (10M captions) achieves better accuracy than CLIP trained with 50M real images (50M captions).
2 Standard Self-supervised Learning on Synthetic Images
A typical visual representation learning algorithm takes an image dataset  $\{\mathbf{x}_i\}_{i=1}^N$  as input, and yields an image encoder  $F: \mathbf{x} \to \mathbf{e}$ , which embeds an image  $\mathbf{x}$  into a vector  $\mathbf{e}$ . In this paper, we instead try to produce a good  $F$  by using a generative model  $G$  rather than a real image dataset. Specifically, we focus on text-to-image generative models  $G: (\mathbf{t}, \mathbf{z}) \to \mathbf{x}$ , which maps a pair of text  $\mathbf{t}$  and latent noise  $\mathbf{z}$  to an image  $\mathbf{x}$ . While there are several top performing text-to-image models [59, 67, 88, 7, 36, 3], we conduct our exploration with the Stable Diffusion [61] since it is publicly available and widely used. The version we used is v1-5.
2.1 Synthetic images from Stable diffusion
Stable diffusion [61] is a denoising diffusion probabilistic model [73, 31] that runs the diffusion process in the latent space of an autoencoder. It improves the sample quality and text-image alignment
Figure 2: Performance of linear probes on ImageNet as a function of the guidance scale of Stable Diffusion generation. Left: using SimCLR as pre-training; Right: using MAE as pre-training. In both cases, we see pre-training on synthetic images that are generated by Stable Diffusion with a guidance scale between 6 and 8, gives a significant boost over training only on real images. We used the CC3M dataset for these experiments.
via classifier-free guidance [32], which linearly combines conditional score estimate  $\epsilon (\mathbf{t},\mathbf{z}_{\lambda})$  and unconditional estimate  $\epsilon (\mathbf{z}_{\lambda})$  with the guidance scale  $w$  at each step  $\lambda$ :
$$
\tilde {\boldsymbol {\epsilon}} (\mathbf {t}, \mathbf {z} _ {\lambda}) = w \boldsymbol {\epsilon} (\mathbf {t}, \mathbf {z} _ {\lambda}) + (1 - w) \boldsymbol {\epsilon} (\mathbf {z} _ {\lambda}) \tag {1}
$$
The Stable Diffusion model  $G_{sd}$  relies on text sources to generate images. Instead of collecting a corpus of captions from scratch, we use the text part of existing uncurated image-text pair datasets, such as CC3M [71] and CC12M [9]. Formally, given an image caption dataset  $\{\mathbf{t}_i\}_{i=1}^N$ , we generate one image per caption, forming a synthetic image dataset of the same size.
2.2 Self-supervised learning on synthetic images
Recent representative self-supervised learning algorithms are mostly from two families: (1) contrastive learning which encourages invariance between embeddings of different augmentations of the same image ; (2) masked image modeling where model uses unmasked patches to predict masked patches (although there are other methods that fall into neither category, such as BYOL [25] and DINO [6]). For our study, we choose SimCLR [10] from the former family and MAE [26] from the latter due to their simplicity and strong performance. We focus on the Vision Transformer architecture [18], and use captions from CC3M [71] except when noted.
SimCLR [10]. We directly train SimCLR with ViT-B/16 on the synthetic image dataset, and measure the representation quality by linear probing evaluation on ImageNet [15]  ${}^{1}$  . One factor to consider is the classifier-free guidance scale  $w$  ,as it trades off between diversity and quality of the synthesized images and thus can affect the learned representations. To study this, for each  $w$  in the set  $\{ 2,3,4,6,8,{10},{12}\}$  ,we generate a copy of size  $N$  (one image per caption) to train SimCLR. Figure 2(left) visualizes the influence of  $w$  . The optimal  $w$  is around 8 (both 8 and 10 give an accuracy of 62.0%). This is different from the FID metric where  $w = 2$  is the optimal.
The captions  $\{\mathbf{t}_i\}_{i = 1}^N$  used to generate synthetic images are also paired with  $N$  real images. We train a SimCLR model with these real images. This model achieves  $60.4\%$  accuracy, experiencing a  $13\%$  drop in linear accuracy compared to pre-training on ImageNet. Such gap has been generally observed for uncurated pre-training data [77]. However, both interestingly and surprisingly, synthetic images with  $w = 8$  have  $1.6\%$  higher accuracy than real images (62.0% v.s.  $60.4\%$ ).
MAE [26]. Following the default hyperparameters in MAE [26], we train a ViT-B/16 model for each guidance scale  $w$ . Figure 2(right) reports the linear probing results. The accuracy of synthetic images increases quickly with  $w$  after 2, and gradually drops when  $w$  is large, e.g.,  $w \geq 10$ . The optimal guidance scale for MAE is 6, and this is different from SimCLR where the accuracy peaks at 8 or 10. This suggests that different methods may require different  $w$ . With  $w = 6$ , synthetic images have a  $4.2\%$  better accuracy than real images.
While the linear probing accuracy of MAE is lower than that of contrastive methods, its effectiveness often comes with fine-tuning. When fine-tuning pre-trained MAE models on ImageNet, we found synthetic images are still able to outperform real images. For instance, synthetic images with  $w = 6$  is  $0.3\%$  higher than real images (82.9% v.s. 82.6%).
<sup>1</sup>We verify our SimCLR implementation by pre-training on ImageNet. It achieves 74.3% linear probing accuracy. As a comparison, SimCLR in [11] with the same architecture and epochs achieved 73.9%.
Figure 3: Training self-supervised methods on synthetic images can be better than, or on par with, real images of the same sample size. Left: CC3M dataset; Right: CC12M dataset
Figure 4: We compare our pipeline (C) to that of (A) SimCLR; (B) CLIP. In SimCLR, the real image is augmented to give two views which are contrasted against each other through the same encoder. For CLIP, a real image and corresponding real caption are passed into image and text encoder, the image is augmented (usually more weakly than for SimCLR) followed by a contrastive loss. In our pipeline, each real caption is passed into Stable Diffusion (SD) to generate a number of synthetic images. These synthetic images are then augmented as in SimCLR, and treated as positives for each other in a multi-positive contrastive loss.
Other SSL methods. To test if synthetic images can be generically applied to different self-supervised learning methods, we try three more representative approaches: BYOL [25], MoCo-v3 [11], and DINO [6]. We do not tune  $w$  for each method, and instead apply the optimal  $w (= 8)$  discovered for SimCLR. The results on CC3M and CC12M are visualized in Figure 3. Synthetic images significantly improve over real for MAE, DINO, and SimCLR, and performs on par with real for BYOL, and slightly worse for MoCo-v3 (which could be attributed to not tuning the guidance scale  $w$ ).
3 Multi-Positive Contrastive Learning with Synthetic Images
Text-to-image generative models offer a new way to compose positive samples for contrastive learning. Given an image caption, we can create multiple diverse samples by starting the reverse diffusion process with different latent noise  $\mathbf{z}$ . Since these images are produced using the same prompt, they possess similar visual semantics, making them suitable for use as multiple positive samples for each other in contrastive learning. This property is unique to generative models, since collecting multiple images for each caption in large scale is infeasible. Figure 4 compares our StableRep pipeline with that of SimCLR and CLIP.
Multi-positive contrastive loss. We describe multi-positive contrastive learning as a matching problem. Consider an encoded anchor sample  $\pmb{a}$ , and a set of encoded candidates  $\{\pmb{b}_1, \pmb{b}_2, \dots, \pmb{b}_K\}$ .
We compute a contrastive categorical distribution  $\mathbf{q}$  that describes how likely  $a$  is to match each  $b$ :
$$
\mathbf {q} _ {i} = \frac {\exp \left(\boldsymbol {a} \cdot \boldsymbol {b} _ {i} / \tau\right)}{\sum_ {j = 1} ^ {K} \exp \left(\boldsymbol {a} \cdot \boldsymbol {b} _ {j} / \tau\right)} \tag {2}
$$
where  $\tau \in \mathcal{R}_{+}$  is the scalar temperature hyper-parameter, and  $a$  and all  $b$  have been  $\ell_{2}$  normalized. Intuitively, this is a  $K$ -way softmax classification distribution over all encoded candidates. Assume there is at least one candidate that the anchor  $a$  matches. Then we know the ground-truth categorical distribution  $\mathbf{p}$  is:
$$
\mathbf {p} _ {i} = \frac {\mathbb {1} _ {\operatorname {m a t c h} \left(\boldsymbol {a} , \boldsymbol {b} _ {i}\right)}}{\sum_ {j = 1} ^ {K} \mathbb {1} _ {\operatorname {m a t c h} \left(\boldsymbol {a} , \boldsymbol {b} _ {j}\right)}} \tag {3}
$$
where the indicator function  $\mathbb{1}_{\mathrm{match}(\cdot ,\cdot)}$  indicates whether the anchor and candidate match. Then the multi-positive contrastive loss is the cross-entropy between the ground-truth distribution  $\mathbf{p}$  and the contrastive distribution  $\mathbf{q}$ :
$$
\mathcal {L} = H (\mathbf {p}, \mathbf {q}) = - \sum_ {i = 1} ^ {K} \mathbf {p} _ {i} \log \mathbf {q} _ {i} \tag {4}
$$
This is a generalized form of the widely-used single-positive contrastive loss [54], where  $\mathbf{p}$  reduces to a one-hot vector. This loss is closely related to that in [39], but a key distinction here is that we have no image class labels, and only assume images generated from the same caption are matched.
The PyTorch-like pseudocode of the batched multi-positive contrastive learning algorithm is described in Algo. 1. Each batch consists of  $n * m$  images, meaning that we sample  $m$  images for each of the  $n$  captions. Here we still apply data augmentation, even though images from the same caption are different. This is to reduce overfitting since we perform many epochs of training over pregenerated synthetic images. However, if in the future the image generator is capable of producing images fast enough, then we can draw batches online and data augmentation may not be necessary. The
multi-positive contrastive learning algorithm is also generic such that SimCLR can also be described by it - we begin by randomly selecting a set of  $n$  images and subsequently apply  $m$  (set as 2) crops to each of the chosen images. However, in our StableRep we only utilize a single crop from each image.
Algorithm 1 Multi-Pos CL: PyTorch-like Pseudocode
f: encoder: backbone + proj mlp  
# tau: temperature  
# minibatch x: [n, m, 3, h, w]  
# n captions, m images per caption  
for x in loader:  
    x = augment(x)  
    x = cat(unbind(x, dim=1)) # [n*m, 3, h, w]  
    h = f(x)  
    # compute ground-truth distribution p  
    label = range(n * m) % n  
    p = (label.view(-1, 1) == label.view(1, -1))  
    p fill(diagonal(0) # self masking  
        p /= p-sum(1)  
    # compute contrastive distribution q  
    logits = h @ h.T / tau  
    logits fillsdiagonal(-1e9) # self masking  
    q = softmax(logits, dim=1)  
    H(p, q).backward()  
def H(p, q): # cross-entropy  
    return - (p * log(q)).sum(1).mean()
Notes: h. T is h's transpose. The  $\ell_2$  normalization operator is included in the encoder f.
We perform StableRep pre-training on synthetic images synthesized from texts in the CC3M (2.7 million samples) [71], CC12M (10 million) [9], or RedCaps datasets (11.6 million) [16]. We then evaluate the frozen representations by (1) linear probing on ImageNet-1k and other smaller scale image classification benchmark, and (2) few-shot image recognition that measures the generalization ability of the representations.
Backbone. We use ViT models [18] as the backbone for our approach StableRep. On top of the CLS token, we apply a 3-layer MLP projection head with hidden layers of 4096 dimensions and an output of 256 dimensions. Batch Normalization [33] is used in this projection head.
Training. In most of our experiments, we adopt a batch size of 8192 images (i.e.  $m*n = 8192$ ). This way the computation of each batch is equivalent to SimCLR with a batch size of 4096, because each image in SimCLR has two crops. We use AdamW optimizer [46] with a learning rate of 0.0032 and weight decay of 0.1, and set  $\beta_{1},\beta_{2}$  as 0.9, 0.98 respectively. We pre-generate 10 images for each text prompt. In each iteration, we randomly sample 6 out of the 10 for each sampled caption to form the training batch, i.e.,  $m = 6$  in Algo. 1. Recall that for SimCLR  $m = 2$ . As a result, one epoch training of StableRep is computationally equivalent to 3 epochs of SimCLR. To provide easy comparison, we report SimCLR-equivalent epochs for StableRep in all of our analysis.
4.1 Main results on CC12M and RedCaps
In this section, we perform StableRep on images synthesized by either CC12M or RedCaps. For StableRep, we first removed duplicate captions from each dataset, resulting in a reduced number of captions: from 10.0M to 8.3M for CC12M and from 11.7M to 10.5M for RedCaps. We compared StableRep to SimCLR, which was trained on either synthetic or original real images. We also included CLIP with a synthetic and a real version<sup>2</sup>. For SimCLR and CLIP, we did not perform de-duplication for either real or synthetic setting. We train for 35 epochs for all methods using ViT-B/16 (for StableRep, this refers to 35 SimCLR-equivalent epochs). We observed that CLIP started to overfit around 30 epochs. But StableRep did not overfit with this schedule (see Table 6c for results with longer training). For StableRep, we additionally apply random downsample augmentation (see Appendix A.1 for details and how such downsample affects different methods).
ImageNet. Table 1 presents the results of linear probing on ImageNet. For StableRep, we preprocess a BatchNorm layer without affine transformation to the linear classifier (see Appendix A.5 for more details). We observed that training SimCLR on synthetic images yields an improvement of  $2.2\%$  top-1 accuracy on CC12M and  $1.0\%$  on RedCaps when compared to real images. However, the accuracy of CLIP drops by  $2.6\%$  on CC12M and  $2.7\%$  on RedCaps when trained on synthetic images (see Section 5 for more discussion). On the other hand, our method StableRep outperforms CLIP trained on real images, with improvements of  $3.2\%$  and  $2.6\%$  for CC12M and RedCaps, respectively.
Real	SimCLR	Syn	StableRep
SimCLR	CLIP	CLIP	CLIP
acc.	61.5	70.3	63.7	67.8	73.5
(a) CC12M
Real	Syn
SimCLR	CLIP	SimCLR	CLIP	StableRep
acc.	61.8	71.9	62.8	69.2	74.5
Linear classification on more datasets. We followed the approach of SimCLR [10] and BYOL [25] to assess the generality of our learned representations across different image domains. Specifically, we performed linear classification on 11 image classification datasets introduced by [40]. The results are reported in Table 2, and the relative performance is consistent with that on ImageNet. Notably, our proposed method, StableRep, achieves the highest accuracy on all of the 11 datasets.
(b) RedCaps
Table 1: Comparison under the linear probing protocol on ImageNet [15]; measuring top-1 accuracy on a frozen pre-trained backbone. We compare our StableRep with SimCLR [10] and CLIP [58] with either synthetic or real images, on CC12M [9] and RedCaps [16]. All models are pre-trained with 35 epochs using ViT-B/16 [18].
CFAR-10	CFAR-100	Aircraft	Cars	DTD	Flowers	Peds	SUN397	Caltech-101	Food-101	VOC2007	Average
Real	SimCLR	88.3	70.3	47.1	45.5	76.2	92.5	70.1	65.4	83.8	75.0	81.2	72.3
CLIP	94.0	79.0	53.2	75.8	75.7	96.0	86.7	72.5	92.7	81.6	86.1	81.2
Syn	SimCLR	84.8	65.2	51.0	53.2	74.5	93.3	74.2	65.0	81.7	74.8	81.8	72.7
CLIP	87.3	69.5	53.5	79.5	75.8	95.4	85.8	69.2	90.9	78.3	84.5	79.1
StableRep	96.2	84.1	58.3	80.9	78.1	97.2	87.5	73.0	94.6	83.6	87.2	83.7
Table 2: Linear probing experiments on image datasets from various domains. Pre-training is conducted on CC12M, with either synthetic or real images. Best results for each dataset are highlighted with bold.
2We verified our CLIP implementation by comparing to prior work [51] on CC12M. With ViT-B/16, our CLIP achieved  $40.2\%$  zero-shot and  $70.3\%$  linear accuracy on ImageNet (v.s.  $36.0\%$  and  $69.0\%$  in [51]).
CIFAR-10	CIFAR-100	Aircraft	Cars	DTD	Flowers	Pets	SUN397	Caltech-101	Food-101	Average
Real	SimCLR	64.0	70.4	40.7	50.9	82.2	92.1	74.4	94.0	90.4	70.4	73.0
CLIP	77.5	82.1	62.0	90.9	83.3	97.6	91.1	97.2	98.2	87.0	86.7
Syn	SimCLR	50.0	58.9	45.2	54.2	79.8	92.0	74.6	92.9	89.1	71.0	70.8
CLIP	63.1	73.5	61.3	92.5	81.7	96.9	91.5	96.7	96.8	82.5	83.7
StableRep	92.3	91.8	62.6	91.8	86.4	98.2	91.7	97.3	98.8	87.3	89.8
Few-shot image classification. Prior work [80, 83, 17] has shown that representation learning is the key for few-shot image classification. A simple classifier on top of frozen representation is sufficient to achieve strong results. We perform 5-way, 5-shot classification following the setup in [83, 19]. As shown in Table 3, StableRep stands out on 9 out of the 10 datasets.
Table 3: Few-shot experiments. We report 5-way, 5-shot classification performance. Best results for each dataset are highlighted with bold.
MAE
IN1k, Real	StableRep
cc12m, 35ep	cc12m, 105ep	redcaps, 35ep	redcaps, 105ep
mIoU	48.1	48.8	49.4	47.3	48.4
Semantic segmentation. We fine-tune pre-trained StableRep models on ADE20k [91] using UperNet [86]. For this evaluation, StableRep is pre-trained for 35 or 105 epochs. Table 4 shows that StableRep trained on synthetic data is able to outperform MAE trained on the real ImageNet images, despite StableRep has no masked image modeling which benefits dense prediction tasks.
4.2 Ablation analysis
For simplicity, ablation studies in this section do not use the random downsample augmentation in pre-training or presupend an extra BatchNorm layer to the linear classifier.
The design choice of  $m$  (number of synthetic images per caption) is one of the key design choices for our approach. Therefore we study the following two factors relevant to  $m$  on CC3M captions (2.7 million after de-duplication).
Table 4: ADE20k semantic segmentation (mIoU) using UperNet. StableRep models are trained by 35 or 105 SimCLR-equivalent epochs.
l	1	2	4	6	8	10
acc.	61.2	64.2	65.6	66.0	66.2	66.2
(a) Given a generation budget  $T$ , we use  $T / l$  captions and generate  $l$  images per caption. When  $l = 1$  we train a SimCLR model.