StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners
Authors: Yonglong Tian
Lijie Fan
Phillip Isola
Huiwen Chang
Dilip Krishnan
Venue: NeurIPS 2023 poster (2023)
Keywords: representation learning
synthetic images
text-to-image models
TL;DR: We present that learning representations by synthetic data from text-to-image models can be very effective at large scale, even outperforming CLIP

Abstract: We investigate the potential of learning visual representations using synthetic images generated by text-to-image models. This is a natural question in the light of the excellent performance of such models in generating high-quality images. We consider specifically the Stable Diffusion, one of the leading open source text-to-image models. We show that (1) when the generative model is properly configured, training self-supervised methods on synthetic images can match or beat the real image counterpart;
(2) by treating the multiple images generated from the same text prompt as positives for each other, we develop a multi-positive contrastive learning method, which we call StableRep. 
With solely synthetic images, the representations learned by StableRep surpass the performance of representations learned by SimCLR and CLIP using the same set of text prompts and corresponding real images, on large scale datasets. 
When we further add language supervision, \name~trained with 20M synthetic images (10M captions) achieves better accuracy than CLIP trained with 50M real images (50M captions).

StableRep: Synthetic Images from Text-to-lmage
Models Make Strong Visual Representation Learners
Yonglong T
Lijie Fan 1.2.
Phillip Isola?
Huiwen Chang
Dilip Krishnan'
GGoole Research,
MIT CSAIL,
"equal contribution
Code: ttettts://gitee com/ google rttp://errreeerr/////eeeeee
Abstract
We investigate the potential of learning visual representations using synthetic images generated by text-to-image models. This is a natural question in the light the excellent performance of such models in generating high-quality images. We consider specifically the Stable Difftusion, one of the leading open source text-to- image models. We show that (1) when the generative model is configured with proper classifier-free guidance scale, training self-supervised methods on synthetic images match beat the real image counterpart; (2) by treating the multiple images generated from the same text prompt as positives for each other, we develop multi-positive contrastive learning method, which we call StableRep. With solely synthetic images, the representations learned by StableRep surpass the performance of representations learned by SimCLR and CLIP using the same of text prompts and corresponding real images, on large scale datasets. When we further add language supervision, StableRep trained with 20M synthetic images achieves better accuracy than CLIP trained with 50M real images.
Engine
Generative Models Stable Diffusion (SD)
Real data
Synthetic Data
Real
Encoder
Embedding
Synthetic data
Encoder
Embedding
(A) Traditional Representation Learning
(B) Representation Learning with Synthetic Data
Figure 1: Left: traditional visual representation learning relies dataset of real images train an image embedding function. Right: view generative models datasets that allow sample images from the data distribution. our study, we leverage text-to-image models (Stable Diffusion [61]) and treat multiple images synthesized from the same prompt as positives for contrastive representation learning.
Introduction
Data has assumed paramount role as the key component for the success of modern machine learning systems. Such systems, especially foundation models in various domains, heavily rely vast and diverse datasets to acquire knowledge, make accurate predictions, and generate content. The quality, quantity, and diversity of the data significantly impacts the performance and effectiveness of these
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

models, as they learn from the collective information encapsulated within the data. In this data-centrie era, a central question is: how can we collect such large amounts of varied data to train AI models?
As an example, suppose we are trying to solve a new computer vision problem, and need to collect data (images) for it. An ideal situation is to place a camera anywhere in the wold and capture whatever we need. But in reality, collecting data is historically not easy. In the 1990s, researchers needed to take photos by themselves to create datasets for objects [52] and faces [68. 24]. To collect data the 2000s, people crawled the Internet [15]. Noisy, uncurated data collected in such a manner can exhibit domain gaps with the real world problem and reflect imbalances due to societal bias. Removing or reducing such imperfection in data of high volume by human labeling is costly and can be prohibitive.
However, what if data collection could be simplified to the utterance of a natural language command, specifying what you want? What if, for hardly any cost, you could take a photo every few millisec- onds? This sounds fanciful, but modern text-to-image generative models are approaching this vision. It has long been a dream that someday we could use these as our data sources, rather than taking photos [75, 30, 35]. In this paper, we study if this now practical option in the context of large scale visual representation learning.
To achieve this, we choose to work with Stable Diffusion [61], one of the leading open source tont-to-image models. We synthesize images by prompting Stable Diffusion with text from large scale image-text datasets, such CC12M [9] and RedCaps [16]. Surprisingly. reveals that when the classifier-free guidance scale is properly configured for Stable Diffusion, it is able synthesize images which training self-supervised methods can perform at par with or better than training on real images of the same sample size. Inspired by the idea of contrastive self-supervised learning, which promotes intra-image invariance, approach that promotes intra-caption invariance. We achieve this by treating the multiple images generated from the same text prompt as positives for each other and use them contrastive loss (see Figure 1). Despite training with solely synthetic images, this approach, called StableRep, even outperforms state-of-the-an methods such as CLIP [58] using the same text set, but with corresponding real images, on various representation evaluation benchmarks.
investigation
develop a representation learning
multi-positive
Intuitively, one reason that synthetic data can be better than real data is because we are able to achieve a greater degree of control in the sampling, such as via the guidance scale in Stable Diffusion, or via text prompts and latent noise variables. Furthermore, generative models have the potential to generalize beyond their training data and therefore provide a richer (synthetic) training set than the corresponding real data alone. Our key contributions are:
1. We discover that training modern self-supervised methods on synthetic images from Stable Diffu- sion can be surprisingly effective. The learned representations are often better than representations learned from real images of the same sample size.
2. We develop StableRep. novel representation learning approach by capturing invariance between images generated from the same text prompt, and propose a multi mulli-positive contrastive loss.
3. With StableRep, are able to achieve 76.7% linear accuracy on ImageNet with ViT-B/16, using solely synthetic images.
4. When coupled with language supervision, our StableRep trained with 20M synthetic images (10M captions) achieves better accuracy than CLIP trained with 50M real images (50M captions).
2 Standard Self-supervised Learning on Synthetic Images
A typical visual representation learning algorithm takes an image dataser {x, ) an image encoder F x - e, which embeds image into vector e. In this paper, we instead try to produce a good F by using a generative model G rather than a real image dataset. Specifically, we focus text-to-image generative models G (t, x) - x, which maps pair of text t and latent noise z to an image x. While there are several top performing text-to-image models [59. 67, 88, 7, 36, 3]. we conduct our exploration with the Stable Diffusion [61] since is publicly available and widely used. The version we used is v1-5.
==1 as input, and yields
2.1 Synthetic images from Stable diffusion
Stable diffusion [61] is a denoising diffusion probabilistic model [73, 31] that runs the diffusion process in the latent space of autoencoder. It improves the sample quality and text-image alignment

---

SimCLR
62.0
56.2
ImageNet Linsr ACE
62.0
Synthotic
56.0
55.4
55,8
55.4
61.5
Real
55.0
61.0
Synthetic Real
60.5
60.4
53.0
60.0
59.7
52.0
51.0
10
12
guidance scale W
guldance scale
Figure 2: Performance linear probes on ImageNet as function the guidance scale of Stable Diffuusion generation. Left: using SimCLR as pre-training; Right: using MAE as pre-training. In both cases, we see pre-training synthetic images that generated by Stable Diffusion with a guidance scale between 6 and 8, gives significant boost over training only real images. We used the CC3M dataset for these experiments.
via classifier-free guidance [32], which linearly combines conditional score estimate e(t, zx) and unconditional estimate E(2x) with the guidance scale w at each step A:
é(t. zA) = we(t, (t, ZA) + (1 w))(z))
(1)
The Stable Diffusion model Ged relies text sources to generate images. Instead of collecting corpus of captions from scratch, we use the text part of existing uncurated image-text pair datasets, such CC3M [71] and CC12M [9]. Formally, given image caption dataset {t; ),N we generate one image per caption, forming synthetic image dataset of the same size.
2.2 Self-supervised learning on synthetic images
Recent representative self-supervised learning algorithms mostly from two families: (1) con- trastive learning which encourages invariance between embeddings of different augmentations the same image (2) masked image modeling where model uses unmasked patches to predict masked patches (although there are other methods that fall into neither category, such as BYOL [25] and DINO [6]). For study, choose SimCLR [10] from the former family and MAE [26] from the latter due to their simplicity and strong performance. We focus on the Vision Transformer architecture [18], and use captions from CC3M [71] except when noted.
SimCLR [10]. We directly train SimCLR with ViT-B/16 on the synthetic image dataset, and measure the representation quality by linear probing evaluation on ImageNet [15] One factor to consider is the classifier-free free guidance scale w, as it trades off between diversity and quality of the synthesized images and thus affect the learned representations. To study this, for each w in the set (2, 3, 4, 6, 8, 10, 12), we generate a copy of size N (one image per caption) to train SimCLR. Figure 2(left) visualizes the influence of w. The optimal is around 8 (both 8 and 10 give an accuracy of 62.0%). This is different from the FID metric where w is the optimal.
The captions [t used to generate synthetic images are also paired with N real images. We train a SimCLR model with these real images. This model achieves 60.4% accuracy, experiencing a 13% drop in linear accuracy compared to pre-training on IetgeNet. Such gap has been generally observed for uncurated pre-training data [77]. However, both interestingly and surprisingly, synthetic images with w 8 have 1.6% higher accuracy than real images (62.0% v.s. 60.4%).
MAE [26]. Following the default hyperparameters' in MAE [26], train VIT-B/16 model for each guidance scale w. Figure (rright) reports the linear probing results. The accuracy of synthetic images increases quickly with w after 2, and gradually drops when large, e.g., 2 10. The optimal guidance scale for MAE is 6, and this is different from SimCLR where the accuracy peaks at 8 or 10. This suggests that different methods may require different w. With w synthetic images have 4.2% better accuracy than real images.
While the linear probing accuracy of MAE is lower than that of contrastive methods, its effectiveness often comes with fine-tuning. When fine-tuning pre-trained MAE models on ImageNet, we found synthetic images are still able outperform real images. For instance, synthetic images with is 0.3% higher than real images (82.9% v.s. 82.6%).
We verify our SimCLR implementation by pre-training on ImageNet. It achieves 74.3% linear probing accuracy. As comparison, SimCLR in [1I] with the same architecture and epochs achieved 73.9%.
3

---

Synthetic vs. Real on CC3M
Synthetic vs. Real CC12M
-0.2
64
Synthetic
@+0.2
-0.5
*
Synthetic
R-0.3
Real
Real
ImageNet Linear
+1.6
62
+2.1
58
+2.4
60
+2.1
56
58
+5.2
56-
52-
54
50
52
MAE
DINO
SimCLR
BYOL
MoCo v3
MAE
DINO
SimCLR
BYOL
MoCo v3
Figure 3: Training self-supervised methods on synthetic images can be better than, or on par with, real images of the same sample size. Left: CC3M dataset; Right: CC12M dataset.
(A) SimCLR
(B) CLIP
(C) StableRap (Ours)
Encoder
Encoder
Encoder
Encoder
Encoder
Encoder
Encader
Aug
Aug
Aug
Aug
Weak Aug
"Clay Animation, Adorable Dragon"
Real Image
Real Image
Real Caplion
"Clay Anination, Dragan"
Figure 4: We compare our pipeline (C) to that of (A) SimCLR; (B) CLIP. In SimCLR, the real image is augmented give two views which contrasted against each other through the encoder. For CLIP, real image and corresponding more weakly than SimCIR R) followed by a contrastive loss. In our pipeline, each real caption is passed into Stable Diffusion (SD) to generate number of synthetic images. These synthetic images are then augmented as in SimCLR, and treated as positives for each other in multi-positive contrastive loss.
caption passed into image and text encader, the image is augmented (usually
Other SSL methods. To test if synthetic images can be generically applied to different self-supervised learning methods, we try three more representative approaches: BYOL [25]. MoCo-v3 [11], and DINO [6]. We do not tune w for each method, and instead apply the optimal w (= 8) discovered for SimCLR. The results on CC3M and CC12M are visualized in Figure Synthetic images significantly improve over real for MAE, DINO, and SimCLR, and performs on with real for BYOL, and slightly worse for MoCo-v3 (which could attributed to not tuning the guidance scale w).
3 Multi-Positive Contrastive Learning with Synthetic Images
Text-to-image generative models offer way to compose positive samples for contrastive learning. Given an image caption, we can create multiple diverse samples by starting the reverse diffusion process with different latent noise Since these images are produced using the same prompt, they possess similar visual semantics, making them suitable for use as multiple positive samples for each other in contrastive learning. This property is unique to generative models, since collecting multiple images for each caption in large scale is infeasible. Figure 4 compares our StableRep pipeline with that of SimCLR and CLIP.
Multi-positive contrastive loss. We describe multi-positive contrastive learning as matching problem. Consider encoded anchor sample a, and set of encoded candidates {b, by.... bx

---

We compute a contrastive categorical distribution that describes how likely a is to match each b:
exp(a bz/T)
Σ 1== exp(a by/T)
where 7 E R.4 the scalar temperature Intuitively, this is K-way softmax classification distribution over all encoded candidates. Assume there least one candidate that the anchor a matches. Then we know the ground-trutt categorical distribution p is:
and and all have been E normalized.
p: = match(a,
2j=1 match(a.b,)
where the indicator function mmach((--)) indicates whether the anchor and candiate match. Then the multi-positive contrastive loss is the cross-entropy between the ground-truth distribution p and the contrastive distribution q:
K = H-H(p.)) = Σ
pi log qi
(4)
==1
This is generalized form of the widely- used single-positive contrastive loss [54], where p reduces to one-hot vector. This loss is closely related to that in [39], but a key distinction here is that we have no image class labels, and only assume im- ages generated from the same caption are matched.
Algorithm Multi-Pos CL: PyTorch-like Pscudocode
Fi encader: backbone pro. m1p tau: cumperature
minibatch x: [n, m, 3, h, w]
cap:ions, images per caption for x Loader:
augment (x)
cat (unbind din-1)) [nsm, 3, w] f (x)
The PyTorch-like like pseudocode of the batched multi-positive contrastive learn- ing algorithm is described in Algo. images, meaning that we sample m images for each of the captions. Here still ap- ply data augmentation, even though im- ages from the same caption different. This is to reduce overfitting since we per- form many epochs of training over pre- generated synthetic images. However, if in the future the image generator ca- pable of producing images fast enough, then can draw batches online and data augmentation may not be necessary. The multi-positic contrastive learning algorithm is also generic such that SimCLR can also be described by it we begin by randomly selecting a set of n images and subsequently apply m (set as 2) crops to each of the chosen images. However, our StableRep we only utilize single crop from each image.
compute ground-truth distribution label = range (n * ml &
(label. (label.viek)) view 1) label. viewtl, -111
Each batch consists of
p. rill_ dig.ddaagaaaaa + self maaking
/- p.sum
compute contrastive distribution logi:s - h.I / tau logizs fill folfffiiiiggman _diagonal (-1e9) # self masking aoftmax (logits, dim-1)
(p, backward
def H (p, q) t cross-ent.ropy
return (p * lag mum (1) .nean (1
Notes: h.T is h's transposc. The &z normalization operator included in the encoder f.
Experiments
We perform StableRep pre-training on synthetic images synthesized from texts in the CC3M (2.7 million samples) [71], CC12M (10 million) [9], or RedCaps datasets (11.6 million) [16]. We then evaluate the frozen representations by (1) linear probing on ImageNet- 1k and other smaller scale image classification benchmark, and (2) few-shot image recognition that measures the generalization
ability of the representations.
Backbone. We use VIT models [18] as the backbone for our approach StableRep. On top of the CLS token, we apply a 3-layer MIP projection head with hidden layers of 4096 dimensions and an output of 256 dimensions. Batch Normalization [33] is used in this projection head.
5

---

Training, In most of our experiments, we adopt batch size of 8192 images (i.e. m 8192). This way the computation of each batch is equivalent SimCLR with a batch size of 4096, because each image in SimCLR has two crops. We use AdamW optimizer [46] with a learning rate of 0.0032 and weight decay of 0.1, and set $1, 82 as 0.9, 0.98 respectively. We pre-generate 10 images for each text prompt. In each iteration, we randomly sample 6 out of the 10 for each sampled caption to form the training batch, i.e., m = 6 in Algo. I. Recall that for SimCLR m 2. As a result, one epoch training of StableRep is computationally equivalent to 3 epochs of SimCLR. To provide easy comparison, we report SimCLR-equivalen equivalent epochs for StableRep in all of our analysis.
4.1 Main results on CC12M and RedCaps
In this section, perform StableRep on images synthesized by either CC12M RedCaps. For StableRep. we first removed duplicate captions from each dataset, resulting in reduced number of captions: from 10.0M to 8.3M for CC12M and from 11.7M 10.5M for RedCaps. We compared StableRep to SimCLR, which was trained on either synthetic or original real images. We also included CLIP with synthetic and real version 2. For SimCLR and CLIP, we did not perform de-duplication for either real or synthetic setting. We train for 35 epochs for all methods using V (for StableRep, this refers 35 SimCLR-equivalent epochs)) We observed that CLIP started to overfit around 30 epochs. But StableRep did not overfit with this schedule (see Table 6c for results with longer training). For StableRep, we additionally apply random downsample augmentation (see Appendix A.1 for details and how such downsample affects different methods).
ImageNet. Table presents the results of linear probing ImageNet. For StableRep, we prepend BatchNorm layer without affine transformation to the lincar classifier (see Appendix A.5 for more details). We observed that training SimCLR synthetic images yields improvement of 2.2% top-1 accuracy on CC12M and 1.0% on RedCaps when compared to real images. However, the accuracy of CLIP drops by 2.6% on CC12M and 2.7% RedCaps when trained on synthetic images (see Section 5 for more discussion). On the other hand, our method StableRep outperforms CLIP trained on real images, with improvements of 3.2% and 2.6% for CC12M and RedCaps, respectively.
Real
Syn
Real
Syn
SimCLR CLIP SimCLR CLIP StableRep
SimCLR CLIP SimCLR CLIP
StableRep
acc.
61.5
70.3
63.7
67.8
73.5
acc.
61.8
71.9
62.8
69.2
74.5
(a) CC12M
(b) RedCaps
Table Comparison under the linear prohing protocol on ImageNet [15]; measuring top-1 accuracy on a frozen pre-trained backbone. We compare our StableRep with SimCLR [10] and CLIP [58] with either synthetic or real images, CC12M [9] and RedCaps [16]. All models are pre-trained with 35 epochs using VIT-B/16 [18].
Linear classification on more datasets. We followed the approach of SimCLR [10] and BYOL [25] to assess the generality of our learned representations across different image domains. Specifically, we performed linear classification on 11 image classification datasets introduced by [40]. The results reported in Table 2, and the relative performance is consistent with that ImageNet. Notably, our proposed method, StableRep, achieves the highest accuracy on all of the 11 datasets.
CIFAR-10
CIFAR-100 100
Aireraft
Flowers
SUN397
Caltech-10r 101
Food 101
VOC20N7
Average
Can
DTD
Peta
Real
SimCLR
88.3
70.3
47.1
45,5
76.2
92.5
70.1
65.4
83.8
75.0
81.2
72.3
CLIP
94.0
79.0
53.2
75.8
75.7
96.0
86.7
72.5
92.7
81.6
86.1
81.2
SimCLR 84.8
65.2
51.0
53.2
74.5
93.3
74.2
65.0
81.7
74.8
81.8
72.7
Syn
CLIP
87.3
69.5
53,5
79.5
75.8
95.4
85.8
69.2
90.9
78.3
84.5
79.1
StableRep
96.2
84.1
58.3
80.9
78.1
97.2
87.5
73.0
94.6
83.6
87.2
83.7
Table 2: Linear probing experiments on image datasets from various domains. Pre-training is conduceted on CC12M, with either synthetic or real images. Best results for each dataset are highlighted with bold.
We verified our CLIP implementation by comparing to prior work [51] on CC12M. With VIT-B/16, our CLIP achieved 40.2% zero-shot and 70.3% linear accuracy on ImageNet (v... 36.0% and 69.0% in [51]).

---

CIFAR-10 -1
CIFAR- 100
Airozaft
Flowrs
SUN397
Caltech- 10]
Food. 101
Average
Car
DTD
Pass
Real
SimCLR
64.0
70.4
40.7
50.9
82.2
92.1
74,4
94.0
90.4
70.4
73.0
CLIP
77.5
82.1
62.0
90.9
83.3
97.6
91.1
97.2
98.2
87.0
86.7
SimCLR
50.0
58.9
45.2
54.2
79.8
92.0
74.6
92.9
89.1
71.0
70.8
Syn
CLIP
63.1
73.5
61.3
92.5
81.7
96.9
91.5
96.7
96.8
82.5
83.7
StableRep
92.3
91.8
62.6
91.8
86.4
98.2
91.7
97.3
98.8
87.3
89.8
Table 3: Few-shot experiments, dataset are highlighted with bold.
We report 5-way, 5-shot classification performance. Best results each
Few-shot image classification. Prior work [80, 83, 17] has shown that representation learning is the key for few-shot image classification. A simple classifier on top of frozen representation is sufficient achieve strong results. We perform 5-way, 5-shot classification following the setup in [83, 19]. As shown in Table 3, StableRep stands out on 9 out of the 10 datasets.
MAE
StableRep
INIk, Real
cc12m, 35ep
cc 121 105ep
redcaps, 35ep redcaps, 105ep
mloU
48.1
48.8
49.4
47.3
48.4
Table 4: ADE20K semantic segmentation (mloU) using UperNet. StableRep models are trained by 35 105 SimCLR-epee LunCLR-equivalent epochs.
Semantic segmentation. We fine-tune pre-trained StableRep models on ADE20k [91] using Uper- Net [86]. For this evaluation, StableRep is pre-trained for 35 105 epochs. Table shows that StableRep trained on synthetic data able to outperform MAE trained on the real ImageNet images, despite StableRep has no masked image modeling which benefits dense prediction tasks.
4.2 Ablation analysis
For simplicity, ablation studies in this section do not use the random downsample augmentation in pre-training or prepend an extra BatchNorm layer the linear classifier.
The design choice m (number of synthetic images per caption) is one the key design choices for approach. Therefore study the following two factors relevant to CC3M captions (2.7 million after de-duplication).
8
10
772
2
8
10
acc.
61.2
64.2
65.6
66.0
66.2
66.2
acc.
60.5
68.7
69.6
69.6
69.8
69.5
(a) Given generation budget Т, use T/I cap- tions and generate ( iimagss per caption. When = 1, we train SimCLR model.
(b) Given hatch size of C, we form each batch by sampling C/m captions and m images per caption. We "abuse"
to represent SimCLR.
Table 5: Ablation experiments CC3M. ImageNet linear probing results with design choices relevant data generation parameter I, and batch sampling parameter m.
Fixed generation budget. We first study the question: given a fixed value for the number of total synthetic images generated (T), should we generate more images per caption (l), and therefore use fewer captions (T/l) or the reverse. We assume image budget of T 2.7 million. During training, we use the same total batch size (8192) for all I, and set the sampling parameter m as l. Table 5a presents the results. There is clear benefit of generating more than 1 image per caption, e.g., / 8 improves over I = by 4.8%. But this benefit saturates around = 10. We thus generate 10 images per caption for our final experiments.
How to form the batch. Suppose we have generated 10 images for each of the 2.7 million captions. Now given fixed batch size, i.e., m C (recall that n is the number of captions, is the number of images per caption, inside each batch), a larger m encourages stronger invariance of images from the same caption, while larger incorporates more negatives and thus encourages better separability

---

74
73.
7
76
40.0
5x Less
40.2
Imagetett zero-shot AcC.
Synthetic
35.0
23.4
Real
70
69.3
50.1
ImageNet.
66
25.0
2.9
35.0
63.2
StableRep+
22.6
20.0
60.6
Real CLIP
2
3
10
1
10
20
50
guidance scale
Number Captions (M)
Figure 5: ImageNett zero-shot accuracy with different Stable Diffusion generation guidance scale w, using CLIP as pre-training.
Figure 6: ImageNet linear probing accuracy compari- son hetween StahleRept on synthetic images CL.IP on real images on LAION subsets. For this experiment, only 2 images generated for each caption.
of representations. To study this trade-off, we vary the sampling parameter m from 2 to 10 while keeping C/ /.m. As shown in Table 5b, The linear probing accuracy similar between 4 and m = 10 (peak at m = with 69.8% accuracy), showing the robustness of StableRep w.r.t. m. We choose 6 as our default setup. We "abuse" m to represent SimCLR.
After the above study, we continue to ablate the following factors on CC12M and RedCaps.
Case
IN
avg.
Size
IN
avg.
Epochs
CC12M RedCaps
small
72.8
82.2
VIT-B/16
72.8
82.2
72.8
73.7
large
70.8
80.4
VIT-L/16
74.7
82.9
70
75.0
76.3
mixed
71.9
81.5
105
75.7
76.7
(a) Guidance scale u. Smaller w (b) Model size. Our approach yields better linear accuracy.
(c) Training epochs. Longer train- ing further improves accuracy.
scales up with model size.
Table 6: Ablation experiments by pre-training on CC12M or RedCaps. We report linear probing accuracy on ImageNet (IN) and/or average accuracy over the 11 fine-grained classification datasets (avg.). The colored cell indicates the default setup each dataset: ViT-B/16 trained for 35 epochs with small guidance scale
Guidance score for training. We consider three configurations for the classifier free guidance scale w: (1) large scale w {8, 10); (2) small scale E 3); (3) mixed scale (2,3, 4,5, 5, 6, 8, 10, 12). As shown in Table 6a, small scale gives the best linear transfer accuracy on ImageNet and fine-grained classification datasets. This is possibly because smaller w leads to larger intra-caption variation between generated images, which enforces StableRep to learn stronger invariance. This is different from SimCLR which requires larger (recall Section 2. SimCLR only models intra intra-image invariance and thus higher image quality (larger w) helps more.
Model scale. We switch the backbone architecture to ViT-L/16. Table 6b presents the results. The accuracy improves by 1.9% on ImageNet linear probing and 0.7% on the average over fine- ane-grained classification datasets. We found that pre-training with ViT-I. was unstable. The loss kept exploding to NaN, and we resumed from the checkpoint before NaN. But this led to a higher convergent loss than ViT-B (VTT-L loss is lower before exploding). This may partly be due to the usage of BatchNorm.
Longer training. To investigate the scaling behavior StableRep w.r.t training compute, we further increase the pre-training computation budget to 2x and 3x epochs, and report the linear probing accuracy on ImageNet in Table 6c. The results indicate that StableRep scales well with longer training, e.g., improving by 2.2 for 2x and 2.9 for 3x CC12M pre-tral mre-training, and by 2.6 for 2x and 3.0 for 3x on RedCaps pre-training.
Adding Language Supervision
How would training CLIP using synthetic images work? We study this question by generating a copy (one image per caption) for each guidance scale in 21, 3, 4, 6, 8, 10] and training CLIP using

---

Real caption
Synthesized Image
Real image
Figure 7: Examples of misalign- ment between input text and synthe- sized image, which can lead to subop- timal performance for CLIP trained synthetic images. Upper: require headhammer shark hut Stable Diffu- sion often generates sharks without headhammer; Lower; "Andrex Pup- pies" is brand of toilet rolls.
"Hand drown textured vintage lobel, retro badge hammerhead vector Illustration and inspirational lettering."
BEARD
"Andrex Puppies On Roll
Andrex
each copy. Figure 5 plots the zero-shot ImageNet accuracy. Contrary SSL methods, CLIP favors lower w. With the optimal w = 2, CLIP achieves 34.90 4.9% zero-shot accuracy. This is 5.4% lower than training real images (40.2%). Such gap may be explained by misalignment between the generated images and the input text, shown in Figure 7. This is especially true for fine-grained classes.
We can add language supervision to StableRep by adding 0.5 (Liz + L12s) to StableRep loss, where Lin,, Li2 are image-to-text and text-to-image contrastive losses described by Eq. 4. Adding supervision improves StableRep from 72.8% to 74.4% CC12M and from 73.7% to 75.4% on RedCaps for ImageNet linear probing. We term StableRept. We then further scale StableRept to randomly selected 50M subset of LAION-400M [70]. For this experiment, we only generate 2 images per caption with w' = 2, and train CLIP with real images and StableRep+ with synthetic images using different scales of random subsets of the 50M data. We plot the results in Figure 6. StableRep+ consistently achieves better accuracy than CLIP. Notcably, StableRep+ with 10M captions outperforms CLIP with 50M captions, yielding 5x time caption efficiency (2.5x image efficiency).
5.1 Fairness and compositionality
We further study the fairness and compositional understanding of the learned models on FairFace [37]
and ARO [89] benchmarks, respectively. The results presented in Table 7.
FairFace
ARO relation acc.
pre-train data
mean acc.
best-class acc. worst-class acc.
redcaps c2m
CLIP
Real
28.2
60.2
0.3
46.4
Syn
30.4
64.0
3.1
50.0
StableRep+
Syn
37.2
74.9
10.0
47.3
CLIP
Real
9.3
31.1
0.4
59.0
Syn
22.3
52.4
1.0
56.0
StableRept
Syn
27.3
64.4
2.1
52.3
Table 7: Results of fairness and compositionality evaluation.
Fairness.
We perform zero-shot classificaton on FairFace. We jointly classify both races and genders, different classes the same
e.g., treating Black male, Black female, Indian female, and time. For ec cc12m models, CLIP with real data only achieved 0.3% accuracy with Southeast Asian male class, and CLIP wth synthetic data improves this class to 3.1%, while StableRept furthers it to 27.2%. For redcaps models, real CLIP only has 0.4% accuracy for East Asian Male, while StableRep+ improves this class to 22.8%. In summary, training with synthetic data is able to improve the worst class accuracy. However, a obvious geographic bias still exists all models.
The results of compositionality evaluation are less clear. While training with synthetic data on cc 2m slightly improves the relational understanding, an accuracy drop is observed in models trained with synthetic data redcaps. An in-depth investigation may be further needed.
Compositionality.
6 Related Work
Text-to-Image generative models. Text-to-image models trained large image and text pairs have recently enabled the creation of rich and diverse images encompassing many genres and themes [7, 61, 67, 88). The resulting creations have become sensation, with Stable Diffusion having

---

millions of downloads and many tools for image manipulation built on top [66, 38, 90]. Most of these models are built on denoising diffusion models [31. 73] with some notable exceptions [8. 7]. In this paper, we leverage this latest generation of diffusion- based pre-trained generative models for the task of representation learning.
Visual representation learning, Early approaches for visual representation learning often relied on pretext tasks such as inpainting [56] to train image encoders. More recent advancements have shown that mask image modeling, a form of self-supervised training, can be highly effective. In particular, Masked Autoencoder (MAE) [26] has demonstrated significant improvements in downstream fine- tuning performance. Another line of research focuses on contrastive learning, which aims to learn visual representations by maximizing agreement between two augmented views of the same image while distinguishing it from negative examples [10. 78, 54, 84, 27, 79J. Mcanwhile CLIP [58] and its subsequent works [51] leverage contrastive learning to train image representations using language supervision, leading to impressive transferability across various tasks.
Learning from synthetic data. It has been common train machine learning models with synthetic data in different domains [72, 81, 14, 63, 44, 64, 50, 43, 76, 87, 29, 49]. In computer vision, synthetic images have been used as source for training models, such as optical flow [48, 23], autonomous driving [1], semantic segmentation [12, 62], object detection [65, 57], human pose estimation [82, 34] or classification [2. 69, 28]. The closest set of work are the ones that conduct representation learning on synthetic images [60, 45, 4, 35). In [60]. a model is trained to perform multi-task learning on synthetic images. The main method in [45, 4, 35] is to manipulate the latent variable of deep generative models [45, 4, 35] or image generation procedures [4], to form meaningful synthetic images for their representation learning methods. Our method falls into this category, but we use text-to-image diffusion models, which have also been explored by [2, 28, 69]. The key difference is that they conducted supervised learning while we use synthetic data for pre-training representations.
7 Conclusion, Limitations and Broader Impact
We have shown that solely synthetic data generated from state of the art text-to-image models can be used to train powerful visual representations. By harnessing the stochastic nature of Stable Diffusion in combination with a multi-positive contrastive loss, our approach yields a representation that surpasses the performance achieved through training on real data alone. Through series of experiments, we establish that pre-training with synthetic datasets of varying scales yields impressive results across different downstream tasks, including linear probing and few-shot classification. Interestingly, we discover that even vanilla eeffsuuprviiee methods trained on synthetic data can either outperform or achieve comparable results to those trained on real data.
Despite demonstrating the potential of training with synthetic data, this paper acknowledges its limitations. Firstly, we have yet to comprehend the reasons behind the effectiveness of training self-supervised methods on synthetic images compared to an equal amount of real images. It is possible that this observation is confined our particular evaluation methodology. Furthermore. the current image generation process remains slow, with approximately 0.8s per image on A100 GPU or 2.2s per image on a V100 GPU while xFormers is enabled. Consequently, we are not able to train StableRep models with non-repetitive images synthesized online. Additionally, we have not addressed the issue of semantic mismatch between the input prompts and the generated images, which may impact the quality and usefulness of the synthetic data. Moreover, synthetic data has the potential to exacerbate biases due to mode collapse and a predisposition to output "prototypical" images. Lastly, image attribution becomes challenge when working with synthetic data.
Broader impacts. This paper focuses on the fundamentals of visual representation learning, and we believe it will be beneficial to the practice of this field. Our method presents an immediate application by reducing the reliance on collecting a vast amount of real images for learning representations. This approach brings potential benefits in terms of effeciffecss and minimizing biases introduced through human collection and curation processes. However, is important to acknowledge that our method relies on text-to soxt-to-image generative models trained on large-scale, uncurated web data. Such data may conceal social biases and errors that would have been exposed through human curation. Additionally, we must recognize that the text prompts we employed are not completely bras-free; the selection of prompts influences the synthesized images. Thus, the choice of prompts assumes a role similar to the selection of real images for self- visual representation learning.
10

---

Acknowledgements
We would like to thank anonymous reviewers and Shumeet Baluja for reviewing our manuscript and providing many helpful comments and suggestions. We also appreciate the helpful discussions and the general supports from the VisCam teammates in Google Research.
References
[1] Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, and Carsten Rother. Augmented reality meets computer vision: Efficient data generation for urban driving scenes. IJC,, 2018. 10
[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David Fleet. Synthetic data from diffusion models improves imagenet classification. arXiv preprint AXXi::2304.08466 2023. 10
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0132. 2022. 2
[4] Manel Baradad Jurjo, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. In NeurlPS, 2021. 10
[5] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. Food-101- minning discriminative components with random forests. In ECCV, 2014. 19
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
4, 16, 18
[7] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- -suan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint vrXiv:2301.00704 2023. 2, 9,
10
[8] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022. 10
[9] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. 2,
5,
[10] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. In ICML, 2020. 3, 6, 10, 16, 17, 18
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training supe-supe vision transformers. In ICCV, 2021. 3, 4, 16, 18
[12] Yuhua Chen, Wen Li, Xiaoran Chen, and Lue Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach. In CVPR, 2019. 10
[13] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures the wild. In CVPR, 2014. 19
[14] Yabo Dan, Yong Zhao, Xiang Li, Shaobo Li, Ming Hu, and Jianjun Hu. Generative adversarial networks (gan) based efficient sampling of chemical composition space for inverse design of inorganic materials. NPJ Computational Materials, 2020. 10
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2, 6
[16] Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint a1X1v.2111.11411, 2021. 2, 5,
11

---

[17] Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv preprint arXiv:7909.02729) 2019.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint PrXiv:2010.11922 2020. 5, 6
[19] Mohamed El Banani, Karan Desai, and Justin Johnson. Learning visual representations via language-guided sampling. In CVPR, 2023. 7, 18
[20] Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? In CVPR, 2021. 18
[21]
Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010. 19
[22] Li Fei-Fei, Robert Fergus, and Pietro Perona. One-shot learning of object categories. TPAMI, 2006. 19
[23] Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazıraa, Vladimir Golkov, Patrick Van der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. arXiv preprini
2015. 10
[24] Athinodoros Georghiades, David J Kriegman, and PN Belhurneur. Illumination cones for recognition under variable lighting: Faces. In CVPR, 1998. 2
[25] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to supervised learning. In NeurlPS, 2020. 3,4, 6, 16, 17, 18
[26] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022. 3, 10
[27] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv:1911.0577 2019. 10
[28] Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.0757. 2022. 10
[29] Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi. Generate, annotate, and learn: Nlp with synthetic text. TACL, 2022. 10
[30] Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford Neal. The "wake-sleep" algorithm for unsupervised neural networks, Science, 1995. 2
[31] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 2, 10
[32] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022. 3
arXiv preprint
[33] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML. 2015. 5
[34] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, 2013. 10
[35] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning, arXiv preprint
2021. 2, 10
12

---

[36] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. arXiv preprint arXiv:2303.055/1 2023. 2
[37] Kimmo Kärkkäinen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint 190i::1900.04913, 2019. 9
[38] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani, Imagic: Text-based real image editing with diffusion models. arXiv preprint 100002210.
2022. 10
[39] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv:2004.1333 2020.
[40] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In CVPR, 2019.
[41] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshop, 2013. 19
[42] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech Report, 2009. 19
[43] Varun Kumar, Ashutosh Choudhary, and Eunah Cho. Data augmentation using pre-trained transformer models. arXiv preprint arXiv:2003.02225,, 2020. 10
[44] Jason Li, Ravi Gadde, Boris Ginsburg, and Vitaly Lavrukhin. Training neural speech recognition systems with synthetic speech augmentation. arXiv preprint 10X0v:111.000700 2018. 10
[45] Hao Liu, Tom Zahavy, Volodymyr Mnih, and Satinder Singh. Palm up: Playing in the latent manifold for unsupervised pretraining. arXiv preprint 1rXiv:2210.109111 2022. 10
[46] lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6
[47] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint
2013, 19
[48] Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR, 2016. 10
[49] Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. arXiv preprint 22Xx:22002.05388.. 2022. 10
[50] Masato Mimura, Sei Ueno, Hirofumi Inaguma, Shinsuke Sakai, and Tatsuya Kawahara. Leverag- ing sequence-t-sequueee speech synthesis for enhancing acoustic-to-we -o-word speech recognition. In SLT, 2018 10
[51] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In ECCV, 2022. 6, 10, 18
[52] Sameer A Nene, Shree K Nayar, Hiroshi Murase, al. Columbia object image library (coil-20). Tech Report, 1996.
[53] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing. 2008. 19
[54] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arX.v:1807.03748, 2018. 5, 10
13

---

[55] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012. 19
[56] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue. Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016. 10
[57] Xingchao Peng, Baochen Sun, Karim Ali, and Kate Saenko. Learning deep object detectors from 3d models. In CVPR, 2015. 10
[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021. 2, 6, 10, 18, 19
[59] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0612. 2022.
[60] Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-task feature learning using synthetic imagery. In CVPR, 2018. 10
[61] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 1, 9, 20
[62] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016. 10
[63] Andrew Rosenberg, Yu Zhang, Bhuvana Ramabhadran, Ye Jia, Pedro Moreno, Yonghui Wu, and Zelin Wu. Speech recognition with augmented synthesized speech. In ASRU, 2019.
[64] Nick Rossenbach, Albert Zeyer, Ralf Schlüter, and Hermann Ney. Generating synthetic audio data for dreention-based speech recognition systems. In ICASSP, 2020. 10
[65] Artem Rozantsev, Vincent Lepetit, and Pascal Fua. On rendering synthetic images for training an object detector. Computer Vision and Image Understanding, 2015. 10
[66] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, arXiv preprint arXiv:2208. /2242, 2022. 10
[67] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022. 2, 9
(68] Ferdinando S Samaria and Andy C Harter. Parameterisation of stochastic model for human face identification. In IEEE workshop applications of computer vision, 1994. 2
[69] Mert Bulent Sariyildiz, Karteck Alahari, Diane Larlus, and Yannis Kalantidis. Fake till you make it: Learning transferable representations from synthetic imagenet clones. In CVPR, 2023.
10
[70] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk. Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Loion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021. 9
[71] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACI., 2018. 3,5
[72] David Silver, Julian Schrittwisser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 2017. 10
[73] Jaseha vised learning using nonequilibrium thermodynamics. In ICML, 2015. 2, 10
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper-
14

---

[74] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02550, 2020. 20
[75] Richard Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 1991. 2
[76] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Perey Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-oolloiing model. Stanford Center for Research Foundation Models., 2023. 10
[77] Yonglong Tian, Olivier J Henaff, and Aäron van den Oord. Divide and contrast: Self-supervieed learning from uncurated data. In ICCV, 2021. 3
[78] Yonglong Tian, Dilip Krishnan, and Phillip Isola. arXiv: 05849, 2019. 10,
Contrastive multiview coding.
[79] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NeurlPS, 2020. 10
[80] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need? In ECCV, 2020. 7
[81] Allan Tucker, Zhenchen Wang, Ylenia Rotalinti, and Puja Myles. Generating high-fidelity synthetic patient data for assessing machine learning healthcare software. NPJ digital medicine, 2020. 10
[82] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, 2017. 10
[83] Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Re- visiting nearest-neighbor classification for few-shot learning. arXiv preprint arXiv:1911.04623 2019.
[84] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. 10
[85] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 19
[86] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. ECCV, 2018.
[87] Yiben Yang. Chaitanya Malaviya. Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji- Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey, Generative data augmentation for commonsense reasoning. arXiv preprint arXiv:2004.1155 115460 2020. 10
[88] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022. 2, 9
[89] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In ICLR. 2022. 9
[90] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05553, 2023. 10
[91] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. ISCV, 2019. 7
15

---

A Implementation Details
A.1 Downsample augmentation
Synthetic images have constant high resolutions (e.g., 512x512 for Stable Diffusion). We find this leads to domain gap when transferring to situations involving low resolution images, such as CIFAR- 10 CIFAR-100. To address this issue, we introduce Random Downsample augmentation, which randomly resizes images to a resolution of 64 or 128 (equally probable) and then resizes them back to 224. During pre-training, we apply this augmentation with probability of 0.05 and prepend it to other augmentations.
In Table 8, we ablate the effects of applying this random downsample augmentation to different pre- training methods. This augmentation brings significant improvements on CIFAR-10 and CIFAR- 100 datasets, while maintaining the performance on other datasets. On average this augmentation is more beneficial for pre- trainini with synthetic images than real ones.
Dowasample
CIFAR-100
Calech 101
CIFAR-10
Aireraft
Flowers
SUN397
Food-10 101
Average
Cars
DTD
Pets
SimCLR
70.3
47.1
45.5
76.2
92.5
70.1
65.4
83.8
75.0
81.2
72.3
Real
SimCLR
V
92.3
75.4
47.8
44.4
77.1
91.8
69.2
65.1
84.7
74.9
81.2
73.1 (+005)
CLIP
94.0
79.0
53.2
75.8
75.7
96.0
86.7
72.5
92.7
81.6
86.1
81.2
CLIP
95.8
82.9
51.5
76.5
74.7
95.3
87.2
72.5
92.7
81.7
86.2
81.5 (4-0.3
SimCLR
84.8
65.2
51.0
53.2
74.5
93.3
74.2
65.0
81.7
74.8
81.8
72.7
SimCLR
92.6
76.2
50.4
53.2
74.7
93.3
74.7
64.8
86.0
74.7
81.2
74.7 (+220)
Syn
CLIIP
87.3
69.5
53.5
79.5
75.8
95.4
85.8
69.2
90.9
78.3
84.5
79.1
CLIP
93.4
78.8
52.4
79.6
75.1
95.0
85.0
69.5
90.9
78.4
84.7
80.3 (+1.2)
StableRep
90.7
74.4
57.6
80.3
79.0
96.7
87.1
73.2
94.0
83.5
87.2
82.2
StableRep
v
96.2
84.1
58.3
80.9
78.1
97.2
87.5
73.0
94.6
83.6
87.2
83.7 (+1.5)
Table 8: We ablate the effects of Random Downsamp) augmentation lincar probing benchmarks from various domains. This augmentation brings significant improvements on CIFAR-10 and CIFAR-100 datasets, while maintaining the performance on other datasets.
A.2 Standard self-supervised learning
We follow the default settings for standard self-supervised learning algorithms, and present the training details in Table 9 and Table 10. We use the linear (r scaling rule: Ir = base_rr x bs2/256. For BOL [25], we did not follow the hyperparameters (blr 1.0e-4, wd 0.03) in [11], found our setting here yielded better accuracy. For DINO [6], we did not use the multi-crop strategy and only pre-trained the model with two 224x224 crops.
config
MAE Adam 1.5e-4
SimCLR
optimizer
AdamW
base learning rate
2.0e-4
weight decay
0.05
0.1
optimizer momentum batch size learning schedule epochs
B1 (2=0.99 0.95
81, 82=0.9 0.98
4096
4096
cosine decay
cosine decay
300 (cc3m) / 80 (cc12m)
100 (cc3m) 35 (cc 12m)
warmup epochs
10 (cc3m) / 4 (ce 12)
(cc3m) 1 (ccl2m)
augmentation
RandomResizedCrop. Flip
SimCLR Aug. [10]
Table 9: Self-supervised pre-training settings. MAE and SimCLR.
16

---

config
DINO
BYOL/MoCo-V3
optimizer
AdamW W 5.0e-4 0.04 to 0.4, cosine 31 82099, 0.999 4096
AdamW
base learning rate
1.5e-4
weight decay
0.1
optimizer momentum batch size learning rate schedule epochs
81, 01,92=0.9,0.88 0.95
4096
cosine decay 100 (cc3m) / 35 (ccl 2m)
cosine decay 100 (cc3m) / 35 12ml
warmup epochs
(cc3m) 2 (ce 12m)
(cc3m) 2 (cc12m) 12m)
momentum update X
0.996 to I, cosine
0.996 to 1, cosine
augmentation
BYOL Aug. [25]
BYOL Aug. [25]
teacher temp.
0.04 to 0.07 in warmup
student temp.
0.1
Table 10: Self-supervised pre-training settings. DINO, BYOL and MoCo v3.
A.3 StableRep pre-training
The hyperparameterss for StableRep is presented in Table 11. Indeed, they are the same as that in SimCLR. The difference is that the Mase_rrii in StableRep is for 512 images while in SimCLR it is for 256 images, because each image StableRep only has one single crop. We ended up using a batch size of 8256 images, since trained model with 32 GPUs and 8192 is not divisible over 3 The computation for StableRep has been converted to SimCLR-equivalent epochs.
config
StableRep
batch size
8256 (m AdamW 2.0e-4
1376)
optimizer
base learning rate
peak learning rate
base Ir x bsz/512 0.1
weight decay optimizer momentum learning rate schedule epochs
B1 0.9. 0.98 cosine decay
35 70 105
warmup epochs augmentation
1.2 / 2.3 3.5
Downsample Aug. + SimCLR Aug. [10]
Table 11: StableRep pre-training settings.
A.4 CLIP training
config
CLIP
batch size
8192
optimizer
Adam W
peak learning rate
le-3
weight decay
0.5
optimizer momentum
B1 00.9,,9900000 0.98
learning rate schedule
cosine decay 35
epochs
warmup epochs
augmentation
RandonnRsiceCCCppcccl0000.,, 1.0))
Table 12: CLIP training settings.
17

---

Patch
Input
Embedding
Vision Transformer
Text Transformer
Vocab
Text
Model
size
resolution
dimension
Layens
Width
Heads
Layers
Width
Heads
size
length
VIT-B/16
16
224
512
12
768
12
12
512
8
49,408
77
Table 13: CLIP encoder details.
We follow the hyperparameter setting used in [51] since is better than that from the original CLIP [58] paper. Table 12 summarizes the training details, and Table 13 presents the architecture of CLIP encoders. With this training setup, able produce 40.2% ImageNet zero-shot accuracy when training CLIP on CC12M dataset. As a comparison, [51] reports 36.0% using the same architecutre.
A.5
ImageNet linear probing
We follow prior work [11, 6] to train the linear classifier. has been generally observed that regular- ization such weight decay hurts the performance [78]. Following [78, 11], we weight decay as O, and only use RandomResizedee
and RandomHor wetoorrioonntteee zontalFlip as data augmentation.
We sweep the base_Ir Ir over 0.21, 0.5, 1, 5, 10, 20, 50) 10-2.
config
value
batch size
1024
optimizer
SGD
base learning rate
sweep 0
weight decay
optimizer momentum learning rate schedule epochs
0.9
cosine decay
90
augmentation
RandomResizedCrop. Flip
Table 14: ImageNet linear probing settings.
For StableRep trained with 35 epochs, find that adding extra BatchNorm layer without affine transformation improves and stablizes the linear probing results. However, this additional BatchNorm does not help when StableRep is trained with longer schedule, e.g., 105 epochs. We conjecture that BatchNorm is helpful when StableRep is not convergent, and present the comparison in Table 15.
w/ BN
w/o BN
StableRep. 35 epochs
73.5
71.4
StableRep, 105 epochs
75.2
75,4
Table 15: ImageNet linear probing results w/ or w/o extra BatchNorm layer for the linear classifier.
A.6 Fine-grained linear classification
Following [10, 25, 20], fit regularized multinomial logistic regression model top of the frozen CLS token. In training and testing, we do not perform any data augmentation; images are resized to 224 pixels along the shorter side using bicubic resampling, followed by center crop of 224x224. We minimize the cross-entropy objective using L-BFGS with l--regularization. We select this fzrreuulaiiztionn constant on the validation set over logarithmically spaced values between 10-6 and 10 The maximum number of L-BFGS iterations is set to 500.
The details about the fine- grained classification datasets are presented in Table 16.
A.7 Few-shot image classification
Following the settings [20, 19), we evaluate the 5-way 5-shot performance on different datasets. We do not use data augmentation; images resized to 224 pixels along the shorter side using
18

---

Dataset
Metric
Categories
Train Size
Test Size
CIFAR-10 [42]
Accuracy
10
50,000
10,000
CIFAR-I00 [42]
Accuracy
100
50,000
10,000
Aircraft [47]
Mean per class
100
6,667
3,333
Cars [41]
Accuracy
196
8,144
8,041
DTD [13]
Accuracy
47
3,760
1,880
Flowers [53]
Mean per class
102
2,040
6,149
Pets [55]
Mean per class
37
3,680
3,669
SUN397 [85]
Accuracy
397
19,850
19,850
Caltech-101 [22]
Mean per class
102
3,060
6,085
Food-101 [5]
Accuracy
101
75,750
25,250
VOC2007 [21]
Mean per class
20
5,011
4,952
Table 16: Details of fine-grained linear classification datasets.
bicubic resampling, followed by center crop of x224. We report the mean accuracy of 600 randomly sampled tasks (also known as episodes). For each task, images are randomly sampled from the combination of training, validation and testing We sample 15 query images for each class in every task for evaluation purpose.
B
Additional Results
B.1
Fine-grained classification
CIFAR-10 10
CIFAR- 100
Alterutt
Flowers
SUNG97
Caltecii 101
VOC2N07
Average
Cara
DTD
Peta
Food
Pre-traine training on Redcaps for 35 epochs
Real
SimCLR
90.2
72.0
46.8
42.8
77.9
94.6
83.0
61.2
82.7
81.3
80.9
73.9
CLIP
94.2
78.9
52.9
74.9
73.9
97.8
91.6
66.2
91.6
89.2
85.4
81.5
SimCLR
85.1
65.4
48.7
53.7
74.6
95.0
79.6
61.8
84.5
79.7
80.4
73.5
Syn
CLIP
88.7
71.4
53.7
77.3
76.0
96.9
88.2
67.3
90.3
83.7
84.5
79.8
StableRep
96.7
84.6
57.2
78.8
79.0
98.4
90.9
70.7
94.9
88.1
86.6
84.2
Longer training for StableRep
redcaps cmm1mm
35 epochs 105 epochs
96.2
84.1
58.3
80.9
78.1
97.2
87.5
73.0
94.6
83.6
87.2
83.7
96.7
84.7
59.2
83.5
80.1
97.3
88.3
74.3
94.7
85.1
87.9
84.7
35 epochs 105 epochs
96.7
84.6
57.2
78.8
79.0
98.4
90.9
70.7
94.9
88.1
86.6
84.2
96.9
85.6
60.2
83.9
80.0
98.5
91.7
72.5
94.8
89.4
87.8
OpenAl's CLIP trained on WIT-400M (numbers copied from [58])
WIT- 96.2
83.1
59.5
86.7
79.2
98.1
93.1
78.4
94.7
92.8
89.2
86.5
400M
Table 17: Linear transfer results on fine-grained datasets. results are with VIT-B/16. Results StableRep marked Upper: different methods pre-trained on RedCaps. Middle: StableRep with different training schedules. Lower: OpenAl's CL.P trained on WTT-400M dataset. Our StahleRep trained with synthetic images only is approaching the performance of CLIP trained with 400 millions of real images.
In Table 17, we further present the fine the-grained linear classification results by models from RedCaps or models that are trained longer (2x 3x longer). When pre-training on RedCaps, StableRep achieves the best average accuracy. Longer training of StableRep further improves transferability. Notably, our StableRep trained with synthetic images only is approaching the performance of OpenAl's CLIP trained with 400 millions of real images.
19

---

B.2 Few-shot image classification
We further summarizes the few-shot image classification results in Table 18. The 95% confidence interval is provided. StableRep stands out on the majority of the evaluated datasets.
CIFAR-10 10
CIFAR- 100
Airorat
Rowers
SUN397
Caltech 101
Food 101
Average
Cars
DTD
Pots
Pre-training cc12m
Real
SimCLR
64.0a0.7
70.4
40.7a0.9
50.920.
82.2.06 92.1a0s 74,4
94.0a0.4
90.
70.4
73.0
CLIP
77.5 20.7 62.0+1.0
90.9
83.3e06
97.6e02
91.1
97.2am.2
98.2an.2
87.0aas
86.7
SimCLR
50.0tos
58.9
45.2
54.2.0.
79.8
92.0eos
74.6
92.9.0.4
89. 1906
71.0e0.
70.8
Syn
CLIP
13.1
73.5
61.3a1.0
92.5+04
81.7=06
96,9203
91.5a0.5
96.720.2
96.820.3
82.5
83.7
StableRep
92.3 91.8
62.6s1.a
91.8+0.5
86,4 98.2sa.2 91.7aa.5
97.3an.2
98.8 87.3
89.8
Pre-training redcaps
Real
SimCLR 62.320.6
69.4
39.6an.9
51.0a0.s
82.7=06 94.8s04
85,4
91.8a0.5
88.520.6
79.1au)
74.5
CLIP
80.6+a. 85.3+0.5
54.5+0.9
88.5+06
82.6-06 99.0+a. 94.5ee.4
95.9+0.
97.8+0.2
94,4.0.3
87.3
SimCLR
52.9206
60.8
40.920.9
53.2.0.8
79.5.06
94.320.
78.320.7
92.0a0.
88.940.5
75.920.7
71.7
Syn
CLIP
65.7+06
75.7 55.2+1.0
90.1 82.6ea6 98.2+02 92.0ees 96.3+0, 96.940,3 88. +
84.1
StableRep
92.7.0.3 92.9.0.4 57.3a1.0
89.4.0.6 86.2.0. 99.2.0.1 94.5s0. 96.810.3 98.920.2
91.820.
90.0
Table 18:
Few-shot image classification results with 95% confidence interval provided. All models here are trained for 35 epochs. Upper: pre-training on CC12M dataset. Upper: pre-training on RedCaps dataset.
C Image Generation
C.1 Implementation details
We use Stable Diffusion [61] v1.5. During sampling, we generate images by 50 DDIM [74] steps. To accelerate the generation process, we leverage xFormers library for efficient attention computation, which brings down the sampling time ~0.88 per image on a single A100 GPU and 2.33 per image on V100 GPU.
Image resolution. The image resolution may affect the quality of representations learned by self- supervised learning algorithms. We try to make a relative fair comparison by storing all synthetic and real images in similar resolutions. The synthetic images generated by Stable Diffusion are 512x512; we resized them to 256 256 before storing them on the disk. The real images have various sizes, ranging from less than hundred of pixels in shorter side thousands of pixels; we resize the shorter side of all real images to 256.
C.2 Generation examples
Some examples of synthetic images are visualized in Figure
D
Computation
Synthesis. The slowest part of the StableRep pipeline is the image generation. We use 512 V100 GPUs to synthesize images, which takes 13 hours for every ten million images.
Pre-training. Each of our StableRep models with ViT-B/16 is trained on 4 nodes, each of which has 8 A 100 GPUs and 96 CPU cores. It takes 20 hours complete 35 MennL--euui epochs of training on CC12M and 233 hours on RedCaps. For ViT-L/16, we use 64 A A00 80GB GPUs spread over 8 nodes.
20

---

10
12
20
40
60
real
at the
player
Illustration of cal embracing a planet
jumping
Figure 8: Examples of synthetic images. We show examples for 4 different text prompts. For each prompt, we provide examples synthesized with different guidance scale as well as the original real image.
21


Reviews:
[{"review_id": "quVloNICWm", "reviewer": "Reviewer_HH2w", "summary": "This paper studies pre-training by using generated image from diffusion models. It presents StableRep that generate different images with the same caption by using stable diffusion models. The model is hence pre-trained by using the generated samples and contrastive loss. Extensive experiments demonstrate the effectiveness of the proposed method by using synthetic images only.", "questions": "1. For general vision-language pre-training methods, image labels are usually very difficult to obtain. This paper studies the use of synthetic data, where the labels of synthetic images can be obtained during image synthesis. As such, we can simply pre-train the model by generating images together with the labels in the task-specific label spaces, which could improve the model performances [24]. As such, it is not clear why the contrastive pre-training is necessary or better when using synthetic data for pre-training. More discussion and experiments are expected.\n2. According to the experimental results, pre-training on synthetic data helps to improve the performances on most of the datasets while may introduce degradation some of the datasets (e.g. CIFAR-10). How about combining the synthetic and real datasets together for pre-training?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper studies the pre-training by using synthetic data only. Experimental results show that pre-training by using synthetic data only outperforms the pre-training over real images.\n2. This paper is generally clear and well-written.", "weaknesses": "The pre-training over generated images by using diffusion models have been studied in [24]. In [24], it seems direct pre-training on synthetic images could helps to improve the classification results without the proposed pre-training pipeline. As the comparison between the proposed method and the reported results in [24] is missing, it is difficult the determine the effectiveness of the proposed pre-training pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. In the following we address your concern in detail.\n\n> the comparison between the proposed method and the reported results in [24] is missing\n\n\nThis is a reasonable comparison, which we have included in the question 1 of \"global rebuttal\" shared to all reviewers (please find it around the top of the page). \n\nOur StableRep clearly outperforms [24] (and additionally [a]), and even outperforms supervised training using Real ImageNet on both downstream transfer and few-shot evaluation.\n\n> As such, we can simply pre-train the model by generating images together with the labels in the task-specific label spaces, which could improve the model performances [24]. As such, it is not clear why the contrastive pre-training is necessary or better when using synthetic data for pre-training. More discussion and experiments are expected.\n\nFirstly, as shown in aforementioned experiments, our StableRep has higher accuracy than [24], despite that we do not specifically use ImageNet labels.\n\nSecondly, using target label space for synthesis means we have to know the downstream task beforehand. This means we have to synthesize a different dataset and train a different model for each task, which clearly limits the applicability of this approach. In contrast, we are aiming at learning a general purpose representation which hopefully can be easily transferred to a broader range of tasks. The aforementioned comparison between [24] and our StableRep also suggests that our representation has stronger transferability.\n\n> According to the experimental results, pre-training on synthetic data helps to improve the performances on most of the datasets while may introduce degradation some of the datasets (e.g. CIFAR-10)\n\nWe have identified this issue. This is due to the image resolution: cifar-10/100 has very low resolution image (32x32 pixels), while other evaluation datasets have much higher resolutions. To test this hypothesis, we also downsampled the other 9 evaluation datasets to 32x32 pixels before we evaluate. The results and explanations are presented in question 2 of the \"global rebuttal\" (please find it around the top of the page). A fix may be to randomly downsample the synthetic image to very low resolution during StableRep training, which we leave as future work given the limited rebuttal time.\n\n> How about combining the synthetic and real datasets together for pre-training?\n\nThis is a good suggestion, and the results of combining synthetic and real datasets are as below.\n\nFor SimCLR:\n| method | pre-train data | cifar-10 | cfiar-100 | ImageNet LP | Downstream avg | few-shot avg| \n|--|:--:|:--:|:--:|:--:|:--:|:--:|\n| SimCLR | Real | 88.3 | 70.3 | 61.5 | 72.3 | **73.0** |\n| SimCLR | Syn  | 84.8 | 65.2 | 63.7 | 72.7 | 70.8 |\n| SimCLR | Syn+Real | **88.7** | **71.9** | **64.1** | **74.9** | 71.9 |\n\nFor CLIP:\n| method | pre-train data | cifar-10 | cfiar-100 | ImageNet LP | Downstream avg | few-shot avg| \n|--|:--:|:--:|:--:|:--:|:--:|:--:|\n| CLIP | Real | **94.0** | 79.0 | 70.3 | 81.2 | 86.7 |\n| CLIP | Syn | 87.3 | 69.5 | 67.8 | 79.1 | 83.7 |\n| CLIP | Syn+Real | 93.9 | **80.2** | **73.3** | **83.0** | **87.3** |\n\nFor StableRep (StableRep items here only use two views per caption, and are trained with only 15 epochs): \n| method | pre-train data | cifar-10 | cfiar-100 | ImageNet LP | Downstream avg | few-shot avg| \n|--|:--:|:--:|:--:|:--:|:--:|:--:|\n| StableRep (2 views) | Syn | 88.5 | 71.5 | 67.9| 78.0 | 81.4 |\n| StableRep (2 views) | Syn+Real | **92.2** | **76.1** | **69.1** | **79.3** | **84.2** |\n\nWith these results, we conclude:\n- the cifar-10/100 resolution issue causes both CLIP and SimCLR to drop if only trained on synthetic data, but combining both synthetic and real datasets significantly alleviate this issue (and often surpass real-only).\n- combining synthetic and real datasets uniformly improve all methods\n\nWe hope that these discussion and results help address your concern and lead to a favorable increase of the score. Please don’t hesitate to let us know for any additional comments or suggestions.\n\n\n[a] Fake it till you make it: Learning transferable representations from synthetic ImageNet\n\n"}, {"review_id": "PIMdNwssuv", "reviewer": "Reviewer_AnD7", "summary": "This paper investigates how synthetic data generated with the text-to-image diffusion model Stable Diffusion can be leveraged for representation learning.\nTo this end, the paper analyzes established representation learning approaches such as SimCLR and CLIP, but trained on the synthetically generated data. Further, it introduces a novel method,\ndubbed \"StableRep\", specifically designed for representation learning from a generative model, to generate multiple \"positive\" image instances for downstream representation learning.\nExperiments demonstrate the potential of synthetic data for representation learning and mostly perform on-par with or better than when trained on real data only.", "questions": "- Can we mix synthetic and real data? How does this affect the outcome of training approaches like CLIP and SimCLR or StableRep?\n- Other sampling hyperparameters than guidance scale w are not discussed. How does the choice of solver change the results? Same question for the number of denoising steps / function evaluations...\n- small note: In Eq. 1, t could be misinterpreted as timestep variable, as often used in the context of diffusion models.\n- Sec. 4, l. 229-233: Do the instabilities for larger models only occur for synthetic data?\n- Sec. 5, l. 240-245: This section speculates about the inferior results when training CLIP on synthetic data only (vs real data only). Why not plot the imagenet zero-shot accuracy vs clip-image similarity for different guidance scales? Using a better model than SD 1.5 might also help clarify this point.", "limitations": "yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a well-written paper, which approaches a timely research question: To what extent can large-scale, generative models which have been trained on internet-scale data be leveraged for other tasks, by augmenting or replacing real data. The experiments are well-designed and hint at the potential of using synthetic data. StableRep is a nice method specificially designed for generative models that can generate multiple positive examples for a given prompt. The experiments are encouraging for future research in this area.", "weaknesses": "While experiments demonstrate potential, there remain a few unaddressed points: How does the choice of generative model influence the results for representation learning? In particular, Stable Diffusion is conditioned on CLIP text features, which themselves are pretrained through CLIP's contrastive objective. There exist, however, other generative text-to-image variants that use non-contrastive features for conditioning, such as T5-representations (e.g., Imagen [https://github.com/deep-floyd/IF], IF [https://github.com/deep-floyd/IF]).\nFurther, the paper does not discuss how the size of the pretraining dataset for both CLIP (for conditioning SD) and the training data for SD change performance of StableRep (the pretraining data is much larger than CC3M/12M). In addition, the possibility of data poisoning w.r.t to the results presented in Tab. 2 and Tab. 3 is not discussed (for example, SD's training data might have contained examples from the linear probing datasets). The performance gap between CIFAR-10/100 and the other datasets remains unexplained.\nl. 1014-107 need more detail on hyperparameter settings like number of steps, learning rate used, convergence of the model.\n\nFor further comments, please refer to the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging our method is nice and that our experiments are encouraging, and we appreciate the constructive feedback. We address your concerns or questions one by one as below.\n\n> How does the choice of generative model influence the results for representation learning? \n\nThis is a good point, and we provide the results of using DeepFloyd-IF instead. Given that DeepFloyd-IF is slow and rebuttal time is limited, we: (1) only used its stage 1 & 2 models; (2) never tuned its hyper-perameter. We train StableRep with 2 views and and present the results as below:\n\n| method | ImageNet LP |\n|--|:--:|\n| StableRep w/ SD | 65.8 |\n| StableRep w/ IF | 65.5 |\n\nThis suggests StableRep also works with models that are *not* conditioned on CLIP features.\n\n> Further, the paper does not discuss how the size of the pretraining dataset for both CLIP (for conditioning SD) and the training data for SD change performance of StableRep.  In addition, the possibility of data poisoning w.r.t to the results presented in Tab. 2 and Tab. 3 is not discussed.\n\nUnfortunately, we cannot train CLIP and Stable Diffusion (SD) with various different data sizes to ablate this. In terms of poisoning we perform an surrogate sanity check: we compute how close the downstream datasets are to the StableRep's training set. Specifically, for an image $I$ in a downstream dataset, we find its closest image in StableRep's training set by using consine similarity of some pre-trained SimCLR feature extractor $f$ (Recall that SimCLR is optimized to maximize cosine similarity of the same image). We define this highest similarity ($s$) as this image's similarity with StableRep's training set $D$. Concretely, $s_i = \\max_{X \\in D} cos(f(I_i), f(X))$. We evaluate four datasets, and provide the statistics (avg, min, and max) of this highest similarity metric of each image. \n\n| Train Set | stats | aircraft|cars|flowers|food101|\n|--|:--:|:--:|:--:|:--:|:--:|\n| StableRep Train | avg | 0.891 | 0.836 | 0.836 | 0.791 |\n|| min | 0.652 | 0.258 | 0.523 | 0.443 |\n|| max | 0.985 | 0.980 | 0.967 | 0.958 |\n| ImageNet  Train | avg | 0.901 | 0.870 | 0.823 | 0.807 |\n|| min | 0.700 | 0.629 | 0.586 | 0.560 |\n|| max | 0.980 | 0.998 | 0.988 | 0.977 |\n\nThe results show downstream datasets are closer to ImageNet (which has been widely used for pre-training and then evaluating on these downstream datasets) than to StableRep's train set, therefore we believe the training set of StableRep is not poisoned.\n\nPlease refer to our response to reviewer FSjQ for more relevant discussions.\n\n> The performance gap between CIFAR-10/100 and the other datasets remains unexplained\n\nWe have identified this is because CIFAR-10/100 have very low resolution images. More detailed results and explanations are provided in the question 2 of the \"global rebuttal\" (please find it around the top of this page).\n\n> Can we mix synthetic and real data? \n\nYes.\n\nFor SimCLR:\n| method | pre-train data | cifar-10 | cfiar-100 | ImageNet LP | Downstream avg | few-shot avg| \n|--|:--:|:--:|:--:|:--:|:--:|:--:|\n| SimCLR | Real | 88.3 | 70.3 | 61.5 | 72.3 | **73.0** |\n| SimCLR | Syn  | 84.8 | 65.2 | 63.7 | 72.7 | 70.8 |\n| SimCLR | Syn+Real | **88.7** | **71.9** | **64.1** | **74.9** | 71.9 |\n\nFor CLIP:\n| method | pre-train data | cifar-10 | cfiar-100 | ImageNet LP | Downstream avg | few-shot avg| \n|--|:--:|:--:|:--:|:--:|:--:|:--:|\n| CLIP | Real | **94.0** | 79.0 | 70.3 | 81.2 | 86.7 |\n| CLIP | Syn | 87.3 | 69.5 | 67.8 | 79.1 | 83.7 |\n| CLIP | Syn+Real | 93.9 | **80.2** | **73.3** | **83.0** | **87.3** |\n\nFor StableRep (StableRep only trained with 2 views and 15 epochs): \n| method | pre-train data | cifar-10 | cfiar-100 | ImageNet LP | Downstream avg | few-shot avg| \n|--|:--:|:--:|:--:|:--:|:--:|:--:|\n| StableRep (2 views) | Syn | 88.5 | 71.5 | 67.9| 78.0 | 81.4 |\n| StableRep (2 views) | Syn+Real | **92.2** | **76.1** | **69.1** | **79.3** | **84.2** |\n\nCombining synthetic and real datasets uniformly improve all methods\n\n\n> -Other sampling hyperparameters than guidance scale w\n\nThis is a good suggestion, and the comparison are as below:\n\n| method | solver | step | ImageNet LP|\n|--|:--:|:--:|:--:|\n| baseline | DDIM  | 50  | 65.8 |\n|| DDIM | 25  | 64.9 |\n|| DDIM | 100 | 65.8 |\n|| DPM | 50  | 66.0 |\n\nHalving the sample steps brings about 0.9% drop, while doubling the steps has no effect. Switching from the ODE solver DDIM to SDE solver DPM slightly improves over the baseline by 0.2%.\n\n> Do the instabilities for larger models only occur for synthetic data?\n\nNo. Large vision transformers are notoriously unstable during training in general. This paper [a] has more discussion about it. \n\n> This section speculates about the inferior results when training CLIP on synthetic data only (vs real data only). Why not plot the imagenet zero-shot accuracy vs clip-image similarity for different guidance scales?\n\nWe worry that plot w/ clip score can be misleading. The performance of CLIP trained with synthetic images drops when guidance scale increases, and meanwhile clip score increases with guidance scale. Putting togther, it becomes that the higher the clip score, the lower the CLIP performance. This is misleading and only part of the story. We know that guidance scale trades off between data diversity and image-text alignment. While image-text alignment increases with guidance scale, data diversity level reduces a lot. As a result, the disadvantage brought by diversity dropping outweighs the advantage brought by better image-text alignment.\n\nWe are more than happy to discuss more if you have further questions. We hope our response will encourage the reviewer to consider increase the rating if they have addressed your concern.\n\n[a] Scaling Vision Transformers to 22 Billion Parameters\n\n"}, {"review_id": "D9aiEM7RR6", "reviewer": "Reviewer_qM14", "summary": "The authors presents a novel method for learning visual representations using synthetic data. The authors leverage text-to-image generative models (Stable Diffusion) to synthesize images from textual prompts, which are then used to train a self-supervised visual representation model. The synthetic data generation process is guided by a set of diverse and non-repetitive textual prompts, which helps in creating a wide variety of images. The authors demonstrate that their approach outperforms traditional self-supervised learning methods that rely on real images, especially when the amount of real data is limited. They also propose a multiple positive samples based contrastive learning approach called StableRep to utilize different synthetic images generated from a source text prompt, which outperforms training with simply real or synthetic images on a variety of downstream tasks. ", "questions": "My main concern is with the motivation behind using synthetic images as well as the limited commentary and no evaluation of the downstream effects of using data from generative models for training. I would be willing to improve my rating if the authors address these issues (as discussed in weaknesses).", "limitations": "The authors have addressed the limitations of their work in Section 7, but it could use further commentary. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. **Important Problem:** The authors study a very important problem, how to utilize synthetic images for training vision and vision-language models. They also show success at this central problem (with caveats, see weaknesses). They are one of the first studies showing successful use of synthetic data from generative models on standard computer vision benchmarks.\n\n2. **Novel approach:** The multi-positive StableRep approach proposed by the authors for multi-caption invariance learning in self-supervised models is a novel idea and seems to work very well in practice leading to significant improvements over simply using synthetic images for training. \n\n3. **Ablations:** The authors perform extensive ablations on the role of guidance scale, and also include a study on how additional language supervision could be used to increase caption efficiency while training with multiple positive synthetic images. The results indicate that proposed approaches (StableRep and StableRep+) indeed are more efficient versus training on real images.", "weaknesses": "1. **Limited motivation behind why synthetic images:** It is unclear to me as a reader why Synthetic Images are being used in Section 2.2 as training data for pre-training. It seems that there is no real benefit from using them on state-of-the-art contrastive models like BYOL and MoCo-V3 in terms of linear accuracy. Plus, the key issue for training is the lack of real world labelled data as mentioned by the authors' central question \"how can we collect such large amounts of varied data to train AI models?\" (L21). There is plenty of real-world unlabelled data that is available and can be used to train self-supervised models. The real benefit that synthetic images present is unlimited labelled data. However no fine-tuning comparisons or supervised learning comparisons are provided except for fine-tuning MAE, which could very well be explained by the stochasticity in model training (82.9% vs 82.6%). Section 2.2 does not do much to justify the usage of synthetic images for training in a manner that addresses the authors' central question, and follow up sections justify it for efficiency but not performance.\n\n2. **Limits of generative model:** While the authors include a limitation section, and mention briefly the issues that affect generative models like Stable Diffusion, there needs to be more rigorous evaluation of these issues. Conceptually, these methods suffer from issues like limited compositional understanding. There have been countless studies on the social biases reflected in these models. Very lately (after the NeurIPS deadline), there was an important work showing that models trained on data from generative models can suffer from model collapse, where tails of the original data distribution disappear from the subsequent trained model [1]. All these issues are quite major and should be part of this study, since it is one of the first to utilize synthetic data and show some improvements in results and could potentially effect future research in this area. In particular, the authors should include results on compositional benchmarks (like ARO, CREPE, Winoground, SugarCREPE etc) for their CLIP models trained with StableRep, discuss the fairness of the self-supervised models such as worst-class accuracy and geographic bias etc. \n\n### References \n\n1. Shumailov I, Shumaylov Z, Zhao Y, Gal Y, Papernot N, Anderson R. The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv preprint arXiv:2305.17493. 2023 May 27.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work as novel and acknowledging that we study a very important problem. \n\n> Limited motivation behind why synthetic images: It is unclear to me as a reader why Synthetic Images are being used in Section 2.2 as training data for pre-training. It seems that there is no real benefit from using them on state-of-the-art contrastive models like BYOL and MoCo-V3 in terms of linear accuracy\n\nThe whole point of Section 2.2 is to analyze and study how good synthetic images are for self-supervised training, thus is **not** about chasing numbers. It's actually quite surprising to find that using synthetic images are on par at (or often better than) real images. We have improved the training for MoCo v3, and now the results are even closer between real and synthetic images (see below). This is an surprising finding, not to mention that synthetic images are much better than real images for MAE for linear probing.\n\n|  | data | cc3m | cc12m|\n|--|:--:|:--:|--:|\n| MoCo v3 | real | 64.8 | 66.6 | \n|         | syn  | 64.3 | 66.4 |\n| BYOL    | real | 64.0 | 65.9 |\n|         | syn  | 64.2 | 65.6 |\n\nLearning from synthetic data is quite and interesting topic, and there actually have been a whole line of work exploring synthetic images for training (such as [3],[24],[58],[31]). One advantage of using synthetic data is that we have multiple ways of freedom to control the way we generate the data. Sec 2.2 shows one freedom the guidance scale, while Sec 2.3 shows another freedom which is leveraging the stochasticity to generate multiple images per caption. We hope our work can inspire future exploration in this direction.\n\n> no supervised learning comparisons\n\nWe have include supervised learning comparisons in the question 1 of the \"global rebuttal\" shared to all reviewers. Feel free to let us know if you mean something else.\n\n> no fine-tuning comparisons\n\nWe add additional ImageNet fine-tuning results for SimCLR:\n|  | Pre-train data | ImageNet fine-tuning |\n|--|:--:|:--:|\n| SimCLR | Real | 81.8|\n|        | Syn | **82.2**|\n\nWe tried Stable Diffusion 2.1 for synthetic images, and SimCLR and MAE with SD 2.0 achieved 82.1 and 83.0 ImageNet fine-tuning, both higher than pre-train on the real images (81.8 for SimCLR and 82.6 for MAE)\n\n> which could very well be explained by the stochasticity in model training (82.9% vs 82.6%)\n\nNot really, we fine-tuned multiple times and the averaged results remain the same. The performance gap in fine-tuning is usually small, please refer to Table 3 in the MAE paper to see that MAE also only beat MoCo v3 and BEiT by 0.4%.\n\n> Section 2.2 does not do much to justify the usage of synthetic images for training in a manner that addresses the authors' central question, and follow up sections justify it for efficiency but not performance.\n\nWe believe we have justified the performance in our paper (acknowledged by other reviewers too). Also see question 1 in \"global rebuttal\".\n\n> All these issues are quite major and should be part of this study, since it is one of the first to utilize synthetic data and show some improvements in results and could potentially effect future research in this area. In particular, the authors should include results on compositional benchmarks (like ARO, CREPE, Winoground, SugarCREPE etc) for their CLIP models trained with StableRep, discuss the fairness of the self-supervised models such as worst-class accuracy and geographic bias etc.\n\nThis is a great suggestion! We tried ARO benchmark for compositionality test, and FairFace for evaluating representation fairness. We generally found that training on Synthetic data is able to improve the performance on both compositionality and fairness.\n\nFor ARO (compositionality):\n|  |pre-train data| Relation accuracy |\n|--|:--:|:--:|\n| CLIP | Real | 46.4 |\n|      | Syn  | **50.0** |\n| StableRep+ | Syn | 47.3 |\n\nFor FairFace (fairness):\n|  |pre-train data| mean accuracy | best-class accuracy | worst-class accuracy |\n|--|:--:|:--:|:--:|:--:|\n| CLIP | Real | 28.2 | 60.2 | 0.3 | \n|      | Syn  | 30.4 | 64.0 | 3.1 |\n| StableRep+ | Syn | **37.2** | **74.9** | **10.0** |\n\nCLIP w/ real data only achieved 0.3% accuracy with \"southeast asian male\" class, and CLIP w/ synthetic data improves this class to 3.1%, while our StableRep+ improves this class to 27.2%. The worst class for StableRep+ is \"middleastern male\", which CLIP w/ real and w/ synthetic struggle too (they achieved 6.9% and 6.2%). This suggests a geographic bias and StableRep+ generally improves the fairness.\n\nPlease don’t hesitate to let us know for any additional comments on the paper or the ARO or FairFace experiments.\n\n"}, {"review_id": "X6RPKNhCgT", "reviewer": "Reviewer_DyTC", "summary": "In this paper, the authors investigate the potential of learning visual representations using synthetic data generated by text-to-image models. The authors choose Stable Diffusion for exploration and extensive experiments demonstrate that self-supervised models trained on synthetic data can perform better or at par with training on real data. The authors also propose a multi-positive contrastive variant called StableRep to allow multiple images generated from the same text prompt as positives for each other.", "questions": "I am concerned about the questions mentioned above. Given the current status of the paper, I am leaning towards borderline accept and hope the authors could address my concerns.", "limitations": "The authors have addressed the limitations and broader impacts in Sec. 7 (main text) and Sec. D (supplementary material) in detail, which look good to me.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well motivated. Given the recent progress of large-scale text-to-image generative models like Stable Diffusion, it is timing to investigate the effectiveness of these models in generating high-quality images to assist discriminative tasks.\n\n- The paper is generally well-written and easy to follow.\n\n- The experiments are extensive and the results seem promising.", "weaknesses": "- The authors use large-scale image-text datasets like CC3M, CC12M, RedCaps for study. These datasets have well-collected image captions. Although the authors claim that the proposed method can reduce the reliance on collecting large-scale real images for learning representations, the cost of collecting these image captions is also somewhat expensive and should not be ignored. Thus, a more cost-effective and interesting setting should be to generate a synthetic ImageNet dataset using category labels as text prompts (e.g., a photo of [category]) since it does not require any human effects of collecting captions. Then self-supervised models could be pre-trained on the real ImageNet dataset as well as the synthetic ImageNet dataset for comparisons. I am curious about whether the self-supervised methods trained on synthetic ImageNet still have advantages over training on real ImageNet.\n\n- It seems that the synthetic data only have performance advantages for linear probing experiments. For few-shot experiments, self-supervised methods trained on synthetic data still have a large performance gap with training on real data, which limits the practical applications of using synthetic data to some extent.\n\n- The authors only evaluate the proposed method on classification downstream tasks. What about dense prediction downstream tasks like object detection or semantic segmentation? Will the synthetic data still have the advantages on these kinds of downstream tasks?\n\n- According to ablations in Sec. 4.2, generating multiple images per caption yields better performance. However, this may be due to simply increasing the number of views. To eliminate the interference of the number of views, the authors should also conduct an ablation by replacing multiple generated images per caption with the same number of cropped views per real image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable feedback. Below we try to address your concerns:\n\n> Thus, a more cost-effective and interesting setting should be to generate a synthetic ImageNet dataset using category labels as text prompts (e.g., a photo of [category])...I am curious about whether the self-supervised methods trained on synthetic ImageNet still have advantages over training on real ImageNet.\n\nThis is a good suggestion. We have conducted the experiments and include the results in question 1 of the \"global rebuttal\" shared to all reviewers (it is located around the top of this page).\n\nIn terms of Synthetic ImageNet v.s. Real ImageNet, we observe that:\n\n- When conduct supervised cross-entropy training, Real data significantly outperforms Synthetic data on either downstream linear classification or few-shot evaluation.\n- When conduct self-supervised learning, i.e., SimCLR, Synthetic ImageNet is comparable to (or just slightly worse than) Real ImageNet on downstream linear probing benchmarks. Generally, different prompting strategies result in different performance, and this indicates the importance of designing prompts.  \n\nWe also observe that, StableRep consistently outperforms both supervised training and SimCLR with synthetic ImageNet, demonstrating the effectiveness of our pipeline. While collecting image captions can be more expensive than using labels, we argue that we can explore large language models to help us synthesize these captions, which may becoming increasingly accessible and cheaper in the near future.\n\n> It seems that the synthetic data only have performance advantages for linear probing experiments. For few-shot experiments, self-supervised methods trained on synthetic data still have a large performance gap with training on real data, which limits the practical applications of using synthetic data to some extent.\n\nThis is not completely true. \n\nFirstly, our StableRep outperforms training with real data on 8 out of the 10 few-shot benchmarks, and the average accuracy (over all 10 datasets) only lags behind by 0.8%. \n\nSecondly, the impression of underperfomance (for synthetic training) indeed comes from cifar-10/100. We have identified this as image resolution issues (please check our experiments in question 2 of the \"global rebuttal\"). Intuitively, the real training images contain low-resolution and blurring images, which help the model generalize to extremely low resolutions, such as 32x32 for cifar. In contrast, the synthetic data only contains high resolution images. A potential fix could be randomly downsampling images when training models on synthetic data.\n\n> The authors only evaluate the proposed method on classification downstream tasks. What about dense prediction downstream tasks like object detection or semantic segmentation?\n\nThis is a great suggestion,  and we have evaluated the quality of representation on ADE20k semantic segmentation dataset.  Specifically, we freeze the backbone, and train the decoder part of an UperNet [a]. We used the default parameters in the mmsegmentation library [b]. The results are as below:\n\n| method | pre-train data | mean IoU (%) | pixel acc. (%)|\n|--|:--:|:--:|:--:|\n| clip | real, cc12m | 33.3 | 75.9 |\n| simclr | real, cc12m | 35.2 | 77.6 |\n| simclr | syn, cc12m  | 33.3 | 76.0 |\n| StableRep | syn, cc12m | **38.0** | **78.4** |\n| StableRep | syn, RedCaps | 37.1 | 77.8 |\n\nWe observe that our StableRep consistently outperforms all other methods.\n\n\n> According to ablations in Sec. 4.2, generating multiple images per caption yields better performance. However, this may be due to simply increasing the number of views. To eliminate the interference of the number of views, the authors should also conduct an ablation by replacing multiple generated images per caption with the same number of cropped views per real image.\n\nThis is a good suggestion for ablation study. To keep the comparison with StableRep sensible, we keep the total amount of crops per-batch as 8192, and vary the number of crops per image. In other words, if we use k views per image for SimCLR, then we only sample 8192 / k images per batch. Note that this setup is different from the SwAV paper [c] (which is the origin of the idea of > 2 views per image), where they actually increased the total number of crops per batch. The comparison between SimCLR and StableRep is:\n\n| method | pre-train data | 2 view | 4 view | 6 view |\n|--|:--:|:--:|:--:|:--:|\n|SimCLR| Real | 59.5 | 60.0 | 59.3 |\n|StableRep| Syn | **68.7** | **69.6** | **69.6** |\n\nAnother ablation is to compare Synthetic SimCLR and StableRep with 2 views. Both are using synthetic images and two views. The difference is that the two views of synthetic SimCLR come from the same image, while the two views of StableRep come from two different images. The comparison shown below clearly demonstrates that most of the improvement comes from using different images of the same caption, rather than just more views.\n\n| method | pre-train data | 2 view | \n|--|:--:|:--:|\n|SimCLR   | Syn | 60.5 | \n|StableRep| Syn | **68.7** | \n \nPlease don’t hesitate to let us know if you have more comments or questions. We respectfully hope that you can consider raising the score if our response could address existing concerns.\n\n[a] Unified Perceptual Parsing for Scene Understanding\n[b] MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark\n[c] Unsupervised Learning of Visual Features by Contrasting Cluster Assignments\n\n"}, {"review_id": "jDXXIS3X0w", "reviewer": "Reviewer_FSjQ", "summary": "The paper investigates the potential of using synthetic images generated by text-to-image models to train self-supervised image embedding models. Methodologically, the authors propose StableRep, a multi-positive contrastive learning method that treats multiple images generated from the same text prompt as positive examples for each other. The paper demonstrates two key findings. First, when the generative model is appropriately configured, self-supervised methods trained on synthetic images can achieve comparable or superior performance to real image counterparts. Secondly, when language supervision is incorporated, synthetic data become more efficient. ", "questions": "See Weaknesses. ", "limitations": "Limitations are included in Section 7. ", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. **Interesting topic:** The paper investigates the potential of using synthetic images generated by text-to-image models as strong visual representation learners. This exploration addresses an important and timely question. \n\n2. **New contrastive learning method:** The proposed StableRep is new and tailored for synthetic data. \n\n3. **Comprehensive evaluation:** The paper evaluates the representations learned by StableRep on large-scale datasets and compares them with strong baselines. ", "weaknesses": "1. **Potentially entangled comparison in terms of the amount of data:** The only concern I have for this paper is the claim of data amount. When claiming \"synthetic and real images of the same amount\", the authors refer to the amount of data for representation learning. However, Stable Diffusion is trained on LAION-2B. Therefore, 50M synthetic images may contain information from 2B images, making this comparison less meaningful. I'd like to see more discussion on this issue. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for considering our work as interesting and novel! The concern you raised is a great question, and we will try to answer it from two aspects:\n\n 1. While the dataset distillation hypothesis is unavoidable, we have a different interpretation from the \"distribution\" perspective. Suppose there is an underlying distribution of all natural images, e.g., every single scene on this planet projected into an image via a pinhole camera. We call such underlying distribution $D$.  LAION-2B is just a set of 2B images sampled from $D$, and we used LAION-2B to estimate this underlying distribution and then yielded a stable diffusion model, which can be viewed as a \"parameterized distribution\" $D_{sd}$ that approximate $D$. In theory, $D$ is still much more powerful than $D_{sd}$. Then if we are about to sample some random training images fro representation learning from either $D$ or $D_{sd}$, which one should I pick? One may choose $D$ because it's theoretically more powerful, but we argue $D_{sd}$ may be a good (or even better) option, why? Because $D_{sd}$ allows us to have more fine-grained control over how we sample those images: (1) guidance scale allows us to trade off between image quality and diversity so we can tune it, but for $D$ it's very expensive or even infeasible to tune it -- once the data is collected it's collected; (2) the stochasticity of diffusion models allows us to sample multiple images that correspond to the same underlying semantics described by an caption, but doing so with $D$ is almost infeasible; (3) and potentially others. So in summary, $D_{sd}$ is a weaker distribution and randomly sampling from it may actually be disadvantageous, rather than than advantageous implied by the distillation hypothesis. But through the control of (1) and (2), we can sample from $D_{sd}$ in a more efficient way than $D$ allows us to do.\n\n2. Practically, we conduct an surrogate sanity check to get a rough sense of how much information the synthesized training data contains about the downstream task. Specifically, we compute how close the downstream datasets are to the StableRep's synthetic training set. Concretely, for an image $I$ in a downstream dataset, we find its closest image in StableRep's training set by using cosine similarity of some pre-trained SimCLR feature extractor $f$ (Recall that SimCLR is optimized to maximize cosine similarity of the same image). We define this highest similarity (noted as $s$) as this image's similarity with StableRep's training set $D$. Concretely, $s_i = \\max_{X \\in D} cos(f(I_i), f(X))$. We evaluate four datasets, and provide the statistics (avg, min, and max) of this highest similarity metric of each image. \n\n| Train Set | stats | aircraft|cars|flowers|food101|\n|--|:--:|:--:|:--:|:--:|:--:|\n| StableRep Train | avg | 0.891 | 0.836 | 0.836 | 0.791 |\n|                 | min | 0.652 | 0.258 | 0.523 | 0.443 |\n|                 | max | 0.985 | 0.980 | 0.967 | 0.958 |\n| ImageNet  Train | avg | 0.901 | 0.870 | 0.823 | 0.807 |\n|                 | min | 0.700 | 0.629 | 0.586 | 0.560 |\n|                 | max | 0.980 | 0.998 | 0.988 | 0.977 |\n\nAs a comparison, we also include the distance to ImageNet training set as a baseline. We observe that:\n- generally downstream datasets are closer (by avg metric) to ImageNet training set than our StableRep train set, except for flowers.\n- there is no value surpass 0.99 for StableRep train set, while there is 0.998 for ImageNet train set. \n\nFrom this perspective, we may conjecture that the ImageNet train set contains more information about the downstream dataset than the StableRep dataset. However, the StableRep still generalizes better than methods trained on real ImageNet train set, as illustrated in question 1 of the \"global rebuttal\" shared to all viewers.\n"}]

Author rebuttal:
We thank all reviewers for their insightful comments and feedback! We are glad to see the reviewers found:

 1. the paper is well-written and easy to follow (by reviewer HH2w, AnD7, DyTC)
 2. this paper approaches an interesting timely research question / very important problem (by AnD7, DyTC, FSjQ, qM14).
 3. the approach is novel (by qM14, FSjQ) and nice (by AnD7).
 4. this paper shows well-designed experiments / extensive ablation study / comprehensive evaluation / encouraging and promising results (by all reviewers).


We provide answers to shared questions/concerns here (and respond to remaining individual questions to each reviewer separately): 


***1. directly use labels (e.g. from ImageNet) to synthesize image?***


We acknowledge that the challenging part of synthesizing with labels is how to design the text prompts. We include four variants here:

- **Language Enhancement (LE)** proposed by [24]: it generates 200 prompts for each ImageNet class.
-  **c, hc, in place** proposed by [a]: it composes ImageNet class name *c* with the hypernyms of the class *hc* and Places class name *place*.
- **c, dc, in place** proposed by [a]: it composes ImageNet class name *c* with the definition of the class *dc* and Places class name *place*.
- **a photo of [category]** proposed by us (also suggested by Reviewer DyTC).

For each of above items, we perform both supervised cross-entropy (xent) training and SimCLR training. We compare them with our StableRep, as well as training on the Real ImageNet dataset. All methods are trained for roughly 300 ImageNet-equivalent epochs for a fair comparison. We report: (1) linear probing accuracy on ImageNet (ImageNet LP); (2) the average accuracy of the linear transfer on 12 downstream datasets; (3) the average accuracy of few-shot evaluation on 11 datasets. Results are as follows:

|  | pre-train data | method | ImageNet LP | Downstream avg | few-shot avg| 
|--|:--:|:--:|:--:|:--:|:--:|
| Supervised | Real ImageNet | xent | - | 78.6 | 85.5 |
| Unsupervised | Real ImageNet | SimCLR | **74.3** | 74.3 | 77.8 |
| LE [24] | Synthetic | xent | 64.7 | 71.5 | 77.5 |
|    | Synthetic | SimCLR | 66.8 |74.0 | 71.9 |
| c, hc, in place | Synthetic | xent | 67.1 |73.0 | 77.2 |
|  | Synthetic | SimCLR | 66.1 | 73.2 | 71.1 |
| c, dc, in place | Synthetic | xent | 68.6 | 73.2 | 77.2 |
|  | Synthetic | SimCLR | 65.8 | 73.2 | 69.2 |
| a photo of c | Synthetic | xent | 67.9 | 72.6 | 76.5 |
|  | Synthetic | simclr | 66.6 | 74.1 | 71.3 |
| StableRep | Synthetic | StableRep |72.8 | **82.2** | **85.9** |


We observe that:
- Our StableRep clearly outperforms all models trained on synthetic imagenet (even on ImageNet linear probing despite synthetic ImageNet explicitly leverages ImageNet labels), and even outperforms supervised/unsupervised training using Real ImageNet on both downstream transfer and few-shot evaluation. 
- StableRep is only inferior to SimCLR trained with ImageNet on the ImageNet linear probing benchmark, mostly because StableRep here hasn't converged yet. In fact, our convergent StableRep with longer training can reach 75.7 (using cc12m captions) or 76.7 (using RedCaps captions). In contrast, SimCLR with real ImageNet does not improve much with longer training, indicating the item here is already convergent.


***2. StableRep (or training with synthetic data) underperforms training with real data on cifar-10/100 evalution, while outperforms on the other evaluation datasets ?***


We have identified this issue. This is due to the image resolution: cifar-10/100 has very low resolution image (32x32 pixels), while other evaluation datasets have much higher resolutions. To test this hypothesis, we also downsampled the other 9 evaluation datasets to 32x32 pixels before we evaluate. The results for our StableRep and CLIP with real data are as below:

| method | downsample? | aircraft | cars| dtd| flowers| pets| sun397| caltech101 | food101 | voc2007| Average |
|--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| real CLIP |  | 53.2| 75.8 | 75.7 | 96.0 | 86.7 | 72.5 | 92.7 | 81.6 | 86.1 | **80.0** |
| real CLIP | ✅ | 10.2| 35.7 | 48.7 | 71.6 | 70.4 | 58.5 | 77.9 | 66.8 | 83.5 | **58.1** |
| Stablerep |  | 57.6| 80.3 | 79.0 | 96.7 | 87.1 | 73.2 | 94.0 | 83.5 | 87.2 | **82.1** | 
| Stablerep | ✅ | 5.2 | 24.3 | 53.3 | 53.2 | 65.4 | 46.2 | 70.7 | 52.0 | 83.3 | **50.4** |


While StableRep outperforms Real CLIP on all of these 9 tasks before downsampling, it underperforms Real CLIP on 8 out 9 tasks (sometimes significantly) after the downsampling. This can be explained by the training data. While Stable Diffusion only synthesizes high resolution images for StableRep training, the real image set often contain many low-resolution or blurred images which in turn help real CLIP's transfer on data of very low resolutions, e.g., 32x32 for cifar-10/100. A fix may be to randomly downsample the synthetic image to very low resolution during StableRep training, which we leave as future work given the limited rebuttal time.

Please don’t hesitate to let us know for any additional feedback. Thanks!


[a] Fake it till you make it: Learning transferable representations from synthetic ImageNet


Meta-review: This submission was reviewed by five knowledgeable referees, who raised concerns about novelty, missing experimental details, a potential data poisoning, the observed performance gap on CIFAR-10/100, and the limits of generative models. The reviewers also raised important questions such as the effect of the choice of generative model on the performance, and the amount of data used to pretrain stable diffusion. The rebuttal and subsequent discussion between authors and reviewers adequately addressed the reviewers concerns. After discussion, reviewers agree that the paper presents a valuable contribution and they all recommend to accept. The AC agrees with the reviewers' assessment and therefore recommends to accept.
Decision: Accept (poster)