Learning to See by Looking at Noise
Authors: Manel Baradad
Jonas Wulff
Tongzhou Wang
Phillip Isola
Antonio Torralba
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: unsupervised representation learning
image generation
contrastive training
learning from noise

Abstract: Current vision systems are trained on huge datasets, and these datasets come with costs: curation is expensive, they inherit human biases, and there are concerns over privacy and usage rights. To counter these costs, interest has surged in learning from cheaper data sources, such as unlabeled images. In this paper we go a step further and ask if we can do away with real image datasets entirely, instead learning from procedural noise processes. 
We investigate a suite of image generation models that produce images from simple random processes. These are then used as training data for a visual representation learner with a contrastive loss. In particular, we study statistical image models, randomly initialized deep generative models, and procedural graphics models.
Our findings show that it is important for the noise to capture certain structural properties of real data but that good performance can be achieved even with processes that are far from realistic. We also find that diversity is a key property to learn good representations.

Learning to See by Looking at Noise
Manel Baradad" MIT CSAIL mbaradadłmit..en edu
Jonas Wulff MIT CSAIL jyulff@csail.min .di
Tongzhou Wang
MIT CSAIL tongzhouêmit, .dd
Phillip Isola MIT CSAIL
Antonio Torralba MIT CSAIL
phillipi@mit. edu
torralba@mit.cc edu
Abstract
Current vision systems are trained on huge datasets, and these datasets come with costs: curation expensive, they inherit human biases, and there are concerns over privacy and usage rights. To counter these costs, interest has surged in learning from cheaper data sources, such as unlabeled images. In this paper, we go a step further and ask if can do away with real image datasets entirely, by learning from procedural noise processes. We investigate a suite of image generation models that produce images from simple random processes. These then used as training data for a visual representation learner with a contrastive loss. particular, we study statistical image models, randomly initialized deep generative models, and procedural graphics models. Our findings show that it important for the noise to capture certain structural properties of real data but that good performance be achieved even with processes that are far from realistic. We also find that diversity is key property for learning good representations.
Introduction
The importance of data in modern computer vision is hard to overstate. Time and again we have seen that better models are empowered by bigger data. The ImageNet dataset [[]. with its 1.4 million labeled images, is widely thought to have spurred the era of deep learning, and since then the scale of vision datasets has been increasing at rapid pace; current models are trained on up to one billion images (2. In this paper, we question the necessity of such massive training sets of real images.
Instead, we investigate a suite of procedural noise models that generate images from simple random processes. These are then used as training data for visual representation learner.
We identify two key properties that make for good synthetic data for training vision systems: 1) naturalism, 2) diversity. Interestingly, the most naturalistic data is not always the best, since naturalism can come at the cost diversity. The fact that naturalistic data help may not be surprising, and suggests that indeed, large-scale real data has value. However, find that what is crucial is not that the data be real but that it be naturalistic, i.e. must capture certain structural properties of real data. be captured in simple noise models (Fig. I
Many of these properties
The implications of our work are severalfold. First, our results call into question the true complexity of the problem of vision - if very short programs can generate and train high-performing vision system, then vision may simpler than we thought, and might not require huge data-driven systems to achieve adequate performance. Second, our methods open the door to training vision systems without reliance on datasets. The value in this is that datasets are encumbered by numerous costs:
"quual contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure I: Which of these image crops come from real photos and which are noise? See footnote for answer These crops are examples of what MoCo v2 3 sees as input in our experiments.
they may be expensive, biased, private, or simply intractable to collect. We do not argue for removing datasets from computer vision entirely (as real data might be required for evaluation), but rather reconsidering what can be done in their absence.
2 Related work
2.1 A short history of image generative models
Out of the R3 dimensional space spanned by x n² color images, natural images occupy a small part of that space, the rest is mostly filled by noise. During the last decades, rescarchers have studied the space of natural images to build models capable of compressing, denoising and generating images. The result of this research is a sequence of generative image models of increasing complexity narrowing down the space occupied by natural images within R
One surprising finding is that natural images follow a power law on the magnitude of their Fourier transform 4 5. This is the basis of Wiener image denoising [6] and scale-invariant models of natural images 7 8 Dead leaves model [8] 9 was an attempt at generating images that explained the power law found in natural images and inspired the Retinex algorithm [10]. The multiscale and self-similar nature of natural images inspired the use of fractals
12
13
as image models.
showed another remarkable property of natural images: the output values of zero mean wavelets to natural images are sparse and follow generalized Laplacian distribution [6] Color and intensity distributions in natural images have also been studied and found to follow rules that deviate from random noise TO. 17]. Research in texture synthesis showed how these statistical image models produced more realistic-
Coding research for TV 5 and image modeling 66
14
15 16
textures [1]. Those required fitting the image model parameters to specific images to sample more "like it". Recently, GANs (20) have shown remarkable image synthesis results [1]] Although GANs need real images to learn the network parameters, we show in this paper that they introduce a structural prior useful encode image properties without requiring any training.
2.2 Training without real data and training with synthetic data
Through the above progression, generative models have become increasingly complex, with more parameters and more training data needed to fit these parameters. The same has happened with vision
systems in general: state-of-the-at systems like BiT (22), CLIP [23]. and SEER (2) obtain their best results on 300 million, 400 million, and billion images respectively. These papers further show that such large data is critical to getting the best performance.
While this may be true, other work has shown that much smaller data is sufficient already get decent performance. A single image can be enough to train, from scratch, a compelling generative model [24 25 or visual representation [26), and, even with no training data all, deep architectures
already encode useful image priors that can be exploited for low-level vision tasks (27 or for measuring perceptual similarity (28). Our results, using an untrained StyleGANv2
29
generate
training data, further affirm the utility of the structural priors in neural net architectures.
An alternative to training with real data is to train synthetic data. This approach has been widely
used in low-level tasks like depth, stereo, or optical flow estimation 33 rendering engines can provide densely annotated data to learn from. Interestingly, for this class of tasks diversity is more important than realism (34,, making procedurally generated scenes an effective alternative to renderings designed by professional 3D artists [35] (3)].
31
32 33
where 31)
Answers: 1.3.4.5.6,8,14 are from ImageNet images.

---

Recent work has also investigated using deep generative models as a souree of synthetic data to train classifiers [7 38 and visual representations [39] or to generate synthetic annotated data for other
downstream tasks 40
42
However, these generative models are still fit to real image datasets
and produce sealistic-loking images as samples.
In this paper we push even further away from realism, generating synthetic data from simple noise processes. The closest prior work in this direction is the pioneering work of [43), which used automatically generated fractals to pre-train neural networks that converge faster than their randomly initialized counterparts. While they demonstrated that fractals can be effective for pre- mre-trinnnng there is still a large gap compared to pre- pre-training real data. We explore a much broader range of noise processes, including many classic models from the image coding and texture synthesis literature.
The use randomized training data has also been explored under the heading of domain randomiza- tion 44, where 3ID synthetic data is rendered under a variety of lighting conditions to transfer real environments where the lighting may be unknown. Our approach can be viewed as an extreme form of domain randomization that does away with the simulation engine entirely: make the training data so diverse that natural image will just look like sample from the noise process.
There is some evidence that biology takes similar approach during the prenatal development of the vision system. "Retinal waves" spontaneous, semi mani-random activations of the retina are thought to entrain edge detectors and other simple structures in the developing mammalian brain [45]
3 A progression of image generative models
Here we provide the details for the image models we will use in this paper. We test suite of generative models of the form ge
where are stochastic latent variables and x is an image. We will treat image generation as a hierarchical process in which first the parameters of a model, 0, are sampled, and then the image is sampled given these parameters and stochastic noise. The parameters 9 define properties the distribution from which we will sample images, for example, the mean color of the image. The sampling process is as follows: 0 ~ p(0),
p(z), and (z)) which corresponds to sampling images from the distribution p(x, x 0) = (0)). The parameters that define the model may be fit to real data not. We will explore the case where the parameters are not fit to real data but instead sampled from simple prior distributions. Next, we describe the generative image models that will evaluate in this paper (Fig. 2)
3.1 Procedural image models
The first class of models belong to the family of procedural image models. Procedural models are capable of generating very realistic-looking images in specific image domains. We include in this set also fractals, although they could make class on their own.
Fractals have been shown to capture geometric properties of elements found in nature [6]. Consequently, image models consisting of renderings of human-designed shapes with fractal structure [4] are likely to reproduce patterns found in natural images.
Fractals:
CG: Simulators and game engines rely on mixture of human-driver design and procedural methods to generate environments simulating real-world properties such as illumination, 3D, and semantics. Here include three CG models popular in computer vision with available datasets: CLEVR [4). DMLab (48) and MineRL [49].
3.2
Dead leaves image models
The second family of models is Dead leaves, one of the simplest image models. We consider simple shapes (leaves) like circles, triangles and squares, which are positioned uniformly at random in the image canvas until it is covered. To produce each shape, we circumscribe following exponential distribution of parameter X. This procedure has been shown to produce images that have similar statistics to natural images, such as having a 1/lf power spectrum [ and non-gaussian distribution of derivatives and wavelet coefficients [50)
a circle with a radius
In this study we will consider four dead leaves models: Dead leaves Squares: Only uses squares axis- aligee. Dead leaves Oriented: Squares are randomly rotated. Dead leaves Shapes: Leaves
3

---

Procedural
TraclaiSDB
b) CLeVR
c) DMLab
d) Minecraft
Dead leaves (squano axis align)
Dead leaves (rotatied squaras)
Dead leaves (alverse shapes)
n) Dead leaves (lextures squares)
Stotelical
Spectrum
Wavelets marginal model (WMMM)
k) Spectrum and colors
Spectrum, color and WMM.
SlylGANN
StyleGAN: default int
1M noise and wavelets
laplacen envelope and bas
bed wavelets
Foature
Figure Samples from our synthetic datasets: a-d) CG synthetic images, e-h) Dead-leave models, i-l) Statistical image models, m-p) StyleGAN -ased models with different weight initializations, and q-r) Feature visualizations.
can be circles, triangles and rectangles. Dead leaves Textured: uses square leaves filled with texture sampled from the statistical image models described the next section.
3.3 Statistical image models
The third family of models is statistical image models with increasing complexity. Several generative models can be composed by using different combinations of properties.
Spectrum: The magnitude of the Fourier transform of many natural images follows a power law, 1/lfi", where is a constant close images constrained have FT magnitude following 1/ 1/(1/z 1 (fy11% lo) with and b being random numbers uniformly sampled in the range [0.5, 3.5]. This model introduces a bias towards horizontal and vertical image orientations typical of natural images [5I]. To generate color images first sample three random orthogonal color directions and then generate power-law-noise on each channel independently. Samples from this model are shown Fig. i).
In this generative model, will sample random noise
Wavelet-margine model (WMM): Following 122 we generate textures by modeling their his- tograms of wavelet coefficients. To produce texture, create marginal histograms for the coeffi- cients e at N scales (i E {O...N - 1 and 4 orientations following a Generalized normal distribution centered at zero, thus p(c) x exp(( (ppp(((l///a))) ).). Each scale i represents the image down-scaled by a factor of 2 and the parameters and 3; for each scale are a; = 4 and B, ~ 0.4 + L/(0, 0.4). In practice we use N and 4 for generating 128 x 128 and 256 256 resolution images respectively. Once we have sampled marginal distribution of waveler coefficients for each of the three channels, we do histogram matching iteratively starting from a Gaussian noise image following [53] Fig. 2) shows samples from this model.

---

Color histograms: Here we take generative model that follows the color distribution of the dead-leaves model. First we sample number of regions N ~ 3 + [0, 20) their relative sizes S ~ 0.001 + (0, 1) and color at uniform. This results in color distribution different from uniform.
Combining all these different models allows capturing color distributions, spectral components, and wavelet distributions that mimic those typical from natural images. Fig. Fig-2(k) shows the result of sampling from a model that enforces random white noise have the power-law spectrum and the color histogram according to this model. Fig. 21) shows samples from a model incorporating all of those properties (spectrum, color and WMM). Those models produce intriguing images but fail to capture the full richness of natural images as shown in Fig. 2 il)).
3.4 Generative adversarial networks
The fourth family of models is based on the architecture of GANs. Commonly, the parameters of GANs trained generate realistic samples of given training distribution. Here, do not use adversarial training or any training data. Instead, we explore different types of initializations, study the usefulness of the GAN architecture as prior for image generation, and show that effective data generators can be formed by sampling the model parameters from simple prior distributions. We use an untrained StyleG [29], and modify its initialization procedure to obtain images with different characteristics. This results in four classes of StyleGAN initializations:
StyleGAN is the default initialization. Fig. m) shows samples from this model. They lack high-frequency image content since the noise maps are not applied at initialization.
In this model, we increase high-frequency image content by sampling the noise maps as f/f" noise with ~ (0.5, 2). which models the statistics of natural images 4. Additionally, the convolutional filters all layers are randomly sampled from a bank of x 3 wavelet filters, and each sampled wavelet multiplied by a random amplitude ~ NNO0, 1). Note that using
StyleGAN-Highh-ee
Wavelets as spatial filters is common practice when hand-designing networks [54] 55 and seems to
well capture the underlying general structure of visual data. The samples in Fig. 2 n) show that this
model generates high- frequeccy structures which are fairly uniformly distributed across the image.
StyleGAN-Sparree Natural images exhibit high degree of sparsity. In this model, we increase the sparsity of the images through two modifications. First, we modulate the 1/f" noise maps using Laplacian envelope. We sample 4 x grid of i..dd Laplacian noise, resize to the desired noise map resolution using bicubic upsampling and multiply this envelope with the original sampled 1 1/f" noise. Second, at each convolution, we add a random bias U (-0.2, 0.2), which, in conjunction with the nonlinearities, further increases sparsity. Fig. O)) shows that the images created by this model indeed appear sparser. Yet, they are still lacking discernible image structures.
EayleGAN--rienteee Oriented structures are crucial component of natural images. We found that an effective way introduce such structures to the previous models tie the wavelets. i.c to use the same wavelet for all output channels. Under tied wavelets, the standard convolution becomes yk = Ei (ax,1 (ar * f() + bg. where YI denotes output channel k; .y denotes input channel 1, by is bias term, d ~ N (0, 1) is random amplitude multiplier and the wavelet fi depends only on the input channel, but shared across all output channels. As can be seen in Fig. pp), this creates
visible, oriented structures in the output images.
3.5 Feature visualizations
The final class of models we study is feature visualizations [56]. CNN's can be used to produce novel images by optimizing the output of single or multiple units with respect to the input image, which is initialized with noise. Although these methods are commonly used to interpret and visualize the internal representation of a network, here we can use it as an image generative process. Following the procedure in [57]. we obtain feature visualizations by optimizing the value of a single or combination of two units of a neural network. We select those units from the layer that is typically used as feature extractor (i.e. the output of the penultimate linear layer), as we have empirically found this to yield more image-like visualizations compared to shallower layers. We create two datasets using this technique. Feature vis. Random: ResNe with the default random initialization, shown in Fig. 4 and, 2) Feature vis. Dead leaves: ResNet50 trained with dead leaves with diverse shapes, using MoCo v2 (3] and 1.3M sampled images. Samples are shown in Fig. 1
5

---

60
Baselines
50
Procedural CG
Dead leaves
Performance (%)
40
Stutisticul models
Feature visualizations
Untrained StyleGAN
Dead leaves
Spectrum + Color WMM
30
Feature vis. Random
Squares
Dead leaves OOiieted
Shppes
StyleGAN Sparse
SkyleGAN: High freg.
StyleGAN Qhiented
Rew Plxes
Spectrum Color
20
Mean colps
Imagonst 1k
Spectrum
FractaDD
SlyleGAN
Dead lesves
Feature vis
Minecraft
Imagenet
ONN
CLeVA
DML
WMM
Dead
Places
STL10
Figure 3: Top-1 accuracy for the different models proposed and baselines for Imagenet- 100 [60] The horizontal
axis shows generative models sorted by performance. The dashed lines represent approximated upper and lower bounds in performance that one can expect from a system trained from samples of generic generative image model.
50
Natural
80
Specialized
Structured
Busiiio
Presiiaal
70
3.5
top-1 accuracy
40
Festars ruanl audldd
Unritdd BHSKIAN
30
40
20
RerThaa
7
Vienn m
20
10
Fil
Figure 4: Average top-1 accuracy across the tasks each of the three groups tasks VTAB 61 The
horizontal axis shows the different image generative models sorted by the average performance across all tasks. DMLab and CLEVR trained networks are not tested, as these datasets are present on some VTAB tasks.
Experiments
To study the proposed image generation processes, we train an AlexNet-based encoder using the Alignment and Uniformity loss proposed [58], which is form of contrastive loss theoretically equivalent to the popular InfoNCE loss [59]. We generate 105k samples using the proposed image models at 128 128 resolution, which are then downsampled to 96 96 and cropped at random to 64 64 before being fed to the encoder. After unsupervised training, performance (without finetuning) on the representation right before the projection layer, following standard practice [58 (5). We fix common set of hyperparameters for all the methods under test to the values found to perform well by the authors [58]. Further details of the training are provided in the Mat.Mat.
evaluate linear training
evaluate performance using Imagenet- 1 100 and the Visual Task Adaptation Benchmark [6I] VTAB consists of 19 classification tasks which are grouped into three categories: a) Natural, consisting of images of the world taken with consumer cameras b) Specialized, consisting in images of specialized domains, such medical aerial photography and c) Structured, where the classification tasks require understanding specific properties like shapes or distances. For each of the datasets in VTAB, we fix the number of training and validation samples to 20k at random for the datasets where there are more samples available.
As an upper- bound for the maximum expected performance with synthetic images, we consider the same training procedure but using the following real datasets: 1) Places365 (62) consisting of wide set of classes, but different domain 2) STL-100 63 consisting of only 10 classes of natural images and 3) Imagenet1k (.. superset Imagenet As baselines we use mean image colors, raw pixels and features obtained by untrained Alexnet (denoted CNN Random).

---

Pretraining dataset
Real images
Procedural images
Other
1-1K
Places
CNN Random
FractalDB
Dead leaves
Feature Vis DDead leaves
Spectrum +C+WMM
StyleGAN -Oriented
4-Mixed
Test
Shapes
I-IK
67.50 55.59
4.36
23.86
20.00
28.49
35.28
38.12
40.06
I-100
86.12
76.00
10.84
44.06
38.34
49.48
56.04
58.70
60.66
Table I: Performance linear transfer for ResNet50 pre-trained different image models using MoCo-v2, on Imagenet-IK (I-IK) and Imagenet- (1-100), The 4-Mixed pretraining dataset corresponds to training with 4 of our datasets together, each with 1.3M images: Dead leaves Shapes, Feature Vis. Dead leaves, Spectrum +
C + WMM and StyleGAN-Oriented.
4.1 Image model comparison
Figures B and show the performance for the proposed fully generative methods from noise on Imagenet 00 and VTAB (Tables can be found in the Sup.Mat.). The results both datasets show increased performance for Natural dataset (Imagenet100 and the Natural tasks in VTAB) that match in Fig. On the other hand, Structured
the qualitative complexity and diversity of samples and Specialized tasks do not benefit as much from natural data (as seen in the middle and right-most plots in Fig. 4). and our models perform similarly for the tasks under test in this setting.
4.2 Large-scale experiments
Finally, we test one of the best-perforning methods of each type on large-scale experiment using a Resnet50 encoder instead of AlexNet. We generate 1.3M samples of the datasets at 256 x 256 resolution, and train using the procedure described in MoCo v2 [3] with the default hyperparameters for Imagenet (details of the training can be found in the Sup.Mat.).
The results in Table show that the relative performance for the experiments with the blex encoder is approximately preserved (except for dead leaves, which underperforms FractalDB- Despite using the MoCo v2 hyperparameters found be good for Imagenetlk (which may not be optimal for our datasets) and not using any real data, our best performing model achieves 38.15 12% top-1 accuracy on linear classification on top the learned features. Furthermore, we achieve 40.06% top-1 accuracy by combining four of our datasets: Dead leaves Shapes, Feature Visualizations Dead leaves, Statistical (Spectrum + Color + WMM) and StyleGAN Oriented.
Additionally, our image models allow training on arbitrary amounts of samples that can be generated on the fly. For StyleGAN Oriented we found that training with the procedure described above but continuously sampling instead of using the fixed 1.3M images yields an improved top-1 accuracy of 38.94%.
5 What properties make for good generated data?
As we have seen, representations learned from noise-like images can be surprisingly powerful. Why this the case? Intuitively, one might think that the features learned from these datasets themselves resemble noise. Interestingly, this is not the case. Fig. shows the diversity of feature activations learned with the different datasets, extracted using the same procedure described in Section[.33 Though some datasets fail to capture certain properties (e.g. the Dead leaves Squares dataset only reacts axis-aligned images), feature visualizations qualitatively show that our datasets contain sufficient structure to learn variety of complex features.
Yet, not all of our datasets reach the same performance, which raises the question of what makes generated dataset good. Which properties are strongly correlated with performance? In the following, we investigate the statistical properties of individual images (5.1) and datasets whole (5.2)
5.1 Properties of individual images
Across our datasets, the image appearance varies significantly (see Fig. 2]: They contain different structures and shapes, different color profiles, different levels of "realism". and different effects
7

---

WMM
leaves Squares
lsaves
Feature
Dead leaves Texturee
SIYAGAN
Places
Rendom
Criented
Dead leaves
freq
Feature vis.
Spectrum
Dead leaves
Specium
StvleGAN
CLaYR
Colore WilM
Minecratt
Figure 5: Feature visualizations for AlexNet-baset encoder trained on some of the proposed datasets, for the 3rd and 5th (last) convolutional layer, using the procedure in [
65
Procedural CG
65
*
Procedural
65
Procedural CG
Dead leaves
Dead leaves
Dead leaves
Statistical
Statistical
55
StyleGAN
StA
55
StyleGAN
Accuracy
Feature Vis
Accuracy
Feature Vis
Accuracy
Feature Vis
Baselines
45
45
35
35-
35.
5
10 15
1.00
1:25
1.50
1.75
2.00
2.25
0.20
0.25
0.30
0.35
0.40
Distance in L space
Estimated
Perceptual variation
(a) Color distance to ImageNet
(b) Sparsity of images
(c) Variation within images
Figure 6: Different properties of the images have different impacts on accuracy. (a) Having a similar color distribution to natural images is important. (b) The spectral properties of dataset images are weakly correlated with accuracy, as long as they are within the right range. (c) There exists a sweet spot for variation within the images; having higher variation results in less successful contrastive lcarning and decreased performance.
such occlusions. How much of the difference in performance can be attributed such low-level, per-image appearance characteristics?
Color profile. The first and most basic question is how important the color characteristies of our datasets are, since even in such basic dimension as color, the datasets differ greatly (see Fig. (g To performance,
test the impact of color characteristies dataset (here, ImageNet- 100) and each of our pretraining datasets. First, we extract I values from 50K images from all datasets and ImageNet- 100 The color distribution of each dataset can then be modeled as three-dimensional Gaussian, and the color difference between two datasets can be computed as the symmetric KL divergence between those distributions. Fig. (|a) shows that the color similarity of most of our datasets to natural images is fairly low; at the same time, we see clear negative correlation between performance and color distance (r -0.57..
measure color similarity between the test
Image spectrum. is well known that the spectrum of natural images can be modeled as a heavy- [0.5, 2.0] [5]] How well do our tailed function A/I with A scaling factor and typically performance? Fig. bbb) shows that datasets resemble these statistics, and how much does this impact the statistics of most of datasets fall within this range, except for Stylegan-default. However, while an exponent a E [1.4, 1.7] seems to benefit performance, the correlation is weak.
Image coherence. Contrastive Learning utilizes different views positive pairs during training, which are commonly generated using random augmentations of the same input image. For this to work, the images need a degree of global coherence, i.e. two views of the same image should be more
"In this and the next section, all measures were computed on 50K random images from the respective datasets, and, where applicable, compared against the reference statistics of ImageNet- 100. To avoid distortions, correlations always computed without taking the datasets ImageNet- 100 and ImageNetIk into account.
8

---

log/Voluma) x10-
annvolua x10->
Accuracy
Accursyy
45-
Procedural CG
Procedural
leaves
Dead
Dead leaves
Statistical
Statistical
statactical
StyleC&NN
StyleGAN
StyleCANN
Baselines
Vis
-9.5
Festure
Baseines
Boselines
25
20
40 ImageNet-100
0.6 Recall ImageNet-100
Precision wrt. ImageNet-100
(a) FID
(b) Precision vs volume
(c) Recall vs volume
Figure 7: Properties of the datasets as a whole. (a) Resembling the test data is the best predictor of performance. Since a perfect overlap with the lest data cannot be achieved, recall and precision are negatively correlated; (b) a high precision corresponds to datasets with small volume, resulting in low performance. (c) Recall is positively correlated with both volume and accuracy.
similar than views from different images. To test the amount of coherence in datasets and how this impacts downstream accuracy, we first compute the average perceptual variation within a dataset as * > (f (In) .9 (In)). where f.g are two random crops of the same image and LPIPS is the perceptual distance from (28) Fig. [[c) shows the results. Interestingly, there seems to be a sweet spot of perceptual variation around 0.37; for datasets with lower variation, perceptual variation is strongly correlated with accuracy (r = 0.75). Going beyond this sweet spot decreases accuracy, since now two augmentations of the same image are too dissimilar to provide good information. Finding a similar effect, [64] showed that there is a sweet spot for augmentation strength in contrastive learning.
5.2 Properties of the datasets
Beyond the properties of individual images, it is important to ask which properties of a dataset as whole explain its level of performance. The first obvious predictor for good performance on a given test task is the distance of the training data distribution to the test data. We can quantify this using the Fréchet Inception Distance (65), which measures the distance between two sets of images by fitting Gaussians to the distribution of Inception features and measuring the difference between those Gaussians. As shown in Fig. 7 (),, FID is strongly negatively correlated (r = -85) with accuracy.
An ideal pretraining dataset would therefore resemble the test dataset as closely as possible, achieving a low FID as well as high precision and recall to the test dataset (66). Interestingly, the trend we observe in our data points is in a different direction: Precision is negatively correlated with accuracy (r
0.67), and the data shows a strong negative correlation between precision and recall (r 0.88). This can be explained by the fact that our methods are not capable of perfectly reproducing the entirety of ImageNet, leaving two possible scenarios: Either a dataset concentrated in a small part of the image manifold (images from CLEVR and DMLAB look naturalistic. yet are severely restricted in their diversity compared to natural images), or the dataset overshoots the manifold of test images, containing as much diversity as possible, in the limit encompassing the test dataset as a subset. Should a dataset be as realistic (i.e. maximize precision), or as diverse as possible?
An extreme version of the first case would be a training dataset that concentrated around a few points of the test dataset, which would achieve high precision but low recall, and generally have low diversity. Given a distribution of features, we can measure the diversity of the dataset as [Σ], i.c. the determinant of the covariance matrix in this feature space. Here, as for the FID (see above). Fig. b)) shows precision vs. log-volume for all datasets; color indicates the performance particularly diverse and are negatively correlated with log-volume (r = -084). As volume increases, precision decreases, yet performance benefits.
the same inception features
ImageNet-Too TOO. As can be seen, datasets with high precision tend to not be
Considering Recall, the picture changes. As shown Fig. Cc) higher recall is (after a certain point) positively correlated with both log-volume (r -79) and accuracy (r = -83). Interestingly, this does not mean that precision is irrelevant - when controlling for recall, precision again positively correlated with accuracy, albeit more weakly (r .222. The reason for this behavior is that in case

---

of our datasets, precision and recall are negatively correlated, and thus present trade-off. If such trade-off necessary when designing datasets for pretraining, our results indicate that maximizing recall might be more important than maximizing precision. Interestingly, despite a completely different setting, the same effect was observed in [34].
Conclusion
Can we learn visual representations without using real images? The datasets presented in this paper are composed of images with different types of structured noise. We have shown that, when designed using results from past research on natural image statistics, these datasets can successfully train visual representations. We hope that this paper will motivate the study of new generative models capable of producing structured noise achieving even higher performance when used in a diverse set visual tasks. Would it be possible to match the performance obtained with ImageNet pretraining? Maybe in absence of a large training set specific to a particular task, the best pre-training might not be using standard real dataset such as ImageNet.
Although might be tempting to believe that the datasets presented in this paper might reduce the impact of dataset bias, bias might still exist (although they might not come from social biases, they might still impact differently subsets of the test data) and it will be important to characterize it for each particular application.
Acknowledgments: Manel Baradad was supported by the LaCaixa Fellowship and Jonas Wulff was supported by grant from Intel Corp. This research was partially conducted using computation resources from the Satori cluster donated by IBM to MIT.
References
[1] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):111-222 2015.
[2] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self- supervised pretraining of visual features in the wild, 2021.
[3] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, 2020.
[4] G. J. Burton and lan R. Moorhead. Color and spatial structure in natural scenes. Appl. Opt., 26(1):157-17,, Jan 1987.
[5] E. R. Kretzmer. Statistics of television signals. Bell System Technical Journal, 31(4)7751-763 1952.
[6] P Simoncelli. Statistical modeling of photographic images. In Alan Bovik, editor, Handbook of Image and Video Processing, chapter 4.7, pages 431-441. Academic Press, May 2005. 2nd edition.
[7] David Field. Scale-invariance and Self-similar 'Wavelet' Transforms: an Analysis Natural Scenes and Mammalian Visual Systems., pages 151-193. Oxford University Press., 1993.
[8] Daniel L. Ruderman. Origins of scaling in natural images. Vision Research, 37(23):3385-3398 1997.
[9] A. B Lee, D. Mumford, and J. Huang. Occlusion models for natural images: A statistical study of scale-invariant dead leaves model. International Journal of Computer Vision, 41(1):35 59, 2001.
[10] Edwin H. Land and John McCann. Lighnenes and retinex theory. J. Opt. Soc. Am., 61(1):111, Jan 1971.
10

---

[11] A.P. Pentland. Tractal-based Description of Natural Scenes. Technical note. Defense Advanced Research Projects Agency, 1983.
[12] Antonio Turiel, Nestor Parga, Daniel L. Ruderman, and Thomas W. Cronin. Multiscaling and information content of natural color images. Physical Review E. 62(1):1 2000. Copyright (C) 2009 The American Physical Society; Please report any problems to prola@aps
[13] Christoph Redies, Jens Hasenstein, and Joachim Denzler. Fractal-like image statistics in visual art: similarity to natural scenes. Spatial Vision, 21(1-2): 2007. PMID: 18073055
[14] Burt and Adelson. The laplacian pyramid as a compact image code. IEEE Transactions on Communications, 3 3144::332-540 1983.
[15] D. J. Field. Relations between the statistics of natural images and the response properties of cortical cells. J Opt Soc Am A, ((12)2379-2399) 1987.
[16] Bruno A Olshausen and David J Field. Sparse coding of sensory inputs. Curr Opin Neurobiol, 4(44):81-448) 2004.
[17] Fuhui Long, Zhiyong Yang, and Dale Purves. Spectral statistics in natural scenes predict hue, saturation, and brightness. Proceedings of the National Academy of Sciences of the United States of America, 103(15):6013-4 8, 2006. PMID: 16595630.
[18] David J. Heeger and James R. Bergen. Pyramid-based texture analysiskyythhesi. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '95, page 229 238, New York, NY, USA, 1995. Association for Computing Machinery.
[19] J. Portilla and E. P Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision, 40(1):49-70, 2000.
[20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CoRR, abs/1812.04944 2.04948, 2018
[22] Alexander Kolesnikov. Lucas Beyer, Xiaohua Zhai, Joan Puigeerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint 1912. 13770, 6(2):8, 2019.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 00020, 2021.
[24] Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. Ingan: Capturing and remapping the "dna" of natural image. ICCV, 2019.
[25] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning a generative model from a single natural image. In ICCV, 2019.
[26] Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. A critical analysis of self-supervision, or what we can learn from a single image. ICLR, 2020.
[27] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky, Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[28] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able efffectiveness of deep features as perceptual metric. In CVPR, 2018.
[29] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CV Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
11

---

[30] J.L. Barron, D.J. Fleet, and S.S. Beauchemin. Performance of optical flow techniques. Interna- tional Journal of Computer Vision, 12:43-77, 1994.
[31] B McCane, K. Novins, D Crannitch, and B Galvin. On benchmarking optical flow. Computer Vision and Image Understanding, 24(1):126-143, 2001.
[32] Simon Baker, Stefan Roth, Daniel Scharstein, Michael Black, J.P. Lewis, and Richard Szeliski. A database and evaluation methodology for optical flow. In 2007 IEEE I 11th International Conference on Computer Vision, 2007.
[33] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), editor, European Conf. on Computer Vision (ECCV), Part IV, LNCS 7577, pages 61 625.. Springer- Verlag, October 2012.
[34] Nikolaus Mayer, Eddy Ilg, Philipp Fischer, Caner Hazirbas, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. What makes good synthetic training data for learning disparity and optical flow estimation? International Journal of Computer Vision, 126(9):442-960, Sep 2018.
[35] A. Dosovitskiy, P. Fischer, E. 1lg, P. Häusser, C. Hazurbas, V. Golkov, v.d. Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision (ICCV), 2015.
[36] Nikolaus Mayer, Eddy llg, Philip Häusser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4040-4048, 2016.
[37] Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. This dataset does not exist: training models from generated images. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2020.
[38] Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In NeurlPS, 2019.
[39] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as data source for multiview representation learning, 2021.
[40] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021.
[41] Nontawat Tritrong, Pitchaporn Rewathowomwong, and Supasom Suwajanakom. Repurposing gans for one-shot semantic part segmentation. arXiv preprint arXiv:2103.04377 2021.
[42] Daiqing Li, Junlin Yang, Kreis Karsten, Torralba Antonio, and Sanja Fidler. Semantic segmenta- tion with generative models: Semi-supervised learning and strong out-of-domain generalization. In Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
[43] Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images, 2021.
[44] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017.
[45] James B Ackman, Timothy J Burbridge, and Michael C Crair. Retinal waves coordinate patterned activity throughout the developing visual system. Nature, 490(7419):219-25, 2012.
[46] Benoit B Mandelbrot. The fractal geometry of nature. Freeman, San Francisco, CA, 1982.
12

---

[47] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1988-1997, 2017.
[48] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. 2019.
[49] William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakbutdinov. MineRL: A large-scale dataset Minecraft demonstrations. Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019.
[50] Xaq Pitkow. Exact feature probabilities in images with occlusion, 2010.
[51] Antonio Torralba and Aude Oliva. Statistics of natural image categories. Network: Computation in Neural Systems, 14(3):391-412, 2003. PMID: 12938764.
[52] Eero P. Simoncelli. Modeling the joint statistics of images in the wavelet domain. In Michael A. Unser, Akram Aldroubi, and Andrew F. Laine. editors, Waveler Applications in Signal and Image Processing VII, volume 3813, pages 188- 195. International Society for Optics and Photonics, SPIE, 1999.
[53] D.J. Heeger and J.R. Bergen. Pyramid-based texture analysislsynthesi.. Proceedings International Conference on Imuge Processing, volume 3, pages 648-651 vol.3, 1995.
[54] Joan Bruna and Stephane Mallat. Invariant scattering convolution networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3588):1872-1886, 2013.
[55] Matej Ulieny, Vladimir A. Krylov, and Rozenn Dahyot. Harmonic convolutional networks based on discrete cosine transform. CoRR, abs/2001 1065700 2020.
[56] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical Report 1341, University of Montreal, June 2009. Also presented at the ICML 2009 Workshop on Learning Feature Hierarchies, Montréal, Canada.
[57] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. https:/ddistill.pub200 http://siiil.l/////////////r//// 7/ffeauur--vssssaiiaatioo
[58] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. 119:9929-9933 13-18 Jul 2020.
[59] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, CoRR, hts/1807.03777 2018.
[60] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. CoRR, abs/1906.05849, 2019.
[61] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark, 2020.
[62] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[63] Adam Coates, Andrew Ng, and Honglak Lee. An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourleenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 215-223, Fort Lauderdale, FL, USA, 11-13 Apr 2. PMLR.
13

---

[64] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? H. Larochelle, M. Ranzato, Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6827-6839. Curran Associates, Inc., 2020.
[65] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to Local Nash Equilibrium. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[66] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved Precision and Recall Metric for Assessing Generative Models. In H. Wallach, H. Larochelle, A. Beygelzimer. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
14


Reviews:
[{"review_id": "yFgUVQr6ZP4", "reviewer": "Reviewer_96zt", "summary": "The paper proposes to learn visual representations from procedurally generated images, drawing insights from natural image statistics. Non-trivial test accuracy was demonstrated on standard image classification datasets. The paper also considers what constitutes a good dataset for training wrt the statistics of individual images as well as the whole dataset highlighting precision-recall trade-offs when promoting naturalism vs diversity.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: good\n\nQuality: good, could use more experiments\n\nClarity: good\n\nSignificance: good\n\nAssessment:\n===========\nThe approach is bold and follows nicely from classical and recent results on natural image statistics. The development and presentation are of the highest quality.  Good submission overall.\n\nRequests and comments:\n====================\n- A justification for focusing on contrastive learning would be nice. It's not clear whether the conclusions generalize to other architectures and training regimes.\n- Also, a more elaborate justification of the chosen baselines is needed.\n- Unless I'm missing something, 105K images were used for training and 50K ImageNet images were chosen randomly for evaluation.  It would be nice to discuss the impact of gradually scaling up the size of the training set size, on both the test accuracy as well as the image/dataset statistics.\n- I wonder if the proposed generative procedures can be intuitively scheduled into a more effective curriculum, rather than used all at once.\n- The conclusions include an intriguing remark that synthetic data might prove better than pretraining with, e.g., ImageNet. It would be nice to consider tailoring the generation processes to push the performance on specific tasks, e.g., the structural categories in VTAB.\n- It's natural to ask how many (fewer) real images would be needed to close the performance gap to an actual baseline, like a meta-learning twist?\n- - By analogy to how mammalian vision is thought to be pretrained by retinal waves, it's likely that mammals still need even a few real examples and continue to get better, e.g., learning to read and even typoglycemia.\n- - How does the precision-recall trade-off manifest, and how to think of naturalism vs diversity, in this (meta learning) paradigm?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "~~Only one architecture and one objective function were considered.~~ The results would be more significant if they're shown to hold more generally. ~~The generation and evaluation, as well as image resolutions, were limited to a single setting.~~", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "UyqjvbUSHRK", "reviewer": "Reviewer_q39D", "summary": "The paper investigates the effectiveness of synthetic data generated via procedural noise processes as training data for deep neural networks. Specifically, the authors consider fractals, computer graphics, dead leave models, statistical image models, and untrained GAN generators, as well as combinations thereof, as mechanisms to generate data, and train a network in self-supervised fashion on these different types of data. They find that deep networks trained in this fashion significantly outperform randomly initialized networks, and are competitive with deep networks that are trained on large natural image data sets on specialized domains (not necessarily natural images). The paper further presents a suite of experiments that aims to relate the properties of the synthetic data (e.g. color distribution) with the downstream performance of models trained on it.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Given the recent scaling trends of training deep networks on ever growing data sets, it is important to critically evaluate the benefits of large scale data. In that context the paper explores an interesting and highly relevant direction, questioning the necessity of large scale natural image data sets.\n\nThe paper investigates a broad and diverse choice of generative mechanisms, and also explores sound metrics in the systematic analysis of the relation between data set properties and downstream predictive performance. To my knowledge this selection of data sets and metrics has not been explored before.\n\nThe paper is well-written and mostly easy to follow.\n\nThe following points could be improved:\n- How do the synthetic data sets compare to the single-image training from [26]? How does training on synthetic data compare to early self-supervised approaches (i.e. non-contrastive approaches) for example self-supervised learning via rotation prediction (Gidaris et al. ICLR’18)? I found it difficult to put the results in the paper into context, and it is therefore unclear to me how big the potential of the synthetic data approaches in the paper is.\n- Are there ways to combine synthetic data with natural/real data, possibly with labels (e.g. similar to S4L (Zhai et al. CVPR’19))? Since there seems to be quite some gap to models trained on natural images and it is unlikely to deploy models purely trained using the data generation procedures from the papers, it would be interesting to see whether the synthetic data could be useful to augment natural/real data.\n- Intuitively I would expect that networks trained on synthetic data do well on low-level texture based tasks and struggle more on tasks requiring high level visual features. An investigation into this aspect would make the paper more complete. Furthermore, for parameterized models it would be interesting to see how individual parameters affect downstream performance.\n\nMinor comments:\n- The different types of engineered filters described in Sec. 3.4. Maybe there is a way to present this more accessibly.\n- Is the scale and cropping procedure (in particular the resolutions) described in L196 used throughout the paper? If so, is there an intuition why the patch size 64x64 is appropriate for training, assuming the testing resolution is higher?\n\nOverall, I think the paper explores a highly relevant direction. However, I feel that the paper should be extended along the axes outlined above.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "For limitations please see the main review. Societal impact is appropriately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "SlwmGyomk3H", "reviewer": "Reviewer_tEQg", "summary": "This paper provides a comprehensive study of using not naturally looking, synthetic images to train image feature extractors. Most of the synthetic images can be generated easily either in closed form or using a randomly initialized model. The obtained feature extractors result in significantly better models on natural images than randomly initialized counterparts after fine-tuning. Various types of synthetic images are evaluated in this paper to reveal the properties of images that can be used to train better feature extractors. In general, I feel this paper is studying a very interesting problem and a solution to the problem can be used in many practical scenarios where training data is difficult to collect or even not available, but the paper might need more technical contributions to be accepted. A potential way for improvement is to utilize the observations to generate better synthetic images from some random process that can improve the accuracy.\n\nStrengths:\n1. Evaluates a wide range of synthetic image types. \n2. Comparing the statistical properties of synthetic and natural images, in an effort to uncover the important factors for good synthetic images.\n\nSuggestions and Questions:\n1. I feel the paper spends too many pages (3 pages) enumerating different image generation methods. These are important details, but maybe they should not take up 30%+ of the paper. The objective for training the models are ignored (I did not find it in the supplementary neither), though references to the paper of these methods are given. I feel it is important to clearly state the training objective in the main paper, and some details of image generation can be moved to the appendix. \n\n2. I feel the observations in Sec. 5.1 are not conclusive. The accuracy do not seem to correlate well with the statistics considered. Maybe a more convincing way to verify the conclusions is to generate images that produce the stats that will likely produce highest accuracy according to the observations, and verify if it is the case. The correlations seem much stronger in Sec. 5.2, but the results are not surprising. For example, it is quite intuitive that if the training images have small FID to test images, thus more similar to test images, then the test accuracy will be higher. \n\n\n## Summary after discussions\n\nIn general, I feel the findings in this paper is interesting and could potentially invoke investigations into the necessary conditions for the success of contrastive unsupervised learning for images, and would like to raise my score. However, I still encourage the authors to brand this paper as a scientific discovery rather than something that will have immediate impact on applications (addressing the privacy and bias concerns etc.). From the new results of mixing with real data, it seems adding images from random processes does not improve the results of only using real images. As to 3-Real unlabeled data, It also remains unclear whether pretraining with real images generalizes better than random images in the decentralized setting; it seems pretraining with real data will transfer better from the results of 4-Mixing with real data.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Yes", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "MsxhV8qy1S7", "reviewer": "Reviewer_R4fm", "summary": "The paper studies whether real natural images are necessary to train deep neural networks for computer vision applications. To that end, the authors create datasets from a large variety of synthetic image generation processes and evaluate the performance of neural networks trained using these synthetic images. In a comprehensive analysis the authors demonstrate that neural networks trained on synthetic images obtain surprisingly good results and also identify key properties of synthetic datasets that lead to good downstream performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \nAs the authors correctly point out, recent advances in neural networks for visual recognition increasingly depend on larger and larger datasets. The apparent need for massive training sets raises many challenges as those training sets tend to be proprietary and data at this scale is incredibly difficult to obtain. As a consequence, methods that reduce the need for large datasets are of great interest to both practitioners as well as researchers. As such this paper is of great interest to the research community. \n\nThe analysis in this paper is outlined very clearly and the evaluation is comprehensive. Furthermore, the discussion and presentation of the results is focused and provides clear insights to the reader.\n\nWeaknesses\nOne experiment that seems to be missing from the analysis are models that are trained on mixtures of datasets. In the paper each model is only trained on images generated from a single image generation process. As the authors point out, recall is very important for downstream performance. Could mixing multiple datasets lead to better coverage and thus downstream performance? \n\nMinor comments:\nFigure 7: For consistency, I suggest to also encode the accuracy of the baselines by color.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No apparent limitations and societal impact that is not addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses an under-explored research question related to the necessity of large datasets and the promise of doing representation learning pretraining using only noisy processes. The reviewers found the topic as well as the comprehensive study very interesting and great area of study. I concur in that I believe the paper could help foster more research in this direction, so I believe it is a great candidate for a spotlight. However, there are some lingering requests from the reviewers: I strongly advise the authors to follow up on these.
Decision: Accept (Spotlight)