Learning to See by Looking at Noise

1 Introduction . The importance of data in modern computer vision is hard to overstate . Time and again we have seen that better models are empowered by bigger data . The ImageNet dataset [ 1 ] , with its 1.4 million labeled images , is widely thought to have spurred the era of deep learning , and since then the scale of vision datasets has been increasing at a rapid pace ; current models are trained on up to one billion images [ 2 ] . In this paper , we question the necessity of such massive training sets of real images . Instead , we investigate a suite of procedural noise models that generate images from simple random processes . These are then used as training data for a visual representation learner . We identify two key properties that make for good synthetic data for training vision systems : 1 ) naturalism , 2 ) diversity . Interestingly , the most naturalistic data is not always the best , since naturalism can come at the cost of diversity . The fact that naturalistic data help may not be surprising , and it suggests that indeed , large-scale real data has value . However , we find that what is crucial is not that the data be real but that it be naturalistic , i.e . it must capture certain structural properties of real data . Many of these properties can be captured in simple noise models ( Fig . 1 ) . The implications of our work are severalfold . First , our results call into question the true complexity of the problem of vision – if very short programs can generate and train a high-performing vision system , then vision may be simpler than we thought , and might not require huge data-driven systems to achieve adequate performance . Second , our methods open the door to training vision systems without reliance on datasets . The value in this is that datasets are encumbered by numerous costs : ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . they may be expensive , biased , private , or simply intractable to collect . We do not argue for removing datasets from computer vision entirely ( as real data might be required for evaluation ) , but rather reconsidering what can be done in their absence . 2 Related work . 2.1 A short history of image generative models . Out of the R3n2 dimensional space spanned by 3× n2 color images , natural images occupy a small part of that space , the rest is mostly filled by noise . During the last decades , researchers have studied the space of natural images to build models capable of compressing , denoising and generating images . The result of this research is a sequence of generative image models of increasing complexity narrowing down the space occupied by natural images within R3n2 . One surprising finding is that natural images follow a power law on the magnitude of their Fourier transform [ 4 , 5 ] . This is the basis of Wiener image denoising [ 6 ] and scale-invariant models of natural images [ 7 , 8 ] . Dead leaves model [ 8 , 9 ] was an attempt at generating images that explained the power law found in natural images and inspired the Retinex algorithm [ 10 ] . The multiscale and self-similar nature of natural images inspired the use of fractals [ 11 , 12 , 13 ] as image models . Coding research for TV [ 5 ] and image modeling [ 6 , 14 , 15 , 16 ] showed another remarkable property of natural images : the output values of zero mean wavelets to natural images are sparse and follow a generalized Laplacian distribution [ 6 ] . Color and intensity distributions in natural images have also been studied and found to follow rules that deviate from random noise [ 10 , 17 ] . Research in texture synthesis showed how these statistical image models produced more realistic-looking textures [ 18 , 19 ] . Those required fitting the image model parameters to specific images to sample more `` like it '' . Recently , GANs [ 20 ] have shown remarkable image synthesis results [ 21 ] . Although GANs need real images to learn the network parameters , we show in this paper that they introduce a structural prior useful to encode image properties without requiring any training . 2.2 Training without real data and training with synthetic data . Through the above progression , generative models have become increasingly complex , with more parameters and more training data needed to fit these parameters . The same has happened with vision systems in general : state-of-the-art systems like BiT [ 22 ] , CLIP [ 23 ] , and SEER [ 2 ] obtain their best results on 300 million , 400 million , and 1 billion images respectively . These papers further show that such large data is critical to getting the best performance . While this may be true , other work has shown that much smaller data is sufficient to already get decent performance . A single image can be enough to train , from scratch , a compelling generative model [ 24 , 25 ] or visual representation [ 26 ] , and , even with no training data at all , deep architectures already encode useful image priors that can be exploited for low-level vision tasks [ 27 ] or for measuring perceptual similarity [ 28 ] . Our results , using an untrained StyleGANv2 [ 29 ] to generate training data , further affirm the utility of the structural priors in neural net architectures . An alternative to training with real data is to train on synthetic data . This approach has been widely used in low-level tasks like depth , stereo , or optical flow estimation [ 30 , 31 , 32 , 33 ] , where 3D rendering engines can provide densely annotated data to learn from . Interestingly , for this class of tasks diversity is more important than realism [ 34 ] , making procedurally generated scenes an effective alternative to renderings designed by professional 3D artists [ 35 , 36 ] . 2Answers : 1,3,4,5,6,8,14 are from ImageNet images . Recent work has also investigated using deep generative models as a source of synthetic data to train classifiers [ 37 , 38 ] and visual representations [ 39 ] , or to generate synthetic annotated data for other downstream tasks [ 40 , 41 , 42 ] . However , these generative models are still fit to real image datasets and produce realistic-looking images as samples . In this paper we push even further away from realism , generating synthetic data from simple noise processes . The closest prior work in this direction is the pioneering work of [ 43 ] , which used automatically generated fractals to pre-train neural networks that converge faster than their randomly initialized counterparts . While they demonstrated that fractals can be effective for pre-training , there is still a large gap compared to pre-training on real data . We explore a much broader range of noise processes , including many classic models from the image coding and texture synthesis literature . The use of randomized training data has also been explored under the heading of domain randomization [ 44 ] , where 3D synthetic data is rendered under a variety of lighting conditions to transfer to real environments where the lighting may be unknown . Our approach can be viewed as an extreme form of domain randomization that does away with the simulation engine entirely : make the training data so diverse that a natural image will just look like a sample from the noise process . There is some evidence that biology takes a similar approach during the prenatal development of the vision system . “ Retinal waves '' – spontaneous , semi-random activations of the retina – are thought to entrain edge detectors and other simple structures in the developing mammalian brain [ 45 ] . 3 A progression of image generative models . Here we provide the details for the image models we will use in this paper . We test a suite of generative models of the form gθ : z→ x , where z are stochastic latent variables and x is an image . We will treat image generation as a hierarchical process in which first the parameters of a model , θ , are sampled , and then the image is sampled given these parameters and stochastic noise . The parameters θ define properties of the distribution from which we will sample images , for example , the mean color of the image . The sampling process is as follows : θ ∼ p ( θ ) , z ∼ p ( z ) , and x = gθ ( z ) , which corresponds to sampling images from the distribution p ( x , θ ) = p ( x|θ ) p ( θ ) . The parameters θ that define the model may be fit to real data or not . We will explore the case where the parameters are not fit to real data but instead sampled from simple prior distributions . Next , we describe the generative image models that we will evaluate in this paper ( Fig . 2 ) . 3.1 Procedural image models . The first class of models belong to the family of procedural image models . Procedural models are capable of generating very realistic-looking images in specific image domains . We include in this set also fractals , although they could make a class on their own . Fractals : Fractals have been shown to capture geometric properties of elements found in nature [ 46 ] . Consequently , image models consisting of renderings of human-designed shapes with fractal structure [ 43 ] are likely to reproduce patterns found in natural images . CG : Simulators and game engines rely on a mixture of human-driven design and procedural methods to generate environments simulating real-world properties such as illumination , 3D , and semantics . Here we include three CG models popular in computer vision with available datasets : CLEVR [ 47 ] , DMLab [ 48 ] and MineRL [ 49 ] . 3.2 Dead leaves image models . The second family of models is Dead leaves , one of the simplest image models . We consider simple shapes ( leaves ) like circles , triangles and squares , which are positioned uniformly at random in the image canvas until it is covered . To produce each shape , we circumscribe it in a circle with a radius following an exponential distribution of parameter λ . This procedure has been shown to produce images that have similar statistics to natural images , such as having a 1/|f |α power spectrum [ 8 ] and non-gaussian distribution of derivatives and wavelet coefficients [ 50 ] . In this study we will consider four dead leaves models : Dead leaves - Squares : Only uses squares axis-aligned . Dead leaves - Oriented : Squares are randomly rotated . Dead leaves - Shapes : Leaves can be circles , triangles and rectangles . Dead leaves - Textured : uses square leaves filled with a texture sampled from the statistical image models described in the next section . 3.3 Statistical image models . The third family of models is statistical image models with increasing complexity . Several generative models can be composed by using different combinations of properties . Spectrum : The magnitude of the Fourier transform of many natural images follows a power law , 1/|f |α , where α is a constant close to 1 [ 4 ] . In this generative model , we will sample random noise images constrained to have FT magnitude following 1/ ( |fx|a + |fy|b ) with a and b being two random numbers uniformly sampled in the range [ 0.5 , 3.5 ] . This model introduces a bias towards horizontal and vertical image orientations typical of natural images [ 51 ] . To generate color images we first sample three random orthogonal color directions and then generate power-law-noise on each channel independently . Samples from this model are shown in Fig . 2 ( i ) . Wavelet-marginal model ( WMM ) : Following [ 52 ] , we generate textures by modeling their histograms of wavelet coefficients . To produce a texture , we create marginal histograms for the coefficients ci at N scales ( i ∈ { 0 ... N −1 } ) and 4 orientations following a Generalized normal distribution centered at zero , thus p ( ci ) ∝ exp ( ( −|ci|/αi ) βi ) . Each scale i represents the image down-scaled by a factor of 2i , and the parameters αi and βi for each scale are αi = 42 i and βi ∼ 0.4 + U ( 0 , 0.4 ) . In practice we use N = 3 and N = 4 for generating 128 × 128 and 256 × 256 resolution images respectively . Once we have sampled a marginal distribution of wavelet coefficients for each of the three channels , we do histogram matching iteratively starting from a Gaussian noise image following [ 53 ] . Fig . 2 ( j ) shows samples from this model . Color histograms : Here we take a generative model that follows the color distribution of the dead-leaves model . First we sample a number of regions N ∼ 3 + bU ( 0 , 20 ) c , their relative sizes S ∼ 0.001 + U ( 0 , 1 ) and color at uniform . This results in a color distribution different from uniform . Combining all these different models allows capturing color distributions , spectral components , and wavelet distributions that mimic those typical from natural images . Fig . 2 ( k ) shows the result of sampling from a model that enforces random white noise to have the power-law spectrum and the color histogram according to this model . Fig . 2 ( l ) shows samples from a model incorporating all of those properties ( spectrum , color and WMM ) . Those models produce intriguing images but fail to capture the full richness of natural images as shown in Fig . 2 ( i-l ) .

Summary: This paper provides a comprehensive study of using not naturally looking, synthetic images to train image feature extractors. Most of the synthetic images can be generated easily either in closed form or using a randomly initialized model. The obtained feature extractors result in significantly better models on natural images than randomly initialized counterparts after fine-tuning. Various types of synthetic images are evaluated in this paper to reveal the properties of images that can be used to train better feature extractors. In general, I feel this paper is studying a very interesting problem and a solution to the problem can be used in many practical scenarios where training data is difficult to collect or even not available, but the paper might need more technical contributions to be accepted. A potential way for improvement is to utilize the observations to generate better synthetic images from some random process that can improve the accuracy.  Strengths: 1. Evaluates a wide range of synthetic image types.  2. Comparing the statistical properties of synthetic and natural images, in an effort to uncover the important factors for good synthetic images.  Suggestions and Questions: 1. I feel the paper spends too many pages (3 pages) enumerating different image generation methods. These are important details, but maybe they should not take up 30%+ of the paper. The objective for training the models are ignored (I did not find it in the supplementary neither), though references to the paper of these methods are given. I feel it is important to clearly state the training objective in the main paper, and some details of image generation can be moved to the appendix.   2. I feel the observations in Sec. 5.1 are not conclusive. The accuracy do not seem to correlate well with the statistics considered. Maybe a more convincing way to verify the conclusions is to generate images that produce the stats that will likely produce highest accuracy according to the observations, and verify if it is the case. The correlations seem much stronger in Sec. 5.2, but the results are not surprising. For example, it is quite intuitive that if the training images have small FID to test images, thus more similar to test images, then the test accuracy will be higher.    ## Summary after discussions  In general, I feel the findings in this paper is interesting and could potentially invoke investigations into the necessary conditions for the success of contrastive unsupervised learning for images, and would like to raise my score. However, I still encourage the authors to brand this paper as a scientific discovery rather than something that will have immediate impact on applications (addressing the privacy and bias concerns etc.). From the new results of mixing with real data, it seems adding images from random processes does not improve the results of only using real images. As to 3-Real unlabeled data, It also remains unclear whether pretraining with real images generalizes better than random images in the decentralized setting; it seems pretraining with real data will transfer better from the results of 4-Mixing with real data.