Learning to See by Looking at Noise

1 Introduction . The importance of data in modern computer vision is hard to overstate . Time and again we have seen that better models are empowered by bigger data . The ImageNet dataset [ 1 ] , with its 1.4 million labeled images , is widely thought to have spurred the era of deep learning , and since then the scale of vision datasets has been increasing at a rapid pace ; current models are trained on up to one billion images [ 2 ] . In this paper , we question the necessity of such massive training sets of real images . Instead , we investigate a suite of procedural noise models that generate images from simple random processes . These are then used as training data for a visual representation learner . We identify two key properties that make for good synthetic data for training vision systems : 1 ) naturalism , 2 ) diversity . Interestingly , the most naturalistic data is not always the best , since naturalism can come at the cost of diversity . The fact that naturalistic data help may not be surprising , and it suggests that indeed , large-scale real data has value . However , we find that what is crucial is not that the data be real but that it be naturalistic , i.e . it must capture certain structural properties of real data . Many of these properties can be captured in simple noise models ( Fig . 1 ) . The implications of our work are severalfold . First , our results call into question the true complexity of the problem of vision – if very short programs can generate and train a high-performing vision system , then vision may be simpler than we thought , and might not require huge data-driven systems to achieve adequate performance . Second , our methods open the door to training vision systems without reliance on datasets . The value in this is that datasets are encumbered by numerous costs : ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . they may be expensive , biased , private , or simply intractable to collect . We do not argue for removing datasets from computer vision entirely ( as real data might be required for evaluation ) , but rather reconsidering what can be done in their absence . 2 Related work . 2.1 A short history of image generative models . Out of the R3n2 dimensional space spanned by 3× n2 color images , natural images occupy a small part of that space , the rest is mostly filled by noise . During the last decades , researchers have studied the space of natural images to build models capable of compressing , denoising and generating images . The result of this research is a sequence of generative image models of increasing complexity narrowing down the space occupied by natural images within R3n2 . One surprising finding is that natural images follow a power law on the magnitude of their Fourier transform [ 4 , 5 ] . This is the basis of Wiener image denoising [ 6 ] and scale-invariant models of natural images [ 7 , 8 ] . Dead leaves model [ 8 , 9 ] was an attempt at generating images that explained the power law found in natural images and inspired the Retinex algorithm [ 10 ] . The multiscale and self-similar nature of natural images inspired the use of fractals [ 11 , 12 , 13 ] as image models . Coding research for TV [ 5 ] and image modeling [ 6 , 14 , 15 , 16 ] showed another remarkable property of natural images : the output values of zero mean wavelets to natural images are sparse and follow a generalized Laplacian distribution [ 6 ] . Color and intensity distributions in natural images have also been studied and found to follow rules that deviate from random noise [ 10 , 17 ] . Research in texture synthesis showed how these statistical image models produced more realistic-looking textures [ 18 , 19 ] . Those required fitting the image model parameters to specific images to sample more `` like it '' . Recently , GANs [ 20 ] have shown remarkable image synthesis results [ 21 ] . Although GANs need real images to learn the network parameters , we show in this paper that they introduce a structural prior useful to encode image properties without requiring any training . 2.2 Training without real data and training with synthetic data . Through the above progression , generative models have become increasingly complex , with more parameters and more training data needed to fit these parameters . The same has happened with vision systems in general : state-of-the-art systems like BiT [ 22 ] , CLIP [ 23 ] , and SEER [ 2 ] obtain their best results on 300 million , 400 million , and 1 billion images respectively . These papers further show that such large data is critical to getting the best performance . While this may be true , other work has shown that much smaller data is sufficient to already get decent performance . A single image can be enough to train , from scratch , a compelling generative model [ 24 , 25 ] or visual representation [ 26 ] , and , even with no training data at all , deep architectures already encode useful image priors that can be exploited for low-level vision tasks [ 27 ] or for measuring perceptual similarity [ 28 ] . Our results , using an untrained StyleGANv2 [ 29 ] to generate training data , further affirm the utility of the structural priors in neural net architectures . An alternative to training with real data is to train on synthetic data . This approach has been widely used in low-level tasks like depth , stereo , or optical flow estimation [ 30 , 31 , 32 , 33 ] , where 3D rendering engines can provide densely annotated data to learn from . Interestingly , for this class of tasks diversity is more important than realism [ 34 ] , making procedurally generated scenes an effective alternative to renderings designed by professional 3D artists [ 35 , 36 ] . 2Answers : 1,3,4,5,6,8,14 are from ImageNet images . Recent work has also investigated using deep generative models as a source of synthetic data to train classifiers [ 37 , 38 ] and visual representations [ 39 ] , or to generate synthetic annotated data for other downstream tasks [ 40 , 41 , 42 ] . However , these generative models are still fit to real image datasets and produce realistic-looking images as samples . In this paper we push even further away from realism , generating synthetic data from simple noise processes . The closest prior work in this direction is the pioneering work of [ 43 ] , which used automatically generated fractals to pre-train neural networks that converge faster than their randomly initialized counterparts . While they demonstrated that fractals can be effective for pre-training , there is still a large gap compared to pre-training on real data . We explore a much broader range of noise processes , including many classic models from the image coding and texture synthesis literature . The use of randomized training data has also been explored under the heading of domain randomization [ 44 ] , where 3D synthetic data is rendered under a variety of lighting conditions to transfer to real environments where the lighting may be unknown . Our approach can be viewed as an extreme form of domain randomization that does away with the simulation engine entirely : make the training data so diverse that a natural image will just look like a sample from the noise process . There is some evidence that biology takes a similar approach during the prenatal development of the vision system . “ Retinal waves '' – spontaneous , semi-random activations of the retina – are thought to entrain edge detectors and other simple structures in the developing mammalian brain [ 45 ] . 3 A progression of image generative models . Here we provide the details for the image models we will use in this paper . We test a suite of generative models of the form gθ : z→ x , where z are stochastic latent variables and x is an image . We will treat image generation as a hierarchical process in which first the parameters of a model , θ , are sampled , and then the image is sampled given these parameters and stochastic noise . The parameters θ define properties of the distribution from which we will sample images , for example , the mean color of the image . The sampling process is as follows : θ ∼ p ( θ ) , z ∼ p ( z ) , and x = gθ ( z ) , which corresponds to sampling images from the distribution p ( x , θ ) = p ( x|θ ) p ( θ ) . The parameters θ that define the model may be fit to real data or not . We will explore the case where the parameters are not fit to real data but instead sampled from simple prior distributions . Next , we describe the generative image models that we will evaluate in this paper ( Fig . 2 ) . 3.1 Procedural image models . The first class of models belong to the family of procedural image models . Procedural models are capable of generating very realistic-looking images in specific image domains . We include in this set also fractals , although they could make a class on their own . Fractals : Fractals have been shown to capture geometric properties of elements found in nature [ 46 ] . Consequently , image models consisting of renderings of human-designed shapes with fractal structure [ 43 ] are likely to reproduce patterns found in natural images . CG : Simulators and game engines rely on a mixture of human-driven design and procedural methods to generate environments simulating real-world properties such as illumination , 3D , and semantics . Here we include three CG models popular in computer vision with available datasets : CLEVR [ 47 ] , DMLab [ 48 ] and MineRL [ 49 ] . 3.2 Dead leaves image models . The second family of models is Dead leaves , one of the simplest image models . We consider simple shapes ( leaves ) like circles , triangles and squares , which are positioned uniformly at random in the image canvas until it is covered . To produce each shape , we circumscribe it in a circle with a radius following an exponential distribution of parameter λ . This procedure has been shown to produce images that have similar statistics to natural images , such as having a 1/|f |α power spectrum [ 8 ] and non-gaussian distribution of derivatives and wavelet coefficients [ 50 ] . In this study we will consider four dead leaves models : Dead leaves - Squares : Only uses squares axis-aligned . Dead leaves - Oriented : Squares are randomly rotated . Dead leaves - Shapes : Leaves can be circles , triangles and rectangles . Dead leaves - Textured : uses square leaves filled with a texture sampled from the statistical image models described in the next section . 3.3 Statistical image models . The third family of models is statistical image models with increasing complexity . Several generative models can be composed by using different combinations of properties . Spectrum : The magnitude of the Fourier transform of many natural images follows a power law , 1/|f |α , where α is a constant close to 1 [ 4 ] . In this generative model , we will sample random noise images constrained to have FT magnitude following 1/ ( |fx|a + |fy|b ) with a and b being two random numbers uniformly sampled in the range [ 0.5 , 3.5 ] . This model introduces a bias towards horizontal and vertical image orientations typical of natural images [ 51 ] . To generate color images we first sample three random orthogonal color directions and then generate power-law-noise on each channel independently . Samples from this model are shown in Fig . 2 ( i ) . Wavelet-marginal model ( WMM ) : Following [ 52 ] , we generate textures by modeling their histograms of wavelet coefficients . To produce a texture , we create marginal histograms for the coefficients ci at N scales ( i ∈ { 0 ... N −1 } ) and 4 orientations following a Generalized normal distribution centered at zero , thus p ( ci ) ∝ exp ( ( −|ci|/αi ) βi ) . Each scale i represents the image down-scaled by a factor of 2i , and the parameters αi and βi for each scale are αi = 42 i and βi ∼ 0.4 + U ( 0 , 0.4 ) . In practice we use N = 3 and N = 4 for generating 128 × 128 and 256 × 256 resolution images respectively . Once we have sampled a marginal distribution of wavelet coefficients for each of the three channels , we do histogram matching iteratively starting from a Gaussian noise image following [ 53 ] . Fig . 2 ( j ) shows samples from this model . Color histograms : Here we take a generative model that follows the color distribution of the dead-leaves model . First we sample a number of regions N ∼ 3 + bU ( 0 , 20 ) c , their relative sizes S ∼ 0.001 + U ( 0 , 1 ) and color at uniform . This results in a color distribution different from uniform . Combining all these different models allows capturing color distributions , spectral components , and wavelet distributions that mimic those typical from natural images . Fig . 2 ( k ) shows the result of sampling from a model that enforces random white noise to have the power-law spectrum and the color histogram according to this model . Fig . 2 ( l ) shows samples from a model incorporating all of those properties ( spectrum , color and WMM ) . Those models produce intriguing images but fail to capture the full richness of natural images as shown in Fig . 2 ( i-l ) .

Summary: The paper proposes to learn visual representations from procedurally generated images, drawing insights from natural image statistics. Non-trivial test accuracy was demonstrated on standard image classification datasets. The paper also considers what constitutes a good dataset for training wrt the statistics of individual images as well as the whole dataset highlighting precision-recall trade-offs when promoting naturalism vs diversity.