Procedural Image Programs for Representation Learning
Authors: Manel Baradad
Chun-Fu Chen
Jonas Wulff
Tongzhou Wang
Rogerio Feris
Antonio Torralba
Phillip Isola
Venue: NeurIPS 2022 Accept (2022)
Keywords: procedural images
representation learning
generative image models
contrastive learning

Abstract: Learning image representations using synthetic data allows training neural networks without some of the concerns associated with real images, such as privacy and bias. Existing work focuses on a handful of curated generative processes which require expert knowledge to design, making it hard to scale up. To overcome this, we propose training with a large dataset of twenty-one thousand programs, each one generating a diverse set of synthetic images. These programs are short code snippets, which are easy to modify and fast to execute using OpenGL. The proposed dataset can be used for both supervised and unsupervised representation learning, and reduces the gap between pre-training with real and procedurally generated images by 38%.

Procedural Image Programs for Representation Learning
Manel Baradad', Chun-Fu (Richard) Chen', Jonas Wulff Tongzhou Wang', Rogerio Feris', Antonio Torralba', Phillip Isola*
MITT CSAIL, JJPMorg Chase Bank, N.A., XXyla, Inc, *MIT-IBM Watson Al Lab
Abstract
Learning image representations using synthetic data allows training neural networks without some of the concerns associated with real images, such as privacy and bias. Existing work focuses on handful of curated generative processes which require expert knowledge to design, making it hard to scale up. To overcome this, we propose training with a large dataset twenty-one thousand programs, each one generating diverse set of synthetic images. These programs short code snippets, which are easy to modify and fast to execute using OpenGL. The proposed dataset be used for both supervised and unsupervised representation learning and reduces the gap between pre- wee-training with real and procedurally generated images by 38%. Code, models, and datasets available at: https; //github. com http://aaadd/saaaereeeee
Introduction
Training neural networks using data from hand-crafted generative models a novel approach that
allows pre-training without access to real data. This technique has been shown to be effective downstream tasks for different modalities, such as images [ videos [3], and text [4]. Despite its big potential, previous work for images focuses textured polygons 3), or dead-leaves and statistical processes
handful of generative processes, like fractals [5),
To achieve good performance with the original set of procedures, expert knowledge is required to make the generated images match simple image statistics. This makes it hard to improve over existing work, these generative models have already been carefully studied and tuned. What would happen we used a large set of procedural image programs without curation, instead of focusing on a small set of programs?
To this end, we collect a large-scale set of programs, each producing a diverse set of images that display simple shapes and textures (see Figures I [2 for examples). We then use these programs to generate images and train neural networks with supervised and unsupervised representation learning methods. These networks can then be used for different downstream tasks, reaching state-of waat--of-the-ar performance for methods that do not pre-train using real images.
The generative programs of this collection are coded in a common language, the OpenGL shading language, that encapsulates the image generation process in few lines of code. Previous work uses ad-hoc slow methods for each generative process, which makes it hard to integrate them into the same framework. Compared existing approaches, the rendering of our programs is performed at high throughput (hundreds of frames per second with single modern GPU), allowing generation on the fly while training.
As the dataset consists of large collection generative processes, we can use each code snippet as a latent class, which allows learning representations using supervised approaches. In contrast, previous methods require ad-hoc techniques for clustering the generative parameters into discrete classes (5).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

b)
vold oo Coeec petyt, Preetoord syety on rirroyyed soreec indermnn ve
s-3--dert padd azeeriinte ax flaar B. diforr
Places3665
(eeec yoz-11)oet) prlgtragte darorrceeeoo deco teecibtut sh-rdl (6,3))
Imagenet-11 1k Top- Accuracy
Shaders 23k MixUp
40
Sky leyAA Oriented
30
FrectaiD8-1k
Ineeiiee stel ancEsnntabcon Ita), aegil Pudliiry yu 2ird eunter seprecttooote 24,34 pipn 2303:-0
20
Ocart Scavess
ene CeCleattitoc Peccllaag Lleath lol stpnts. esl i- 813 Jo. ahooee//arae
Fandom
Figure 1: a) We propose learning image representations using a vast collection (21k) of short OpenGL fragment shaders. The code and rendered images for two of them are shown. b) Top-1 Accuracy on ImageNet-Ik 1k for linear classification with ResNet-50 pre-trained with procedurally generated images from previous methods (Dead-Leaves Mixed 2 FractalDB D and StyleGAN Oriented (2) in gray) and our best generative image model, Shaders-21K k with MixUp (in red). The lower and upper-bound performance (in black) is for a random initialization and pretraining with Places365 (6.
Together with the collection of programs, the main contributions of the paper are:
A study how performance scales as function of the number of generative image programs, for supervised and unsupervised approaches. Our experiments show that downstream performance scales logarithmically with the number of generative programs (see Figure 3 and Section 4
State-of-the-art performance for representation learning without real data. When the number of programs is high enough, our approach outperforms previous methods by a large margin.
An analysis of the properties that make for a good image generation program in Section 5 This provides insights on how design new programs that maximize performance.
Previous work [2] showed that performance saturates fast when generating more samples from their data distribution (i.e. more images from their curated of programs), while our work shows that more samples from our distribution (i.e. more programs) do not saturate performance at the current scale. This points out clear direction on how further improve performance: collect or learn to generate more image programs.
To the best of our knowledge, this is the first work to pair large collection of mmage-generation processes and short programs that is amenable to experimentation. This set of procedural programs has the potential to be testbed to answer key questions linked to these paired modalities, some of which we answer in this paper.
2 Previous work
Pre-training with synthetic data Pre-training neural networks is well-known technique to improve performance and convergence speed during training. Typically, pre-trained networks are either used starting point in the optimization transfer the knowledge from related tasks to the of interest 7
When available, pre-training uses real data close possible to the task of interest. If the data available for the task is scarce, a common practice is to train on a dataset of natural images, typically ImageNet [8. These pre-trained models are then used on wide variety of tasks, achieving state-of- the-art results on tasks with different degrees of similarity to ImageNet. Despite this, it is unclear how much of the performance gain is due to knowledge transfer from the pre-training task the optimization procedure being simpler to engineer when starting from a pre-trained model (
Alternatively, other works achieve tate-offtheean performance by pre-training on synthetic data. These approaches generate realistie-looking images paired with ground-truth annotations, which are then used as pre-training data. Currently, this approach is a standard technique obtain state-of-the the

---

art performance for low-level vision tasks such as flow estimation [0, and depth prediction [2 between others.
Contrary to these approaches, recent works have studied whether it is necessary to use real-looking data for pre-training. The motivations for doing several folds, from targeting domains where rendering would require expert knowledge (like medical imaging), deploying agnostic models to downstream tasks, or making the generative process interpretable and optimizable [
In this spirit, the work of M introduced a way to train image models using renders with fractal structures. This procedure has been shown to work with different neural architectures [5] and has been further improved with better sampling and colorization strategies [13]. In the same spirit, 2 tested different generative processes to learn general image representations. Their best- rest-performing models are based on neural networks, which are not interpretable and require expert knowledge to design manually.
Representation learning Representation learning is at level of maturity that shallow networks on top of pre-trained models outperform supervised approaches trained from scratch [14 15]. Training on a large corpus with supervision is currently the best representation learning practice in terms of performance [16].
On the other hand, unsupervised approaches allow training with unlabeled data, which is usually easier to acquire than labeled data. Although both outperform models trained with smaller data collections, this comes at the price of more severe ethical issues, as models inherit the concerns associated with the datasets they are trained on.
The most widely used method for unsupervised representation learning is contrastive learning [17], which has seen consistent improvements in recent years [14] [8] Contrastive learning is well established and has been shown perform well in wide range of image domains, including medical
imaging [19] and satellite imagery [20]. Because of this, in this work we focus on contrastive learning as an unsupervised approach learning image representations.
Alternatively, early work on unsupervised representation learning focused on pretext tasks that different from contrastive training. such context prediction [2]] or rotation prediction [22] Although these did not achieve the levels of performance of contrastive learning, recent work has shown that of these pretext tasks, masked autoencoding with image prediction loss, coupled with Vision Transformers (33, can outperform contrastive training in large-scale settings [24). We do not study these novel approaches their robustness and broad applicability to different tasks and small training budgets is yet to be studied.
3 The Shaders-1k/21k dataset
The collection of programs we propose consists set of generative processes of the form: g x. Each generative program corresponds to the latent class gg. from which we can produce images by sampling the stochastic variables from prior model. The constant i indexes the programs available, while / corresponds to the numerical constants of the generative program. In the proposed collection, each d corresponds to OpenGL fragment shader. Existing similar methods 2 focus on a single type of program (i = 1, with variable 0), while this work focuses on a large collection of generative procedures (i {l,..., 21k}) with fixed 0.
In previous work, the generative programs were designed ad hoc in different programming languages, making it hard to scale or mix different approaches. To overcome this, we fix a common programming language and choose the OpenGL Shading Language. high-level programming language that allows specifying rendering programs with syntax based on the C programming language. Furthermore, it amenable to GPU acceleration, making it particularly suitable to quickly produce images using a standard deep learning stack.
3.1 Program collection
To obtain the desired OpenGL fragment shaders, we query the web for code snippets corresponding to fragment shaders. We obtain them from two sources: Twitter and Shadertoy 225 The programs
obtained from Twitter use the TwiGL. syntax as described in (6), which allows for extremely compact code (typically less than 280 characters) by abstracting away standard functionalities. On the other
3

---

Property
Avg.
Q5
Q95
T
283.8
160
662
chars
S
4307.2 2
499
13.6k
JPEG
T
70.1
15.0
149.5
(KB)
S
53.1
8.8
142.0
gzip
T 57.5
9.8
124.0
9075
(KB/img)
S
42.0
3.6
117.4
T 886.9
331.0
1.3k
FPS
S
984.4
293.5
1.4k
Table I: Dataset properties across shaders for TwiGL (T) and Shadertoy (S), reported as av- erage and 5/95-quantiles over all shaders of each subset. Statistics per shader have been computed with 400 samples resolution of 384 x 384, and rendering time is computed us- ing single Nvidia GeForce GTX TITAN X, including transfer to general memory.
Figure 2: Example images for single random timestep / for several random shaders. The first three columns correspond TwiGL (Shaders- -kkdders--k,, while the last three columns are shaders from Shadertoy (only included on Shaders-2Ik). As it is apparent, the complexity, of TwiGL shaders tends be superior.
shaders from
hand, Shadertoy a repository of OpenGL shaders using the standard OpenGL syntax. Shaders in Shadertoy are generally longer (on average 4k characters) and they produce simpler images than those from TwiGL, as the latter are typically coded by more expert users.
The only stochastic variable we consider in practice the collected shaders is the timestep (z = t E R). The parameter t allows generating continuous video stream, which in lots of cases is periodic. We sample at 4 frames per second for as many samples as required through all experiments. We add an increment to sampled uniform At [0, 0.25], avoid exact duplicates for the cases where the shader is periodic.
After downloading the code snippets from these two sources, obtain initial set of 52k shaders that compile with the OpenGL shading language syntax. We first perform duplicate removal by rendering single image using the same t for all scripts and removing shaders that produce the same images. Additionally, we remove scripts for which different t's produce the same image (i.e. they produce static images). With this, obtain final set of 1.089 fragment shaders from TwiGL and 19.994 from Shadertoy, which are unique and generate a diverse set images when varying t.
In Table we report the main characteristics of the shaders and the images they produce. Combining all the shaders from both sources, images can be rendered at 979 frames per second at a resolution of 384 x 384, using single modern GPU and including transfer to general memory. When stored to disk as JPEG, the average size per image is 54 kB, compared to 70 kB for ImageNet-Il 1k when resized to the same resolution.
Beaaders-1/2111 From a qualitative analysis and the compression results of the generated images in Table 1 we further conclude that TwiGL shaders are more curated and of higher quality than those obtained from Shadertoy (see Figure [ examples of both). To distinguish and experiment with both sources, we refer to the dataset composed of only TwiGL shaders as Shaders- 1k and the dataset consisting of available shaders from both sources as Shaders-21k.
Experiments
4.1 Supervised vs Unsupervised representation learning
To test downstream performance when pre-training with Shaders- Ik and Shaders-2Ik, we perform an initial set of experiments with and without latent class supervision. We test three representation learning methodologies: supervised classification with cross-entropy loss (CE), supervised contrastive learning (SupCon) (2), and unsupervised representation learning (SimCLR) [4].

---

Method
Dataset
S.CLR
CE
S.Con
50
SimCLR
Random init.
17.10
17.10
17.10
Imagenet 100 Accuracy
SupCon
Baselines
Places
52.20
50.7
55.22
Stylegan
I-100
60.30
78.2
80.18
Places
S.GAN O.
44.60
D.leaves M.
30.56
FracDB-Ik
33.12
26.5
32.98
S-lk
43.26
41.2
40.54
10 100 1k
10k: 21K
S-21k
47.74
40.3
48.10
Table 2: Top-1 accuracy with linear evalu- ation ImageNet- 100, for a ResNet-18 18 pre-trained with supervised methods (CE and SupCon) and an unsupervised method (SimCLR), with 100k examples and all the classes (if available). Underlined re- sults are upper bounds for real images.
Figure 3: Linear classification performance on ImageNet- 100 using ResNet- 18 trained with three representation learning methods and 100k images while varying the number of shaders. As dashed lines, we show perfor- mance with SimCLR trained on StyleGAN Oriented (state-of-the-ar baseline), Places365 (upper bound), and random network (lower bound).
To compare these training methodologies, train ResNet-18 with 100k samples generated for each dataset at a resolution of 256 x 256. We follow the training procedure and public implementation described in 27 for the three methods. We train for 200 epochs with batch size of 256, with the rest of the hyperparameters set to those found to work well for ImageNer- -00 in the original paper. The images after augmentations are fed the network resolution of 64 x 64. After training the encoder, we evaluate using the linear protocol described in [17], which consists of only training linear classifier on top of the representations learned. We evaluate on ImageNet- a subset of ImageNet- 1k [ defined in (28) using the averaged pooled features from the last convolutional layer, which have dimensionality of 512.
As shown in Table 2 we compare performance using the shader datasets against Places365 (6) which was proposed in [2] as upper bound for real images when testing on ImageNet; two procedurally generated datasets, FractaIDB I and Dead Leaves with mixed shapes 2 and the state of the art for training with similar data, StyleGAN Oriented 2
Shaders-22 2 using SupCon achieves the best performance over existing methods that do not use real data, but it is closely followed by the same dataset when using SimCLR. Both methods performing similarly is expected in this case, as the number of class positives per batch decreases with the number of classes in the dataset. In this experiment, the batch size is relatively small (256) compared the number of classes (21k). As the number of classes increases, the SupCon loss tends to be equivalent to SimCLR loss, and in the limit of having many classes training samples, they the same.
Using this setting, we test performance for an increasing number of generative programs, sampled at random from Shaders-21k. In this experiment, the number of images is kept constant at 100k while the number of shaders increases (i.c. with more shaders, there are fewer images per shader). As seen in Figure 3 we observe that supervision is helpful in the middle regime where there more than 20 shaders but less than 1000 shaders, but fails to outperform unsupervised approaches outside this regime. We also note from this sweep that performance is not saturated at 2 lk shaders, and it scales logarithmically with the number of shaders.
Finally, we also note from this sweep that using the full set of Shaders- 1k outperforms a random subset of Shaders-21k of same size. Shaders-I1 1 achieves higher top-1 accuracy on ImageNet- (43.26) compared to random subset of 1089 shaders obtained from Shaders-21k (42.26) This further verifies that the shaders Shaders- 1k are on average of higher quality than those on Shaders-21k, as hypothesized in Section
5

---

Pre-train Dataset
1-1k
I-100
VTAB Nat.
VTAB Spec.
VTAB Struet.
Random init
4.36
10.84
10.98
54.30
22.64
Places365 [6]
55.59
76.00
59.72
84.19
33.58
ImageNet-Ik
67.50
86.12
65.90
85.02
35.59
StyleGAN O.
38.
58.70
54.19
81.70
35.03
StyleGAN O.
(MixUp)
31.73
53.44
51.26
81.39
33.21
FractalDB-Ik 1k 1
23.86
44.06
38.80
76.93
31.01
Dead-leaves Mixed 2
20.00
38.34
35.87
74.22
30.81
S-1k
16.67
34.56
32.39
75.28
28.23
S-1k MixUp
38.42
60.04
53.24
82.08
30.32
S-21k
30.25
51.52
45.23
80.75
32.85
S-21k MixUp
44.83
66.36
57.18
84.08
31.84
Table Top-1 accuracy with linear evaluation on ImageNet- 1k/100 and the VTAB suite of datasets (averaged over the Natural, Specialized, and Structured categories), for a ResNer-50 trained with MoCo V2 with 1.3M samples for each the pre-training datasets. Underlined results correspond to an upper bound (training with natural images different than the evaluation distribution) and the previous state-of-the-arn without real data StyleGAN Oriented [2].
4.2 Large scale unsupervised training
Following the observation that contrastive training performs better than classification to pre-train feature extractors, we extend the previous approach to a large-scale setting. In this setting, we train ResNet-50 using MoCo v2 (18), with images generated 384 384 resolution. We train for 200 epochs with 1.3M images, with a batch size of 256, and set the rest of the hyperparameters to those found to work best ImageNet- lkk in the original paper. In Table we show downstream performance for ImageNet- and ImageNet- 100 and the 19 datasets of the Visual Task Adaptation Benchmark 229 averaged over the natural, specialized and structured categories.
When using data directly rendered from Shaders- 1/21k, the performance is low compared to existing methods. We observe that the contrastive loss is lower for Shaders- 1 compared to ImageNet- 1k (6.2 and 6.6), while top-1 accuracy for the contrastive task is much worse for Shaders- 1 (12.5% and 92.2% respectively). This points out that the network is able to solve the contrastive task properly on average (i.e. is highly confident on most the 64k negatives the MoCo buffer), while it is highly confused by few of the negative samples, which correspond images from the shader.
We hypothesize that this caused by large capacity network being able learn shortcut solutions for the contrastive task [30]. which results in learned features that do not transfer to downstream tasks. As can be seen from examples in Figure 2 it is casy for a large capacity network to distinguish different shaders via shortcut solutions, like background color salient features, even after MoCo v2 image augmentations.
To overcome this, we propose mixing samples from different shaders in order to produce interpolated samples in pixel space. This removes some of the shortcut solutions while preserving the complexity of the underlying processes being mixed. We investigate three well-known mixing strategies: CutMix 31, MixUp [22, and producing samples from a GAN 33 that has been trained on the shaders. MixUp has been previously studied in the context of unsupervised contrastive learning (34), showing improved performance in the case where the train time distribution is aligned with the test time distribution.
Using FID computed with 50k samples with respect to ImageNet-100,, we conclude that MixUp outperforms the other two strategies (details in Supp. Mat.). This metric was found to correlate with downstream performance in (2] and allows searching for good hyperparameters without training. For the MixUp strategy, we found that mixing 6 frames with weights sampled from Dirichlet distribution with a; = 1 yields the best FID, while rendering time is still affordable. For fair comparison against previous methods, MixUp is performed offline before training starts, so that the amount of images seen during training time is the same for all datasets (i.e. 1.3M fixed images, with the same MoCo v2 augmentations applied during training across all experiments).

---

Query
Random: 5-NN Acc: 4.3%
5-21k MixUp: 5-NN Acc: 43.2%
Places: 5-NN Acc: 57.0%
Figure 4: 5 nearest neighbors on ImageNet- 100 for ResNet-50 randomly initialized (left) or trained with MoCo v2 on Shaders-21k with MixUp (middle) and Places365 (right). Reported accuracy corresponds to 5-NN accuracy on ImageNet-100.
As seen in Table 3] training with Mix greatly outperforms the raw shaders. Shaders-2 lk with MixUp achieves the best performance overall on 1k00000 and outperforms the previous state-of-the-arr by substantial margin: an extra 6.71% top-1 accuracy on ImageNet-Ik, This represents a 38% relative improvement for the gap between real images different than ImageNet-1K Ik (Places365), and the previous state-of-the-arr for procedural images, StyleGAN Oriented (2). In Figure
4 we show hearest-neighbor, retrieval on Imagenet- 100 for our best model against random initialized network and the Places365 network (additional results for other networks can be found in Supp.Mat.). As can be seen, nearest neighbors successfully retrieves good candidates given query, capturing different perceptual aspects of the image that are not captured with a randomly initialized model.
We note that StyleGAN Oriented [2] does not improve with the MixUp strategy, achieving worse downstream performance than with the original samples. This is consistent with the FID metric, as the samples with MixUP yield worse (higher) FID than the samples without (38.74 vs 37.74).
In the case of VTAB, Shaders-1k/2111 with MixUp outperforms the baselines for natural and special- ized tasks, in the case of specialized tasks being close to Places performance. On the other hand, Chaders-1k/211 without MixUp outperform their MixUp counterparts on structured tasks, although performance for structured tasks is generally low for methods trained with contrastive learning. This is because structured tasks require information about counts, position, or size of the elements in the images, and this information is lost given the invariances that the data augmentations impose (35). From the results per dataset (see Supp. Mat.), it can be seen that Shaders- 1k performs well (even better than ImageNet) on datasets that resemble shaders, is the case of the dSprites dataset, which consists of 2D shapes procedurally generated.
Finally, we the fact that the Shaders dataset allows rendering novel samples during train time to train with the same procedure, but with live generator (i.e. the samples for each of the 200 epochs different). This methodology requires twice as much time to train with the same compute power, as the GPU is shared by the rendering and the training. Despite this, it only improves performance marginally, achieving top-1 accuracy on ImageNet-1k//li This shows that, similarly to other works, scaling up the number of samples per shader (and not the number of shaders) has limited benefits, as this corresponds 200 times more samples (corresponding to the 200 epochs) than the results presented in Table B
of 45. 24% and 66.42%, respectively.
5 What makes for good generative image program?
single shader: To test what properties of a single shader are useful for downstream performance, we train a ResNet-18 with SimCLR as described in Section forr each shader in Shaders- 1k. We use 10k images from each of the programs in Shaders- 1k, train the ResNet-18 for 100 epochs, and then compute performance with linear evaluation for 100 epochs on random subset of 10k images from ImageNet-100 As shown in Figure there substantial variability in performance between each of the shaders, and one of the main qualitative drivers for performance is the diversity between samples.
7

---

6.0%
6.5%
7.2%
16.1%
16.1%
18.2%
25.3%
25.4%
25.4%
Imageeee 100 Accuracy
10
Shader rank
Figure 5: Left: Performance for the shaders in Shaders- 1k trained with SimCLR on 10k images, evaluated on 10k samples of ImageNet- 100. Left-to-right: images for different shaders sorted by performance (on top of each block). Right: I-100 accuracy of shaders Shaders- Ik, sorted by rank.
25
25
Imaegret1100
10
10
10
150 100 0.2
0.4
0.6
0.8
1500 1000
500
10
100
LPIPS
FPS
(kB/img)
Figure 6: I-100 Performance against FID, LPIPS self-similarity, rendering time (frames per second), and gzip compression (for 400 images). Performance increases with lower FID (activations more similar ImageNet images), lower self self-similarity, higher rendering time (low FPS), and higher gzip file size, though the correlation is weak except for LPIPS self-similarity.
With the performance computed for each of the shaders in Shaders- 1K, train two shallow networks, consisting of two linear layers with ReLu and a hidden dimension of to predict performance per shader. The inputs to the shallow network are the features extracted with the ResNet-50 trained with MoCo v2 for Shaders- 1k with MixUp in Section 4.2 We train using an Li-loss to regress top-1 accuracy for each shader, splitting them 85-15 into train-val. Of the two trained networks, one gets information from 50 frames (as the concatenated average, minimum, and maximum of the activations) while the other only takes single image input. We these networks for the rest of the experiments this section, depending on whether the phenomena studied requires predicting performance from single multiple frames.
With the network that gets information from 50 frames, we rank all remaining shaders Shaders- 21k. In Figure 7 we show the shader with the best predicted performance over all Shaders-21k k, which is more qualitatively diverse than the top-performing in Shaders- (see Figure Figures)and its top-1 empirical accuracy is 36%, considerably higher than the 26% achieved for the best shader in Shaders- lk.
Additionally, in Figure we show scatter plots for several simple properties of the generated images for each shader against their performance. As can be seen, the metric that best correlates with performance (r 0.46) is LPIPS intra-image distance, matching the findings in [2] LPIPS (36) is a perceptual similarity metric that has been found to align better with human judgment than other alternatives. LPIPS intra-image is defined as the average LPIPS distance for two random crops covering 50% of given image and was first proposed (37. Consequently, we conclude that shaders that produce images with high self-sime self-similarity tend to perform poorly.

---

Top predicted
S-21k selection
#1
#2
Pred. S-21k
Rand. S-21k
Imagenet- Accuracy
S-1k
40
S-1k
30
25
15
n
Figure 7: 1-100 Top-1 accuracy with 100k images and SimCLR with different sets of shaders selected: 1) at random from Shaders-2ll lk and 2) by predicted performance from all Shaders-22 2 that remain unseen while training the predictor. As reference, we also plot the performance of the top shaders of Shaders- Ik, but we note that this is a more limited set than the full Shaders-21k. The right-most columns are images from the top predicted shaders from Shaders 21k, the best best-performing one achieving 36% top-1 accuracy empirically, compared to 26% for the best in Shaders- Ik.
Figure 8: Walks on & space for a StyleGAN trained with Shaders- Ik, starting at a random 2 (center column of each block) and following the trajectory that minimizes (first two columns) maximizes (last two columns) performance. Factors that drive the performance include color diversity, filling the whole canvas, and intra-diversity, following the findings of the metries in Figure 6
On the other hand, although FID and gzip compression have a low correlation with performance, the plots show that there is Pareto frontier for both metrics, where shaders cannot achieve high accuracy with low compression or high FID. Performance as a function rendering time also suggests that there is big margin for improvement, the correlation between rendering time and performance is weak and there are lots of shaders that perform well while their rendering time is low.
A collection of shaders: In the same spirit, consider what makes collection of shaders perform well or bad when treated as a set, and whether is possible to achieve increasingly better performance with fixed number of shader programs. In Figure 7 we compare using variable number of shaders selected either by predicted performance (with the same predictor as before that takes 50 frames) at random. As seen in Figure 7 the greedy approach of cach time selecting the shader with the best- rest-predicted performance outperforms by a big margin random selection.
Finally, as qualitative experiment of the properties of the Shaders- 1k dataset that make for good shader, we explore the space StyleGAN v2 trained with Shaders- with the shallow predictor that only takes features from single image. We explore StyleGAN space by tracing walks in the embedding space z, in the direction that minimizes or maximizes performance. As Figure shows, samples that improve performance obtained after traversing in the direction that minimizes performance tend collapse to repetitive shapes, scay- images, or big regions of the image of solid color.
more colorful and fill the canvas. On the other hand, samples

---

Conclusions
This paper proposes using large collection of procedural image programs image representations without real images. As experiments show, training with shaders outperforms existing methods that do not use real data. Using these procedural image models is competitive against natural images. especially in domains far away from the natural image domain, as demonstrated in the specialized tasks in VTAB.
Acknowledgement
Manel Baradad was supported by the LaCaixa Fellowship, and this research was supported by grant from the MIT-IBM Watson Al lab. Rogerio Feris was supported in part by Darpa LwLL. Experiments were partially conducted using computation resources from the Satori cluster donated by IBM to MIT, and the MIT's Supercloud cluster.
Disclaimer
This paper was prepared for information purposes by the teams of researchers from the various institutions identified above, including the Global Technology Applied Research group of JPMorgan Chase Bank, N.A.. This paper is product of the Research Department of JPMorgan Chase Bank, N.A. or its affiliates. Neither JPMorgan Chase Bank, N.A. nor any of its affiliates make any explicit or implied representation or warranty and none of them accept any liability in connection with this paper, including, but limited to, the completeness, accuracy, reliability of information contained herein and the potential legal, compliance, tax or accounting effects thereof. This document is not intended as investment research investment advice, or recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or be used in any way for evaluating the merits of participating in any transaction.
References
[1] Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images. CoRR, abs/2101.08515, 2021.
[2] Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking at noise. In Advances in Neural Information Processing Systems, 2021.
[3] Deqing Sun, Daniel Vlasic, Charles Herrmann, Varun Jampani, Michael Krainin, Huiwen Chang, Ramin Zabih, William T Freeman, and Ce Liu. Autoflow: Learning better training set for optical flow. In CVPR, 2021.
[4] Kundan Krishna, Jeffrey P. Bigham, and Zachary C. Lipton. Does pretraining for summarization require knowledge transfer? CoRR, abs/2109.0499 2021.
[5] Kodai Nakashima, Hirokatsu Kataoka, Asato Matsumoto, Kenji Iwata, and Nakamasa Inoue. Can vision transformers learn without natural images? In CoRR:2103. 2021.
[6] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[7] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. CoRR, abs/1911.022 2019.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 255. leee, 2009.
[9] Kaiming He, Ross B. Girshick, and Piotr Dollár. Rethinking imagenet pre-training. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4917-4926, 2019.
10

---

[10] A. Dosovitskiy, P. Fischer, E. 1lg, P. Häusser, C. Hazırbas, V. Golkov, P. v.d. Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In IEEE International Conference Computer Vision (ICCV), 2015.
[11] Zachary Teed and Jia Deng. RAFT: recurrent all-pairs field transforms for optical flow. In Computer Vision ECCV 2020- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part H1 volume 12347 of Lecture Notes in Computer Science, pages 402-419. Springer, 2020.
[12] Cho- Ying Wu, Jialiang Wang, Michael Hall, Ulrich Neumann, and Shuochen Su. Toward practical supervupe monocular indoor depth estimation. CoRR, abs/21 2021.
[13] Connor Anderson and Ryan Farrell. Improving fractal pre-training. In WACV, pages 1300-1309, 2022.
[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple frame- work for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020.
[15] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021.
[16] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint arXiv: 1912.1172.100 6(2):8, 2019.
[17] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807 807.03748, 2018.
[18] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momen- tum contrastive learning. CoRR, abs/2003.04297, 2020.
[19] Krishna Chaitanya, Ertune Erdil, Neerav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advances in Neural Information Processing Systems, 33, 2020.
[20] Oscar Mañas, Alexandre Lacoste, Xavier Giró-i Nieto, David Vazquez, and Pau Rodríguez. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. In Proceed- ings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 9414-9423, October 2021.
[21] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In International Conference on Computer Vision (ICCV), 2015.
[22] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference Learning Representations, 2018.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. CoRR, abs/2111.06377,, 2021.
[25] Shadertoy. Shadertoy. httpe://ww..haddettoy....l
2021. Accessed: 2022-05-15.
[26] TwiGL. Twigl. httpp://giihubb.com/oaaaaa
2021. Accessed: 2022-05-15.

---

[27] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc., 2020.
[28] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. CoRR, abs/1906.05549. 2019.
[29] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. CoRR, abs/ Abss/191000486,, 2019.
[30] Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shorteut solutions? NeurlPS, 2021.
[31] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision (ICCV), 2019.
[32] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018.
[33] Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. arXiv preprint wrXiv:2106.05255 2021.
[34] Zhiqiang Shen, Zechun Liu, Zhuang Liu, Marios Savvides, and Trevor Darrell. Rethinking image mixture for unsupervised visual representation learning. CoRR, abs/2003.05434 2020.
[35] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations, 2021.
[36] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[37] Vun Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In I. Guyon, U. Luxburg. S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 465-476. Curran Associates, Inc., 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] https
//github. cow htmm/mbaradad// comm/bbaraaaa//saaaeeeen. shaders21k
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? (Yes] For novel techniques, we report hyperparameters. For well-known methods, we use the default set of hyperparameters, described in the references
12

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We run large scale experiments (ImageNet- 1k scale), and multiple runs are computationally unfeasible
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We include total amount of compute to generate the data. For the compute requiered is the same as the original methods.
methods, we use standard hyperparameters, and
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] We use code-snippets available online, and we will only release the references to the available websites.
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] We use code-snippets available online, and we will only release the references the available websites.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We have checked the output of the generative image programs through our experiments and we have not found offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "bj1juJ4LBJ6s", "reviewer": "Reviewer_mJZE", "summary": "This paper studies procedural image programs. Procedural image programs are image representation learning methods based on synthetically generated unnatural images (e.g., fractals and noise). The authors collected OpenGL fragment shader codes to generate diverse synthetic images and trained their image representation model on this dataset. This paper evaluated the proposed method on ImageNet and verified that their approach outperforms existing other unnatural image generation methods.", "questions": "I left questions in the \"Strengths and Weaknesses\" section.", "limitations": "The authors have adequately addressed the limitations and potential negative social impact of their work.", "rating": 3, "confidence": 1, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. This paper aims to train image representation models without natural images, a challenging research problem.\n\nWeaknesses\n\n1. The problem statement in the introduction paragraph is that using real images causes privacy and bias problems (lines 1-2). However, this paper only shows the performance of their approach on the ImageNet dataset. Please provide more experimental results that support the original claim, alleviating the privacy problem and the bias problem.\n2. This paper is hard to follow. Please add more descriptions\n    1. Please provide more descriptions of existing approaches, such as the Fractal and other image generation methods. As the research topic of this paper covers unfamiliar concepts and methods, more description for background knowledge is required.\n    2. Section 3.1 is hard to follow. What is the time paramter, t?\n3. More experimental results are required.\n    1. What happens if we use only a part of the collected programs? Can we categorize the programs? Can we measure the performance in each category? Is the Fractal method included in the collected programs?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XMBmS3Zpx2j", "reviewer": "Reviewer_vdrw", "summary": "Manuscript presents a method to train neural networks over synthetic data. The work collected a large collection of 21K OpenGL programs for rendering diverse set of synthetic images to be used for representation learning. Experiments show that the collected programs can be used for pretraining in both supervised and unsupervised fashion. New state of the art results are obtained for pretraining with procedurally generated images and are observed to compete closely against pretraining with real data samples.", "questions": "- While the authors clearly mention that their approach involves generating images from several processes (but with fixed $\\theta$), it is not clearly described how it is different from the existing works [1, 2] which focus on a single program ($i=1$) but with varied parameters $(\\theta)$ ? In other words, although it may be intuitive to see that more processes are better than a single process, how to quantify (even loosely) the bigger or better quotient given the degrees of freedom are different ($\\theta$ versus $i$ in lines 94-100).", "limitations": "- As discussed in the weaknesses section, the major limitation of the work is that fails to go beyond the collection of large-scale image generating programs. Authors have attempted to study the collection and draw useful properties for crafting such programs that are more useful, but in my opinion that needs more work.\n- Outright I do not see any adverse impacts of the present work on the society.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Collected a large-scale repository of image programs for easily generating variety of synthetic images. These data samples can help pretraining while avoiding the unfair biases and other contaminations that arise from natural datasets.\n- Because of the large scale, the collection enables to study how the performance scales with the size and variety of the synthetic data.\n- Representation learning experiments have been performed in both the supervised and unsupervised scenarios.\n\nWeaknesses\n- The major contribution of the manuscript is somehow not very strong (despite being very important). In other words, in my opinion, large-scale collection of procedural image programs alone may not be sufficient enough to excite the NeurIPS community.\n- It is easy and trivial to see that more and diverse set of synthetic data samples improve the pretraining quality. They can outperform the existing small-scale collections and close the gap between the performance of synthetic and real image datasets.\n- I believe studying the effectiveness of the individual programs would be an interesting and useful contribution. The presented analysis on the properties of the programs that leads to better learning is not very conclusive or effective. In the end the reader may not feel very clear about the desired properties (except for weak observations such as LPIPS, more colors, etc.) and the study feels incomplete or lacking. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IC50ys2knWU", "reviewer": "Reviewer_CTGk", "summary": "This paper proposed a dataset for learning representations without access to real images, the dataset is composed of twenty-one thousand OpenGL programs is proposed for supervised and unsupervised representation learning, the synthetic data can help reduce the concerns about using real images in training such as human face etc. \nExperiments show that pretraining on the proposed dataset can bring performance improvement over previous synthetic dataset, and the investigation on what makes for a good generative image program seems interesting.\n", "questions": "Q1: It would be good if the paper could also include some visualization results of the learned representations like in [R1], and maybe the linear readout performance of the representations per layer? I think this could give the reader more sense of how well the learned representation is.\n\nQ2: Should the augmentations on synethic data be the same as real data? IMO, the data augmentations used by MoCov2 on ImageNet is choosen because the augmentation can encode some prior knowledge about the real images, so probably when we are training the model on synthetic images, the choice of augmentations should be different.\n\nQ3: There are efforts of reducing privacy concerns in the real data such as PASS [R2], it would be good if the paper could discuss the connection to these types of works and what is the different between using synthetic images and modified real images. \n\nQ4: It would also be interesting to see how well the learned representation on the proposed dataset transfers by finetuning the representation on a target dataset, because as shown in [R3], the linear readout performance does not always correlates with the full finetuning results.\n\n[R1] A critical analysis of self-supervision, or what we can learn from a single image, ICLR 2020\n\n[R2] PASS: An ImageNet replacement for self-supervised pretraining without humans, NeurIPS 2021 Datasets and Benchmarks\n\n[R3] Masked Autoencoders Are Scalable Vision Learners, CVPR 2022\n", "limitations": "1. There is no section about the limitation and negative societal impact in the main paper and the supplementary, but IMO learning representation on synthetic data does not bring any negative impact in general, however, regarding this work, I am curious about the license of the all the shader used in the dataset, do all of them allow for research purposes?\n2. One limitation for learning representation using synthetic data is the performance is not comparable with training on real images, but I think by studying what factors help learning a good representation using synthetic data, this paper is a meanful step towards makeing representation learned on synthetic data usable.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: The goal of learning representation without real images is practically important.\n\nS2: The analysis of the properties for good image generation programs could be of interest for people who are working on image generation programs for representation learning.\n\nW1: The evaluation of the learned representations on the proposed dataset is only limited to linear readout, while I believe using full finetuning can better show the difference between representations and are closer to a real use case.\n\nW2: There are works that aims to learn representations using only one image [R1] or using filtered real images [R2], I think the paper should include discussion with these works because the goal is the same: learning the representation without ethical concerns.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept