Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper studies procedural image programs. Procedural image programs are image representation learning methods based on synthetically generated unnatural images (e.g., fractals and noise). The authors collected OpenGL fragment shader codes to generate diverse synthetic images and trained their image representation model on this dataset. This paper evaluated the proposed method on ImageNet and verified that their approach outperforms existing other unnatural image generation methods.
strengths_and_weaknesses: Strengths

1. This paper aims to train image representation models without natural images, a challenging research problem.

Weaknesses

1. The problem statement in the introduction paragraph is that using real images causes privacy and bias problems (lines 1-2). However, this paper only shows the performance of their approach on the ImageNet dataset. Please provide more experimental results that support the original claim, alleviating the privacy problem and the bias problem.
2. This paper is hard to follow. Please add more descriptions
    1. Please provide more descriptions of existing approaches, such as the Fractal and other image generation methods. As the research topic of this paper covers unfamiliar concepts and methods, more description for background knowledge is required.
    2. Section 3.1 is hard to follow. What is the time paramter, t?
3. More experimental results are required.
    1. What happens if we use only a part of the collected programs? Can we categorize the programs? Can we measure the performance in each category? Is the Fractal method included in the collected programs?
questions: I left questions in the "Strengths and Weaknesses" section.
limitations: The authors have adequately addressed the limitations and potential negative social impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Manuscript presents a method to train neural networks over synthetic data. The work collected a large collection of 21K OpenGL programs for rendering diverse set of synthetic images to be used for representation learning. Experiments show that the collected programs can be used for pretraining in both supervised and unsupervised fashion. New state of the art results are obtained for pretraining with procedurally generated images and are observed to compete closely against pretraining with real data samples.
strengths_and_weaknesses: Strengths
- Collected a large-scale repository of image programs for easily generating variety of synthetic images. These data samples can help pretraining while avoiding the unfair biases and other contaminations that arise from natural datasets.
- Because of the large scale, the collection enables to study how the performance scales with the size and variety of the synthetic data.
- Representation learning experiments have been performed in both the supervised and unsupervised scenarios.

Weaknesses
- The major contribution of the manuscript is somehow not very strong (despite being very important). In other words, in my opinion, large-scale collection of procedural image programs alone may not be sufficient enough to excite the NeurIPS community.
- It is easy and trivial to see that more and diverse set of synthetic data samples improve the pretraining quality. They can outperform the existing small-scale collections and close the gap between the performance of synthetic and real image datasets.
- I believe studying the effectiveness of the individual programs would be an interesting and useful contribution. The presented analysis on the properties of the programs that leads to better learning is not very conclusive or effective. In the end the reader may not feel very clear about the desired properties (except for weak observations such as LPIPS, more colors, etc.) and the study feels incomplete or lacking. 
questions: - While the authors clearly mention that their approach involves generating images from several processes (but with fixed $\theta$), it is not clearly described how it is different from the existing works [1, 2] which focus on a single program ($i=1$) but with varied parameters $(\theta)$ ? In other words, although it may be intuitive to see that more processes are better than a single process, how to quantify (even loosely) the bigger or better quotient given the degrees of freedom are different ($\theta$ versus $i$ in lines 94-100).
limitations: - As discussed in the weaknesses section, the major limitation of the work is that fails to go beyond the collection of large-scale image generating programs. Authors have attempted to study the collection and draw useful properties for crafting such programs that are more useful, but in my opinion that needs more work.
- Outright I do not see any adverse impacts of the present work on the society.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a dataset for learning representations without access to real images, the dataset is composed of twenty-one thousand OpenGL programs is proposed for supervised and unsupervised representation learning, the synthetic data can help reduce the concerns about using real images in training such as human face etc. 
Experiments show that pretraining on the proposed dataset can bring performance improvement over previous synthetic dataset, and the investigation on what makes for a good generative image program seems interesting.

strengths_and_weaknesses: S1: The goal of learning representation without real images is practically important.

S2: The analysis of the properties for good image generation programs could be of interest for people who are working on image generation programs for representation learning.

W1: The evaluation of the learned representations on the proposed dataset is only limited to linear readout, while I believe using full finetuning can better show the difference between representations and are closer to a real use case.

W2: There are works that aims to learn representations using only one image [R1] or using filtered real images [R2], I think the paper should include discussion with these works because the goal is the same: learning the representation without ethical concerns.

questions: Q1: It would be good if the paper could also include some visualization results of the learned representations like in [R1], and maybe the linear readout performance of the representations per layer? I think this could give the reader more sense of how well the learned representation is.

Q2: Should the augmentations on synethic data be the same as real data? IMO, the data augmentations used by MoCov2 on ImageNet is choosen because the augmentation can encode some prior knowledge about the real images, so probably when we are training the model on synthetic images, the choice of augmentations should be different.

Q3: There are efforts of reducing privacy concerns in the real data such as PASS [R2], it would be good if the paper could discuss the connection to these types of works and what is the different between using synthetic images and modified real images. 

Q4: It would also be interesting to see how well the learned representation on the proposed dataset transfers by finetuning the representation on a target dataset, because as shown in [R3], the linear readout performance does not always correlates with the full finetuning results.

[R1] A critical analysis of self-supervision, or what we can learn from a single image, ICLR 2020

[R2] PASS: An ImageNet replacement for self-supervised pretraining without humans, NeurIPS 2021 Datasets and Benchmarks

[R3] Masked Autoencoders Are Scalable Vision Learners, CVPR 2022

limitations: 1. There is no section about the limitation and negative societal impact in the main paper and the supplementary, but IMO learning representation on synthetic data does not bring any negative impact in general, however, regarding this work, I am curious about the license of the all the shader used in the dataset, do all of them allow for research purposes?
2. One limitation for learning representation using synthetic data is the performance is not comparable with training on real images, but I think by studying what factors help learning a good representation using synthetic data, this paper is a meanful step towards makeing representation learned on synthetic data usable.
review_title: null

Meta-review: metareview: This work presents a method for training neural nets on synthetic data. This data is collected from a collection of thousands of OpenGL programs that rendered images, which are then used for representation learning. The big advantage of this approach is that it avoids of a lot of the biases that are present in natural image datasets. The proposed method is competitive for supervised and unsupervised scenarios. I find the results, especially those with finetuning (as done during the rebuttal period) relatively compelling. 

While I agree with reviewer vdrw that, on the whole, the major contribution (an image collection) is not very strong, I still think this kind of approach will be widely interesting to the NeurIPS community. Precisely because the work shows carefully (albeit empirically only) that procedurally generated datasets could be useful for representation learning, especially if you want to avoid the various pitfalls of natural image sets.
confidence: Less certain
Decision: Accept