Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper provides a comprehensive study of using not naturally looking, synthetic images to train image feature extractors. Most of the synthetic images can be generated easily either in closed form or using a randomly initialized model. The obtained feature extractors result in significantly better models on natural images than randomly initialized counterparts after fine-tuning. Various types of synthetic images are evaluated in this paper to reveal the properties of images that can be used to train better feature extractors. In general, I feel this paper is studying a very interesting problem and a solution to the problem can be used in many practical scenarios where training data is difficult to collect or even not available, but the paper might need more technical contributions to be accepted. A potential way for improvement is to utilize the observations to generate better synthetic images from some random process that can improve the accuracy.

Strengths:
1. Evaluates a wide range of synthetic image types. 
2. Comparing the statistical properties of synthetic and natural images, in an effort to uncover the important factors for good synthetic images.

Suggestions and Questions:
1. I feel the paper spends too many pages (3 pages) enumerating different image generation methods. These are important details, but maybe they should not take up 30%+ of the paper. The objective for training the models are ignored (I did not find it in the supplementary neither), though references to the paper of these methods are given. I feel it is important to clearly state the training objective in the main paper, and some details of image generation can be moved to the appendix. 

2. I feel the observations in Sec. 5.1 are not conclusive. The accuracy do not seem to correlate well with the statistics considered. Maybe a more convincing way to verify the conclusions is to generate images that produce the stats that will likely produce highest accuracy according to the observations, and verify if it is the case. The correlations seem much stronger in Sec. 5.2, but the results are not surprising. For example, it is quite intuitive that if the training images have small FID to test images, thus more similar to test images, then the test accuracy will be higher. 


## Summary after discussions

In general, I feel the findings in this paper is interesting and could potentially invoke investigations into the necessary conditions for the success of contrastive unsupervised learning for images, and would like to raise my score. However, I still encourage the authors to brand this paper as a scientific discovery rather than something that will have immediate impact on applications (addressing the privacy and bias concerns etc.). From the new results of mixing with real data, it seems adding images from random processes does not improve the results of only using real images. As to 3-Real unlabeled data, It also remains unclear whether pretraining with real images generalizes better than random images in the decentralized setting; it seems pretraining with real data will transfer better from the results of 4-Mixing with real data.

main_review: Yes
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes to learn visual representations from procedurally generated images, drawing insights from natural image statistics. Non-trivial test accuracy was demonstrated on standard image classification datasets. The paper also considers what constitutes a good dataset for training wrt the statistics of individual images as well as the whole dataset highlighting precision-recall trade-offs when promoting naturalism vs diversity.
main_review: Originality: good

Quality: good, could use more experiments

Clarity: good

Significance: good

Assessment:
===========
The approach is bold and follows nicely from classical and recent results on natural image statistics. The development and presentation are of the highest quality.  Good submission overall.

Requests and comments:
====================
- A justification for focusing on contrastive learning would be nice. It's not clear whether the conclusions generalize to other architectures and training regimes.
- Also, a more elaborate justification of the chosen baselines is needed.
- Unless I'm missing something, 105K images were used for training and 50K ImageNet images were chosen randomly for evaluation.  It would be nice to discuss the impact of gradually scaling up the size of the training set size, on both the test accuracy as well as the image/dataset statistics.
- I wonder if the proposed generative procedures can be intuitively scheduled into a more effective curriculum, rather than used all at once.
- The conclusions include an intriguing remark that synthetic data might prove better than pretraining with, e.g., ImageNet. It would be nice to consider tailoring the generation processes to push the performance on specific tasks, e.g., the structural categories in VTAB.
- It's natural to ask how many (fewer) real images would be needed to close the performance gap to an actual baseline, like a meta-learning twist?
- - By analogy to how mammalian vision is thought to be pretrained by retinal waves, it's likely that mammals still need even a few real examples and continue to get better, e.g., learning to read and even typoglycemia.
- - How does the precision-recall trade-off manifest, and how to think of naturalism vs diversity, in this (meta learning) paradigm?
limitations_and_societal_impact: ~~Only one architecture and one objective function were considered.~~ The results would be more significant if they're shown to hold more generally. ~~The generation and evaluation, as well as image resolutions, were limited to a single setting.~~
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper investigates the effectiveness of synthetic data generated via procedural noise processes as training data for deep neural networks. Specifically, the authors consider fractals, computer graphics, dead leave models, statistical image models, and untrained GAN generators, as well as combinations thereof, as mechanisms to generate data, and train a network in self-supervised fashion on these different types of data. They find that deep networks trained in this fashion significantly outperform randomly initialized networks, and are competitive with deep networks that are trained on large natural image data sets on specialized domains (not necessarily natural images). The paper further presents a suite of experiments that aims to relate the properties of the synthetic data (e.g. color distribution) with the downstream performance of models trained on it.
main_review: Given the recent scaling trends of training deep networks on ever growing data sets, it is important to critically evaluate the benefits of large scale data. In that context the paper explores an interesting and highly relevant direction, questioning the necessity of large scale natural image data sets.

The paper investigates a broad and diverse choice of generative mechanisms, and also explores sound metrics in the systematic analysis of the relation between data set properties and downstream predictive performance. To my knowledge this selection of data sets and metrics has not been explored before.

The paper is well-written and mostly easy to follow.

The following points could be improved:
- How do the synthetic data sets compare to the single-image training from [26]? How does training on synthetic data compare to early self-supervised approaches (i.e. non-contrastive approaches) for example self-supervised learning via rotation prediction (Gidaris et al. ICLR’18)? I found it difficult to put the results in the paper into context, and it is therefore unclear to me how big the potential of the synthetic data approaches in the paper is.
- Are there ways to combine synthetic data with natural/real data, possibly with labels (e.g. similar to S4L (Zhai et al. CVPR’19))? Since there seems to be quite some gap to models trained on natural images and it is unlikely to deploy models purely trained using the data generation procedures from the papers, it would be interesting to see whether the synthetic data could be useful to augment natural/real data.
- Intuitively I would expect that networks trained on synthetic data do well on low-level texture based tasks and struggle more on tasks requiring high level visual features. An investigation into this aspect would make the paper more complete. Furthermore, for parameterized models it would be interesting to see how individual parameters affect downstream performance.

Minor comments:
- The different types of engineered filters described in Sec. 3.4. Maybe there is a way to present this more accessibly.
- Is the scale and cropping procedure (in particular the resolutions) described in L196 used throughout the paper? If so, is there an intuition why the patch size 64x64 is appropriate for training, assuming the testing resolution is higher?

Overall, I think the paper explores a highly relevant direction. However, I feel that the paper should be extended along the axes outlined above.

limitations_and_societal_impact: For limitations please see the main review. Societal impact is appropriately addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies whether real natural images are necessary to train deep neural networks for computer vision applications. To that end, the authors create datasets from a large variety of synthetic image generation processes and evaluate the performance of neural networks trained using these synthetic images. In a comprehensive analysis the authors demonstrate that neural networks trained on synthetic images obtain surprisingly good results and also identify key properties of synthetic datasets that lead to good downstream performance.
main_review: Strengths 
As the authors correctly point out, recent advances in neural networks for visual recognition increasingly depend on larger and larger datasets. The apparent need for massive training sets raises many challenges as those training sets tend to be proprietary and data at this scale is incredibly difficult to obtain. As a consequence, methods that reduce the need for large datasets are of great interest to both practitioners as well as researchers. As such this paper is of great interest to the research community. 

The analysis in this paper is outlined very clearly and the evaluation is comprehensive. Furthermore, the discussion and presentation of the results is focused and provides clear insights to the reader.

Weaknesses
One experiment that seems to be missing from the analysis are models that are trained on mixtures of datasets. In the paper each model is only trained on images generated from a single image generation process. As the authors point out, recall is very important for downstream performance. Could mixing multiple datasets lead to better coverage and thus downstream performance? 

Minor comments:
Figure 7: For consistency, I suggest to also encode the accuracy of the baselines by color.

limitations_and_societal_impact: No apparent limitations and societal impact that is not addressed.
review_title: null

Meta-review: The paper addresses an under-explored research question related to the necessity of large datasets and the promise of doing representation learning pretraining using only noisy processes. The reviewers found the topic as well as the comprehensive study very interesting and great area of study. I concur in that I believe the paper could help foster more research in this direction, so I believe it is a great candidate for a spotlight. However, there are some lingering requests from the reviewers: I strongly advise the authors to follow up on these.
Decision: Accept (Spotlight)