Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces PASS, a dataset for pretraining self-supervised representation learning methods. Compared to ImageNet-1k, this dataset has several benefits: 

1. All images follow the CC_BY license hence no license issue.
2. There is no human in the datasets, eliminating a lot of privacy concerns.
3. The author makes an effort to remove undesirable or biased images from the dataset.

The authors apply several state-of-the-art self-supervised representation learning methods on this dataset and evaluated the performance on multiple downstream tasks. They conclude that the performance is comparable to pretraining with ImageNet-1k.

strengths: 1. The paper proposes a reasonable alternative to ImageNet with less privacy, license, and ethical issues.
2. The paper establishes a sensible pipeline for curating a relatively clean dataset from internet images, which could benefit future efforts in building datasets.
3. The paper points out by experiment that current methods have their hyperparameters tailored to ImageNet pretraining, and shows that to achieve better performance on different pretraining datasets, efforts are needed to tune the parameters.

weaknesses: 1. Currently pretraining on PASS still underperforms ImageNet pretraining on the important segmentation and detection task by approximately 1%, which is a gap enough to make people reconsider using PASS. However I do not attribute the gap solely to the lack of persons in PASS, and I believe a major issue is that the hyperparameters are tuned for ImageNet. This is shown in table 6 row 3 where simply matching the class distribution improves detection result by 0.5%. Re-tuning the parameters could probably remedy this problem, but it would require significant efforts. I believe currently this dataset will be most useful mainly when license and privacy are the main concerns. Mainstream adoption might be difficult unless the performance on these important downstream tasks could match ImageNet pretraining.

2. Personally, I believe the current manual filtering is a bit too aggressive, especially in removing all “license plates, signatures, or handwriting” from the images. I understand the rationale of protecting the information, however, it may not be necessary to remove basically all recognizable texts from the dataset as it may influence downstream task performance that requires this knowledge. I would recommend the authors to test their modal pre-trained on PASS in a cross-domain transfer setting, e.g. VTAB[1], to ensure the cross-domain generalizability is not harmed by the strict curation.

[1] A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark, 2019

correctness: The claims are valid in this paper. The dataset construction and evaluation are sound.

clarity: The writing of the paper is clear and the theme of improving ethics and privacy is explained well.

relation to prior work: The related work is well introduced in this paper.

additional feedback: Please refer to the weakness section for comments and suggestions.
review_title: A self-supervised pretraining dataset with more ethical and privacy considerations

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposed the PASS dataset, which has the same number of images as ImageNet-1K and only contains non-human images with CC-BY license. Self-supervised learning experiments show that PASS is diverse and can learn good image representations.

strengths: 1. The authors have constructed PASS very carefully to avoid ethical problems. All images in the PASS dataset has CC-BY license and no copyright issue. In addition, the dataset contains no images of people and has filtered potentially problematic images.

2. The author has conducted extensive evaluations, e.g., training with MoCo-v2, SwAV and DINO. These experiments show that models trained on PASS can generate competitive image representations as those trained on ImageNet / Places205.
weaknesses: 1. The scale of PASS (which has not been labeled) is only comparable to ImageNet-1K (which has been labeled). Given that It is easier to collect a large number of unlabeled images, I would expect that the size of PASS should be much larger than ImageNet-1K. Otherwise, researchers won't evaluate on this dataset given the wide adoption of ImageNet-1K and no clear performance advantage in training with PASS.

correctness: Yes, these claims are correct to the best of my knowledge.
clarity: Yes.
relation to prior work: Yes.
additional feedback: Given that PASS is unlabelled and ImageNet-1K is labelled, it is misleading to call it "an ImageNet replacement" and I would suggest the author to change the title.

--- Post Rebuttal ---

I've read the author's rebuttal and will keep my score unchanged. I still think that it is confusing to call PASS an "ImageNet replacement" because these two datasets have different focuses (self-supervised v.s. supervised) and cannot be directly compared.
review_title: A new image dataset that is diverse and does not have privacy issues.

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: summary and contributions: This paper proposed an unlabelled PASS dataset for self-supervised pretraining. PASS only contains images with CC-BY license and complete attribution metadata to address the copyright issue. Besides, all images on the PASS dataset do not contain humans or body parts at all to address the data protection issue. The authors conduct an extensive downstream task evaluation to discuss performance differences when SSL methods are trained using ImageNet and PASS. This work is of great value and may contribute to the unsupervised learning community.

strengths: (1) From a technical, ethical, and legal perspective, the proposed PASS dataset is very significant and hopefully becomes an ImageNet replacement for self-supervised pretraining. 
(2) This paper has a complete description of the data generation process, including descriptions of the original dataset, the experimental setup, software used, objects involved, instructions to creators, methods for quality control, and so on.
(3) Different self-supervised approaches train well on the PASS dataset, yielding strong image representations.
(4) The experimental results show that pre-trained networks obtained using self-supervised training on the PASS dataset are competitive to the ImageNet dataset on most downstream tasks.
(5) The authors deeply analyze the limitations of this work. 


weaknesses: (1) In Table 2c, PASS is significantly worse than both ImageNet variants on the Pascal VOC07 dataset. An explanation of this phenomenon should be added.
(2) More experimental details in Table 3 should be provided. For example, the number of used annotations, the training details, and so on.
(3) The PASS dataset does not contain humans or body parts at all. Why can PASS pretraining achieve comparable performance to ImageNet pretraining on downstream tasks that involve humans such as human pose estimation?
correctness: Yes, all top-standard.
clarity: Clear and easy to follow
relation to prior work: Yes, very comprehensive
additional feedback: n/a
review_title: This work is of great value and may contribute to the unsupervised learning community.

Meta-review: All reviewers recommended accept. AC didn't find any reason to overturn the consensus recommendation. 
Decision: Accept