Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: - SODA contains 10 million unlabeled images where 20K images are labeled with 6 object categories from 32 different cities.
- The authors have suitably identified a good application for the large-scale dataset without labels, and discuss the scale, diversity, and generalization.
- Dataset collected by crowdsourcing from taxi drivers following the data protection laws by blurring personal information (faces, license plates, etc) and/or omitting GPS coordinates etc. The (small set of) images are exhaustively annotated following a three-step process.
- The dataset has been released and was also part of the CCV2021 SSLAD challenge.
strengths: - SODA contains 10 million unlabeled images where 20K images are labeled with 6 object categories. The images are collected from 27,833 driving hours under varying weather conditions and time-of-day periods from 32 different cities for the task of object detection. This is a comparatively large dataset and is suitable for pre-training and for self/semi-supervision methods on downstream tasks. 
- Problem statement is well motivated.
- Quantitatively better performance on downstream tasks from self-supervised training methods with the proposed dataset as compared with relevant baseline datasets.
- Sound reasoning for the clarification presented in L158-166 regarding the small number of annotated images available in the dataset. 
- Necessary ethical clarifications and privacy compliance have been addressed in L138-143 and Appendix A.
- Supplementary material includes implementation details and also includes improvements from previous versions.
- Self-supervised learning benchmarks in Section 4.3 are conclusive and demonstrate the applicability of the proposed dataset on several relevant tasks including video-based methods. The conclusions drawn are useful insights.
- Diversity statistics of the images presented in Tables 6,7 and 8 in the Supplementary material help verify the claim presented in L219-224 regarding improved diversity of scenes as a factor in improved performance.

weaknesses: - The dataset compilation procedure poses questions on quality control and how crowdsourcing data from volunteer/paid taxi drivers can be uniform. Are the performance gains in the experiments solely based on increased dataset size? Are there any quality control measures in place?
- Results when pre-trained with ImageNet outperform SODA10M for several tasks including the segmentation performance presented in the Supplementary material. The authors should explain this clearly.
correctness: Yes, the claims made in the submission seem correct and the dataset is constructed in a somewhat sound manner (some reservations). The evaluation benchmarks are exhaustive and performed correctly to demonstrate the utility of the proposed dataset in several tasks. The limitations are mentioned in the Supplementary material.
clarity: Yes, the paper is clear, well written, and easy to follow.
relation to prior work: Yes, previous works and relevant datasets have been discussed in the context of the autonomous vehicle vision niche. Further, supervised and self-supervised learning methods have been discussed in light of their progress relevant to baseline experiments.
additional feedback: L70: preforms -> performs
review_title: Useful dataset for a variety of tasks in autonomous driving with appropriate and well motivated experimental validation

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: Raw imaginary data has been collected across 32 cities with a total of 27833 driving hours, which must consume lots of time, investment for these collective efforts. 
strengths: The paper proposed a good idea about using 2K labelled data and 10M unlabeled data. However, after studying the paper and reading the benchmark section of figure 3, it is certainly very underwhelmed. And more details needs to be studied and included to make a strong argument.
weaknesses: * The author claims scale as a one of the unique contributions, as compared to the release of Waymo(with annotations), SODA10M contains 10 million data frames(without annotations). I do not think it is a fair comparison, as the AV industry does not lack raw data, but annotated data.
* There are some experimental analyses of self-supervised/semi supervised approaches included in the paper, but there is no novel contribution from this paper by using these existing toolsets and code implementations.

correctness: Since it is a release of datasets, I hope that more sound details about data collection and annotation have been conducted. Specifically,compared to talking about common knowledge such as collecting data with different weather locations for generalization purposes, discussing how to design the driving fleet across different cities to collect data in the most effective way could also be very interesting points. 
clarity: * The paper repeatedly mentions too many times about its contribution to releasing a dataset.
* I would recommend to cut down some over-emphasized common knowledge, and develop more about the unique contribution of this paper.

relation to prior work: Discussions about emerging methods from semi-supervised and self-supervised learning areas are covered.
additional feedback: NA
review_title: Release of 2K Labelled Data and 10 Million Raw Data

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a dataset collected in China highlighting the scale and diversity of the data collection. With 10M unlabeled images and 20K labeled images, this dataset focuses on the impact towards self and semi- supervised object detection, semantic segmentation, instance segmentation pipelines. The authors also share details about some experiments conducted to compare this dataset with other top datasets in the field. The comparison is effective and the impact is showcased with the results from various top models and techniques. 
strengths: This dataset is among the top largest datasets to be made available to the general public. The diversity of the data collected is extremely important to build any robust downstream perception algorithm. 

The fact that they have blurred out possible identification data in the dataset is very good and shows a clear thought towards ethical implications of the dataset. 

This dataset has been used as part of an ICCV challenge and hence is being widely used by many teams, hence showcasing the quick and far reach of this dataset. 

The domain adaptation intent is good. 

The authors address the questions posed from round 1 of the submissions. 
weaknesses: The paper needs to address the impact of the low frame rate. At highways speeds, 10 secs is a long time. The dataset would not work too well due to the reason mentioned above for any kind of tracking algorithm development and benchmarks. Though that doesn't seem to be the main intent of the authors, it would be of super huge value to have data at a higher frame rate for tracking purposes. 

The fact that annotations are only done on 6 different categories, does not address the concern from the other datasets highlighted in line 12 of the abstract. 

In automotive datasets, from personal experience and other works, we have noticed that position and pose of the camera in data collections can have a big impact on the performance of the models. Especially the difference between these across the different vehicles. The authors have neither mentioned about that in the paper nor the impact of characteristics such as height of the camera from the ground, pitch, yaw on the performance of the algorithms. Also information on the kind of vehicles used would be helpful. In large scale crowdsourced datasets, this is an incredibly tough ask to maintain consistency across all vehicles and it would help to strengthen the paper to mention some information on this. 
correctness: The dataset has mostly been collected in a sound way with keen observation to various noise/varying factors in the environment. The diversity of the data is a key contribution. 
clarity: The paper is mostly well-written with a consistent messaging and clear progression. There are some grammatical and spelling errors to be fixed, and with that taken care of along with addressing some of the other weaknesses, this paper is a top 25% among accepted papers for me. They do use a long appendix to add information to the paper. 
relation to prior work: Yes. 
additional feedback: Overall the paper is good and thanks to the authors to bring this out to the community in general for improvement of self and semi supervised algorithms in the Autonomous driving space.
review_title: Good paper, with beneficial large and diverse dataset. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper releases a Large-Scale 14 2D Self/semi-supervised Object Detection dataset for Autonomous driving, named as SODA10M, which contains 10 million unlabeled images and 20K images labeled with 6 representative object categories. To improve diversity, the images are collected within 27833 driving hours under different weather conditions, periods and location scenes of 32 different cities. We provide extensive experiments and deep analyses of existing popular self-supervised and semi-supervised approaches, and give some interesting findings in autonomous driving scope.
strengths: The paper is well written, and easy to understand.
weaknesses: 1. As described in the paper, the images are capured every 10 seconds per frame, it is hard to understand why. The consecutiveness among different frames is really low under such setting, and it is easy to capture high frame rate images.  As described in line 125, "They have to use the mobile phone or driving recorder (1080P+) to obtain images every 10 seconds per frame" , why "have to", the mobile phone and  driving recorder are easy to capture video, this is really hard to understand. 

2. In the released dataset contains 10 million images and 20K labeled images. However, it is unclear that how the number of labeled images affects the final results in self/semi-supervied object detection task. In line 165, the paper says that "we believe that SODA10M provides sufficient labeled samples to evaluate different self/semi-supervised learning methods," how to evaluate whether the labeled images are sufficient in the self/semi-supervised learning methods?

3. In section 4.3.3, how does it work in video based self-supervised methods? The images are captured every 10s, though time information exists, the images are in-continuous.

4. The task of each table should be clearly given, which results of task are reported in Table.5?


correctness: the evaluation methods and experiments same correct.
clarity: The paper is easy to understand and well written.
relation to prior work: The differences between the work and previous methods same clear.
additional feedback: see weakness.
review_title: SODA10M: A Large-Scale 2D Self/Semi-Supervised Object Detection Dataset for Autonomous Driving

Meta-review: The reviews are mostly positive. Although the dataset has a small number of labels, it can be a useful resource for research on self-supervised learning. The diversity of coverage is definitely a big plus. I am thus happy to accept i.
Decision: Accept