Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a large-scale dataset for object detection in autonomous driving systems. The data is collected from some major cities and provinces in China. The data covers different locations, such as city streets, highways, and country roads. It also covers different weather conditions, and both daytime and night. The authors promised to maintain the dataset in the future and the dataset will also be used in an ICCV challenge.
strengths: 1.	The datasets are much larger than previous ones in this domain and are more diverse. It could facilitate future work in autonomous driving.
2.	The datasets will be open-sourced and will be maintained by the authors.
3.	The authors give a detailed analysis of the statistics of the datasets and how the dataset is annotated. The authors also present benchmark results on the dataset, which could serve as the baselines in future work.

weaknesses: Only 20k images are annotated in the dataset. This could limit the usability of the datasets. The labeled and unlabeled parts could also cover different domains. It would be hard to evaluate the domains that are not annotated.
correctness: The content appears to correct.
clarity: The paper is clearly written, with detailed descriptions of the datasets, annotation, and benchmark results.
relation to prior work: Compared with the previous datasets, SODA10M is larger in size and is more diverse. However, only 20K of them are labeled.
additional feedback: NaN
review_title: review

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The paper introduces the SODA dataset for 2D object detection, containing 10m unlabeled images and 20k labeled RGB images. The authors collect the dataset across different geographical locations, weather and times. The paper shows that RGB detectors (Faster RCNN and RetinaNet) pre-trained on the SODA dataset perform better than Waymo on the SODA10M labeled dataset. Moreover, FCNs trained on SODA10m transfer better compared to the Waymo for semantic segmentation. The paper also builds a benchmark for self-supervised and semi-supervised learning in autonomous driving.

Update :
Additional experiments (with different upstream datasets like nuScenes and BDD100K, finetuning on object detection tasks) shows that the SODA10M dataset performs better on object detection compared to current datasets. However, diversity comparisons in Table 2 shows that the SODA10M dataset is less diverse compared to BDD100k based on period/weather/location. Therefore, I am still not convinced that the dataset is super useful to the community.
strengths: - SODA contains data from different geographical locations (32 cities), weathers and times.
- The paper shows that RGB detectors (Faster RCNN and RetinaNet) pre-trained on SODA dataset perform better compared to Waymo on SODA10M labelled dataset. 
- FCNs trained on SODA10m transfer better compared to the Waymo for semantic segmentation.
- The paper also builds a benchmark for self-supervised and semi-supervised learning in autonomous driving which is super nice.
weaknesses: - SODA has six categories, much less than the nuScenes dataset, which contains 23. Even in evaluation, nuScenes [1] uses ten categories (way more than SODA).

- Although SODA has more total images, 2D/3D annotations are less than that of nuScenes [1], BDD100k [2], and Waymo. The number of boxes for SODA is only 149k, while the number of 2D boxes for nuScenes, BDD100k, and Waymo are 1.4m, 4.2m, and 12m, respectively. Thus, the number of boxes in SODA is orders of magnitude smaller than other released datasets.

- Comparison of diversity missing. It is worth comparing the diversity of SODA, nuScenes, Waymo, and BDD100k to get a sense of how SODA is more diverse than other datasets. In other words, a histogram plot of weather, times of the day, and areas comparing the four datasets similar to Figure 2 (but comparing with the statistics of other datasets) would be nice.

- Faster RCNN and RetinaNet pre-trained on the SODA dataset perform better than pre-training with Waymo on the SODA10M labeled dataset. However, this does not show that the SODA dataset is better compared to Waymo. The detectors pre-trained on SODA10M do not face a domain shift, while the detectors trained on Waymo face the issue of domain adaptation. Therefore, a more fair way to compare the performance is to train these detectors and evaluate a different dataset such as nuScenes. A more thorough evaluation is to train and test on all combinations of source/target datasets similar to Table 2 of [3] (Note that [3] does for 3D object detection and not for 2D object detection.)

- Please also benchmark the Table 3 performance against nuScenes [1] and BDD100k, which are more diverse compared to Waymo open.

- Comparison of driving hours with the current datasets.

- The supervised benchmark of SODA is small compared to other benchmarks. nuScenes has 40k images, Waymo Open has 200k images, while SODA only has 20k. Although I agree that the total number of images is more compared to nuScenes and Waymo, I doubt that the supervised learning community would use the SODA dataset for benchmarking their models.

- One of the major drawbacks of this dataset is that it only builds a 2D object detection benchmark. However, in general, the 2D RGB object detection benchmark is not enough. A dataset should incorporate the Lidar data (maybe Radar), maps, and 3D boxes for evaluation. The vision and machine learning community does not use one dataset for the task of 2D object detection and another dataset for carrying multi-sensor fusion or even 3D object detection. Do the authors plan to extend the dataset to include multi-sensor information as well as 3D detection benchmark?

References- [1] nuscenes: A multimodal dataset for autonomous driving, Caeser et al, CVPR 2020.
                      [2] BDD100K, Yu et al , CVPR 2020.
                      [3] Train in Germany, Test in USA, Wang et al, CVPR 2020.
correctness: Please see the Weaknesses section.
clarity: Yes. The paper is clear.
relation to prior work: The paper does not compare with nuScenes. The experimental evaluation also does not use BDD100k and nuScenes dataset.
additional feedback: - In Implementation Details (Section B of the Appendix), please provide the details of data augmentation, learning rate policy and optimizer of each of the methods. It is also worthwhile to put the data split of each of the benchmark for clarity.

- KITTI resolution in Table 1 should be 1242 x 375
review_title: Building 2D RGB detection benchmark for autonomous driving

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Addressing the problems of requiring extensive annotated data in existing dataset, this paper contributes a large-scale 2D object detection dataset called SODA10M. In specific, SODA10M contains 10 million unlabeled images and 0K labeled images. These images are collected from 32 different cities under different weather conditions, period and location scenes, demonstrating its diversity. Moreover, the authors provide extensive experiments and analyses of various existing methods.
strengths: + A large-scale data with 10 million unlabeled images and 20K labeled images is presented. This dataset could be used to facilitate future research on autonomous driving.
+ The dataset is released tp public, which is nice.
+ Extensive experiments with many existing methods are performed, and authors also conduct in-depth analysis. This is important as it can provide baseline comparison for future research.
+ This paper is well written and organized.
weaknesses: - I think the comparison in Tab. 1 is not fair. Because the most images in SODA10M are unlabeled, and collecting these unlabeled are actually not that hard. It is necessary to point it out in Tab. 1.
- In table, it is better to point out the aims of different datasets. For example, the proposed SODA10M is aimed at supervised, self- and semi-supervised object detection.
- In abstract, the authors claimed that "some insights about how to develop future models" is provided. However, I didn't see in-depth analysis about this claim.
- Since one of the major goal is for unsupervised learning for object detection, why don't develop your own baseline model?
correctness: The datasets are constructed in a sound way and experiments are clear.
clarity: This paper is well written.
relation to prior work: Discussion with related works is sufficient and clear.
additional feedback: Will you release the models used in experiments in this paper?
review_title: A larger-scale dataset for 2D object detection in autonomous driving

Meta-review: This paper receives conflicting reviews. The AC committee spent quite some time discussing the merits and drawbacks of the paper. The main concern is about the paper's claims and arguments. The major drawback is the lack of strong support for this dataset to be a unique benchmark for self-supervised and semi-supervised approaches. The existing quantitative results don't show distinct findings of self-supervised learning methods on the proposed dataset. Even though the experiments on different datasets show that the pre-trained model can benefit from the new dataset, the improvement can hardly be claimed to be significant. The claim of more cities is weak, due to different definitions of cities in different countries. In addition, nuScenes and BDD100K have more unlabeled images than those used in the experiments. The paper doesn't discuss the effects of those unlabeled images in the other datasets.

However, we do encourage the authors to conduct further investigation into the proposed dataset. Besides the study of uniqueness for self-supervised learning, the images from Chinese cities may be a good complement to the existing datasets and there may be interesting findings in domain adaptation study.
Decision: Reject