Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a new large-scale dataset for autonomous driving. The new dataset consists of nearly 1 million unlabeled scenes and 16k annotated scenes which in total provide 1 million point clouds and 7 million images. The annotated scenes include 3D bounding boxes of 5 categories including car, bus, truck, pedestrian and cyclist. It also extends the existing evaluation metric for 3D object detection by also taking the object orientations into account. Experiments show that pre-training on the unlabeled data improves the performance of the network on the 3D object detection tasks. 
strengths: This dataset shows that unlabeled data which are relatively cheap can improve the network’s performance on important downstream tasks where annotations can be expensive to collect for autonomous driving. I think this would motivate future research on how to collect even better or more diverse unlabeled 3D data for better pre-training networks, which improves the reliability of autonomous driving.
weaknesses: It contains very few annotations of 3D boxes compared to existing datasets such as nuScenes and Waymo. The authors show the diversity of the dataset but it is unclear how it compares with existing datasets. 
correctness: They construct this dataset by attaching LiDAR sensors and cameras on a car. The claims in the submission seem to be correct. 
clarity: Yes
relation to prior work: Yes
additional feedback: The authors should put table 4 in their appendix which shows how pre-training on ONCE improves performance on nuScenes and Waymo in the main paper. This is an important experiment and I almost missed it because it was in the appendix.
review_title: A large-scale dataset for autonomous driving

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper introduces the ONCE dataset which contains 1m LiDAR scenes and 7m camera images from 144 driving hours. The dataset is collected across different geographical locations, weather and times.  The paper shows that backbone trained on ONCE is superior compared to the ones trained on Waymo and nuScenes on the downstream KITTI dataset. The paper additionally benchmarks a bunch of self-supervised and semi-supervised methods on the ONCE dataset.
strengths: + ONCE contains more driving scenes and more hours than the current datasets.
+ ONCE also contains data from different geographical location, weathers and times. 
+ Backbone trained on ONCE is superior compared to the ones trained on Waymo and nuScenes on the downstream KITTI dataset. 
+ Insights on use of pixels/voxels/pillars and anchor/center assignments are nice.
+ Bench-marking of self-supervised and semi-supervised methods is nice.
weaknesses: - ONCE has five categories. This is much less compared to the nuScenes dataset, which contains 23. Even in evaluation, nuScenes uses ten categories (way more than ONCE).

- The supervised benchmark of ONCE is small compared to other benchmarks. nuScenes has 40k images, Waymo Open has 200k images, while ONCE only has 16k. The train + Val split is of 8k images, which is similar to KITTI. Although I agree that the number of images is more compared to nuScenes and Waymo, I doubt that the supervised learning community would use the ONCE dataset for benchmarking their models.

- Comparison of diversity missing. It is worthwhile to compare the diversity of ONCE, NuScenes, Waymo, and BDD100k (2D object detection) to get a sense of how ONCE is more diverse than other datasets. In other words, a histogram plot of weather, times, and areas instead of a pie chart comparing the four datasets in Figure 3 would be nice.

- BDD dataset [1] has more RGB images (120m) compared to ONCE. Although it does not have 3D labels, BDD100k has a task for lane detection, semantic segmentation, instance segmentation, and segmentation. A more reasonable alternative to release the ONCE dataset is to annotate 3D labels on the BDD 100k dataset rather than coming up with a new dataset altogether.  Annotating BDD100K also makes sense as BDD100k has 1.8m objects compared to the 417k objects in ONCE.

- An additional feature of the nuScenes dataset (against Waymo dataset) is nuScenes also contains 3D annotations for both RGB and Lidar maps. Does ONCE provide the 3D annotations for RGB images as well?

- Does ONCE have multi-lane data? It is not super clear to me. nuScenes does have multi-lane data.

References-
[1] BDD100K, Yu et al , CVPR 2020.
correctness: Yes, the evaluation seems to be correct.
clarity: Yes, the paper is clear. 
relation to prior work: Somewhat unclear to me.
additional feedback: - In Implementation Details (Section C of the Appendix), please provide the details of data augmentation, learning rate policy and optimizer of each of the methods. It is also worthwhile to put the data split of each of the benchmark for clarity.
review_title: ONCE Dataset Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a large-scale dataset, ONCE, for 3D object detection in autonomous driving scenario. The proposed dataset consists of 1 million LiDAR scenes and 7 million corresponding camera images. Further, ONCE is constructed from 144 driving hours with various areas, periods and weather conditions, which makes the proposed one larger and more challenging than Waymo Open, the existing largest dataset for 3D detection. Moreover, this paper introduces different evaluation protocols for self-supervised, semi-supervised and unsupervised methods. Various models have been benchmarked on this dataset for future comparison. 
strengths: + The authors contribute the largest dataset for 3D object detection in autonomous driving scenario, which requires a lot of efforts. 
+ Compared with existing datasets, ONCE contains more scenes, images, larger size and area.
+ The dataset is released.
+ Various evaluations have been done on many existing detection models. I believe this will be beneficial for future research and comparison.
+ This paper is well written and organized.

weaknesses: - Table.1 shows the comparison of different datasets, which is good. But I think some important information is missing. For example, it is good to see what the aim is for each dataset, self-supervised/semi-supervised/unsupervised.
- The number of annotations is actually less then that of Waymo Open and nuScenes, which limits its usage if one wants to have more training data with annotations, given the large scale of ONCE.
- L166-174, it is not clear about the dataset splits. For example, how do you divide the dataset for each split? What is the information for weather, time and areas in each split? If it needs too much space, the author can put it in Supp. material.

correctness: The dataset is constructed in a sound way. The evaluation and experiment sound reasonable to me.
clarity: This paper is well written and organized.
relation to prior work: Related works have been sufficiently and clearly discussed. References in this paper are appropriate.
additional feedback: - It will be great to see a new model proposed on this dataset. Especially, one of the goal of ONCE is to facilitate research of unsupervised detection. It will be interesting to see one.

- How do you guarantee the quality of annotation?
review_title: A useful large-scale dataset for 3D autonomous driving with diverse scenes

Meta-review: All reviewers recommended acceptance. The AC concurs. 
Decision: Accept