Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This work proposes a new simulated outdoor LIDAR point cloud dataset: VoxelScape. The motivation of this work is to improve the amount of publicly available point-level labeled 3D PCD dataset which can potentially increase the performance for learning-based PCD semantic segmentation tasks. 
strengths: 1. The scale of the dataset is very large. It has 100K scans and 13,340M points in total.

2. This dataset provides simulation for intensity/reflectivity, and these properties are proved to be useful in semantic segmentation tasks;  

3. Fast acquisition of PCDs (2s per frame)
weaknesses: 1. The dataset provides both point-level and B-Box annotations. In addition to semantic segmentation, instance segmentation/object detection can also be done. However, there are no experiments on instance segmentation or object detection, which does not fully exploit the potential of this dataset. B-Box annotations play no role in semantic segmentation tasks.

2. The experiments do not show any comparison between their dataset and other simulated dataset. By applying other simulated dataset to semantic segmentation tasks, the performance may also be improved.

3. The description for SemanticKITTI is not very accurate. SemanticKITTI is a dataset that has both semantic labels and instance labels. Therefore, the B-BOX labels can be achieved by calculating the B-BOX for points with the same instance label.

4. Is the noise of the LIDAR sensor also simulated? How does the noise of the LIDAR influence the results? 

5. What is the source of the basic models for city generation? How do you obtain the material information from them (eg. buildings, pedestrians, cars, trees...) ? 

6. For the Experiment 2, the current setup is training on the  real-world dataset and fine-tuning on the simulated dataset. I am also interested in training on the simulated dataset and fine-tuning on the real-world dataset, since the simulated dataset is easier to obtain. In this case, maybe the amount of real-world dataset can be reduced.

7. It is not very clear what is the definition of “less-frequent scenarios”, please specify it and show some examples.

8. I suggest writing experiment 1 and experiment 2 in two different subsections. It will be clearer to the reader.

correctness: Some of the claims are not accurate. Please see above.
clarity: This work is overall well written and easy to follow. But some details are missing. Please see above.
relation to prior work: The comparison to prior work is not clearly stated. Please see above.
additional feedback: Please see all the weaknesses mentioned.
review_title: A large simulated LIDAR dataset, but the benchmark/baselines are not enough

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes  a large scale 3D point cloud dataset including simulated urban traffic environments with 32 point-wise semantic labels and 3D bounding boxes annotations of 9 object classes. The dataset is constructed with realistic simulation of physical LiDAR sensor properties.  It also contains diverse simulation of less-frequent scenarios. 
strengths: - This work provides a  3D urban traffic dataset with point-wise semantic annotation and 3D bounding boxes annotation, which is large enough to enlarge the self-driving scenes.
-  The dataset collection procedure is clear. The LiDAR simulation renders 360 degrees equirectangular depth map and maps them on a sphere then carve point cloud based on the depth values. The urban city generation procedure utilizes PCG to generate in three stages, which can synthesize the real scenes.
- The whole dataset has 100K point cloud scans with a total number of 13,340 million points. The point coordinates and reflection intensity are contained in each scan. 32 semantic class labels are annotated, which includes common elements in typical scenes.
weaknesses: - There is no explicit data validation procedure in the paper. The data collected in VoxelScape should be validated since it is generated by simulation.
- The experiment is not sufficient. The main objective of the experiment is to validate the applicability of synthetic dataset on real scenes, while only semantic segmentation task is performed. Object detection is one key task in 3D perception. Besides, VoxelScape includes sufficient bounding box labels. Therefore, 3D object detection task should be included in the experiments.
- Baseline methods are not state-of-the-art methods. Methods perform well on SemanticKITTI should be compared in the experiments.

correctness: The VoxelScape is well constructed. The experiment is designed for the applicability of synthetic dataset for real 3D perception tasks, e.g. semantic segmentation on SemanticKITTI, which is not enough to verify the applicability.
clarity: This paper is well written.
relation to prior work: This paper discussed the superiority over past synthetic and real urban traffic dataset.
additional feedback: The dataset is well-designed and includes sufficient synthetic elements that are common in real traffic scenes. It is an interesting attempt. The experiment should be designed more accurately.

I do not find the reproduction code for experiments. If it is possible, please provide the code for the experiments.

\* The experiments should be improved in the future version and include some simple 3D baselines for comparison. VoxelScape is a good dataset, and I suggest that authors could submit the revised version to the 2nd round.
review_title: VoxelScape is a well designed dataset while experiments should be more appropriate.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposes a detailed pipeline for simulating a LiDAR urban scene, and also proposes a large simulated LiDAR dataset with point-wise semantic labels as well as 3D bounding box labels. The main contribution lies in the proposed strategy for procedurally generating urban LiDAR scene, and the generation of point-wise LiDAR intensity that was never explored by previous simulated LiDAR dataset.  The experiments show that both the introduced intensity and the proposed simulated dataset could improve the 3D semantic segmentation performance of realistic dataset (like SemanticKITTI).
strengths: 1. The proposed urban scene generation strategy is detailed and reasonable, which covers most normal scenes and also some rare scenes like roadwalk. 
2. The proposed intensity generation strategy is reasonable to generate similar intensity with LiDAR sensors, and the proposed sub-label strategy is also useful to generate fine-grained even for different parts of a single object.
3. The provided simulated scene looks as good as a real LiDAR scene. 
weaknesses: 1. The experiments are not enough to verify its effectiveness on improving the performance of realistic point cloud semantic segmentation. 
(a) Both of two utilized baseline methods are based on 2D CNN, which achieve much lower performance than current state-of-the-art methods on SemanticKITTI dataset, such as SparseConvNet, KPConv, SPVNas, Cylinder3D (all these methods are worked on 3D space).  (b) With these lower baseline methods, the experiments are not reliable to verify the effectiveness of both intensity and the simulated dataset.  (c) So a clear state-of-the-art results on SemanticKITTI is more convincing to verify the effectiveness of simulated dataset. 
2. Lack of experiments for comparison on Table 5, L283 mentioned that there should be a experiments for training on simulated dataset and testing on real dataset, and I could not find it in Table 5. 
3. The ablation studies are short and not enough, such as how about the performance of jointly training simulated+real? 
4. Lack of experiments for 3D object detection. The authors claim that this dataset also have 3D bounding box labels, but all experiments are based on 3D semantic segmentation, and the 3D bounding box labels are not used in any experiments. 

Typo: L219, "the the"
correctness: Yes, the proposed dataset is constructed in a sound way. 
clarity: Yes, the paper is well organized and written. 
relation to prior work: Yes, it is clearly discussed the differences with previous works. 
additional feedback: The main concerns are the experiments, and the authors are suggested to better design and conduct the experiments to verify the effectiveness of the provided dataset. 

PostRebuttal: 
Thanks for the authors' response, and both the demo and scene generation strategy are good and reasonable. 
But the main concerns are still the experiments as listed before. Moreover, for the new added 3D detection results on KITTI dataset, there are several open-source PointPillar codes (such as OpenPCDet) that already achieves 77%+ 3D mAP in terms of moderate difficutly of Car, by training on 3712 samples and evaluating on 3769 samples. However, in the new added Table 5, the reported performance by this paper is only 62%+ 3D mAP, and the finetuned results (KITTI+VoxelScape) is only 71%+ 3D mAP, so the inferior baseline performance could not provide clear evidence to show that VoxelScape is beneficial on improving the performance of realistic 3D detection dataset like KITTI. 
More strong and reasonable experiments are expected to validate the effectiveness of this virtual dataset.
review_title: Review of VoxelScape

Meta-review: The reviewers unanimously raised concerns about insufficient experiments and gave valuable comments on how to make them stronger. I encourage the authors to revise the paper accordingly and submit it later.
Decision: Reject