Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: Considering the sparse nature of point clouds within distant objects, this paper proposes to generate dense 3D virtual points by nearest interpolation from sparse point based on the instance mask provided by the 2D network to augment the sparse 3D point-cloud. In this way, the point cloud within the distant object will be denser, so that the downstream 3D object detector can localize distant objects better. The method in this paper makes good use of the high resolution of the 2D network for distant objects.
main_review: - The paper uses the advantages of 2D image detection on distant objects to search for possible instance locations in 2D. It then extracts the sparse point clouds inside these instances for densification, making full use of the advantages of 2D information and 3D information. It achieves good results and a certain degree of innovation. The design of the method itself is simple, and the authors have explained the proposed method very clear.
- My concern is that adding a 2-stage instance segmentation network will bring a significant burden to the whole network. If this part of the computation is added to a normal 3D detection network through network scaling, will it improve performance at the same level? As a result, this is not a fair comparison to the non-ensembled methods as the amount of computation is not consistent.
- Another question is that compared to PointPainting, which requires a segmentation network, virtual points need a more complex 2D instance segmentation network. Will this weaken the generalization ability of the model? It's better to verify on different data sets.
- Need to add the analysis of the impact of the quality of 2D instance segmentation on downstream 3D object detection because the quality of 2D instance segmentation cannot be guaranteed under different conditions.
- The authors claim that the framework is a plug-and-play module to any existing or new 2D or 3D detectors, but the paper only verifies one type of 2D/3D detector. It's better to perform experimental verification on different 2D/3D detectors to show its generalization.
- If I understand correctly, the only difference between CenterPoint + PointPainting and CenterPoint + Ours in Table 4 is that virtual points are added, am I right? It needs to be clarified, or it can be modified to CenterPoint + Ours(w/o virtual points) / CenterPoint + Ours. CenterPoint + PointPainting is a little confusing.
- Better add a gain comparison for the category with small size.
limitations_and_societal_impact: No obvious limitations and potential negative societal impact
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new method (MVP) to use multimodal (RGB+LiDAR) sensor data in 3D perception. MVP is simple in principle: compute the 2D segmentation mask, project LiDAR points on the mask, and generate the virtual points by sampling the points in the mask and attaching them with the closest LiDAR depth. Though simple, the authors show empirically the proposed method can bring performance gain and achieve SOTA results on the nuScenes 3D detection task.
main_review: Pros:
1. The paper is well written and easy to read.
2. I personally like that the proposed method is simple and light while effective. The method is sound and reasonable to me.
3. Good experiment results and ablation study support the proposed method.

Cons:
1. The experiments are solely on nuScenes dataset, whose LiDAR is extremely sparse. LiDARs are getting denser and denser nowadays, thus I wonder if the proposed method can still bring gain on datasets with denser LiDARs, e.g. KITTI and Waymo Open Dataset? This can also showcase the generalizability of the proposed method.
2. Could you show more qualitative visualizations on both success and failure cases of the virtual points? As the authors have mentioned in L259, assigning the depth of virtual points to the close-by LiDAR measurement might not be a good estimate in the real world. Though the effectiveness of the MVP is validated by object detection tasks, I would suggest the authors conduct some experiments to show how well the estimate (i.e. take the depth of close-by LiDAR as virtual point depth) can be, for example, take objects with dense LiDAR, randomly mask out most of the LiDAR points, generate virtual points and quantize the closeness by computing the Chamfer distance.

======= 

Post Rebuttal Comments:
Thanks for the authors' effort in putting up the response. The responses address my concerns, though I would suggest including detailed results on the KITTI dataset (include car, pedestrian, and cyclist results.). I decided to keep my original rating.
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper argues that despite being accurate in 3D position, point cloud is more sparse than rgb camera. Thus, the paper proposes to generate virtual points by lifting 2D pixels to 3D space, and helps the detection of far-away objects. Experiments on Nuscenes dataset validate the effectiveness of the proposed method.
main_review: Strengths:

1. The proposed method is well-motivated and the objective is clear.

2. The paper is generally well-written and easy to follow

3. Potentially the proposed point augmentation algorithm can be used on a lot more AD datasets.

Major Weaknesses:

1. The author misses the related work on point cloud augmentation. 

2. The paper lacks theoretical novelty. The proposed point cloud augmentation seems to be a combination of Frustum PointNet and CenterNet, with a point projection and a nearest neighbor search modules. The contribution leans more to the experiment and network tuning part.

3. Experiments only done on Nuscenes. The proposed pipeline is really neat and straightforward, so it should be easy to test on multiple datasets. Results on at least one more public benchmark (E.g. KITTI, Waymo) will make the statement of the paper more convincing. 

Minor Weaknesses:

1. The authors make a statement that 2D detectors have generally higher 2D detection than 3D detectors. However, on KITTI 2D detection benchmark, the highest numbers are almost all achieved by 3D detectors like BANet, SE-SSD, PV-RCNN, etc. Thus, the statement made here is not quite accurate. 

2. The author uses nearest neighbor when doing the inverse projection from image pixel to 3D point. However, this is a very coarse approximation, because not all planes are perpendicular to the z axis. A more accurate approximation would be: To find the K nearest neighbor points, fit a local plane, and then find the intersection point between the camera lifted ray and that plane. In this way, the virtual points would be more smooth in space as they lie on the local plane. 
limitations_and_societal_impact: The authors pointed out the limitations and potential genative societal impact of the work.
review_title: null

Meta-review: This paper approaches the problem off 3D object classification on multimodal sparse LIDAR + dense RGB data by suggesting to use 2D object and region detectors on RGB images, dense 3D “viirtual” point generation by picking 2D points from detected instances nearby sparse 3D points, assuming that the depth of these sampled neighbors is the same as the reference, and then building “dense” 3D “virtual” point clouds to be classified. The technique is evaluated on nuScenes, where it achieves state-of-the-art 3D detection.

Reviewers praised the simplicity of the method and clarity of the paper, as well as the good experimental results. Reviewers had many issues with the paper: computational requirements of the method (FiPr), generalisability to KITTI (FiPr, PYBU, r8dG), flexibility with other detectors (FiPr), performance on small objects (FiPr), literature on point cloud augmentation (r8dG). All these questions were addressed in the rebuttal with additional experimental results and explanations.

One reviewer gave a score of 7 and another one off 6. One reviewer (FiPr) gave a scores of 4 but did not update their reviews after considerable rebuttal and additional experimental work by the authors. While waiting on reviewers FiPr, I am therefore willing to challenge the reviewers and promote this paper to an acceptance.

Decision: Accept (Poster)