Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary: 
This paper describes a new method for Pseudo-Lidar, that is the reliable recovery of a 3D point cloud from 2D inputs and subsequent detection of 3D objects from the point cloud. The authors focus on improving the accuracy of the reconstructed point cloud by formulating a loss in depth, rather than disparity space, and by using sparse true lidar readings to align estimates. These techniques lead to a boost in 3D object detection performance. 

Strengths:
The Pseudo-Lidar method has been well-received, and it appears that this paper makes a nice improvement on the previous in terms of 3D point cloud accuracy. While the image results here are convincing, I would have liked to see an added empirical evaluation of precisely how accurate the resulting 3D reconstructions are, measured against ground truth 3D lidar on a test set. Do the point clouds only look accurate locally (and perhaps near known objects give good shape due to regularity), or are the metric results also quite strong indeed? 

I found the author's technical analysis and method description to be clear and well-motivated. None of the math or formulations are entirely surprising, but they are new to this area, so this appeared as nice sensible progress to me.

This area is closely tied to the self-driving car application, and thus bottom-line performance is the key measuring stick for impact on practitioners. For 3D object detection, the main goal of interest, the authors show up to 20% improvements for their combined method over quite recent and strong PL methods (although the new method uses sparse lidar, which is a great advantage, hence not entirely equal comparison). This is the main impact of the paper, as I see it, and enough reason for acceptance. 

Areas for Improvement:
I found that the authors did not sufficiently recognize that there have been a wide variety of methods utilizing sparse 3D along with dense 2D images to interpolate to full 3D. For example, [A] is one I recall well from 15 years back, but at that time there was a strong community in this area, so I encourage the authors to do a bit more thorough review. 

This paper has the fewest qualitative examples of 3D objects detected among the recent papers I've read. The final pages of the Appendix contain a few more of these visuals, but there are too few in the main paper for the reader to get any intuitive feeling of the physical meaning of your performance improvement. I'd like to see you add several examples, even if small, into the paper to aid in this understanding. 

Decision: weak accept due to the nice clear method that gives a strong improvement on an important area to industry today. 

[A] Statistical Inference and Synthesis in the Image Domain for Mobile Robot Environment Modeling, Luz Abril Torres-Méndez and Gregory Dudek. In Proceedings of the IEEEE/RSJ/GI International Conference on Intelligent Robots and Systems (IROS), Sendai, Japan, 2004.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes to improve the idea of using stereo + lidar -style object detection to form stereo-based 3D object detection, building off of pseudo-lidar. In particular, it proposes to (a) switch the loss function for stereo deep networks from disparity to depth (b) do the stereo cost volume analysis in a depth volume (via resampling) rather than disparity volume and (c) if sparse lidar is available, align the estimated depth with the sparse lidar. Each seems to improve results, and the resulting system achieves good results on KITTI and outperforms past work in this area.

Positives:
+The paper proposes three ideas that seem good and lead to improvements that are demonstrated empirically.
+The paper is well written 
+The experiments are exceptionally thorough
+The ideas seems to me to be of obvious importance, although I realize that I'm likely not qualified to make a statement about this, and this should perhaps be done by a roboticist. 

Negatives:
-Most of the heavy lifting in the case without sparse LIDAR is done by tweaking the loss function rather than the cost, although the remaining gain is still pretty good
-I am not sure if this is a negative, but this is really a 3D vision paper. I do some form of 3D vision, but I really don't feel confident about my ability to assess whether doing stereo matching in a depth cost volume as opposed to disparity is correct -- I really haven't worked on stereo. It seems to work well, but I feel as if the wrong people are being asked to review the paper. I leave questions of venue to the area chair though.

Overall, I am inclined to accept the paper. I am a tiny bit worried about venue and whether the right people will check the work, but I don't think this should be decided by reviewers. However, I think the experiments are quite thorough and the paper is clearly above the bar.

In more detail:

Method:
+The method reads quite well and the idea is clean. I particularly like the graph-based depth correction algorithm, and the LLE-like way of adjusting the estimated depthmap. I have a few small comments below that do not affect my judgment, but I think would improve the paper.
= Small thought: the words systematic bias throughout is primarily referring to a bias for a particular object as opposed to a bias of the system (i.e., any individual object is too far or too close). This seems non-standard to me. A systematic bias would be that everything's too far away by 1m for instance.

Experiments:
+The experiments are exceptionally thorough, and of my pile of ICLR papers, this by far has the most thorough and well-thought-out experimental analysis.
+The system shows systematic improvements on 3 different LIDAR-based object detection systems; I think this is great.
=I'm not sure whether the 64-beam LIDAR can be subsampled to imitate 4-beam LIDAR. I simply don't know enough about the hardware to know if this is a sensible approximation. 
-Table 3 primarily suggests that the vast majority of the hard work in the non-sparse LIDAR is done by the depth loss rather than the depth cost volume. The resulting change is still pretty good (although I suspect that if you stuck in a coordconv in the disparity cost volume, it would handle the fact that you want unequal smoothing).
-The burial of the results on depth prediction results in the appendix with one is a little surprising as is the solitary table on it, but I understand the need to focus on 3D detection.

Small stuff that doesn't affect my review:
1) Framing the problem as having ethical considerations is, in my view, not necessary -- should all network compression papers start arguing that it is of profound ethical importance to figure out your bit quantization? 
2) Last paragraph above Section 4 "gird" -> "Grid"
3) Figure 3 caption "pruple" -> "purple"
4) Figure 4 is suboptimal -- I assume SDN+GDC < SDN < Disparity Net, but this is hard to verify.
5) Calling the network "Disparity Net" is a bit of an issue given that there's DispNet already 
6) "Figure 1 illustrates beautifully how" -> Please don't editorialize like this

-----------------------

Post rebuttal update: I have read the rebuttal and maintain my belief that the paper should be accepted.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes two extensions to the recent work of (Wang et al., 2019) on 3D object detection with pseudo LiDAR data. Wang et al. showed that 3D object detection using stereo images as inputs can be significantly improved if the depth map is projected to 3D and treated like a LiDAR point cloud (i.e., using methods that utilize the LiDAR point cloud). This paper shows that one shortcoming of this approach is given by the fact that the depth uncertainty increases the farther the objects are away. To remedy this, the authors propose to train the stereo estimation network (based on Chang & Chen, 2018) directly with depth outputs, instead of disparity values (inverse depth), by rewriting the loss and converting the cost volume. This already boosts the performance for far away objects. The authors demonstrate that the usage of a (simulated) low-cost 4-beam LiDAR can further facilitate the detection. For this purpose a graph diffusion algorithm is listed that aligns the pseudo LiDAR point cloud from the stereo set-up with the depth estimates from the low-cost LiDAR. Simulating the low-cost LiDAR on the Kitti benchmarks shows that this approach further increases the performance of the object detection methods.

In general, I am in favour of accepting the paper as it shows two orthogonal and interesting additions to the pseudo LiDAR paper of Wang et al. that improve its performance. However, I would like to see some clarifications in the rebuttal.

The proposed stereo network converts a disparity cost volume to a depth cost volume using bilinear interpolation. I agree, that the 3D convolutions are more meaningful (given the spacing of the grid cells) on the latter, but why the detour over the disparity cost volume? It should be possible to build the depth cost volume directly, which would lead to decreased memory consumption and speed up the method without any loss in accuracy?

One assumption of the second contribution (GDC) is that at least one beam of the LiDAR will hit the k-connected local point cloud. Can you give some bounds on the likelihood that this happens, especially for far away objects it could be unlikely, although it is most beneficial for those objects.
Further, I am missing a details on the optimization of (7) and (8). What is meant with slight L2 regularization? In the appendix it is also stated that a slightly different objective is optimized? 
Finally, the notation could also be improved. The authors are using L and G for the LiDAR point cloud and PL and Z for the pseudo LiDAR point cloud and then in the Z' is used for both.

Fig. 4 shows the median error in meters for the different variants of the stereo network. Why has the median been used? Are there severe outliers? If yes, it would also be interesting to quantify those and compare them (e.g., box plots).

In the abstract and in the discussion the authors oversell their results a bit. At the one hand they state that PL++ with GDC performs significantly better than PL++ w/o GDC, on the other hand they also claim that PL++ achieves comparable results to models that have access to the full 64-beam LiDAR data. However, if you compare the differences, then the gaps are for several cases almost as big, or bigger as in the former claim.

Things to improve the paper that did not impact the score:
- In equation (2) you could replace the x with a . (\cdot), or completely remove it
- On page 5: KNN neighbors -> k-nearest neighbors
- Also on page 5: write out W.l.o.g.
- In Tab. 1 it would help to highlight (bold) the best entries per column
review_title: Official Blind Review #3

Meta-review: Three knowledgable reviewers give a positive evaluation of the paper. The decision is to accept.
Decision: Accept (Poster)