Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving
Authors: Yurong You
Yan Wang
Wei-Lun Chao
Divyansh Garg
Geoff Pleiss
Bharath Hariharan
Mark Campbell
Kilian Q. Weinberger
Venue:  (2020)
Keywords: pseudo-LiDAR
3D-object detection
stereo depth estimation
autonomous driving

Abstract: Detecting objects such as cars and pedestrians in 3D plays an indispensable role in autonomous driving. Existing approaches largely rely on expensive LiDAR sensors for accurate depth information. While recently pseudo-LiDAR has been introduced as a promising alternative, at a much lower cost based solely on stereo images, there is still a notable performance gap. 
In this paper we provide substantial advances to the pseudo-LiDAR framework through improvements in stereo depth estimation. Concretely, we adapt the stereo network architecture and loss function to be more aligned with accurate depth estimation of faraway objects --- currently the primary weakness of pseudo-LiDAR. Further, we explore the idea to leverage cheaper but extremely sparse LiDAR sensors, which alone provide insufficient information for 3D detection, to de-bias our depth estimation. We propose a depth-propagation algorithm, guided by the initial depth estimates, to diffuse these few exact measurements across the entire depth map. We show on the KITTI object detection benchmark that our combined approach yields substantial improvements in depth estimation and stereo-based 3D object detection --- outperforming the previous state-of-the-art detection accuracy for faraway objects by 40%. Our code is available at https://github.com/mileyan/Pseudo_Lidar_V2.

Published as a conference paper at ICLR 2020
PSEUDo-LIDAR++: ACCURATE DEPTH FOR 3D OBJECT DETECTION IN AUTONOMOUS DRIVING
Yurong You" Yan Wang"', Wei-LLu Chao" Divyansh Garg', Geoff Pleiss", Bharath Hariharan', Mark Camphell', and Kilian Q. Weinberger' "Comell University, Ithaca, NY "The Ohio State University, Columbus, OH (yy785, yw763, dg595, gp346, bh497, mc288, k @cornel1. chac.2090cn. 2.00.2099cuu.ee
ABSTRACT
Detecting objects such as cars and pedestrians in 3D plays indispensable role in autonomous driving. Existing approaches largely rely on expensive LiDAR sensors for accurate depth information. While recently pseudo-LiDAR has been introduced as a promising alternative, at a much lower cost based solely on stereo images, there is still notable performance gap. In this paper provide substantial advances to the pseudo-LiDAR framework through improvements in stereo depth adapt the stereo network architecture and loss function
estimation. Concretely, to be more aligned with accurate depth estimation of faraway objects currently the primary weakness of LiDAR. Further, we explore the idea to leverage cheaper but extremely sparse LiDAR sensors, which alone provide insufficient information for 3D detection, to di-bias our depth estimation. We propose a depth- propagation algorithm, guided by the initial depth estimates, to diffuse these few exact measurements across the entire depth map. We show on the KITTI object detection benchmark that our combined approach yields substantial improvements in depth estimation and stereo-based 3D object detection - outperforming the previous tfate-of-the-art detection accuracy for faraway objects by 40%. Our code is available at htttps::/ggttbbb..mmmmmmeeeaa Lidar V2
INTRODUCTION
Safe driving in autonomous cars requires curate 3D detection and localization of cars, pedestrians and other objects. This in turn re- quires accurate depth information, which can be obtained from LiDAR (Light Detection And Ranging) sensors. Although highly precise and reliable, LiDAR sensors are notoriously expen- sive: a 64-beam model can cost around $75,000 (USD The alternative is to measure depth through inexpensive commodity cameras. How- ever, in spite of recent dramatic progress in stereo-based 3D object detection brought by pseudo-LiDAR (Wang et al.] (2019)), signif- icant performance gap remains especially for faraway objects (which we want to detect early to allow time for reaction). The trade-off be- tween uffordability and safety creates an ethical dilemma.
An illustration of our proposed depth esti- mation and correction method. The green box is the ground truth location of the car
Figure 1:
KITTI dataset. The red points are obtained with a stereo disparity net- work. Purple points, obtained with our stereo depth net- work (SDN), much closer to the truth. After depth propagation (blue points) with a few (yellow) LiDAR measurements the car is squarely inside the green box. (One floor square is Imx Im.)
Equal contrihutions
'The information is obtained from the automotive LiDAR market report: htwp: wttp:://wwwwwwwwwwwoooooooo decap bads/2018/ 2tddd//201//4//ooee WCw_:CCC-.A.AAAA Ti Btw.DDDAeeeettooArii------ INAI. pdf
com/ wp- content/up pesnttent/ppllodd//211a1.

---

Published as a conference paper at ICLR 2020
In this paper we propose possible solution this remaining challenge that combines insights from both perspectives. We observe that the higher 3D object localization error of stereo- based systems, compared to LiDAR-based ones, stems entirely from the higher enror in depth estimation (after the 3D point cloud is obtained the two approaches are identical (Wang et al. 2019a)). Importantly, this error is not random but systematic: we observe that stereo methods do indeed detect objects with high reliability, yet they estimate the depth of the entire object as either too far or too close. See (eeuue I for an illustration: the red stereo points capture the car but are shifted by about 2m completely outside the ground-truth location (green box). If we can de-bias these depth estimates it should be possible to obtain accurate 3D localization even for distant objects without exorbitant costs.
We start by revisiting the depth estimation routine embedded at the heart of state-of oef-the-art stereo- based 3D detection approach (Wang et al. 2019a). major contributor to the systematic depth bias comes from the fact that depth is typically not computed directly. Instead, one first estimates the disparity - the horizontal shift of a pixel between the left and right images - and then inverts to obtain pixel-wise depth. While the use of deep neural networks has largely improved disparity estimation (Chang & Chen 2018 Cheng et al. 2018 Mayer et al.] 2016, Wang et al. 2019b), designing and learning the networks to optimize the accuracy of disparity estimation simply over- emphasizes nearby objects due to the reciprocal transformation. For instance, a unit disparity error (in pixels) for 5-meter-away object means 10cm error in depth: the length of side mirror. The same disparity error for a 50-meter-awa object, however, becomes a 5.8m error in depth: the length of an entire car. Penalizing both errors equally means that the network spends more time correcting subtle errors on nearby objects than gross errors on faraway objects, resulting in degraded depth estimates and ultimaatlyy poor detection and localization for faraway objects. We thus propose adapt the stereo network architecture and loss function for direct depth estimation. Concretely, the cost volume that fuses the left-right images and the subsequent 3D convolutions are the key components in stereo networks. Taking the central assumption of convolutions - all neighborhoods can be operated in an identical manner- - we propose to construct the cost volume on the grid of depth rather than disparity, enabling 3D convolutions and the loss function perform exactly on the right scale for depth estimation. We refer to our network as stereo depth network (SDN). See Figure for comparison of 3D points obtained with SDN (purple) and disparity estimation (red).
Although our SDN improves the depth estimates significantly, stereo images are still inherently 2D and it is unclear if they can ever match the accuracy and reliability of a true 3D LiDAR sensor. Although LiDAR sensors with 32 or 64 beams are expensive, LiDAR sensors with only 4 beams are two orders of magnitude cheape and thus easily affordable. The 4 laser beams are very sparse and ill-suited to capture 3D object shapes by themselves, but if paired with stereo images they become the ideal tool de-bias our dense stereo depth estimates: a single high-precision laser beam may inform us how to correct the depth of an entire car or pedestrian in its path. To this end, we present a novel cepth-propagation algorithm, inspired by graph-based manifold learning (Weinberger et al., 2005) Roweis& & Saul, 2000 Xiaojin & Zoubin 2002). In nutshell, we connect our estimated 3D stereo point cloud locally by nearest neighbor graph, such that points corresponding to the same object will share many local paths with each other. We match the few but exact LiDAR measurements first with pixels (irrespective of depth) and then with their corresponding 3D points to obtain accurate depth estimates for several nodes in the graph. Finally, we propagate this exact depth information along the graph using a label diffusion mechanism at negligible cost. In Figure we see that the few (yellow) LiDAR measurements are sufficient to position almost all final (blue) points of the entire car within the green ground truth box.
resulting in a dense and accurate depth map
We conduct extensive empirical studies of our approaches the KITTI object detection bench- mark (Geiger et al., 2012. 2013) and achieve remarkable results. With solely stereo images, we outperform the previous state of the art (Wang et al. 2019a) by 10%. Further adding a cheap 4-beam LiDAR brings another 27% relative improvement on some metrics, our approach is nearly on par with those based on a 64-beam LiDAR but can potentially save 95% in cost.
2The Ibeo Wide Angle Scanning (Scal.a) sensor with 4 beams costs $600 (USD). In this paper we simulate the 4-heam LiDAR signal on KITTI henchmark (Geiger et al. 20T2) by sparsifying the original 64-beam signal.
2

---

Published as a conference paper at ICLR 2020
2 BACKGROUND
3D object detection. Most work on 3D object detection operates on 3D point clouds from LiDAR as
input (Li 2017; Li et al. 2016 Meyer et al.
2019b Yang et al. 2018a Du et al.
2018 Shi et al.
2019 Engeleke et al. 201 7 Yan et al. 2018 Lang et al. 2019). Frustum NeintNet (Qi et al. 2018
applies PointNet (Qi et al. 2017a b) to the points directly, while Voxelnet (Zoou & Tuzel 201
quantizes them into 3D grids. For street scenes, several work finds that processing points from the
bird's s-eye view can already capture object contours and locations (Chen et al.) 2017: Yang ct al.
2018b; Ku et al. 20188) Images have also been used, but mainly to supplement LiDAR (Meyer et al.
2019a Xu ct al.
2018: Liang et al. 2018 Chen et al., 2017 Ku ct al. 2018). Early work based solely
on images-
mostly built the 2D frontal-view detection pipeline (Ren et al. 2015 He et al. 2017
Lin et al. 2017)
fell far behind in localizing objects in 3D (Li et al. 2019a Xiang et al. 2015
2017
Chabot et al.
2017:
Mousavian et al. 2017 Chen al. 2015 Xu Chen 2018 | et al.
2016
Pham & Jeon
2017 Chen et al. 2018
Pseudo-LiDAR. This gap has been reduced significantly recently with the introduction of the pseudo- LiDAR framework proposed in (Wang et al. (019a). This framework applies drastically different approach from previous image-based 3D object detectors. Instead of directly detecting the 3D bounding boxes from the frontal view of a scene, pseudo-LiDAR begins with image-based depth estimation, predicting the depth (u, 2) of each image pixel (u, v). The resulting depth map Z is then back- projjecee into a 3D point cloud: a pixel (u, v)) will be transformed to (I, y, 2) in 3D by
(u- cv) x fe
(u cv) x
z Z(u, v),
y
fv
where ev ) is the camera center and fo and fv are the horizontal and vertical focal length. The 3D point cloud is then treated exactly as LiDAR signal any R-DDRed 3D detector can be applied seamlessly. By taking the state-of-the-art algorithms from both ends (Chang & Chen [018]] Ku et al. 2018 Qi et al. 22188), pseudo-LiDAR obtains the highest image-based performance on the KITTI object detection benchmark (Geiger et al. 2012, 2013). Our work builds upon this framework.
Stereo disparity estimation. Pseudo-LiDAR relies heavily on the quality of depth estimation. Essentially, if the estimated pixel depths match those provided by LiDAR, pseudo-LiDAR with any LiDAR based detector should be able to achieve the same performance as that obtained by applying the same detector the LiDAR signal. According to (Wang et al. 2019a), depth estimation from stereo pairs of images (Mayer et al. 2016: Yamaguchi et al. 2014] Chang & Chen 2018)) are more accurate than that from monocular (i.e., single) images (Fu et al. [018] Godard et al. 2017) for 3D object detection. We therefore focus on stereo depth estimation, which is routinely obtained from estimating disparity between images.
A disparity estimation algorithm takes a pair of left-right images J and I, as input, captured from a pair of cameras with a horizontal offset (i.e., baseline) b. Without loss of generality, we assume that the algorithm treats the left image, 4t, reference and outputs a disparity map 1) recording the horizontal disparity to T, for each pixel (u, v). Ideally, (u, v) and I, (u. 0 + D(u, u)) will picture the same 3D location. We can therefore derive the depth map Z via the following transform,
fu x b Z(u, v) D(u,u)
(fu: horizontal focal length).
A common pipeline of disparity estimation is to first construct 4D disparity cost volume Catsp in which Caiso u. d,:) is feature vector that captures the pixel difference between 17(u, v) and I,, v d). It then estimates the disparity D(u, v) for each pixel (u, v) according to the cost volume Caisp- One basic algorithm is to build a 3D cost volume with Caisp v, d) = [|4r(u, v) I,- (u, e+d)ll2 and determine D(u, v)) arg min, Cass (u,, (u. v, d). Advanced algorithms exploit more robust features in constructing Causp and perform structured prediction for D. In what follows, we give an introduction -hrrt-rrtttttt-e--
of PSMNet (Chang & Chen 2018). state-of-the-an art algorithm used in (Wang et al. 2019a).
PSMNet begins with extracting deep feature maps hy and by from J and J,. respectively. It then constructs Cais( U. d,:) by concatenating features of hu hu(u, v) and hy d), followed by layers
Recently Srivastava Descently.Siivastavvaa. et al. 2019) proposed to lift 2D monocular images to 3D representations (c.g., bird's-eye view (BEV) images) and achieved promising monocular-basee 3D object detection results.
3

---

Published as a conference paper at ICLR 2020
PSMNet
2.0
SDN+GDC
1.5
1.0
0.5
Figure Disparity cost volume (left) vs. depth cost volume (right). The figure shows the 3D points obtained from LiDAR (yellow) and stereo (purple) corresponding eye view (BEV). Points from the disparity are stretched out and noisy; while points from the depth cost volume capture the car contour faithfully.
0.0
60
meters)
Figure 4: Depth estimation errors. We compare depth estimation ages, taking 64-beam LiDAR depths as ground truths. We separate pixels according to their true depths (z), Sec the text and appendix for details.
in KITTI, seen from the bird's s- volume
3,769 KITTI validation im-
of 3D convolutions. The resulting 3D tensor Saisp. with the feature channel size ending up being one, then used to derive the pixel disparity via the following weighted combination,
D(u, v) Σ (Satap u, d)) d,
where softmax is performed along the 3rd dimension of Sdisp PSMNet can be learned end-to-end, including the image feature extractor and 3D convolution kernels, to minimize the disparity error
Σ ((D(u,u) - (T(
(M.))A
where is the smooth LI loss, D* is the ground truth map, and A contains pixels with ground truths.
STEREO DEPTH NETWORK (SDN)
A stereo network designed and learned minimize the disparity error (cf. Egquaiinn 4) may over-emphasize nearby objects with smaller depths and therefore perform poorly in estimating depths for faraway objects. To see this, note that Eqqation 2 limplies that for a given error disparity &D, the error in depth dZ increases quadratically with depth:
100
dapth (maters)
60
40
1 1 82 x 228D. x + 8 D=
20
40
60
80
100
(5)
dispariy (piele)
Figure 2: The disparity-to-
The middle term obtained by differentiating Z(D) w.r.t D. particular, using the settings the KITTI dataset (Geiger et al.) 2012] 2013), a single pixel error in disparity implies only 0 in depth depth of 5 meters, but 5.8m error depth of 50 meters. See Figure 2 for mapping from disparity to depth.
depth transform. We set fu 721 (in pixcls) and h 0.54 (in meters) in Eqquation 2 which are the typical values used in the KITTI dataset.
Depth Loss. We propose two changes adapt stereo networks for direct depth estimation. First, learn stereo networks to directly optimize the depth loss
Σ ((Z(u,v) (-, v)). (u.D)EA
Z and Z can be obtained from D and D* using Equation The change from the disparity loss to the depth loss corrects the disproportionally strong emphasis on tiny depth errors of nearby objects necessary but still insufficient change to overcome the problems of disparity estimation.
Depth Cost Volume. To facilitate accurate depth learning (rather than disparity) we need to further address the internals of the depth estimation pipeline. A crucial source of error is the 3D convolutions within the 4D disparity cost volume, where the same kernels are applied for the entire cost volume. This is highly problematic it implicitly assumes that the effect of convolution is homogeneous
4

---

Published as a conference paper at ICLR 2020
Fasture
left fescure map
Image
Dagarity Cort
Depth Goor
Tanue Extrator
right
Come
Clays
Right Imags
KNN grazh
Carmesor
Sparse
Correcsed
Cloud
Figure 5: The whole pipeline of improved stereo depth estimation: (top) the stereo depth network (SDN) constructs depth cost volume from left-right images and is optimized for direct depth estimation; (bottom) the graph-based depth correction algorithm (GDC) refines the depth map leveraging sparser LiDAR signal. The gray arrows indicates the observer's view point. We superimpose the (green) ground-trutt 3D box of car, the same one in Figure The corrected points (blue; bottom right) are perfectly located inside the ground truth box.
throughout- which is clearly violated by the reciprocal depth to disparity relation (Figure 2). For example, it may be completely appropriate to locally smooth two neighboring pixels with disparity 85 and 86 (changing the depth by few em to smooth out surface), whereas applying the same kernel for two pixels with disparity 5 and 6 could casily move the 3D points by 10m or more.
Taking this insight and the central assumption of convolutions - all neighborhoods can be operated upon in an identical manner into account, we propose to instead construct the depth cost volume Cuepth, in which Cdepti (u, v, 2,:) will encode features describing how likely the depth Z(u, v) of pixel (u, v) is z. The subsequent 3D convolutions will then operate on the grid of depth, rather than disparity, affecting neighboring depths identically, independent of their location. The resulting 3D tensor Sdopth is then used to predict the pixel depth similar to Equation 3
Z(u, v)
Σ softmax(-SSapp
v)))) 2.
We construct the new depth volume, Cdepth. based on the intuition that Caepth (u, v, 2,:) and
Caisp u, fo x ::) should lead to equivalent "cost". To this end, we apply bilinear interpolation
construct Cdepth from Caisp using the deptthtoodisparriy transform in Equation 2 Specifically,
consider disparity in the range of [0, 191] following PSMNet (Chang & Chen 20 8) and consider depth in the range of [1m, 80m] and set the grid of depth in Cdopth to be Im. Figure (top) depicts our stereo depth network (SDN) pipeline. Crucially, all convolution operations are operated on Cdepth exclusively, Figure ccommaae the median values of absolute depth estimation errors using the disparity cost volume (i.e., PSMNet) and the depth cost volume (SDN) (see subsection D.5 for detailed numbers). As expected, for faraway depth, SDN leads to drastically smaller errors with only marginal increases in the very near range (which disparity based methods over-optimize). See the appendix for the detailed setup and more discussions.
4 DEPTH CORRECTION
Our SDN significantly improves depth estimation and more precisely renders the object contours (see Figure 3). However, there is fundamental limitation in stereo because of the discrete nature of pixels: the disparity, being the difference in the horizontal coordinate between corresponding pixels, has to be quantized the level of individual pixels while the depth is continuous. Although the quantization error can be alleviated with higher resolution images, the computational depth prediction cost scales cubically with resolution- pushing the limits of GPUs in autonomous vehicles.
We therefore explore a hybrid approach by leveraging a cheap LiDAR with extremely sparse (e.g., 4 beams) but accurate depth measurements correct this bias. We note that such sensors too
5

---

Published as a conference paper at ICLR 2020
sparse to capture object shapes and cannot be used alone for detection. However, by projecting the LiDAR points into the image plane we obtain exact depths on a small portion of "landmark" pixels.
We present graph-based depth correction (GDC) algorithm that efffectively combines the dense stereo depth that has rendered object shapes and the sparse accurate LiDAR measurements. Conceptually, expect the corrected depth map to have the following properties: globally, landmark pixels associated with LiDAR points should possess the exact depths; locally, object shapes captured by neighboring 3D points, back-projected from the input depth map (ef. Equation 1,, should be preserved. Figure 5) (bottom) illustrates the algorithm.
Input Matching. We take as input the two point clouds from LiDAR (I.) and Pseudo-L.iDAR (PL.) by stereo depth estimation. The latter is obtained by converting pixels (u, u) with depth 2 to 3D points (Ius Vu. 2). First, we characterize the local shapes by the directed K- K-ncarest-neighbor (KNN) graph in the PL point cloud (using accelerated KD-Trees (Shevtsov et al., (007)) that connects each 3D point to its KNNS with appropriate weights. Similarly, we can project the 3D LiDAR points onto pixel locations (u, v) and match them to corresponding 3D stereo points. Without loss of generality, we assume that we are given "ground truth" LiDAR depth for the first n points and no ground truth for the remaining m points. We refer to the 3D stereo depth estimates as Z E Ru+ and the LiDAR depth ground truth as G E R".
Edge weights. To construct the KNN graph in 3D we ignore the LiDAR information the first n points and only use their predicted stereo depth in Z. Let Nu denote the set of & neighbors of the jth point. Further, let W E ((=++))((++)) denote the weight matrix, where W., denotes the edge-weight between points and j. Inspired by prior work in manifold learning (Roweis & Saul 2000; Weinberger et al. 2005) we choose the weights to be the coefficients that reconstruct the depth of any point from the depths of its neighbors in Ag. We can solve for these weights with the following constrained quadratic optimization problem:
W arg minw 1 - W 212.
S.L. W1 = 1 and Wi = if & M.
Here 1 E R denotes the all-ones vector. As long as we pick k > 3 and the points are in general position there are infinitely many solutions that satisfy Z = WZ Z. and we pick the solution with the minimum L2 norm (obtained with slight L.2 regularization).
Depth Correction. Let us denote the corrected depth values as Z' E Ru+m with Z' Z'= [Zz; Zpzz and Z E R and Zpz E Rm where Z are the depth values of points with LiDAR ground-truth and Zpz otherwise. For the n points with LiDAR measurements we update the depth to the (ground truth) values Z G. We then solve for Zpr given G and the weighted KNN graph encoded in W. Concretely, we update the remaining depths Z/p,. such that the depth of any point i can still be be reconstructed with high fidelity as weighted sum of its KNNs' depths using the learned weights i.e. i.e. if point i 1 < i i< n is moved to its new depth G, then its neighbors in Ns must also be corrected such that G; R ΣjeN, Zij Further, the neighbors' neighbors must be corrected and the depth of the few n points propagates across the entire graph. We can solve for the final Z' directly with another quadratic optimization:
(8) To illustrate the correction process, imagine the simplest case where the depth of only a single point
z' arg migminz/ I W
s.t. Zi. G.
(n 1) is updated to G| = 211 + 8. A new optimal depth for [Equation 8lis move all the remaining
points similarly, i.c. Z' = Z + 16: as Z = W Z and W1 = we must have W + +0) = Z + 18. In the setting with n > 1. the least-squares loss ensures a soft diffusion between the different LiDAR depth estimates. Both optimization problems in Eqquatinn and Equation 8 can be solved exactly and efficiently with sparse matrix solvers. We summarize the procedure as an algorithm in the appendix.
From the view of graph-based based manifold learning, our GDC algorithm is reminiscent of locally linear embeddings (Roweis& & Saul (000) with landmarks to guide the final solution (Weinberger et al.. 2003). Figure Iillusttrases vividly how the initial 3D point cloud from SDN (purple) of a car in the KITTI dataset is corrected with a few sparse LiDAR measurements (yellow). The resulting points (blue) are right inside the ground-truth box and clearly show the contour of the car. Figure 4 shows the additional improvement from the GDC (blue) over the pure SDN depth estimates (see)subsection D.5 for detailed numbers). The error (calculated only non-landmark pixels) is corrected over the entire image where many regions have no LiDAR measurements. This is because that the pseudo-LiDAR point cloud is sufficiently dense and we choose l to be large enough (in practice, we use k = 10)

---

Published as a conference paper at ICLR 2020
Table 3D object detection results on KITTI validation. We report AP / aD (in %) of the category, corresponding average precision of the bird's-eye view and 3D object detection. We arrange methods according to the imput signals: M for monocular images, for stereo images. L for 64-beam LiDAR, and L# for sparse 4-beam LiDAR. Pl. stands for PSEUDO-LIDAR. Our PSEUDO- ++ (PL++) with enhanced depth estimation SDN and GDC- are in blue. Methods with 64-bcam LiDAR are in gray. Best viewed in color.
=.5 Moderate
IoU= 0.7
Detection algo
Input
Easy
Hard
Easy
Moderate
Hard
3DPP
55.0/46.0
41.37 34,6
34.67 30.1
62.6/6.6
9.
7.6/4.1
MLF-STEREO
S
19.5
S-RCNN
74.1 85.8
66.3
58.9/57.2
68.5/541 54.1
48.3/
41.5/31.1
PL: AVOD
S
88.5
77.5 / 76.4
68.7/61.2
74.9/61.9
56.8 45.3
49.0/39.0
PL: PIXOR*
S 89.0
75.2
67.3/-
73,9/-
54.0/-
46,9/-
PL: P-RCNN
S
88.4
76.6 73.7
69.0/67.8
73.4/62.3
52.7 34.0
PL++: AVOD
89.4
79.0/77.8
79.1
73.0/63.2
46.8
56.0//9.8
PL PIXOR"
S
89.9/-
78.4
74.7/-
79.71-
61.1/-
54.5/-
PL++ P-RCNN
S
89.8 89.7
83.8 / 78.6
77.5 75.1
67.9
64.0/ 50.1
57.3/
PL++: AVOD
87.77 86.9 N0.2790.1
86.87 79.8779.2
76.67 56.2
68.7753.4
PL++: PIXOR*
95.1/-
85.1 /-
78.3/-
84.0/-
71.0/-
65.2
PL++: P-RCNN
L# S
90.3 90.3
87.7 / 86.9
84.6/ 84.2
88.2 75.1
76,9 63.8
73.4/57.4
AVOD
I.+
90.57
89.47 89.2
88.57 88.2
89.47822
86.5
79.37 73.5
67.1
PIXOR"
L+M
94.2/-
86.7 /-
86.1
85.2
81.2/
76.1/-
P-RCNN
97.3/ 89.9 / 89.8
89.4 89.3
90.2 / 89.2
87.9 / 78.9
85.5
such that the KNN graph is typically connected (or consists of few large connected components). See (subseccto D.6 for more analysis. For objects such as cars the improvements through GDC are far more pronounced, these typically are touched by the four LiDAR beams and be corrected effectively.
5 EXPERIMENTS
5.1 SETUP
We refer our combined method (SDN and GDC) for 3D object detection as PSEUDO-LIDAR+ (PL++ in short). To analyze the contribution of each component, we evaluate SDN and GDC independently and jointly across several settings. For GDC we set from (simulated) 4-beam LiDAR, unless stated otherwise.
and consider adding signal
Dataset, Metrics, and Baselines. We evaluate on the KITTI dataset (Geiger et al. 2013 2012). which contains 7,481 and 7.518 images for training and testing. We follow (Chen et al. 2015) to separate the 7,481 images into 3,712 for training and 3,769 validation. For each (left) image, KITTI provides the corresponding right image, the 64-beam Velodyne LiDAR point cloud, the camera calibration matrices, and the bounding boxes. We focus on 3D object detection and bird's seeee--we (BEV) localization and report results on the validation set. Specifically, we focus on the "car" category, following Chen et al. (2017) and Xu et al. (2018). We report average precision (AP) with IoU (Intersection over Union) thresholds at 0.5 and 0.7. We denote AP for the 3D and BEV tasks by AP:p and AP KITTI defines the easy, moderate, and hard settings, in which objects with 2D box heights smaller than or occlusion/truncation levels larger than certain thresholds are disregarded. We compare to four stereo-based detectors: LIDADO (PL in short) (Wang et al., (019a)). 3DOP (Chen et al. 2015), S-RCNN (Li et al. 2019b), and MLF-STEREO (Xu & Chen 2018).
Stereo depth network (SDN). We use PSMNET (Chang & Chen 2018 as the backbone for our stereo depth estimation network (SDN). We follow Wang et al. (2019a) to pre-train SDN on the synthetic Scene Flow dataset (Mayer et al., 2016) and fine-tune on the 3.712 training images of KITTI. We obtain the depth ground truth by projecting the corresponding LiDAR points onto images. We also train PSMNET in the same way for comparison, which minimizes disparity error.
3D object detection. We apply three algorithms: AVOD (Ku et al. 2018). PIXOR (Yang et al. 2018b), and P-RCNN (Shi et al., 2019). All utilize information from LiDAR and/or monocular images. We use the released implementations of AVOD (specifically, AVOD- and P-RCNN. We implement PIXOR ourselves with slight modification to include visual information (denoted
7

---

Published as a conference paper at ICLR 2020
Table 2: Results on the car category the test set. We compare PL++ (blue) and 64- LiDAR (gray), using P- P-RCNN, and report APBEV / AD at 1oU=0.7.
Table Ablation study on depth estimation. We report APREV AP (in %) of the car category at IoU= 0.7 on KITTI validation. DL: depth loss.
Input signal
Easy
Moderate
Hard
Sterco depth
Easy
Moderatc
Hard
PL++ (SDN)
75.5 60.4 57.2 44.6 53.4 38.5 66.5/5122 89.5 / 85.9 85.7 / 75.8 79.1 / 68.3
PSMNET PSMNET DL SDN
63.3
56.0 44,9
$2.7 /4..
PL++ (SDN + GDC)
83.8 68.5 73.5
54.7
80.1
56.0/433 61.9 / 46.8
LIDAR
82.0/ 67.9
64.0 / 50.1
57.3 /4.33
Table 4: Ablation study on leveraging sparse Li- DAR. We report APREV / AP (in 56) of the car cate- gory at IoU= 0.7 KITTI validation. L#: 4-beam LiDAR signal alone. SDN L#: pscudo-LiDAR with depths of landmark pixels replaced by 4-beam LiDAR.
Table 5: Results of pedestrians (top) and cy- clists (bottom) KITTI validation. We apply F- POINTNET Qi et al. (2018) and report APbaV AP3D (in %) at 1o0= 0.5, following Wang ct al.
(019a).
Stereo depth
Easy
Moderate
Hard
The best result of each column is in bold font.
PSMNET
43.3
30.1 34.9/27.4 24.0
24.0
Stereo depth
Easy
Moderate
Hard
SDN
48.7 /40,9
40.4/ 32.9
34,9/ 28.8
SDN
67.0767.9
64.07 50.1
57.3
SDN GDC
63.7 /53.6
53,8 44.4
46.8 38.1
L# L#
73.2/
70.5 73.
PSENET
47.6/41.3
29.9725.2
27.07 24.9
SDN
/
55.0/
67.4 5A.1
SDN
49.3
30.4 28.7
2 26.4
SDN GDC
38.2/75.11
73.4/ 57.4
SDN GDC
65.7 /0.08
45.8 / 40.8
42.8 / 38.0
as PIXOR*). We train all models on the 3,712 training data from scratch by replacing the LiDAR points with pseudo-LiDAR data generated from stereo depth estimation. See the appendix for details.
Sparser LiDAR. We simulate sparser LiDAR signal with fewer beams by first projecting the 64-beam LiDAR points onto 2D plane of horizontal and vertical angles. We quantize the vertical angles into 64 levels with an interval of 0.4%, which is close to the SPEC of the 64-beam LiDAR. We keep points fallen into a subset of beams to mimic the sparser signal. See the appendix for details.
5.2 EXPERIMENTAL RESULTS
Results on the KITTI val set. We summarize the main results on KITTI object detection in Table Several important trends can be observed: 1) Our PL++ with enhanced depth estimations by SDN and GDC yields consistent improvement over PL across all settings; 2) PL++ with GDC refinement of 4-beam LiDAR (Input: L# + S) performs significantly better than PL++ with only stereo inputs (Input: S); 3) PL experiences substantial drop in accuracy from IoU at 0.5 to 0.7 for the hard setting. This suggests that while PL detects faraway objects, it mislocalizes them, likely placing them at the wrong depth. This causes the object to be considered a missed detection at higher overlap thresholds. Interestingly, here is where
experience the largest gain from PL: P P-RCNN (APBEV = 52.7) to PL++: PCRCNN (APBEV = 73.4) with input as L# + S. Note that the majority of the gain comes from GDC, as PL++ with the stereo-only version only improving the score to 57.5 BEV- 4) The gap between PL++ and LiDAR is at most 13% A APBEV, even at the hard setting under IoU at 0.7. 5) For IoU at 0.5, with the aid of only 4 LiDAR beams, PL++ (SDN GDC) achieves results comparable to models with 64-beam LiDAR signals.
Results on the KITTI test set. Table 2) summarizes results the category on the KITTI test set. We see a similar gap between our methods and LiDAR as on the validation set, suggesting that our improvement is not particular the validation data. Our approach without LiDAR refinement (pure SDN) is placed at the top position among all the image-based algorithms on the KITTI leaderboard.
In the following, we conduct series of experiments analyze the performance gain by our approaches and discuss several key observations. We mainly experiment with P P-RCNN: we find that the results with AVOD and PIXOR* follow similar trends and thus include them in the appendix.
Depth loss and depth cost volume. To turn a disparity network (e.g.. PSMNET) into SDN, there are two changes: 1) change the disparity loss into the depth loss; 2) change the disparity cost volume into the depth cost volume. In Table 3) we uncover the effect of these two changes separately. On the AP AP (moderate) metric, the depth loss gives us a 6%/2% /2% improvement and the depth cost volume brings another 2 ~ 3% gain")
We note that, the degree of improvement brought by the depth loss and depth cost volume depends on the 3D detector in use. Table 3 suggests that the depth loss provides more gain than the depth cost volume (for
P-RCNN). In Table 6 we, however, see that the depth cost volume provides comparable or even bigger gain
8

---

Published as a conference paper at ICLR 2020
Impact of sparse LiDAR beams. We leverage 4-beam LiDAR correct stereo depth using GDC. However, is possible that gains in 3D object detection come entirely from the new LiDAR sensor and that the stereo estimates are immaterial. In Table 4, study this question by comparing the detection results against those of models using 1) sole 4-beam LiDAR point clouds and 2) pseudo-LiDAR point clouds with depths of landmark pixels replaced by 4-beam LiDAR: i.e., in depth correction, only correct depths of the landmark pixels without propagation. It can be seen that 4-beam LiDAR itself performs fairly well on locating faraway objects but cannot capture nearby objects precisely, while simply replacing peudo-LiDAR with LiDAR at the landmark pixels prevents the model from detecting faraway object accurately. In contrast, our proposed GDC method effectively combines the merits of the two signals, achieving superior performance than using them alone.
Pedestrian and cyclist detection. For a fair comparison to (Wagg et al. POINTNET (Qi et al., 2018) for detecting pedestrians and cyclists. Table 5 methods significantly boosts the performance.
2019a), we apply F- shows the results: our
LiDAR
Pseudo-LiDAR
Pseudo-LiDAR++ (SDN)
Pseudo-LiDAR++ (SDN GDC)
Figure 6: Qualitative Comparison. We show the detection results on KITTI validation scene by P-RCNN with different input point clouds. We visualize them from both frontal-view images and bird's s-eye view (BEV) point maps. Ground-truth boxes are in green and predicted bounding boxes are in red. The observer is at the left-hand side of the BEV map looking to the right. In other words. ground truth boxes on the right are more faraway (i.e., deeper) from the observer, and hence hard to localize. Best viewed in color.
Qualitative visualization. In Figure 6) we show an qualitative comparison of detection results on randomly chosen scene in the KITTI object validation set, using P-RCNN (with confidence > 0.95) with different input signals. Specifically, we show the results from the frontal-view images and the bird's s-eye view (BEV) point clouds. In the BEV map, the observer is on the left-hand side looking to
than the depth loss (for PIXOR and AVOD). Nevertheless, Table 3 and Table 6 both suggest the compatibility of the two approaches: combining them leads to the best performance

---

commodity stereo Published as a conference paper at ICLR 2020
the right. It can be seen that the point clouds generated by PSEUDO-LIDAR (SDN alone SDN +GDC) align better with LiDAR than that generated by PSEUDO-LIDAR (PSMNET). For nearby objects (i.e.. bounding boxes close the left in the BEV map), cloud performs fairly well in localization. However, for faraway objects (i.e., bounding boxes close the right), PSEUDO-LIDAR with depth estimated from PSMNET predicts objects (red boxes) that are deviated from the ground truths (green boxes). Moreover, the noisy PSMNET points also leads to false negatives. In contrast, the detected boxes by our PSEUDO-LIDAR ++, either with SDN alone or with SDN +GDC, align pretty well with the ground truth boxes, justifying our targeted improvement estimating faraway depths.
that P-RCNN with any point
Additional results, analyses, qualitative visualization and discussions. We provide results of PSEUDO-LIDAK ++ with fewer LiDAR beams, comparisons to depth completion methods, analysis on depth quality and detection accuracy, run time, failure cases, and more qualitative results in the appendix. With simple optimizations, GDC runs 90 ms/frame using a single GPU (7.7 ms for KD-tree construction and search).
6 CONCLUSION
In this paper we made two contributions to improve the 3D object detection in autonomous vehicles without expensive LiDAR. First, we identify the disparity estimation as a main source error for stereo-based systems and propose novel approach to learn depth directly end-to-end instead of through disparity estimates. Second, we advocate that one should not use expensive LiDAR sensors learn the local structure and depth of objects. Instead one cameras for the former and a cheap sparse LiDAR to correct the systematic bias in the resulting depth estimates. We provide novel graph propagation algorithm that integrates the two data modalities and propagates the sparse yet accurate depth estimates using two sparse matrix solvers. The resulting system, PSEUDO-LIDAR
(SDN + GDC), performs almost on par with 64-beam LiDAR systems for $75,000 but only requires 4 beams and two commodity cameras, which could be obtained with a total cost of less than $1,000.
ACKNOWLEDGMENTS
This research is supported by grants from the National Science Foundation NSF (I11-1618134, III- 1526012, IIS- 114982,, 172-1222 and TRIPODS-1740822) the Office of Naval Research DOD 110001417-1-21755) the Bill and Melinda Gates Foundation, and the Corell Center for Materials Research with funding from the NSF MRSEC program (DMR-1719875), We are thankful for generous support by Zillow and SAP America Inc.
REFERENCES
Florian Chabot, Mohamed Chaouch, Jaonary Rabarisoa, Céline Teulière, and Thierry Chateau. Deep manta: A coarse-to-fine many-task network for joint 2d and 3d vehicle analysis from monocular image. In CVPR, 2017.
Jia-Ren Chang and Yong-Sheng Chen. Pyramid stereo matching network. In CVPR, 2018.
Xiaozhi Chen, Kaustav Kundu, Yukun Zhu, Andrew Berneshawi, Huimin Ma, Sanja Fidler, and Raquel Urtasun. 3d object proposals for accurate object class detection. In NIPS, 2015.
Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monoeu- lar 3d object detection for autonomous driving. In CVPR, 2016.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR, 2017.
Xiaozii Chen, Kaustav Kundu, Yukun Zhu, Huimin Ma. Sanja Fidler, and Raquel Urtasun. 3d object proposals using stereo imagery for accurate object class detection. IEEE TPAMI, 1259) 2018.
10

---

Published as a conference paper at ICLR 2020
Xinjing Cheng, Peng Wang, and Ruigang Yang. Depth estimation via affinity learned with convolu- tional spatial propagation network. In ECCV, 2018.
Xinxin Du, Marcelo H Ang Jr, Sertac Karaman, and Daniela Rus. A general pipeline for 3d detection of vehicles. In JCRA, 2018.
Martin Engelcke, Dushyant Rao, Dominic Zeng Wang, Chi Hay Tong, and Ingmar Posner. Vote3deep: Fast object detection in 3d point clouds using efficient convolutional neural networks. ICRA, 2017.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, pp. 2002-2011, 2018.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. The International Journal of Robotics Research, 92(11):1231-1233 2013.
Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In CVPR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander. Joint 3d proposal generation and object detection from view aggregation. In IROS, 2018.
Alex Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpil- lars: Fast encoders for object detection from point clouds. In CVPR, 2019.
Bo Li. 3d fully convolutional network for vehicle detection in point cloud. In IROS, 2017.
Bo Li, Tianlei Zhang, and Tian Xia. Vehicle detection from 3d lidar using fully convolutional network. In Robotics: Science and Systems, 2016.
Buyu Li, Wanli Ouyang, Lu Sheng, Xingyu Zeng, and Xiaogang Wang. Gs3d: An efficient 3d object detection framework for autonomous driving. CVPR, 2019a.
Peiliang Li, Xiaozhi Chen, and Shaojie Shen. Stereo r-cnn based 3d object detection for autonomous driving. In CVPR, 2019b.
Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In ECCV, 2018.
Tsung-Yr Yi Lin, Piotr Dollár, Ross B Girshick, Kaiming He, Bharath Hariharan, and Serge J Belongie. Feature pyramid networks for object detection. CVPR, volume 1, pp. 2017.
Fangchang Ma, Guilherme Venturelli Cavalheiro, and Sertac Karaman. Self-supervised sparse-to-
dense: self-supervised depth completion from lidar and monocular camera. In ICRA, 2019.
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR, 2016.
Gregory P Meyer, Jake Charland, Darshan Hegde, Ankit Laddha, and Carlos Vallespi-Gonzalez. Sensor fusion for joint 3d object detection and semantic segmentation. arXiv preprint arXiv:/904 1466, 2019a.
Gregory Meyer, Ankit Laddha, Eric Kee, Carlos Lasernet: An efficient probabilistic 3d object detector for autonomous driving. In CVPR, 2019b.
Gonzalez, and Carl K Wellington.
11

---

Published as a conference paper at ICLR 2020
Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Košecká. 3d bounding box estimation using deep learning and geometry. In CVPR, 2017.
Cuong Cao Pham and Jae Wook Jeon. Robust object proposals re-ranking for object detection in au- tonomous driving using convolutional neural networks. Signal Processing: Image Communication, 53:1 10-122, 2017.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In CVPR, 2017a.
Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas Guibas. Frustum pointnets for 3d object detection from rgb-d data. In CVPR, 2018.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnct++: Deep hicrarchical feature learning point sets in a metric space. In NIPS, 2017b.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 2000.
Maxim Shevtsov, Alexei Soupikov, and Alexander Kapustin. Highly parallel fast kd-tree construction for interactive ray tracing of dynamic scenes. In Computer Graphics Forum, volume 26, pp. 395-404. Wiley Online Library, 2007.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, 2019.
Siddharth Srivastava, Frederic Jurie, and Gaurav Sharma. Learning 2d to 3d lifting for object detection in 3d for autonomous vehicles. In IROS, 2019.
Paden Tomasello, Sammy Sidhu, Anting Shen, Matthew Moskewicz, Nobie Redmon, Gataryi Joshi, Romi Phadte, Paras Jain, and Forrest landola. Dscnet: Replicating lidar point clouds with deep sensor cloning. arXiv preprint 1811::1111.0700 2018.
Luz Abril Torres-Mendez and Gregory Dudek. Statistical inference and synthesis in the image domain for mobile robot environment modeling. In IROS, 2004.
Tsun-Hsuan Wang, Fu-En Wang, Juan-Ting Lin, Yi-Hsuan Tsai, Wei-Chen Chiu, and Min Sun. Plug- and-play: Improve depth estimation via sparse data propagation. arXiv preprint arXiv: 1812.08350, 2018.
Yan Wang, Wii-Lnn Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q. Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019a.
Yan Wang, Zihang Lai, Gao Huang, Brian H Wang, Laurens van der Maaten, Mark Campbell, and Kilian Q Weinberger. Anytime stereo image depth estimation on mobile devices. In ICRA, 2019b.
Kilian Q Weinberger, Benjamin Packer, and Lawrence K. Saul. Nonlinear dimensionality reduction by semidefinite programming and kernel matrix factorization. In AISTATS, 2005.
Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Data-driven 3d voxel patterns for object category recognition. In CVPR, 2015.
Yu Xiang, Wongun Choi, Yuanqing Lin, and Silvio Savarese. Subcategory-aware convolutional neural networks for object proposals and detection. In WACV, 2017.
Zhu Xiaojin and Ghahramani Zoubin. Learning from labeled and unlabeled data with label propaga- tion. Tech. Rep., Technical Report CMU-CALD-02-100 Carnegie Mellon University, 2002.
Bin Xu and Zhenzhong Chen. Multi-level fusion based 3d object detection from monocular images. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2020
Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In CVPR, 2018.
Koichiro Yamaguchi, David McAllester, and Raquel Urtasun. Efficient joint segmentation, occlusion labeling, stereo and flow estimation. In ECCV, 2014.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018
Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting hd maps for 3d object detection. In Conference on Robot Learning, pp. 146-155, 2018a.
Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In CVPR, 2018b.
Yanchao Yang, Alex Wong, and Stefano Soatto. Dense depth posterior (ddp) from single image and sparse range. In CVPR, 2019.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018.
13

---

Published as a conference paper at ICLR 2020
Appendix
We provide details omitted in the main text.
Appendix A
details on constructing the depth cost volume (section 3 of the main paper).
Appendix B
detailed implementation of the GDC algorithm (section of the main paper).
Appendix
additional details of experimental setups (subsection 5.1 of the main paper).
Appendix
additional results, analyses, and discussions (subsection 5.2 the main paper).
A DEPTH COST VOLUME
With Equation we know where each grid (u, v, 2) in Cdepah corresponds to in Cdisp (may not be on grd). We can then obtain features for each grid in Caepth (i.e., Cdepeh (u, 0, 2, z,:)) by bilinear f0 x
interpolation over features grids of Cdisp around the non-grid location (i.c.. applied the "grid samplesmmppp function in PyTorch for bilinear interpolation.
). We
We use PSMNET (Chang & Chen, 2018) the backbone for stereo depth estimation network (SDN). The only change is to construct the depth cost volume before performing 3D convolutions.
B GRAPH-BASED DEPTH CORRECTION (GDC) ALGORITHM
Here we present the GDC algorithm in detail (see algorithm l). main paper can be easily turned into two (sparse) linear systems and then solved by using Lagrange multipliers. For the first step (i.e...qquaa Eqqattoo 7), we solve the same problem as the main text but we switch the objective to minimizing the 12-norm of W and set Z WZ = as constrain Bl For the second step (i.c., [Equation 8), we use the Conjugate Gradient (CG) to iteratively solve the sparse linear system.
The two steps described in the
Algorithm 1: Graph- based depth correction (GDC). 5." stands for column-wise concatenation.
Input: Stereo depth map Z E R(n+m)x1 the corresponding pseudo-LiDAR (PL) point cloud P p(n+m))x3 and LiDAR depths G gux1 on the first the pixels.
Output: Corrected depth map Z' E x function GDC(Z, P, K)
Solve: W arg minl mitlwe ((++m)
[
s.t. Z - W 2 0,
N, (i.e., the of neighbors of the ith point) according P. Σ, Wyy = for Vi 1,... m.
W 0
Solve: Zpp arg & ||[:; ZP: WIG: 2p2111
return [G; 1p
end
C EXPERIMENTAL SETUP
C.I SPARSE LIDAR GENERATION
In this section, explain how generate sparser LiDAR with fewer beams from 64-beam LiDAR point cloud from KITTI dataset in detail. For every point (x,, Mi, =x) E R of the point cloud in one
'These two problems yield identical solutions but we found the second one is easier to solve in practice. We note that, [Equation is an under-constrained problem, with infinitely many solutions. To identify a single solution, we add small T. regularization the objective (as mentioned in the main text).
14

---

Published as a conference paper at ICLR 2020
scene (in LiDAR coordinate system (æ: front, y: left, up, and (0,0,0) 0) is the location of the LiDAR sensor)), we compute the elevation angle 8g to the LiDAR sensor as
+ +2 +
0 = arg cos
We order the points by their elevation angles and slice them into separate lines by step 0.4", starting from - -23.6 (close to the Velodyne 64-beam LiDAR SPEC). We select LiDAR points whose elevation angles fall within [-2.4" -2.0% -2.0%) - -044%) to be the 2-beam LiDAR signal. and similarly 2.44 - (
--- -0.4%) ) [0.0%, 0.4%) to be the 4-beam LiDAR signal. We choose them in such way that consecutive lines has 0.8 interval, following the SPEC of the "cheap" 4-beam LiDAR Lcala. We visualize these sparsified LiDAR point clouds from the bird's "erd's-eyeeoe view on one example scene in Figure
(a) 2-heam
(b) 4-heam
(c) 64-beam (full)
Figure 7: Bird's-eye views of sparsified LiDAR example scene. The observer is the bottom side
looking up. filter out points invisible from the image. (One floor square is 10m 10m.)
C.2 3D OBJECT DETECTION ALGORITHMS
In this section, we provide more details about the way we train 3D object detection models on pseudo-LiDAR point clouds. For AVOD, we use the same model as in (Wang et al. 2019a). For P-RCNN, we use the implementation provided by the authors. Since the P-RCNN model exploits the sparse nature of LiDAR point clouds, when training it with Ascudo-LiDAR input, sparsify the point clouds into 64 beams using the method described in subsection C.I For PIXOR implement the same base model structure and data augmentation specified by Yang et al. but without the "decode fine-tune" step and focal loss. Inspired by the trick in (Liang et al. 2018). add another image feature (ResNet-18 by He al. (2016)) branch along the LiDAR branch, and concatenate the corresponding image features onto the LiDAR branch at each stage. We train PIXOR using RMSProp with momentum 0.9, learning rate 10-5 (decay by 10 after 50 and 80 epochs) for 90 epochs. The BEV evaluation results are similar to the reported results (see Table T).
will first
2018b)
D ADDITIONAL RESULTS, ANALYSES, AND DISCUSSIONS
D.1 ABLATION STUDY
In Table 6 and Table we provide more experimental results aligned with experiments in subsec tion 5.2 of the main paper. We conduct the same experiments on two other models, AVOD and
PIXOR and observe similar trends of improvements brought by learning with the depth loss (from PSMNET to PSMNET +DL), constructing the depth cost volume (from PSMNET +DI. SDN), and applying GDC to correct the bias in stereo depth estimation (comparing SDN +GDC with SDN).
We note that, in Table 7 results of AVOD (or PIXOR*) with SDN + L# are worse than those with L# at the moderate and hard settings. This observation is different from that in Table 4 where P-RCNN with SDN L# outperforms P-RCNN with L# in out comparisons. We hypothesize that this because P-RCNN takes sparsified inputs (see subseccion C.2) while AVOD and PIXOR* take dense inputs. In the later case, the four replaced LiDAR beams in SDN + L# will be dominated by the dense stereo depths that SDN L# is worse than L##
15

---

Published as a conference paper at ICLR 2020
D.2 USING FEWER LIDAR BEAMS
In PL++ (i.e., SDN + GDC), we use 4-beam LiDAR to correct the predicted point cloud. In Table 8 investigate using fewer (and also potentially cheaper) LiDAR beams for depth correction. We observe that even with beams, GDC can already manage to combine the two signals and yield better performance than using 2- 2-beam LiDAR or pseudo-LiDAR alone.
Table 6: Ablation study stereo depth estimation. We report APREV APso (in %) of the category
1oU= 0.7 on the KITTI validation set. DL stands for depth loss.
PIXOR
AVOD
Depth Estimation
Easy
Moderate
Hard
Easy
Moderate
Hard
PSMNET
73.97-
54.0/-
46.9/-
56.8 / 45.3 49.0 39.0 74.9/61.9
PSMNET + DL
75.8/-
56.2
51.9/-
60.5
57.1 / 44.8
49.2 / 38.4
SDN
79.7
61.1/-
54.5/
63.2
63.7 / 46.8
56.0 39.8
Table Ablation study on leveraging sparse LiDAR. We report AP AP3 (in %) the car category at IoU= 0.7 on the KITTI validation set. 1## stands for 4-beam LiDAR signal. SDN +I.# means we replace the depth of portion of pscudo-LiDAR points (i.e., landmark pixels) by L.#.
PIXOR
AVOD
Depth Estimation
Easy
Moderate
Hard
Easy
Moderate
Hard
SDN
79.71-
61.17-
54.57
77.07 63.2
63.7 / 46.8
56.07 39.8
L#
72.01-
64.77-
63.6/-
77.0 62.1
68.8 / 54.7
67.1 53.0
SDN + L#
75.6/-
59.4/-
53.2/-
84.1 66.0
67.0 53.1
58.8 46.4
SDN + GDC
84.0/-
71.0/-
65.2/-
86.8 70.7
76.6 / 56.2
68.7 53.4
Table 8: Ablation study on the sparsity of LiDAR. We report APBEV / AD (in %%) of the car category at
ToU= 0.7 on the KITTI validation set. I.# stands for using sparse LiDAR signal alone. The number in brackets indicates the number of beams in
P-RCNN
PIXOR
Depth Estimation
Easy
Moderate
Hard
Easy
Moderate
Hard
SDN
67.9
64.0 50.1
57.3/45.3
79.77-
61.17-
54.57-
L# (2)
69.2/46.3
62.8/ 41.9
61.3/40.0
66.87-
55.57-
53.3/-
L# (4)
71.3/ 56.1
/1.5
72.0/-
64.7/-
63.6/-
SDN + GDC (2)
87.2 73.3
72.0 56.6
57.1
82.07-
65.37-
61.77-
SDN + GDC (4)
88.2 75.1
76.9 / 63.8
73.4 57.4
84.0/-
71.0/-
65.2
Table 9: Comparison of GDC and PNP for 3D object detection. We report APpev AP (in %) of the car category at loU= 0.7 on the KITTI validation set, using SDN PNP or SDN GDC for depth estimation and P-RCNN PIXOR" for detection.
P-RCNN Moderate
PIXOR Moderate
Input signal
Easy 86.3 / 72.1 88.2 / 75.1
Hard
Easy
Hard
SDN PNP
73.3 / 58.9 67.2 54.2 79.1 84.0/-
64.2/- 71.0/-
54.0/-
SDN GDC
76.9 63.8
73.4 / 57.4
65.2/-
D.3 DEPTH CORRECTION vs. DEPTH COMPLETION
We compare our GDC algorithm for depth correction to depth completion algorithms, which aim to "densify" LiDAR data bevond the beam lines (Wang et al. 2018 Tomasello al. 2018 Ma et al. 2019; Yang et al. 2019 Cheng et al. 2018 Torres-Mende depth completion approaches take as input 64-beam LiDAR and single image, while our focus is on fusing much sparser 4-beam LiDAR and stereo depths. As such, the two problems are not
& Dudek 2004 We note that most
Mtorres-Mender & Dudek (2044) use MRFs and may thus require less (or even no) training data compared to deep learning algorithms: property shared by GDC.
16

---

Published as a conference paper at ICLR 2020
0.150
GDC
SDN SDN GDC+GDC
0.125
PnP
median error meters)
2
0.100
median error
0.075
10-1
0.050
0.025
10
0.0
0.2
0.4
0.6 0.8 1.0
2
fraction of ground-truth trtnhth-rrthhto LiDAR points used
shortest distance LIDAR (in meters)
Figure 8: Comparison of GDC and PNP for depth cor- rection. We report the median of absolute errors on the KITTI validation set. See text for details.
Figure 9: Median depth estimation errors w.r.f. the shortest distances to 4-beam LiDAR points on KITTI validation set.
Table 10: Comparison of 3D object detection using the naive and optimized implementation of GDC. We report APBEV AP (in %) of the car category loU= 0.7 on the KITTI validation set, using P-RCNN for detection.
Easy
Moderate
Hard
Naive Optimized
88.2 75.1
76.9 / 63.8
73.4 / 57.4
75.0
76.3 63.4
73.1 /5..0
commensurate. Also, GDC algorithm is general, simple, inference-time approach that requires no training, unlike prior learning-baan based approaches to depth completion.
Here empirically compare to PNP (Wang et al.) 2018). recently proposed depth completion algorithm compatible with any (even stereo) depth estimation network, similar to GDC. We use SDN for initial depth estimation, and evaluate GDC and PxP by randomly selecting fraction of LiDAR points as provided ground truths and calculating the median absolute depth errors on the remaining LiDAR points. As shown in Figure 8, GDC outperforms PNP by large margin. [Talle shows a further comparison to PNP on 3D object detection. We apply PNP and GDC respectively to correct the depth estimates obtained from SDN, train a P-RCNN or PIXOR* using the resulting pseudo-LiDAM points on the KITTI training set, and compare the detection results on the KITTI validation set. either case, SDN + GDC outperforms SDN PNP by a notable margin.
D.4 RUN TIME
With the following optimizations for implementation,
1. Sub-sampling pseudo-LiDAR points: keeping at most one point within cubic of size 0. 1m3
2. Limiting the pseudo-LiDAR points for depth correction: keeping only those whose elevation angles are within 3.00, 0.4%) (the range of 4-beam LiDAR plus 0.6%; see subsection C. for details)
3. After performing GDC for depth correction, combining the corrected pseudo-LiDAR points with those outsides the elevation angles of --00% 0.4%)
GDC runs in 90 ms/frame using a single GPU (7.7ms for KD-tree construction and search, 46.5ms for solving W, and 26.9ms for solving Z'r,)) with negligible performance difference (see Table 10). For consistency, all results reported in the main paper are based on the naive implementation. Further speedups can be achieved by CUDA programming for GPUs.
D.5 STEREO DEPTH VS. DETECTION
We quantitatively evaluate the stereo depths by median errors in Figure of the main text (numerical values are listed in Table IT). In "Table 12 we further show mean errors with standard deviation (the large standard deviation likely results from outliers such as occluded pixels around object boundaries).
17

---

Published as a conference paper at ICLR 2020
Table 11: Median depth estimation errors various depth ranges (numerical values of F Figure 4
range (m)
Signal
0-10
10-20
20-30
30-40
40-50
50-60
60-70
PSMNet
0.04
0.11
0.36
0.83
1.24
1.98
2,43
SDN
0.07
0.12
0.30
0.60
0.89
1.31
1.73
SDN GDC
0.07
0.12
0.27
0.51
0.74
1.03
1.53
Table 12: Mean depth estimation errors (with standard deviation) over various depth ranges.
range (m)
Signal
0-10
10-20
20-30
30-40
40-50
50-60
60-70
PSMNet
0.18:10.93
0.36
0.97
2.02:14.05
2.94::5.64
8.61
1..32
SDN
0.2110089
0.16 0.87+2.31
1.80
2.67:16.00
4.27
5.82+11.23
SDN + GDC
0.21 0.35=1.17 0.84+2334
1.74+4.27
2.59:66.06
4.14=8.85 1.72
For both tables, we divide pixels into beams according to their truth depths, and evaluate on pixels the 4-beam LiDAR. The improvement of SDN GDC) over PSMNET becomes larger consider pixels farther away. Table 13 further demonstrates the relationship between depth quality and detection accuracy: SDN GDC) significantly outperforms PSMNET for detecting faraway cars. We note that, for very faraway cars (i.e., 50-70 m), the number of training object instances are extremely small, which suggests that the very poor performance might partially cause by over-fitting.
Further, we apply the same evaluation procedure but group the errors by the shortest distance between each ASEUDO-LIDAN point and the 4-beam LiDAR points in Figure 9 can see that the closer the PSEUDO-LIDAR points are to the 4-beam LiDAR points, the bigger improvement GDC can bring.
D.6 CONNECTED COMPONENTS KNN GRAPHS PSEUDO-LIDAR POINTS BY SDN
Here, provide empirical analysis the relationship between the & choose in building the K- nearest-neighbor graph HEEUOOLLIDA points by SDN and the number connected components of that graph. We show the results KITTI validation set in Fiiurre II It can be seen that with 2 9, the average number of connected components in the graph is smaller than
D.7 FAILURE CASES AND WEAKNESS
There is still a gap between our approach and LiDAR for faraway objects (see Table 13). We further analyze AP BEV different loU in Figure T0, For low IoU (0.2-0.5), SDN (+GDC) is par with LiDAR, but the gap increases significantly at high IoU thresholds. This suggests that the predominant gap between our approach and LiDAR because of mislocalization, perhaps due to residual inaccuracies in depth.
of connected components
60
AP moderate (%))
50
40
PL
30
SDN
20
SDN+GDC
10
LIDAR
10
loU threshold
Figure 10: IoU vs. AP on KITTI validation set on the category (moderate).
Figure 11: A vs. average number of connected components in KNN graphs of PSEUDO-LIDAR points by SDN.
18

---

Published as a conference paper at ICLR 2020
Table 13: 3D object detection at various depth ranges. We compare different imput signals. We report APBEV 3D (in %) the car category at IoU=
KITTI validation set, using P- RCNN for detection. In last two rows we show the number of car objects in KITTI object train and validation sets within different ranges.
Input signal
0-30 m
30-50 m
50-70 m
PSMNET
65.6 54.0
15.87 6.9
0.0 0.0
SDN
68.6 / 56.7
27.4/
0.7 / 0.0
SDN GDC
84.7 67.8
/
2.5/ 1.0
LIDAR
88.5 / 84.0
/9.9/
8.9 3.4
GRJECTS-TRAIN
6903
3768
76
# OBJECTS-VAN VAL
7379
3542
39
D.8 QUALITATIVE RESULTS
In Figure 12 and Figure 14 we show detection results using P-RCNN (with confidence 0.95) with different input signals on four randomly chosen scenes in the KITTI object validation set. Specifically, we show the results from the frontal-view images and the bird' s-eye view (BEV) point clouds. In the BEV map. the observer is on the left-hand side looking to the right. can be seen that the point clouds generated by PSEUDO-LIDAR ++ (SDN alone or SDN +GDC) align better with LiDAR than those generated by PSEUDO-LIDAR (PSMNET). For nearby objects (i.e., bounding boxes close the left in the BEV map), we see that P-RCNN with any point cloud performs fairly well in localization. However, for faraway objects (i.e., bounding boxes close to the right), PSEUDO-LIDAR with depth estimated from PSMNET predicts objects (red boxes) deviated from the ground truths (green boxes). Moreover, the noisy PSMNET points also leads to several false positíves or negatives. In contrast, the detected boxes by our PSEUDO- PSEUDO-LIDAR ++, either with SDN alone or with SDN +GDC, align pretty well with the ground truth boxes, justifying in estimating faraway depths. In Figure 12, we see one failure case for both SEEUDO-LIDAL and PSEUDO-LIDAE ++: the most faraway is missed, while LiDAR signal still detect it, suggesting that for very faraway objects stereo- based methods may still have limitation.
targeted improvement
19

---

Published as a conference paper at ICLR 2020
LiDAR
Pseudo-LiDAR
Pseudo-LiDAR++ (SDN)
Pseudo-LiDAR++ (SDN GDC)
Figure 12: Qualitative Comparison. We show the detection results on a KITTI validation scene by P-RCNN with different input point clouds. We visualize them from both frontal-view images and bird's s-ee view (BEV) point maps. Ground- left hand side of the BEV map looking the right. In other words, ground truth boxes on the right are more faraway (i.e., deeper) from the observer, and hence hard to localize. Best viewed color.
boxes are in green and predieted bounding boxes are in red. The observer is at the
20

---

Published as a conference paper at ICLR 2020
LiDAR
Pseudo LiDAR
Pseudo-LiDAR++ (SDN)
Pseudo-LiDAR++ (SDN + GDC)
Figure 13: Qualitative Comparison another example. The same setup in Figure 12
21

---

Published as a conference paper at ICLR 2020
LiDAR
LiDAR
Pseudo-LiDAR++ (SDN)
Pseudo-LiDAR++ (SDN + GDC)
Figure 14: Qualitative Comparison another example. The same setup in Figure 12
22


Reviews:
[{"review_id": "SklURuLnFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes two extensions to the recent work of (Wang et al., 2019) on 3D object detection with pseudo LiDAR data. Wang et al. showed that 3D object detection using stereo images as inputs can be significantly improved if the depth map is projected to 3D and treated like a LiDAR point cloud (i.e., using methods that utilize the LiDAR point cloud). This paper shows that one shortcoming of this approach is given by the fact that the depth uncertainty increases the farther the objects are away. To remedy this, the authors propose to train the stereo estimation network (based on Chang & Chen, 2018) directly with depth outputs, instead of disparity values (inverse depth), by rewriting the loss and converting the cost volume. This already boosts the performance for far away objects. The authors demonstrate that the usage of a (simulated) low-cost 4-beam LiDAR can further facilitate the detection. For this purpose a graph diffusion algorithm is listed that aligns the pseudo LiDAR point cloud from the stereo set-up with the depth estimates from the low-cost LiDAR. Simulating the low-cost LiDAR on the Kitti benchmarks shows that this approach further increases the performance of the object detection methods.\n\nIn general, I am in favour of accepting the paper as it shows two orthogonal and interesting additions to the pseudo LiDAR paper of Wang et al. that improve its performance. However, I would like to see some clarifications in the rebuttal.\n\nThe proposed stereo network converts a disparity cost volume to a depth cost volume using bilinear interpolation. I agree, that the 3D convolutions are more meaningful (given the spacing of the grid cells) on the latter, but why the detour over the disparity cost volume? It should be possible to build the depth cost volume directly, which would lead to decreased memory consumption and speed up the method without any loss in accuracy?\n\nOne assumption of the second contribution (GDC) is that at least one beam of the LiDAR will hit the k-connected local point cloud. Can you give some bounds on the likelihood that this happens, especially for far away objects it could be unlikely, although it is most beneficial for those objects.\nFurther, I am missing a details on the optimization of (7) and (8). What is meant with slight L2 regularization? In the appendix it is also stated that a slightly different objective is optimized? \nFinally, the notation could also be improved. The authors are using L and G for the LiDAR point cloud and PL and Z for the pseudo LiDAR point cloud and then in the Z' is used for both.\n\nFig. 4 shows the median error in meters for the different variants of the stereo network. Why has the median been used? Are there severe outliers? If yes, it would also be interesting to quantify those and compare them (e.g., box plots).\n\nIn the abstract and in the discussion the authors oversell their results a bit. At the one hand they state that PL++ with GDC performs significantly better than PL++ w/o GDC, on the other hand they also claim that PL++ achieves comparable results to models that have access to the full 64-beam LiDAR data. However, if you compare the differences, then the gaps are for several cases almost as big, or bigger as in the former claim.\n\nThings to improve the paper that did not impact the score:\n- In equation (2) you could replace the x with a . (\\cdot), or completely remove it\n- On page 5: KNN neighbors -> k-nearest neighbors\n- Also on page 5: write out W.l.o.g.\n- In Tab. 1 it would help to highlight (bold) the best entries per column", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1lWwstRFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to improve the idea of using stereo + lidar -style object detection to form stereo-based 3D object detection, building off of pseudo-lidar. In particular, it proposes to (a) switch the loss function for stereo deep networks from disparity to depth (b) do the stereo cost volume analysis in a depth volume (via resampling) rather than disparity volume and (c) if sparse lidar is available, align the estimated depth with the sparse lidar. Each seems to improve results, and the resulting system achieves good results on KITTI and outperforms past work in this area.\n\nPositives:\n+The paper proposes three ideas that seem good and lead to improvements that are demonstrated empirically.\n+The paper is well written \n+The experiments are exceptionally thorough\n+The ideas seems to me to be of obvious importance, although I realize that I'm likely not qualified to make a statement about this, and this should perhaps be done by a roboticist. \n\nNegatives:\n-Most of the heavy lifting in the case without sparse LIDAR is done by tweaking the loss function rather than the cost, although the remaining gain is still pretty good\n-I am not sure if this is a negative, but this is really a 3D vision paper. I do some form of 3D vision, but I really don't feel confident about my ability to assess whether doing stereo matching in a depth cost volume as opposed to disparity is correct -- I really haven't worked on stereo. It seems to work well, but I feel as if the wrong people are being asked to review the paper. I leave questions of venue to the area chair though.\n\nOverall, I am inclined to accept the paper. I am a tiny bit worried about venue and whether the right people will check the work, but I don't think this should be decided by reviewers. However, I think the experiments are quite thorough and the paper is clearly above the bar.\n\nIn more detail:\n\nMethod:\n+The method reads quite well and the idea is clean. I particularly like the graph-based depth correction algorithm, and the LLE-like way of adjusting the estimated depthmap. I have a few small comments below that do not affect my judgment, but I think would improve the paper.\n= Small thought: the words systematic bias throughout is primarily referring to a bias for a particular object as opposed to a bias of the system (i.e., any individual object is too far or too close). This seems non-standard to me. A systematic bias would be that everything's too far away by 1m for instance.\n\nExperiments:\n+The experiments are exceptionally thorough, and of my pile of ICLR papers, this by far has the most thorough and well-thought-out experimental analysis.\n+The system shows systematic improvements on 3 different LIDAR-based object detection systems; I think this is great.\n=I'm not sure whether the 64-beam LIDAR can be subsampled to imitate 4-beam LIDAR. I simply don't know enough about the hardware to know if this is a sensible approximation. \n-Table 3 primarily suggests that the vast majority of the hard work in the non-sparse LIDAR is done by the depth loss rather than the depth cost volume. The resulting change is still pretty good (although I suspect that if you stuck in a coordconv in the disparity cost volume, it would handle the fact that you want unequal smoothing).\n-The burial of the results on depth prediction results in the appendix with one is a little surprising as is the solitary table on it, but I understand the need to focus on 3D detection.\n\nSmall stuff that doesn't affect my review:\n1) Framing the problem as having ethical considerations is, in my view, not necessary -- should all network compression papers start arguing that it is of profound ethical importance to figure out your bit quantization? \n2) Last paragraph above Section 4 \"gird\" -> \"Grid\"\n3) Figure 3 caption \"pruple\" -> \"purple\"\n4) Figure 4 is suboptimal -- I assume SDN+GDC < SDN < Disparity Net, but this is hard to verify.\n5) Calling the network \"Disparity Net\" is a bit of an issue given that there's DispNet already \n6) \"Figure 1 illustrates beautifully how\" -> Please don't editorialize like this\n\n-----------------------\n\nPost rebuttal update: I have read the rebuttal and maintain my belief that the paper should be accepted.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkl-5f5CYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper describes a new method for Pseudo-Lidar, that is the reliable recovery of a 3D point cloud from 2D inputs and subsequent detection of 3D objects from the point cloud. The authors focus on improving the accuracy of the reconstructed point cloud by formulating a loss in depth, rather than disparity space, and by using sparse true lidar readings to align estimates. These techniques lead to a boost in 3D object detection performance. \n\nStrengths:\nThe Pseudo-Lidar method has been well-received, and it appears that this paper makes a nice improvement on the previous in terms of 3D point cloud accuracy. While the image results here are convincing, I would have liked to see an added empirical evaluation of precisely how accurate the resulting 3D reconstructions are, measured against ground truth 3D lidar on a test set. Do the point clouds only look accurate locally (and perhaps near known objects give good shape due to regularity), or are the metric results also quite strong indeed? \n\nI found the author's technical analysis and method description to be clear and well-motivated. None of the math or formulations are entirely surprising, but they are new to this area, so this appeared as nice sensible progress to me.\n\nThis area is closely tied to the self-driving car application, and thus bottom-line performance is the key measuring stick for impact on practitioners. For 3D object detection, the main goal of interest, the authors show up to 20% improvements for their combined method over quite recent and strong PL methods (although the new method uses sparse lidar, which is a great advantage, hence not entirely equal comparison). This is the main impact of the paper, as I see it, and enough reason for acceptance. \n\nAreas for Improvement:\nI found that the authors did not sufficiently recognize that there have been a wide variety of methods utilizing sparse 3D along with dense 2D images to interpolate to full 3D. For example, [A] is one I recall well from 15 years back, but at that time there was a strong community in this area, so I encourage the authors to do a bit more thorough review. \n\nThis paper has the fewest qualitative examples of 3D objects detected among the recent papers I've read. The final pages of the Appendix contain a few more of these visuals, but there are too few in the main paper for the reader to get any intuitive feeling of the physical meaning of your performance improvement. I'd like to see you add several examples, even if small, into the paper to aid in this understanding. \n\nDecision: weak accept due to the nice clear method that gives a strong improvement on an important area to industry today. \n\n[A] Statistical Inference and Synthesis in the Image Domain for Mobile Robot Environment Modeling, Luz Abril Torres-Méndez and Gregory Dudek. In Proceedings of the IEEEE/RSJ/GI International Conference on Intelligent Robots and Systems (IROS), Sendai, Japan, 2004.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three knowledgable reviewers give a positive evaluation of the paper. The decision is to accept.
Decision: Accept (Poster)