Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR
Authors: Ziyue Feng
Longlong Jing
Peng Yin
Yingli Tian
Bing Li
Venue: CoRL2021 Poster (2021)
Keywords: Self-supervised
Monocular
Depth Prediction
Sparse LiDAR

Abstract: Self-supervised monocular depth prediction provides a cost-effective solution to obtain the 3D location of each pixel. However, the existing approaches usually lead to unsatisfactory accuracy, which is critical for autonomous robots. In this paper, we propose FusionDepth, a novel two-stage network to advance the self-supervised monocular dense depth learning by leveraging low-cost sparse (e.g. 4-beam) LiDAR. Unlike the existing methods that use sparse LiDAR mainly in a manner of time-consuming iterative post-processing, our model fuses monocular image features and sparse LiDAR features to predict initial depth maps. Then, an efficient feed-forward refine network is further designed to correct the errors in these initial depth maps in pseudo-3D space with real-time performance. Extensive experiments show that our proposed model significantly outperforms all the state-of-the-art self-supervised methods, as well as the sparse-LiDAR-based methods on both self-supervised monocular depth prediction and completion tasks. With the accurate dense depth prediction, our model outperforms the state-of-the-art sparse-LiDAR-based method (Pseudo-LiDAR++) by more than 68% for the downstream task monocular 3D object detection on the KITTI Leaderboard.

Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR
Ziyue Feng'
Longlong Jing"
Peng Yin
Yingli Tian?
Bing Li¹
Cllemson University
"The City University of New York Carnegie Mellon University
Abstract: Self-superviee monocular depth prediction provides a cost-effective solution to obtain the 3D location of cach pixel. However, the existing approaches usually lead to unsatisfactory accuracy, which is critical for autonomous robots. In this paper, we propose FusionDepth, novel two-stage network to advance the self-supervised monocular dense depth learning by leveraging low-cost sparse (e.g. 4-beam) LiDAR. Unlike the existing methods that use sparse LiDAR mainly in a manner of time-consuming iterative post-processing, our model fuses monoc- ular image features and sparse LiDAR features predict initial depth maps. Then, an efficient feed-forward refine network is further designed to correct the errors in these initial depth maps in pseudo-3D space with real-time performance. Exten- sive experiments show that our proposed model significantly outperforms all the oitt--of-thearrt self-supervised methods, as well as the Lhae meth- ods on both self-supervised monocular depth prediction and completion tasks. With the accurate dense depth prediction, our model outperforms the state-of of-the- art Maurse-LDDARbaasee
-aseed
[1]) by more than 68% for the downstream task monocular 3D object detection on the KITTI Leaderboard. Code is available at https: //g: /tgtthb......./ com/ Aww.AAaabbba/aa..a.aaaaaaaawallt / Fusi onDepth
method (Pseudo-LiDARR)
Keywords: Self-supervised, Monocular, Depth Prediction, Sparse LiDAR
Introduction
Obtaining the 3D location of objects is an essential task for autonomous robots. However, accu- rate dense depth perception with LiDAR is normally expensive, thus limit for mass production. The depth prediction from monocular images [2, 3] is cost-effective and attracting more and more attention from both research and industry communities.
Many methods have been proposed and remarkable progress has been achieved in recent years [4,5, 6, 7]. Unlike other computer vision tasks, it is impractical to obtain large-scale dense depth labels. Therefore, self-supervised monocular depth prediction has been a promising solution. Typically networks are trained predict both the depth and ego-motion of the camera, while the re-projection photo-metric loss is calculated as an intermediary constraint to optimize the networks. However, these methods usually suffer from multiple challenges due to the loss function design. The most significant one is that the re-projection constraint assumes the scene is static and without occlusions between the neighboring frames. In fact, most of the vital objects (c.g., vehicles, pedestrians, and cyclists in the driving scenario) are dynamic, and the occlusions are almost inevitable.
To handle these challenges, potential direction is to perform the monocular depth prediction with other low-cost sensors like sparse (e.g. 4-beam) LiDAR. Compared to the 64-beam Li- DAR, the cost of 4-beam LiDAR is at least two orders lower while providing very sparse yet accurate depth. It is too sparse to be directly used for high-level perception tasks but poten- tially can be used with images guide the network for better dense depth prediction Pseudo- LiDAR++ [1] employed the sparse LiDAR in the post-processing using graph-based depth cor- rection (GDC) module to improve the performance of stereo 3D detection. However, this approach has two significant limitations: 1) the quality of its predicted dense depth is non-optimal since the sparse LiDAR data is not utilized in the depth prediction network; 2) the GDC proc-srons
*Corresponding author: Bing Li <bl148clomsor
edu>
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

did not utilize the visual context information, and is too slow (1 to 2 FPS) for real-time ap- plications like autonomous robots. Pursing an accurate and real-time self-supervised monoc- ular depth prediction, we propose a two-stage network for depth prediction by fully utilizing the sparse LiDAR points and monocular images in both the feature and the prediction levels. Our framework can learn the complementary information from the two distinct types of fea- tures for the dense depth prediction tasks. To overcome the sparsity issue of the sparse Li- DAR, we transform the sparse LiDAR points into pseudo dense representations, which are more suitable for networks to extract features, and then fuse the features with the image fea- tures to predict the initial depth. To further im- prove the quality of initial depth, we train a Re-
Monocurar Image
Dense Dopth Map
Eporse LDAR
Figure I: vides sparse yet accurate points which cannot be directly used for high-level downstream tasks such detection due the sparsity. The monocu- lar depth prediction can be significantly improved by effectively fusing the features of the sparse Li- DAR via self-superrvsed learning. With our high- quality predicted depth, the performance of the downstream perception tasks such as monocular 3D object detection can be remarkably improved.
The sparse LiDAR (e.g. 4-beam) pro-
fineNet to efficiently correct the high-order er- rors in the 3D space to obtain high-quality dense depth maps.
The ultimate goal of depth prediction is to provide 3D information for downstream tasks such as monocular 3D) detection and re-construction. However, the relation between the performance of depth prediction and the high-level downstream tasks has not yet been explored. To thoroughly evaluate our proposed method, we report the performance for both low-level tasks, including self- supervised dense depth prediction and completion, and a downstream high-level perception task: monocular 3D object detection. On all these tasks, our proposed model FusionDepth significantly outperforms the state-of-the-ar methods that rely do rely on sparse LiDAR. To summarize, our key contributions are as follows:
We propose novel two-stage self-supervised network predict and refine dense depth maps by fusing the features of 2D monocular images and 3D sparse LiDAR points. Our experiments demonstrates that our model achieves state-of- of-the-art performance in the depth prediction and completion tasks on the KITTI dataset.
To overcome the sparsity issue of the sparse LiDAR, we propose to transform the sparse points into novel pscudo dense representation (PDR) which can be more effectively fused with monocular image features.
With the improved predicted depth maps, the performance of the downstream task monoc- ular 3D object detection is significantly improved. Our model outperforms the state-of-the- art sparse-LiDAR LDDAR-based 3D detection model (Pseudo-LiDAR++ [1]) by more than 68% the KITTI dataset.
2 Related Work
Self-supervised Monocular Depth Prediction: Early work for depth prediction is usually super- vised methods [4, 3, 8, 9, 10, 11, 5, 6, 7, 12). Pixel-level labeled dense depth rarely available, in recent years, self-supervised methods [13, 14, 15, 16] became more and more popular. They have achieved great success but still suffer from dynamic objects and scale ambiguity. In contrast, we fuse the feature from sparse LiDAR points to help method predict more accurate depth for each pixel.
Depth Prediction with Sparse LiDAR: Recently, many researchers proposed to use few-beam LiDAR for better depth prediction [17, 18. 19, 20, 1]. The Pseudo LiDAR++ [1] achieved excellent performance by a GDC post-processing module to optimize the predicted depth with 4-beam LiDAR data. However, the potential of the sparse LiDAR is not fully discovered since the sparse LiDAR points are unused in the initial depth prediction stage. To effectively utilize the sparse LiDAR and monocular images, we fuse them both feature and prediction level for more accurate depth predictions.

---

POR Feeture Enooder
For Pileer
Inage (T)
Dept
Final Prediction
Image Feature Encoder
DepthNet
Sparse LIDAR (T)
Scale-nvarianl
Spanso
Enhanced Depth
Resprojection Losa
Comoectio
Figure 2:
Overview of our framework: Our proposed two-stage self-supervised FusionDepth model takes a monocular image and the corresponding sparse LiDAR points as input, and predicts dense depth map for each monocular image. At inference time, only the modules inside the gray dashed rectangle are needed.
Depth Completion: The depth completion is task to generate per-pixel dense depth maps from the relative sparse depths. Most depth completion models [21, 22, 23, 24, 25, 26] are trained with labeled data which require intensive human labors. To utilize the massive unlabeled data, self-supervised depth completion methods were developed in recent years [27, 28, 29, 30, 31] []] to generate depth maps from 64-be 14-beams dense LiDAR points. Our proposed method is designed to predict depth maps from 4-beams sparse LiDAR points, however, with the generalizability, our method is also applicable for the depth completion task.
Monocular 3D Object Delection: Monocular 3D object detection to directly predict the 3D coordinates of objects from monocular images. There are mainly two types of methods: RGB image- based and pseudo-LiDAR based. Former employ detection networks like CenterNet [32] to predict the bounding boxes [33, 34, 35, 36] directly from images. The latter perform the detection over the pseudo-LiDAR representation, which lifted from the dense depth prediction [37, 38, 39, 40). Benefited from the 2D-to-3D mapping. the pseudo-LiDAR -based methods achieved much better performance [39]. Our experiments demonstrate that the performance of the monocular 3D object detection [1, 39] can be significantly improved using our depth prediction.
The Proposed Method
The overview of our proposed sulf-super framework is shown in Fig. 2. Our framework predicts dense depth map for each monocular image by taking two types of data input: monocular image and its corresponding sparse LiDAR points. Our framework consists of two steps: initial depth prediction based the fused multi-scale features from both the monocular image and sparse LiDAR points, and depth refinement to correct the high-order errors in the initial depth maps. The details of each component are described in the following sections.
3.1 Initial Depth Generation
For each monocular image H x3, the corresponding sparse LiDAR points are pN captured by few-beam LiDAR (c.g. 4-beam) where N is the number of the points. Each point ps consists of three values X, Y, Z. representing the location in the 3D space. The network predicts an initial depth map for each image based on the fusion of features from image / and the corresponding sparse LiDAR points P.
The main challenge here is to effectively fuse the features from 2D image and the features from set of unordered LiDAR points. When projecting all the sparse LiDAR points into the image plane, only 1.4% of the pixels have corresponding depth values. We observe that simply concatenating the image data and the projected sparse depth map can only negligibly improve the performance due to the sparsity of the representation. To resolve the sparsity issue, we transform the sparse LiDAR

---

points into pseudo dense representations (PDRs), which can be effectively encoded and fused with the monocular image features by our model.
For each image TxxW'3 x3 all its corresponding sparse LiDAR points PNx3 transformed into two-channel PDRs with size of H x W x2, including depth channel to present the absolute depth values of each pixel, and a confidence channel presents the reliability of the corresponding depth. Based on the assumption that the depth values should be similar for most neighboring pixels, we generate the depth channel by first projecting each sparse LiDAR point p; into the image plane at the position of (u,, v.) and then dilate into a circular area with a radius of R. The depth channel (D) is generated by setting the depth value of pixel (æ, y) within this circular area as Z₁:
Pseudo Dense Representations Generation:
(x,y) = V(u, + + (Us - y)²
(1)
D( {
Zz if r y) < R
(2)
0. otherwise
Although the depth channel transfers the sparse LiDAR points into dense representation, it inevitably introduces noises. To mitigate the impact of these noises, we further generate the confidence chan- nel to indicate the reliability of depth for each pixel. The confidence of each pixel is inversely proportional the distance to its circular center:
C(s,y) v) = { + (a.y), v). if y) < R 10,
otherwise
If more than one sparse data point is close to an PDR pixel, the confidence and depth scores gener- ated from these multiple data points will be averaged. These two channels jointly provide alternative dense representations that can be encoded by the convolutional neural networks more effectively.
Net for Initial Depth Map Prediction Based Fused Features. After transforming the sparse LiDAR points into the two-channel PDRs, the features of PDRs and monocular images can be fused together for the initial depth prediction. To enable our network to thoroughly learn the complementary information from two distinct features, we adopt the intermediate fusion to combine the multi-scale deep features layer by layer. As shown in Fig. 2, there are two feature encoders: PDR feature encoder for extracting features that explicitly encodes depth information from the sparse LiDAR pseudo dense representation, and monocular image feature encoder to extract feature which implicitly encodes semantic information from images. The decoder network takes the two types of multi-scale features and concatenates them together to predict the initial depth map at multiple scales. With the effective feature fusion, our DepthNet can predict more accurate dense depths.
PoseNet for Ego-Motion Prediction. The PoseNet is essential for self-supervised monocular depth prediction since its predicted ego-motion makes the cross-frame projection possible, which is used as geometry constraints train the network. It is directly related to the quality of the pixel corre- spondence across frames. To predict accurate ego-motions, as shown in Fig. 2, the PoseNet is also designed to take the complementary monocular images and the PDRs as the input. The ego-motion is formulated as 6 degrees of freedom, consisting the camera rotation and translation.
3.2 Depth Refinement
Our network produces relatively accurate initial depth predictions by fusing the features of monoc- ular images and sparse LiDAR points. However, the network still makes various errors, such as inconsistent depth prediction for different parts of the same object and the systematic depth bias.
To further resolve the inconsistent depth prediction and improve the correction computing efficiency, we propose RefineNet to correct the initial depth errors in the pseudo 3D space [39]. Our RefinNeet a multi-scale fully convolutional network, takes the PDR feature, image feature, and initial depth as input, outputs the refined final depth prediction. To leverage the 3D information predicted from DepthNet, each initial depth map is first transformed into a 3-channel x-y-z map representation. For pixel at location (u. v)) with depth d at the initial depth map, the transformation is based on the camera intrinsics fa. fgy Cz. Cyi
I (u Cp) x d/ fa.
y (v- C.y) d fnz
(4)
z = d,
4

---

Our RefineNet is trained by distilling knowledge from offline depth correction module in the way of self-trrining. As shown in Fig. 2. for each initial depth map, depth correction module is applied to produce more accurate depth named 'Enhanced Depth' with the guidance of sparse LiDAR points, By training with pairs of the pseudo 3D initial depth, PDR and image features, and the enhanced depth, the RefineNet can improve the initial depth quality by correcting its errors. Compared to the depth correction module GDC [1] we used as teacher module, our network is very computationally efficient for real-time systems. Moreover, the conventional depth correction module can still be further applied as post-processing after our RefineNet to improve the accuracy by sacrificing the real-time performance.
3.3 Loss Functions
Our model jointly trained with two loss functions: the re-projection loss Lre utilize the inter- frame geometric constraints and the scale-invariant loss Lei to distill the knowledge from the en- hanced depth maps generated by the offline depth correction model.
The re-projection loss Lie is a linear combination of two parts: the photo-metric loss Lp with a filtering mask 14, and the smoothness loss smooth- The photo-metric loss Lp is to evaluate the pixel-level similarity between the re- re-projected fake image 4-- with the real image I at adjacent time frames, based the photo-metrie reconstruction error pe which consists of the SSIM and L1 distance to penalize the errors of the re-projected image. We choose to use the frames 4 and 1-11 as Ip- The proj() () projects current frame dense depth D2 frame i' with camera intrinsic matrix K and the camera ego-motion estimated by the PoseNet, () is the sampling operator.
Lie 1Tlp + Lsmooth,
(5)
Lp = Σ pelle, kewe)
(6)
Te = Te (proj(( (De. T K)):
(7)
(e(T.: la. Tb) = 2 (SSSSS +))) - 2
(8)
To eliminate the shrinking of the depth map, we further adopt the edge-aware metric from [41] into our smoothness loss function Lsmooth, which is formulated as:
L Lemooth =
|0,di1 e 10244 + +1,dd 18,4,
while di = de de/de is the mean- normalleed inverse depth. This normalization makes the smoothness loss invariant to output scale.
For the photo-metric loss Lp. follow [15] to apply filtering mask μ to filter out the occlusion and stationary pixels, and then interpolate the depth predictions at each scales to the input resolution before computing our re-projection loss to eliminate the 'holes' at the low-texture area. The filtering mask y is formulated as:
c = min pe(I,, +1) < min
(10)
To distill the knowledge from the offline correction model, the scale-invarian loss is employed for optimization, which is formulated as:
Loi = A * VnSii
(11)
Si
Σ (log log wi) - (log log 1)g
(12)
2
where y and y" indicate the predicted depth and enhanced depth respectively, is the number of pixels. The Si loss penalizes the relative depth-differences between each pixel pairs.
Our framework is trained with linear combination of the re-projection loss Tyee and the scale- invariant loss Lee:
Listal = -aLre + BL
(13)
while a and are the weights for re-projection loss Tyne and scale-invariant loss Lag respectively. The implementation details can be found in the supplementary materials.
5

---

Method
Train
Abs Rel
The lower better
Sq Rei
RMSE
RMSE log
61
ligher the better
4
6.
LEGO [2))
M
0.162
1.352
6.276
0.252
0.783
0.921
0.989
PackNet-SIM [16]
M
0.111
0.785
4.601
0.189
0,878
0.960
0.582
MonoDepih [14]
0.13
1.142
5.533
0.230
0.830
0.936
0.970
MonoDapth2 [15]
M+S
0.106
0.818
4.750
0.196
0.874
0.957
0,979
Don [43]
Mc-upp
0.099
0.343
3.113
6.16T
0.897
03966
B1S [44]
Mrsup
0.911
0.555
4.033
0.174
0.967
Gauzilimi ot 145|*
M+L
0.882
0.425
3.75
6.131
0,917
FusionlDepeh (Initial Depth)
M+L.
0.078
0.515
3.67
0.154
0.935
0.975
0.986
PepionDepth (Refined Depth)
M+L
0.074
0.423
3,61
0.150
0.936
0,973
0.986
Struct2Depphh [1o]
M
0.109
0.825
4.150
0.187
0.874
0.958
0.563
GLNett (47)
M
0.099
0.796
4,743
0.186
0.584
0.955
0.979
MonoPL-+
M+L
0,998
0.176
0.967
PepionLepth (Initial Depth GDC)
M+L
3.42
0,148
0.977
Pusionleepth (Kefined Depth GIC)
MiL.
0.063
0.364
3
0.139
0.945
0.978
0.988
Table 1:
Depth prediction on KITTI original dataset: Methods are ranked by absolute relative
error. The best results are in bold. All methods are using resolution of 640x192 pixels. Due to the exceptional time-consuming (around 1-2 FPS), we rank methods with and without iterative refinement separately. M. S. and L respectively indicates Monocular, Stereo, and Sparse LiDAR data, with Sup and respectively indicating supervised training and iterative correction in testing phase. * Only use LiDAR data in training phase, but tested on the KITTI improved dataset, which usually has much lower error value.
Method
Supervised Depth
Easy
KITTI Toxing (AP)
Mod.
Hard
Mathod
Samples
The
Als Rel
henler
lights
Decompled-3D (49)
Pscudo LiDAR-1" [1]
7.02
Tol-MAA [17]
-550
85.3
USS6
MonoPSK [50]
10.76
7.25
3.85
al.
98.4
MonaPI, [38]
7.50
6.10
Spree::)Aroa (19
100
6033
4.003
96.
93.3
SS3D
7.68
FusicnDeephh
100
0034
4.11
9X0
97.8
98.3
MinodDUS [52)
7.94
PosionDepihh + GDC
100
0073
4.11
930
97.0
98.3
MID-RPN [53]
14.76
9.71
7.42
Deese [19]
970
AM3D [54]
16.50
10,74
9.52
PistonlNeph
Fustenlepphh
970
983
PatchNet [39]
15.68
11.12
10:17
MonoPI (1]
9.50
Table 2: Depth prediction with random- sampled LiDAR points: Comparison of per- formances on the KITTI dataset [48] with meth- ods that also rely on sparse LiDAR points. The input point clouds are randomly sampled from 64-beam LiDAR points. Our FusionDepth out- performs all other methods with a large gap even without refinement.
FustonDepth(Cuur))
23:21
16.33
+68.9%
+75.0%
-74.0%
Table 3: 3D detection performance evaluation
for the Car category on the KITTI dataset test- ing set. @037
*The depth module of Pseudo LiDAR++ [1] is stereo, which can greatly improve the de- tection result. For fair comparison, we re- place the with Monodepth2 [15], and refer as MonoPL++ [1]
4 Experiments
4.1 Depth Prediction
Dataset. Following the state-of-the-art methods [15, 16, 13], evaluate our FusionDepth perfor- mance of dense depth prediction on the Eigen split [55] of the KITTI original dataset. We did not evaluate the KITTI testing benchmark which is for vision-only methods. The 4-beams data sampled from original 64-beams LiDAR data same as Pseudo LiDAR++ [1].
Results. To extensively evaluate the performance, we compare with four types of methods includ- ing: (I) self-supervised monocular-basee methods (M) [56, 42, 41, 57, 58, 15, 16. 13]. (2) self- supervised stereo-based methods (S) [59, 60, 14, 61, 62), (3) supervised methods (Sup) [43, 44], and (4) methods that use LiDAR signal as guidance (L) [1. 45]. The performance comparison with the state-of-the-ar methods for these groups is shown in Table 1. Note that the initial depth mod- ule in Pseudo LiDAR++ [1] is supervised and stereo. For fair comparison, we replace it with the ofae--of-the-ar unsupervised monocular depth module Monodepth2 [15]. We will refer this model as MonoPL++.
Due to the limitations of the re-project mhoto-metric loss, the self- monocular (M) [56. 42, 41, 57, 58, 15, 16, 13] and stereo-based methods (S) [59, 60. 14, 61, 62) usually have Abs Rel over 0.1. With the advantage of using the sparse LiDAR, the performance of our initial depth maps

---

Method
FPS
Mod.
Easy
Mod.
Car
Hard
Easy
Pedestrian
Mod.
Hard
Easy
Cycha Mod.
Hard
Depth
Rei
Mensoleph
1.35
10227
JusbonDepthiaattal Depth) FusionDlephh Relinod Degeth) Della (96)
100
9,02
22.29
22.20
15,42
15.37
6.49
6,50
6,00
7.27
5.14
7.25
7.51
+00%
+28%
+20%
+29%
+306
-32%
+326
+217%
+233%
+206%
29%
18
3.30
FusionDepthdnitall Depth-GDCi
PusionDepdhi Refined Depth+ Dalka (9)
44.55
33,59
28.87
14.46
23.28
13.38
6.37
-98%%
+32%
550%
+4/6
+185%
+1725
+140%
+205
+368%
+229%
76%
Table 4: Monocular 3D object detection result with PatchNet [39] on KITTI dataset, AP@0.7 for cars, AP00.5 for pedestrians and eyclists. Our FesionDepth can greatly improve the performance both with or without GDC.
already better than all these methods and even outperforms the supervised methods (Sup) [43, 44]. With our RefineNet, our performance is further improved and outperforms all the sparse-LiiDAR- based methods [1, 45]. Using the GDC for post post-processing, our final results significantly outperform all other methods, including the above mentioned most recent work MonoPL++ [1] which has access to the same amount of the sparse LiDAR points and same post-processing. The results indicate an effective fusion of sparse LiDAR points and monocular images achieves predictions. More quantitative and qualitative comparison and error analysis can be found in the supplementary materials.
accurate dense depth
To extensively compare with the state-of-the-art methods under the same settings, as shown in Ta- ble 2, we compare with methods that were originally designed to use sparse LiDAR for depth pre- diction, including full-MAE [17], Parse-a-Line [18], Sparse-to- Re-Dense [19), and MonoPI.+ [1]. For each group of experiments, the same amount of sparse LiDAR points is used for fair compar- ison. Under the same settings, our proposed model consistently significamtly outperforms all the state-of-the-art methods. These results further confirm the effectivvness of our proposed method.
4.2 Monocular 3D Object Detection
The ultimate goal of the monocular depth prediction is to provide 3D representation for downstream tasks. To demonstrate the impact of our improvement the depth metrics to downstream tasks, evaluate the performance of the monocular 3D object detection task with our predicted depth maps as input on the KITTI detection dataset. Following the state-of-the-art methods [39], we report 3D Average Precision (AP) as the evaluation metrics.
Table 4 shows the performance comparison for the monocular 3D object detection task. The same detection model is employed for all the experiments while the only difference is the input depth. The monodepth2 [15] is used baseline model. Our model use the same CNN backbone (ResNet- 18 [63]) as Monodepth2 [15]. By effectively using the sparse LiDAR, the detection performance can be improved by more than 40% compared to the baseline which only uses monocular images. Compared to the recently proposed MonoPL++ [1]. our model significantly outperforms it by more than 28 in terms of the mAP over all the three categories. Without the time- consumin iterative refinement module, our model is 50 times faster than the MonoPL++ [1].
Furthermore, our RefineNet obtains significantly increased performance on the detection metrics than the depth metrics. The improvement of applying RefineNet on the depth score is only around 3.4% in terms of the relative error, while the improvement is more than 23% on the detection score. We observe that our improvement in the dense depth prediction can yield a even more significant improvement in the downstream task. This indicates the importance of using the downstream tasks to evaluate the quality of the learned dense depth.
Table 3 further shows the comparison of our method with other state-of-the-an for monocular 3D detection tasks on the car category. As an unsupervised learning method, our method significantly outperforms all the state-of-the-arr methods by large margin, including the sparse-LiDAR based MonoPI++ [1]. Note that the performance of our model can be further improved by extending it to the supervised setting. More qualitative analysis can be found in the supplementary materials.
7

---

Pscudo Dense Representation
Camen- IDAR Fusion in Intaal Prodiction Relnement Input I evel Curpur Level
Lower
Feature I aeel RefineNer GDC
Alr Rei
Sq Rel
RMSE
RMSFluy
Evalnaring Pscudo Dense Regresentation (PDR)
0.115
0.882
4,701
0.190
0.814
0.161
0.726
4,364
0.178
Evalunting Camera LiDAR Fasion Initial Prediction
0.101
0.226
4.361
0.178
0.07
0.10
0.74
4.369
0.078
0.515
3678
0.154
Erakating Relinement
0.515
3678
02077
0425
3420
0.74
I,K3
0.346
3291
0.138
Results the KITTI depth prediction dataset Eigen [55] split. We evaluate the effectiveness of PDR, Camera-LiDAR fusion, and RefineNet.
Table 5:
Ablation Study:
4.3 Depth Completion
We evaluate our FusionDepth on the KITTI depth completion task. Most (over 95%) of the methods in the KITTI depth completion benchmark is under supervised training. Following other state-of- the-art self-supervised methods [27, 28, 29, 30, 31J, we test our model on the KITTI validation set, as shown in Table 6. By effectively utilizing the LiDAR features, our model outperforms all other sele-supe methods by a large gap with all the metries demonstrating the generalizability of our proposed method.
4.4 Ablation Study
To evaluate the effectiveness of each component of our FusionDepth. we perform three groups of experiments to evaluate the impact of pseudo
Mathod
RMSE
IRMSF
iMAR
[P]
1593.37
dense representation, eamera-LiDAR fusion, and
Sparse to dense [28]
1342.33
4.28
1.64
DDP [30]
1310.03
the refinement on the dense depth prediction task.
VOICED [29]
1230.5
3.84
1.29
The results of the ablation studies are shown Table 5. The impact of the LiDAR sparsity and the error analysis by semantic categories can be found in the supplementary materials.
SellDeco [27]
1212.89
3.54
1.29
(eshonDepth(Ours)
1193.92
3,385
Table 6: Self-supervised depth completion: We evaluate our FusionDepth on the KITTI depth completion task validation set, comparing it to the state-of-the-arr self-supervised meth- ods. All metries the lower, the better. The best results are in bold.
Pseudo Dense Representation. When the sparse LiDAR are directly used as input, the perfor- mance for depth prediction is only slightly im- proved by around 6%. However, by directly using our proposed pseudo dense presentation as input, the improvement is doubled (12%), confirming its importance.
PDR Feature and Image Feature Fusion. We conduct three types of feature fusions, including input level, feature level, and output level fusion. The best performance is achieved by performing the feature level fusion of the PDR features and image features from our proposed encoders.
Refinement. The third group shows that the depth prediction performance can be improved by RefineNet evenly either with or without GDC, Although the RefineNet can only slightly improve the performance of dense depth prediction, it can significantly improve the monocular 3D object detec- tion task (+23%), demonstrating the effectiveness of the RefineNet. More qualitative and efficiency analysis of the RefineNet can be found in the supplementary materials.
Conclusion
We proposed a two-stage self-supervised framework FusionDepth that effectively fuses features from monocular camera images and sparse LiDAR data. Our method outperforms all the state-of- the-art methods on both depth prediction and completion tasks. Also shows a remarkable advantage for downstream tasks like monocular 3D object detection.

---

Acknowledgments
This work was supported in part by U.S. DOT UTC grant 69A3551747117.
References
[1] Y. You, Y. Wang. W. Chao, D. Garg, G. Pleiss, B. Hariharan, M. Campbell, and K. Q. Weinherger.
Pscudo-lidar+4: Accurate depth for 3d object detection in autonomous driving. Open
[2] Q. Li, J. Zhu, J. Liu, R. Cao, Q. Li, S. Jia, and G. Qiu. Deep learning based monocular depth prediction: Datasets, methods and applications. arXiv preprint arXiv:201 J. 1.04123.
[3] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from single image using a multi-scalc deep
network.
[4] Y. Cao, Z. Wu, and C. Shen. Estimating depth from monocular images as classification using deep fully convolutional residual networks. IEEE Transactions Circuits and Systems for Video Technology.
[5] D. Xu, W. Wang, Tang, H. Liu, N. Scbe, and E. Ricci. Structured attention guided convolutional neural fields for monocular depth estimation. IEEE Computer Society.
[6] Z. Zhang, C. Xu, J. Yang. Y. Tai, and L. Chen. Deep hicrarchical guidance and regularization learning for end-to-end depth estimation. Pattern Recognition,
[7] Z. Zhang, C. Xu, Yang. J. Gao, and Z. Cui. Progressive hard-mining network for monocular depth estimation. JEEE Transactions on Image Processing.
[8] Y. Gan, X. Xu, W. Sun, and L. Lin. Monocular depth estimation with affinity, vertical pooling, and label enhancement.
[9] X. Guo, Li, S. Yi, J. Ren, and X. Wang. Learning monocular depth by distilling cross-domain stereo networks.
[10] L. He, G. Wang, and Z. Hu. Learning depth from single images with deep neural network embedding focal length. IEEE Transactions Image Processing.
[11] Y. Kuznietsov, Stückler. and B. Leibe. Semi-supervised deep learning for monocular depth map pre- diction. IEEE Computer Society.
[12] H. Zhang, Y. Li, Y. Cao. Y. Liu, C. Shen, and Y. Yan. Exploiting temporal consistency for real-time video depth estimation. IEEE.
[13] C. Shu, K. Yu, Z. Duan, and K. Yang. Feature-metric loss for self-supervised learning depth and
egomotion. Springer:
[14] C. Godard, O. M. Aodha, and G. J. Brostow. Unsupervised monocular depth estimation with left-right consistency. IEEE Computer Society.
[15] C. Godard, O. M. Aodha, M. Firman, and G. J. Brostow. Digging into superviid monocular depth estimation. IEEE
[16] V. Guizilini, R. Ambrus, Pillai, and A. Gaidon. Packnet-sfim: 3d packing for self self-supervised monocular depth estimation. arXiv: 00ti::/900.0666.
[17] C. Cadena, A. R. Dick, and D. Reid. Multi-modal auto-encoders as joint estimators for robotics scene understanding.
[18] Y. Liao, L. Huang. Y. Wang, S. Kodagoda, Y. Yu, and Y. Liu. Parse geometry from line: Monocular depth estimation with partial laser observation. IEEE.
[19] F. Ma and S. Karaman, Sparse-to-dense: Depth prediction from sparse depth samples and a single image. TEFF
[20] J. Uhrig, N. Schneider, L. Schneider. U. Franke, T. Brox. and A. Geiger. Sparsity invariant enns.
[21] Y. Chen, B. Yang. M. Liang, and R. Urtasun. Learning joint 2d-3d representations for depth completion.
IEEE
[22] X. Cheng, P. Wang, C. Guan, and R. Yang. CSPN++: learning context and resource aware convolutional spatial propagation networks for depth completion. AAAI Press.
[23] Park, K. Joo, Z. Hu. C.-K. Liu, and S. Kweon. Non-local spatial propagation network for depth completion. arXiv preprint arXivv2007.1004:
[24] J. Qiu, Z. Cui, Y. Zhang, X. Zhang, S. Liu, B. Zeng, and M. Pollefeys. Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image.
[25] C. Qu, Nguyen, and C. Taylor. Depth completion via deep basis fitting.
[26] Y. Xu, X. Zhu, J. Shi, G. Zhang, H. Bao, and H. Li. Depth completion from sparse lidar data with depth-normal constraints. ICEE.
[27] Choi, D. Jung, Lee, D. Kim, D. Manocha, and D. Lee. Selfdeco: supe-supe monocular depth completion in challenging indoor environments. arXiv preprint M0ii:2011.049777
[28] F. Ma, G. V. Cavalheiro, and S. Karaman. Self-supervised sparse-to to dense: Self- superviee depth com- pletion from lidar and monocular camera. IEEE.
[29] A. Wong, X. Fei. Tsuei, and S. Soatto. Unsupervised depth completion from visual inertial odometry. IEEE Robotics and Automation Letters.
[30] Y. Yang, A. Wong, and S. Soatto. Dense depth posterior (DDP) from single image and sparse range. Computer Vision Foundation TEFF.

---

[31] S. Yoon and A. Kim. Balanced depth completion between dense depth inference and sparse range mea- surements via kiss-gp. arXiv preprint arXiv:2008.05/58
[32] X. Zhou, D. Wang, and Krähenbühl. Objects as points. arXiv preprint arXiv:1904.08850.
[33] X. Zhou, V. Koltun, and P. Krähenbühl. Tracking objects as points. Springer.
[34] Y. Chen, L. Tai, K. Sun, and M. Li. Monopair: Monocular 3d object detection using pairwise spatial relationships. IEEE.
[35] P. Li, H. Zhao, P. Liu, and Cao. Rtm3d: Real-time monocular 3d detection from object keypoints for autonomous driving. arXiv preprint arXiv:2001.03343.
[36] x. Shi, Z. Chen, and T.-K. Kim. Distance-normalized unified representation for monocular 3d object
detection. Springer.
[37] Y. Wang. W. Chao, D. Garg. B. Hariharan, M. E. Campbell, and K. Q. Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.
[38] X. Weng and K. Kitami. Monocular 3d object detection with pscudo-lidar point cloud.
[39] X. Ma, S. Liu, Z. Xia, H. Zhang, X. Zeng, and W. Ouyang. Rethinking pseudo-lidar representation.
Springer.
[40] R. Qian, D. Garg, Y. Wang, Y. You, S. J. Belongie, B. Hariharan, M. E. Campbell, K. Q. Weinberger, and Chao. End-to-end pseudo-lidar for image-based object delection. IEEE.
[41] C. Wang, J. M. Buenaposada, R. Zhu, and S. Lucey. Learning depth from monocular videos using direct methods. IEEE Computer Society.
[42] Z. Yang. P. Wang. Y. Wang. W. Xu, and R. Nevatia. LEGO: learning edge with geometry all once by watching videos. JEEE Computer Society.
[43] H. Fu, M. Gong. C. Wang. K. Batmanghelich, and D. Tao, Deep ordinal regression network for monocular depth estimation. IEFE Computer Society.
[44] J. H. Lee, M.-K. Han, D. W. Ko, and I. H. Suh. From big small: Multi-scale local planar guidance for monocular depth estimation. arXiv:1907.1083:
[45] V. Guizilini. Li. R. Ambrus, Pillai, and A. Gaidon. Robust semi-supervised monocular depth estima- tion with reprojected distances. PMLR.
[46] V. Casser, S. Pirk, R. Mahjourian, and A. Angelova. Depth prediction without the sensors: Leveraging structure for unsupervised lcarning from monocular videos. AAAI Press.
[47] Y. Chen, C. Schmid, and C. Sminchisescu. Self-supervised lcarning with geometric constraints in monoc- ular video: Connecting flow, depth, and camera. IEEE
[48] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. IEEE Computer Society.
[49] Y. Cai, B. Li. Z. Jiao, H. Li, X. Zeng. and X. Wang. Monocular 3d object detection with decoupled
structured polygon estimation and height-guided depth estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10478- 10485, 2020.
[50] J. Ku, M. Mozifian, J. Lee,, A. Harakch, and S. L.. Waslander. Joint 3d proposal generation and ohject detection from view aggregation. IEFF.
[51] F. Jörgensen, C. Zach, and F. Kahl. Monocular 3d object detection and box fitting trained end-to-end using intersection-over-unio loss. CoRR.
[52] A. Simonelli, S. R. Buld, L. Porzi, M. Lopez-Antequea,, and P. Kontschieder. Disentangling monocular
3d object detection. IEEE.
[53] G. Brazil and X. Liu. M3D-RPN: monocular 3d region proposal network for object detection. IEEE,
[54] X. Ma, Z. Wang, H. Li, Zhang, W. Ouyang, and Fan. Accurate monocular 3d object detection via
color-embedded 3d reconstruction for autonomous driving. IEEE
[55] D. Eigen and R. Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. IEEE Computer Society.
[56] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and ego-motion from video. IEEE Computer Suciety.
[57] J. Zhou, Y. Wang. K. Qin, and W. Zeng. Unsupervised high-resolution depth learning from videos with dual networks. IEEE,
[58] S. Pillai, R. Ambrus, and A. Gaidon. Superdepth: Self-supervised, super-resolved monocular depth estimation.
[59] A. Wong and S. Soatto. Bilateral cyclic constraint and adaptive regularization for unsupervised monocular depth prediction. Computer Vision Foundation / IEEE.
[60] F. Tosi, F. Aleoui, M. Poggi, and S. Mattoccia. Learning monocular depth estimation infusing traditional stereo knowledge. Computer Vision Foundation/ / IEEE.
[61] A. Pilzer. S. Lathuilière, N. Sebe, and E. Ricci. Refine and distill: Exploiting cycle-inconsistency and
knowledge distillation for unsupervised monocular depth estimation. Computer Vision Foundation / IEEE
[62] R. Li, S. Wang. Z. Long, and D. Gu. Undeepvo: Monocular visual odometry through unsupervised deep learning.
[63] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. IEEE Computer
Society.
10


Reviews:
[{"review_id": "t-sTEWqCifS", "reviewer": "Reviewer_HJoT", "summary": "This paper proposes a pseudo-Lidar framework from sparse Lidar points and an RGB camera. A two-stage network is utilized which preprocesses the sparse Lidar points and fuses them with RGB information to obtain a coarse depth prediction and then refines this. During training, multiple time frames are used to obtain additional losses to compensate the lack of supervised depth data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although, categorically monocular 3D object detection includes pseudo-lidar based methods, once the depth information is included (sparse or dense), it is not monocular camera estimation anymore, it includes at least two sensors for prediction. This is a repeated mistake in many papers. \n\n\nIf multiple frames can be used, then can't their sparse depth correspondences aggregated over time to produce a denser point cloud which could help depth correction as well?\n\nFor depth prediction, did you submit your results to original KITTI page?\n\nWhat is the difference between your reported results for PseudoLidar++ and KITTI page, can you explain briefly in the paper? (For all tables.)\n\nMinor points:\n\nOur experiments demonstrates-> Our experiments demonstrate", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors have compared a number of methods. Their own proposal, which is a combination of multiple networks, is reported to achieve a high accuracy. Although I could not check it in the original submission page of the datasets. The codes are also provided, which I appreciate. However, I did not have time to check the codes.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- more discussion on the reason of network module selection.\n- explanation of the difference between the dataset benchmark evaluation and your reported results.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "qNrxnHg9Rq", "reviewer": "Reviewer_vPeH", "summary": "This paper predicts depth images based on monocular RGB images and sparse LIDAR data, by fusing both in there so called DepthNet, which is afterwards improved by RefineNet architecture, which uses the result, the features from the sparse LIDAR data and the sparse LIDAR data itself. They show in a extensive evaluation section superior results to prior state-of-the-art methods in different categories. Furthermore, they show how each single change affects the final result in an depth ablation study. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* the paper is well written and explains all important concepts in depth\n* present a novel fusion mechanism by reusing the features from the first network in the second to refine the depth image\n* they claim to be the new state of the art method for depth prediction\n* Extra point for including a point cloud result, as no one can properly judge how well a depth completion method works by looking at a colored image\n\nWeaknesses:\n* The related work section, while citing many papers lacks a deeper comparison to these prior methods, to better clarify the difference between those works and this new work\n* Pseudo Dense Representation: Here they explain how they generate a dense depth map based soley on the sparse LIDAR data. However, they don't answer what happens if more than one sparse data point is close to an evaluated position, this should be added. Is the last point used or is interpolated between the two?\n* Table 1 contains a lot of comparisons to methods, which use different input data. 22 rows use only monocular or stereo data and don't use any sparse LIDAR data as an input, which makes the comparison a bit unfair.\n* I am furthermore confused by the claim that they beat state of the art, if I check out the leaderboard and try to compare it. I already have the problem that they don't provide the results on the test set but on the validation set, for example they report for \"Sparse to dense [30]\" an iRMSE of 4.28, while in the leaderboard: http://www.cvlibs.net/datasets/kitti/eval_depth.php?benchmark=depth_completion it says 2.80. I don't understand why they didn't evaluate the test data online, to make it easier to compare it existing methods. \n* Furthermore, do I struggle with the results report for RSME, as they seem much closer to iRSME values than to actual RSME values for KITTI. \n* Lastly, why are the results for the stereo case of Pseudo Lidar++ not reported, I agree that it makes it more fair to only use monocular to make the comparison more fair. That does not mean though that one should not report the original values, because now I can't judge if this change complete breaks the performance of this algorithm and then gives you edge or if you would have beaten it even with stereo? For me to accept this paper I need to know these values to access if changing Pseudo Lidar++ to monocular is something which the architecture can handle or if you broke Pseudo Lidar++ on purpose to give your method an advantage. As we can only judge other methods as they published them not as we want them to be.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am mostly basing my review on my doubts about the evaluation. If the authors can prove or explain the choices they made in their evaluation then I can gladly switch my recommendation to strong accept. But, at the moment I feel like that the evaluation was designed in a way to make it harder to compare it to the leaderboard and weaken on purpose the other approaches to boost their own method. \n\nBased on that I can not recommend accepting this paper in its current form to CORL.\n\nAfter rebuttal:\nThe authors answered most of my problems with the evaluation and henceforth I have no objection anymore against accepting this paper to CORL. They adjusted the evaluation and positioned themselves better against Pseudo LiDAR++. They also explained why an evaluation on the validation set is justified and extended the related work.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "* Results for Pseudo LIDAR++ with stereo have to be included\n* The depth completion and prediction methods should be evaluated on the test set not the validation set and furthermore the values should be in the same format as in the online leader board, to make it easier for everyone to understand if the claim about state of the art performance is correct\n* The related work section should contain more explanation on how this method compares to existing methods and not just list them. "}}, {"review_id": "e9wLnxGDKL", "reviewer": "Reviewer_SZdB", "summary": "This paper proposes a method for fusing sparse lidar point clouds with RGB images to obtain accurate dense depth predictions. Key to the performance of the approach is the use of pseudo-dense representations of the sparse lidar point cloud which assist the convolutional network in effectively fusing the data. The results show that the approach outperforms a range of current methods in terms of depth completion, monocular depth prediction and stereo depth prediction.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is very well written and polished. The figures are nicely presented and help to explain the approach. The paper seems to be mature and is of publication quality should it be accepted.\n- I think the approach is very interesting and well thought out. What I particularly like about it is that multiple sources, i.e. sparse depth, stereo supervision and monocular training are all combined elegantly into one framework. Many approaches can only train with monocular supervision which is quite limiting. \n- The experimental evaluation is extensive and confirms the claims made in the paper. A good ablation study is done showing how each component contributes to the performance. The method is compared to a multitude of existing approaches. The proposed method performs very well compared to these. Including some other tasks such as mono-only and stereo prediction in the evaluation is useful as it allows the reader to place the method holistically (not just focussing on depth completion). This is definitely very useful as a comparison when making decisions for a real system. (However, see weakness below as I think this has been overdone a bit).\n**Weakness**\n- While the evaluation is quite extensive, I think the proposed approach is more of a depth completion approach for which there are a number of benchmarks (like the KITTI online depth completion benchmark) and many existing methods. However, the paper focuses extensively on monodepth methods in the comparison. This seems a bit misleading to me as it is obvious that mono approaches will perform worse. Maybe using this space to show the performance in other environments would be better. Also, why has the approach not been evaluated on the online benchmark?\n- The approach has only been evaluated on KITTI where there is lots of repeated and regular structure. How does the method work in a general robotic setup such as would be the case for a home robot in an indoor environment? Or a robot in an outdoor field or sidewalk?\nThe improvement gained by the approach on the depth completion evaluation is quite small i.e. the existing methods already perform quite well.\n- There is no comparison between the capacity of the proposed network and those in the literature. Does the method use significantly more layers or weights?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper itself is well written and I like both the manner in which the sparse lidar is effectively fused with the RGB and the natural manner in which multiple sources of supervision can be used. The main weakness though is that this only results in an incremental improvement on the (many) existing methods for depth completion. I do not see what sets this paper apart.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "I would like the authors to address the following points:\n- How does the method perform in more challenging indoor scenes?\n- I think the runtime results should be moved into the main paper as this is quite important for the reader to know\n"}}, {"review_id": "HuH14LfsI-c", "reviewer": "Reviewer_y8K7", "summary": "This paper proposes an architecture and self-supervised learning approach for dense monocular depth estimation when a very sparse set of accurate depth readings are available from a LiDAR. The proposed architecture is a multi-stag architecture that consists of a network that receives input images and densified sparse LiDAR readings and produces and inital depth map. The initial depth maps is refined using a refinement network that essentially distills the GDC depth correction from PseudoLIDAR++ into an efficient network. Training is done using self-supervision based on photometric cues and the output of the offline depth correction module. Experiments are shown on KITTI, including results on a 3D object detection downstream tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Seemingly good results at very high frame-rates. However, I do have some questions about the results (see below).\n- I particularly like that experiments on a down-stream tasks are shown. The improvement shown there is impressive.\n- Reasonable architecture construction that consolidates the individual stages that have been layed out in Pseudo-LiDAR++ into a single efficient end-to-end pipeline. \n\n\nWeaknesses: \n\n- The comparisons in Table 1 are not quite clear to me. For example the BTS is shown with \\delta_1 of 0.904, whereas the original paper shows results somewhere in the range of 0.95 (depending on the employed backbone) - this would actually outperform the proposed work without any LiDAR points required. Similarly DORN in the original paper also shows values of up to 0.93, but the table here only lists it with 0.897. I'd be grateful if the authors could clarify the discrepancy here.\n\n- This is unfortunately shared by many works in this space: the approach is only evaluated on KITTI. No other datasets are shown. Self-supervised approaches promise, in principle, that they can be deployed anywhere (in this work of course conditioned on the availability of LiDAR measurements). It would be great to go beyond KITTI and actually show off the flexibility of this approach. This could be done by showing results on transfer or fine-tuning on other datasets (for example Apolloscape) rather than squeezing every last drop of performance out of KITTI.\n\nOther: \n\n- Is the pose network really required? It seems that temporally consecutive LiDAR point clouds could be aligned using standard geometric approaches to get the pose. This procedure would be guaranteed to be generalization, unlike a pose network which are known to be hard to generalize (have you tried playing a sequence backward? - many pose networks fail to generalize even to this simple task)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work provides a technically well executed pipeline that seems to show good results, especially on the downstream task, which I believe deserve to be seen. However, I do have questions about the evaluation which I hope the authors can resolve.\n\n-- Post rebuttal --\nThe authors resolved my major questions. What remains is the somewhat evaluation only on KITTI. I do believe however that this shouldn't be a reason for rejection. I thus keep my initial rating.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Minor issues:\n\n- l. 92: Previous text refers to [1] as Pseudo-LiDAR++, but here suddenly as PL++ without an introduction of the abbreviation.\n\n- Table 1: \"excepctionall time-consume\" - broken sentence.\n\n- Table 2 is hard to read because of small fonts in combination with misaligned table rulers."}}]

Author rebuttal:

Meta-review: The paper proposes an approach for self-supervised depth estimation from sparse LiDAR scans. The reviewers find the paper to be generally well written and well structured. The reviewers appreciate the experiments on downstream tasks and the good results. Most of the major concerns raised by the reviewers have been addressed in the rebuttal, except for the benchmarking on the KITTI test set. For this the authors have provided the reasoning on why its unfair to compare to methods on the leaderboard. I agree to an extent but it would still be nice to see how well the method performs, getting a lower metric value on the leaderboard will not diminish the contributions of the paper. Nevertheless, I recommend acceptance and I encourage the authors to still benchmark their approach on the KITTI depth completion or depth prediction test set.
Decision: Accept (Poster)