1

Learning Steering Kernels
for Guided Depth Completion
Lina Liu, Yiyi Liao, Yue Wang, Andreas Geiger and Yong Liu

Abstract—This paper addresses the guided depth completiontask in which the goal is to predict a dense depth map given a
guidance RGB image and sparse depth measurements. Recent
advances on this problem nurture hopes that one day we canacquire accurate and dense depth at a very low cost. A major
challenge of guided depth completion is to effectively make useof extremely sparse measurements, e.g., measurements covering
less than 1% of the image pixels. In this paper, we propose
a fully differentiable model that avoids convolving on sparsetensors by jointly learning depth interpolation and refinement.
More specifically, we propose a differentiable kernel regression
layer that interpolates the sparse depth measurements via learned
kernels. We further refine the interpolated depth map usinga residual depth refinement layer which leads to improved
performance compared to learning absolute depth prediction
using a vanilla network. We provide experimental evidence thatour differentiable kernel regression layer not only enables endto-end training from very sparse measurements using standard
convolutional network architectures, but also leads to better
depth interpolation results compared to existing heuristicallymotivated methods. We demonstrate that our method outperforms many state-of-the-art guided depth completion techniques
on both NYUv2 and KITTI. We further show the generalization
ability of our method with respect to the density and spatial
statistics of the sparse depth measurements.
Index Terms—Depth completion, depth interpolation, kernel
regression, steering kernels

I. I NTRODUCTIONEPTH perception plays an important role for humans,
and is also a valuable cue in computer vision and robotics[4]–[6]. Acquiring dense and accurate depth at a low costis particularly important for applications such as autonomous
driving and virtual reality. While existing depth sensors havecontributed significantly to these applications, the goal of
obtaining accurate and dense depth at low cost is still not
fulfilled. For example, 3D LiDAR provides accurate but sparsemeasurements and existing scanners are typically expensive.
Though structured light sensors are more affordable, they often
have a smaller field of view and are constrained to indoor

D

Lina Liu, Yue Wang and Yong Liu are with the State Key Laboratory ofIndustrial Control Technology and Institute of Cyber-Systems and Control,
Zhejiang University, Zhejiang, 310027, China. Lina Liu and Yong Liu were
supported by the National Natural Science Foundation of China under Grant
61836015.
Yiyi Liao and Andreas Geiger are with the Autonomous Vision Group,
Max Planck Institute for Intelligent Systems and University of Tübingen,
Tübingen, 72076, Germany. Yiyi Liao was supported by the German Federal
Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ:
01IS18039A. Andreas Geiger was supported by the ERC Starting Grant
LEGO-3D (850533).
Corresponding authors: Yiyi Liao; Yong Liu (e-mails: yiyi.liao@tue.mpg.de;yongliu@iipc.zju.edu.cn).

(a) Depth prediction from sparse measurements (e.g., [1])Interpolation(b) Depth prediction from interpolated measurements (e.g., [2], [3])Differentiable
KernelRegression(c) End-to-end depth interpolation and refinement (ours)

Figure 1: Illustration comparing different guided depth completion approaches. The task is to predict a dense depth map
given a guidance RGB image and sparse depth measurements.
Learned components are highlighted in yellow.
environment. More recently, learning-based monocular depth
estimation methods have attracted attention due to their low
cost, as only a monocular RGB image is required to estimate
the depth. However, estimating depth from monocular images
is a highly ill-posed problem.
To acquire dense and accurate depth using affordable sensors, we aim for a learning-based solution of guided depth
completion. More specifically, we predict the full resolution
depth map based on a guidance RGB image and a set of very
sparse depth measurements (e.g., less than 1% of the image
pixels), which can be obtained from a low-cost LiDAR with asmall number of laser beams or visual SLAM [7]. Compared
to predicting depth from a monocular RGB image alone, ourmethod is able to resolve spatial ambiguities using metric cues
from the sparse depth measurements.
However, estimating dense depth maps from sparse measurements is not trivial. Focusing on this problem, there are
two dominant lines of research as illustrated in Fig. 1. In
the first class (Fig. 1a), the sparse depth measurements are
represented as a sparse depth map. The pixels without valid
depth are set to a constant number, e.g., 0 [1]. The resulting
sparse depth map is concatenated with the guidance image
and fed into a neural network to predict the dense depth map.
The advantage of this method is that the model can be trainedin an end-to-end manner. However, in this case the networkneeds to resolve the ambiguity between valid and invalid depth
measurements and vanilla convolutional networks with small

2

convolution kernels are not suited for very sparse inputs as
considered in this work [8]. Another line of works avoids thisproblem by addressing the guided depth completion task in
two stages (Fig. 1b). The sparse depth measurements are firstinterpolated into an intermediate dense depth map, which isthen utilized for depth refinement in a second stage [2], [3].
Though each pixel in the interpolated depth image is valid inthis case, existing techniques for constructing the interpolated
depth map are hand-crafted and thus cannot be trained fromdata.
In this paper, we propose a solution to guided depth completion by combining advantages of intermediate dense depth
representations and end-to-end training as illustrated in Fig. 1c.
Our key idea is to replace the hand-crafted interpolation witha differentiable kernel regression layer. Our learned kernel
regression layer transforms the sparse measurements into a
dense representation which is suitable for depth refinement
using a standard convolutional network. More specifically, we
first estimate the interpolated depth map using kernel regression, where the kernel shapes are learned via a neural network
conditioning on the guidance image. Next, we transform theabsolute depth estimation task into residual depth estimation
by refining the interpolated depth map using a refinement
network. Our contributions are summarized as follows:
We propose a differentiable kernel regression layer whichlearns the kernel shapes conditioned on the guidance
RGB image. We show that our learned kernels reduce
the estimation bias comparing to heuristically designed
kernels.
• We integrate our differentiable kernel regression layer
into guided depth completion that enables end-to-end
depth estimation while avoiding convolving directly on
the sparse measurements. With the dense interpolated
map obtained from our kernel regression layer, we areable to redefine the task of estimating the absolute depth
map as estimating the residual between the real and theinterpolated depth map.
• Our experiments demonstrate that the proposed method
achieves superior performance compared to existing
guided depth completion techniques. We further showthat our model generalizes well wrt. varying number ofpoints and different depth observations patterns.

•

The remainder of the paper is organized as follows: Section II gives a review of related work. We introduce ourmethod in Section III and present the experimental results in
Section IV. Finally, we conclude the paper in Section V.
II. R ELATED W ORK
Monocular Depth Estimation: Many works have considered the task of estimating a depth map from a monocular
image. Conventional methods tackle the problem based on
hand-crafted features and graphical models [9], [10]. Morerecently, supervised learning methods achieve state-of-theart performance on monocular depth estimation [11]–[18].
Unsupervised methods also demonstrate promising results on
this task [19], [20].

Despite recent advances on monocular depth estimation, thetask is inherently ill-posed, e.g., the global scale of the depth
map is ambiguous [11]. In this work, we consider the task ofguided depth completion in which scale ambiguities can be
resolved using a set of sparse measurements.
Depth Upsampling: Depth upsampling methods aim atincreasing the spatial resolution of a given depth map. Whilea few works directly upsample the depth map without any
guidance information [21], [22], several approaches demonstrate that it is beneficial to exploit a color or intensity
image at the target resolution as guidance [23]–[32]. Most
approaches falling into this category either use a guided
bilateral filter [23]–[26] or global energy minimization [27]–[32] for this purpose. More recently, deep learning methods
became popular, demonstrating state-of-the-art performance on
the depth upsampling task [33]–[35].
In contrast to depth upsampling methods that take a lowdimensional depth map defined on a regular pixel grid as input,
we consider the depth completion task in which the depth
observations are incomplete, sparse and irregularly distributed.
Depth Completion: Various methods have been proposed
to fill holes in depth images [4], [36]–[41]. These methods
typically assume that the depth is densely observed for mostimage areas but incomplete at specific regions, e.g., at object
boundaries and in reflective areas where depth sensors struggle
to reconstruct depth. Therefore, they are not directly applicableto sparse input observations which we consider in this paper.
Many works have attempted to estimate dense depth maps
given only very sparse and irregular depth measurements. Nonguided methods [8], [42] provide compelling results given
observations at a moderate sparsity level, e.g., 10% of pixels
carry depth information [8]. However, their performance islimited when only very few depth measurements are available,e.g., less than 1% of the pixels being observed. Incorporatinga guidance image can substantially boost depth completion
performance in both moderate [41]–[45] and extremely sparsesettings [1]–[3], [42], [46]–[48]. In this paper, we focus on thesecond, more challenging setting.
In this setting, a major challenge is to effectively incorporate
the extremely sparse measurements into a deep neural network.
A straightforward option is to represent the measurements as asparse depth map where the depth value in unknown regions
is set to 0, and then feed it into the deep networks alongwith the RGB image [1], [46], [47] or an inferred normal map[44]. However, as pointed out in [8], standard convolutional
neural networks degenerate when applied directly on sparse
inputs given a limited kernel size. Though this problem canbe resolved by specifically designing the network architecture
for sparse inputs [8], [42], [49], we seek an alternative bydifferentiably transforming the sparse input into a dense representation that can be fed into standard convolutional neuralnetworks. Following this idea, our earlier work [2] extrapolatesa horizontal 2D laser scanline in gravity direction and takes
this extrapolated depth map as input to a deep neural network.
Similarly, Chen et al. [3] and Shivakumar et al. [48] interpolate
the sparse depth map into a dense depth map as input to the
convolutional neural network. All these methods pre-process

3

the sparse depth measurements heuristically and are not trained
end-to-end.
Instead of constructing a dense depth map using handcrafted interpolation techniques, in this paper, we propose a
differentiable kernel regression layer which allows us to exploit the image distribution for depth interpolation. Moreover,
our model learns to predict the optimal kernel parametersconditioned on the input image end-to-end from data.
Kernel Regression: Kernel regression is a well-establishedregression method [50], [51]. Conventionally, the same kernel
is applied over the entire image, i.e., the kernel is data
and location independent. Takeda et al. [52] propose a dataadaptive kernel regression method that steers the kernel basedon an extra input, e.g., an RGB image. While in [52] thekernel shapes are heuristically designed, we learn the imageconditional kernel shape end-to-end from data. We experimentally demonstrate that our learned kernels are able to reduce
the estimation bias compared to heuristically designed ones.
A few metric learning methods also learn kernel shapes
from data for kernel regression [53], [54]. Weinberger &Tesauro [53] optimize the kernel shape by minimizing a
regression loss. Noh et al. [54] derive an analytical solution
for estimating kernel shapes from Gaussian-distributed data.
Note that none of these approaches uses a deep neural network
as encoder for estimating the kernel shapes. In contrast, we
exploit a deep neural network to directly infer the kernel shapeat each pixel from our guidance image, effectively amortizing
the inference process [55].
III. M ETHOD
Our goal is to estimate a dense depth map given sparse
depth measurements and a guidance RGB image as input.
Existing works for guided depth completion either directly
infer a dense depth map from sparse measurements or basetheir predictions on intermediate dense depth maps that are
obtained using hand-crafted interpolation techniques (e.g.,nearest neighbor). In this work, we instead propose to split
the guided depth completion problem into two differentiable
end-to-end trainable stages as illustrated in Fig. 2: We firstpredict a dense interpolated depth map D̃ from sparse depth
measurements D using kernel regression where the kernel parameters Φ = {Γ, Θ, Σ} are estimated based on the guidance
image I. Here, D = {(x1 , d1 ), · · · , (xP , dP )} denotes the setof sparse depth measurements where xi ∈ R2 is the pixel
location and di ∈ R+ the depth value. In a second step, we
refine D̃ into the final depth map D̂ using a deep residual
network. Our method avoids applying convolutions directly
on extremely sparse inputs while being end-to-end trainable.
In this section, we first give a formal introduction to kernel
regression and demonstrate that existing works for handcrafted depth interpolation [2], [3] are special cases of thiskernel regression framework. Next, we present our differentiable kernel regression module which uses a neural network
for predicting the kernel parameters from the guidance image
instead of heuristically designing the kernels. Finally, we
introduce our residual estimation network, as well as the loss
functions and training details.

A. Kernel Regression
Kernel regression is a non-parametric regression techniquewhich uses a kernel as weighting function. In the context of
depth completion, we aim at developing a regression modelD̃(x) that estimates a depth value at an arbitrary pixel x ∈ R2
from a set of sparse measurements D. Following [52], thekernel regression model at a given pixel x can be formulatedasP
KH (xi − x)di(1)
D̃(x) = Pi=1...P
i=1...P KH (xi − x)which represents the predicted depth at x as a linear combination of known measurements di weighted by the kernel
function KH (xi − x). The kernel shape is crucial for the
performance of the regression model. In practice, they areoften heuristically designed and can be categorized into dataindependent and data-dependent kernels.
Data-independent Kernel: A data-independent kernel determines the weight between xi and x by the relative distancexi − x independent of a guidance image I. In this case, thekernel shape at any pixel is constant across the image domain.
A common kernel choice is the Gaussian kernel:
1
KH (xi − x) = p
2π det (H2 ) (2)(xi − x)T H−2 (xi − x)exp −2Here, H ∈ R2×2 is a constant smoothing matrix which does
not depend on the guidance image I.
We remark that some of existing hand-crafted intermediate
depth maps [2], [3] can be view as interpolated using kernel
regression with data-independent kernels. In our previous
work [2], we expand a single horizontal laser scanline in vertical direction to generate a dense depth map. We can achieve asimilar effect using kernel regression by constructing a Gaussian kernel that is extremely elongated in vertical direction.
In [3], an intermediate dense depth map is constructed using
nearest neighbor interpolation. This operation is equivalent toapplying a hard assignment in Eq. (1), which considers only
the depth value di with the largest KH (xi − x) for estimatingD̃(x). The interpolated depth map of both techniques [2], [3]is constructed without considering a guidance image I.
Data-dependent Kernel: Takeda et al. [52] propose a nonconstant steering kernel which depends on a guidance image I.
More specifically, they use the image gradient ∇I to adaptively“steer” the kernel, resulting in elongated kernels spread along
the image edges
KHi (xi − x) =1pdet (H2i )(xi − x)T H−2i (xi − x)
exp −
2

2π

(3)

where Hi denotes the smoothing matrix which depends on thelocal content of guidance image I at pixel xi .
While Hi is constructed adaptively with respect to image
features in [52], it is still heuristically designed and may not be4Guidance Image

Kernel Parameter Network+

C

K
Interpolated Depth

U-Net

Residual Depth

Final Depth

Global Skip Connection

Sparse Depth

K

C

Kernel Regression

Differentiable Kernel Regression (KernelNet)

Concatenation+ Element-wise Sum

Residual Network (ResNet)

Figure 2: Method Overview. Our differentiable kernel regression module (orange) predicts a dense interpolated depth map D̃
from the guidance image I and the sparse measurements D (visually enhanced). The kernel parameters Φ = {Γ, Θ, Σ} arelearned from I, and the interpolated depth map D̃ is estimated using kernel regression given Φ and D. Given the interpolated
depth map D̃, our residual network (blue) predicts the final depth map D̂ conditioned on the guidance image I. Instead of
directly predicting D̂ using a deep neural network, we learn the residual depth D̂ − D̃ by adding a global skip connection.
Our model avoids directly applying convolutions on the sparse measurements D while being end-to-end trainable.

optimal to reduce the estimation bias between the interpolated
depth map D̃ and the ground truth D.
B. Differentiable Kernel Regression
To overcome the limitation of heuristic steering kernels, we
propose a differentiable kernel regression layer that allows
for learning the smoothing matrix Hi fully driven by theraw data. In contrast to the hand-crafted depth interpolation
based on data-independent kernels, our learned steering kernels allow for adaptively integrating additional informationof the guidance image I. More importantly, we integrate thedifferentiable kernel regression layer into the guided depth
completion framework, allowing for jointly optimizing depth
interpolation and depth refinement end-to-end.
Specifically, we learn the kernel shapes from I to directly
minimize the estimation bias between the interpolated depth
map D̃ and the ground truth D. Note that our kernel shapes
are learned without direct supervision, guided only by theestimation bias, i.e., we do not require ground truth for the
kernel shapes. More formally, we follow the parametrizationof [52] and represent Hi as−1

Hi = hµi Ci 2

(4)where h is the global smoothing parameter and µi is a scalarthat captures the local density of the depth samples. Ci isa covariance matrix that is specified using three parameters(i.e., γi , θi and σi ) which control the scaling, rotation andelongation of Ci respectively:
Ci = γi Uθi Λi UT
θicos θisin θiUθi =− sin θi cos θiσ0Λi = i0 σi−1(5)

Intuitively, a circular kernel with isometric weighting is firstly
elongated by the elongation matrix Λi whose axes lengths are

scaled by σi . Next, the elongated kernel is rotated by the 2D
rotation matrix Uθi . Finally, the kernel is scaled by γi .
In our differentiable kernel regression layer, we propose
to predict the kernel parameters Φ = {Γ, Θ, Σ} from theguidance image I using a kernel parameter network, where
γi = Γ(xi ), θi = Θ(xi ) and σi = Σ(xi ). This is illustrated
in the orange block of Fig. 2. In contrast to [52], we fix h andµi in our formulation as these parameters are redundant wrt.
γi and thus mainly affect the initialization. We set h = 5.0
and µi = 1.0 empirically where a relatively large h is used tocompensate for the sparse observations. Note that each parameter map (Γ, Θ, Σ) has the size of the guidance image I anda data-dependent covariance matrix Ci is specified at every
pixel location xi , determining the steering kernel KHi (xi −x)
in Eq. (3). Given the kernels centered at all sparse depth
measurements D, we then estimate the dense interpolated
depth map D̃ using kernel regression in Eq. (1). The kernelparameter network and the kernel regression operation are bothdifferentiable and thus allow for end-to-end training.
We use a shallow version of [56] as our backbone for learning the kernel parameters. Note that we only use the guidance
image I as input to the kernel parameter network, thus avoiding
convolutions directly on the sparse depth measurements D.
Moreover, our kernel parameter network is independent of thesparse depth measurements. As evidenced by our experiments,
this leads to high generalization performance wrt. the numberof depth measurements as well as the (sparse) observation
pattern. While only pixels with valid depth measurements
provide gradients to update the kernel parameter network, ourapproach is able to learn smooth kernel parameter maps as
illustrated in Fig. 2 and Fig. 6.
We will refer to the entire kernel regression network (illustrated in orange in Fig. 2) as KernelNet in the following.
C. Residual Network
As illustrated in the blue block in Fig. 2, we refine theinterpolated depth to obtain the final depth prediction. The

5dense interpolated depth map allows us to use standard convolutional networks to estimate the final depth map. We use astandard U-Net [56] which aggregates local and global feature
information as our backbone.
As the output of the KernelNet is a dense depth map, therefinement step can be effectively implemented by estimating
the residual depth D̂ − D̃ between the final depth map D̂ andthe interpolated depth map D̃. The residual depth estimation is
implemented by introducing a global skip connection as illustrated in Fig. 2. Note that learning the residual mapping [57]via the global skip connection ensures a performance lower
bound on the final depth, i.e., the final depth should perform
equal or better than the interpolated depth. Another intuition of
the residual formulation is normalization: while the absolutedepth may differ in range across images, the residual depth
is expected to follow a zero-mean distribution which helps
during training [58].
We will refer to the entire residual network (illustrated inblue in Fig. 2) as ResNet in the following.

ground truth probability distribution. A reasonable assumption
is a Dirac delta distribution based on which bin the groundtruth depth falls into after discretization. Thus, a standard
cross-entropy loss can be applied
Ldirac (x) =MX

δ([D(x)] − k) log(pk )

(9)

k=1

where [D(x)] returns the index of the bin D(x) belongs to.
Note that existing works point out that the cross-entropy loss
defined on discrete depth values outperforms naïve depth value
regression [16].
Loss on Final Depth Mean: With only the assumption of
the Dirac delta distribution, the estimated accuracy can belimited by the discretization. Therefore we also minimize the`1 distance between the expectation value in Eq. (6) and theground truth depth:
Lmean (x) = |D(x) − D̂(x)|(10)

In our experiments, we demonstrate that this additional constraint on the expected depth values boosts prediction accuracy.

D. Loss Function
We define our loss function jointly on the interpolated depth
map D̃ and the final depth map D̂. For the interpolated
depth map, the depth at each pixel is estimated using kernel
regression as defined in Eq. (1). For the final depth map,
we adopt a probabilistic estimation method by representing
the depth value as the expectation over a predicted depth
distribution at each pixel. Specifically, we estimate a discreterandom variable at every pixel location x, represented as a setof possible values d¯1 , d¯2 , . . . , d¯M occurring with probabilities
P (x) = {p1 (x), p2 (x), . . . , pM (x)} at pixel x, where M is
the number of the discrete bins. The predicted depth at pixel
location x is calculated as the expected depth value wrt. P (x):D̂(x) =M
X

Loss on Final Depth Gradient: To eliminate depth distortionand blurry predictions at object boundaries, we follow [59] andapply a constraint to the depth gradient. This loss penalizes
the disagreement of depth edges between D and D̂, in bothvertical and horizontal direction
Lgrad (x) = F (|∇(D − D̂)(x)|)(11)

where ∇(D − D̂)(x) is a 2D vector comprising the horizontal
and vertical depth gradients at pixel location x. F (ξ) is arobust activation function formulated asX
F (ξ) =
log(ξi + 1)
(12)id¯k pk (x)

(6)

k=1

In our implementation, d¯1 , d¯2 , . . . , d¯M are fixed and obtainedby uniformly sampling on a provided range interval, while
P (x) is estimated using a softmax layer.
Given the data representation, we formulate our loss fortraining the full model as follows
XL=
w1 Linter + w2 Ldirac + w3 Lmean + w4 Lgrad (7)xwith constant hyperparameters w1 , . . . , w4 . The sum notation
denotes the aggregation of the loss over all pixels. Note that
we have dropped the dependency of the loss functions on the
pixel location x for clarity. We now explain each term in detail.
Loss on Interpolated Depth: We minimize the `1 distance
between the interpolated depth and the ground truth depth:
Linter (x) = |D(x) − D̃(x)|(8)

where ξ is a vector with each entry denoted by ξi .
E. Training
We train the network in three steps. At the first step, wetrain the differentiable kernel regression module using only
Linter , with a learning rate of 0.01 for 10 epochs, resulting ina first approximation of the interpolated depth. Next, we train
the residual depth network to further refine the interpolated
depth using Ldirac , Lmean and Lgrad . Finally, we train thefull model in an end-to-end manner with all loss functions.
IV. E XPERIMENTS
In this section, we first introduce the experimental setup
and compare with state-of-the-art methods for guided depth
completion. Next, we analyze the benefits of our differentiable
kernel regression network and conduct ablation studies wrt.
our architecture design and loss functions. Finally, we explore
the generalization ability of our method in terms of the numberof points and different depth observation patterns.

This loss enables learning the kernel shape directly from data
to reduce the estimation bias.

A. Experimental Setup

Loss on Final Depth Distribution: For the discrete randomvariable at every pixel location x, we do not have access to theDatasets: We evaluate the proposed method on NYUv2 [4]and KITTI [60]. For the indoor dataset NYUv2, we follow

6

DatasetMethodSparse-to-Dense [1]Sparse-to-Dense (SS) [47]
CSPN [46]
NYUv2 Nconv-CNN [49]
DeepLiDAR [44]
D3 -Random [3]Ours
Sparse-to-Dense [1]Sparse-to-Dense (SS) [47]
CSPN [46]KITTI Nconv-CNN [49]
DeepLiDAR [44]
D3 -Random [3]Ours

Error (↓)
rms
rel
0.230
0.044
0.132
0.027
0.117
0.016
0.125
0.017
0.115
0.022
0.157
0.025
0.111
0.015
3.378
0.073
2.245
0.050
2.977
0.044
2.739
0.063
2.126
0.044
3.124
0.060
2.708
0.037

δ1.02
52.3
64.3
83.2
82.2
69.3
84.8
30.0
52.4
70.2
50.4
54.1
46.4
76.4

δ1.05
82.3
87.4
93.4
92.9
88.6
94.1
65.8
81.5
85.7
74.2
82.3
73.1
88.0

Accuracy (↑)
δ1.10
δ1.25
92.6
97.1
95.0
99.0
97.1
99.2
96.7
99.1
99.3
95.3
99.0
97.4
99.3
85.2
93.5
91.2
96.1
91.4
95.7
86.4
94.8
92.0
96.2
85.1
94.1
92.7
96.3

δ1.252
99.4
99.9
99.9
99.8
99.9
99.8
99.9
97.6
98.6
98.0
98.2
98.7
98.2
98.3

δ1.253
99.8
100.0
100.0
100.0
100.0
99.9
100.0
98.9
99.3
99.1
99.3
99.4
99.3
99.2

Table I: Guided Depth Completion on NYUv2 and KITTI. All methods take 500 sampled points as depth measurements.

the official split with 249 scenes for training and 215 scenes
for testing. In particular, the test dataset with 654 images isused for evaluation, and 46k images are sampled from the
raw training data for training, where missing depth values areinpainted with a cross-bilateral filter provided in the official
toolbox. The original image of resolution 640 × 480 pixels
is downsampled and cropped to 304 × 228 pixels as input
following [1], [3], [46].
The outdoor dataset KITTI includes RGB images and depth
images collected by Velodyne HDL64. Following [1], we use
46k images from the training sequences for training, and arandom subset of 3200 images from the test sequences forevaluation. For fair comparison, we also use the bottom crop
(912 × 228) and evaluate on valid pixels as in [1].1
Sparse Measurements: Following [1], [3], [46], we use
500 sparse random depth samples per image (less than 1%of all pixels) on both NYUv2 and KITTI. Note that thissetting is more challenging than the setting of the KITTI
depth completion benchmark [8] which uses an input densityof roughly 10%. We further validate the generalization abilityof our method wrt. different sparse input modalities in theablation study. To this end, we simulate laser scanners withdifferent number of beams on KITTI, by sampling 1/4/8 beams
from the original 64 beams.
Evaluation Metrics: We adopt the standard evaluation metrics. Let di = D(xi ) denote the ground truth depth at a pixel
location xi and dˆi = D̂(xi ) denote the estimated depth. The
evaluation metrics are specified as follows:
q P
1
2
ˆ
• Root Mean Squared Error (rms):i (di − di )NPˆ
|di −di |1• Mean Absolute Relative Error (rel): N
i
di
ˆ
d d
• Threshold δ: percentage of dˆi , s.t. max( i , i ) < δ, δ ∈didˆi{1.02, 1.05, 1.10, 1.25, 1.252 , 1.253 }, see [46].
Here, N denotes the total number of pixels.
1 Note that L
grad is not applicable to KITTI due to the discontinuity of
the ground truth depth D, therefore we only apply Lgrad on NYUv2.

Implementation Details: We adopt the original U-Net [56]as the backbone of our ResNet. Specifically, it contains 4
downsampling steps and 4 upsampling steps, each with a
stride of 2. For KernelNet, we use a shallow version of UNet by reducing both downsampling and upsampling steps to
1. We set the number of the discrete bins M = 401 for all
experiments.
B. Comparison to the State-of-the-Art
Baselines: We compare to the following state-of-the-art
methods on the guided depth completion task.
• Sparse-to-Dense [1] takes sparse measurements as input
where non-observed regions are set to 0. The sparse depth
map and the guidance image are concatenated to estimate
the dense depth map end-to-end.
• Sparse-to-Dense (SS) [47] extends [1] by exploiting selfsupervision (SS) via a photometric loss between neighboring frames.
• CSPN [46] stands for Convolutional Spatial Propagation
Network and improves Sparse-to-Dense [1] by refining
the depth estimation using a recurrent spatial propagation
model.
• Nconv-CNN [49] proposes a normalized convolutional
layer which takes as input sparse measurements anda confidence map for unguided depth completion. The
output of the unguided network is further concatenatedwith the RGB image for guided depth completion.
• DeepLiDAR [44] also takes sparse measurements as
input. It estimates surface normals as the intermediate
representation to produce dense depth and thus additionally requires ground truth surface normals as supervision. While the original implementation uses a synthetic
dataset with ground truth normals for pre-training, weomit this step for fair comparison. We compute the surface normal ground truth from the depth map following[44] where a plane fitting algorithm is first applied on
KITTI to obtain a dense depth map [4].

7•

D3 -Random is an adapted version of Deep Depth Densification (D3 ) [3] which constructs an intermediate dense
depth map from the sparse measurements using nearest
neighbor interpolation as input to a depth completion
network. In the original implementation, more than 500
points are sampled on a regular grid as sparse measurements. This sampling approach can provide richer information due to the even distribution and the larger amountof measurements. For a fair comparison, we retrain andre-evaluate D3 using the same sparse measurement pattern as ours, which we denote as D3 -Random.

We follow the official implementations of all baselines except
for D3 -Random as no official implementation is released2 .
NYUv2 dataset: Table I (top) compares the performance ofdifferent methods on NYUv2. As can be seen, our method
achieves superior performance quantitatively. Fig. 3 shows aqualitative comparison between D3 -Random, our interpolated
depth map and our final depth map. Note that our differentiable
kernel regression layer provides an interpolated depth map that
captures the coarse structure of the scene while the residual
network is able to further refine details.
KITTI dataset: Table I (bottom) and Fig. 4 show our comparison on KITTI. Our method achieves comparable performancewith DeepLiDAR which requires additional supervision andoutperforms the other baselines. As illustrated in Fig. 4,
compared to the baseline, our method provides more reliable
depth estimates for reflective surfaces and thin structures
where fewer and more noisy depth inputs are available.

C. Analysis of Kernel Regression Network
Learned Steering Kernels: We validate the advantages of
our learned steering kernels compared to hand-crafted kernels
in Table II. As we focus on analyzing the advantage of thelearned kernel shapes, for this experiment we directly report
the performance on the interpolated depth map rather than the
final depth map. Specifically, we compare to two baselines fordetermining the kernel parameters. The first baseline adopts
data-independent kernels, i.e., the same scaling, rotation andelongation parameters are applied at all pixel locations. We
perform grid search to determine these parameters. For thesecond baseline, we consider the original steering kernel
regression method which heuristically constructs the steering
kernels based on image gradients [52].
Our results suggest that the steering kernel [52] outperforms
the data-independent kernel, while our learned steering kernel
further improves performance compared to both heuristicallydesigned kernels, demonstrating that our estimated kernelparameters conditioned on the guidance image are able toeffectively reduce the estimation bias.
2Sparse-to-Dense: https://github.com/fangchangma/sparse-to-dense.pytorch,
CSPN: https://github.com/XinJCheng/CSPN/tree/master/cspn_pytorch, NonvCNN: https://github.com/abdo-eldesokey/nconv-nyu and https://github.
com/abdo-eldesokey/nconv, DeepLiDAR: https://github.com/JiaxiongQ/
DeepLiDAR, D3 -Random: https://github.com/Shiaoming/DensefromRGBS(third-party implementation).

Learning Process: We further investigate the learning processof the kernel shapes qualitatively. Fig. 5 visualizes three Gaussian kernels over multiple training iterations, determined by
the learned kernel parameters (γi , θi , σi ) at corresponding pixel
location xi . We also show the heuristically designed kernel
shapes of [52] for comparison. We observe that the elongatedaxes converge to align with the edges of the guidance image.
In contrast, the heuristic kernel shapes [52] are less alignedwith the image edges, suggesting that it is hard to capture thelocal image properties by hand.
Qualitative Analysis: We also visualize the learned kernel
maps in Fig. 6. We observe that the kernel shape is highly
correlated with local statistics of the guidance image. For
example, the scale parameter γi is larger at smooth regions(lighter color) while smaller around edges (darker color). The
rotation parameter θi clearly distinguishes vertical vs. horizontal edges. The elongation parameters σi is larger around theedges and smaller in smooth regions.
D. Ablation Study
Architecture and Loss Function: To discover the contribution of each part of the model, we conduct ablation studies on
NYUv2 in Table III. Specifically, we first perform monoculardepth estimation, taking only the guidance image as input(“RGB”) and predicting the depth map using our residual
network without the global skip connection (“U-Net”). Next,
we add the sparse depth map as an additional channel tothe input (“RGB+Sparse”) similar to [1], [43] in which the
pixels with unknown depth are set to 0. We further add ourdifferentiable kernel regression module and directly estimate
the final depth without the global skip connection (“KernelNet+ U-Net”). Finally, we use our full model by adding the global
skip connection (“KernelNet + ResNet”).
We observe that all variants of “RGB+Sparse” outperform“RGB”, taking advantage of the sparse measurements. More
importantly, we observe noticeable performance gains byreplacing the sparse depth map with our interpolated depth
map from the kernel regression network (KernelNet), demonstrating that standard convolutional networks suffer in thepresence of extremely sparse inputs. The comparison between“KernelNet + U-Net” and “KernelNet + ResNet” demonstrates
the advantage of the residual formulation with the global skip
connection.
In our ablation study on the loss functions, we compare
the performance of Ldirac with a combination of Ldirac and
Lmean . We find that adding Lmean improves the performancein all model variants. The gradient loss Lgrad further improves
the performance on all metrics.
Generalization wrt. Number of Points: We investigate
the generalization ability of our model wrt. the number of
depth input points in Table IV (top). Specifically, we train the
network taking 500 points as sparse measurements and change
the number of observed points at inference time. Table IVshows that the performance is correlated with the number ofobserved points, indicating that the network generalizes well
wrt. different sparsity levels. Note that our method is able toGround Truth

Final Depth D̂

Interpolated Depth D̃

D3 -Random [3]Image I

8

Figure 3: Qualitative Comparison on NYUv2. From top-to-bottom: Guidance image I, depth completion results of D3 Random [3], our interpolated depth map D̃, our final depth map D̂ and the ground truth depth map.
Kernel Regression
Data-independent Kernel
Steering Kernel [52]Learned Steering Kernel

Error (↓)
rms
rel
0.230
0.045
0.221
0.043
0.198
0.034

δ1.02
54.6
55.7
65.5

δ1.05
75.6
77.2
82.9

Accuracy (↑)
δ1.10
δ1.25
87.6
96.7
88.4
96.9
91.6
97.7

δ1.252
99.5
99.5
99.8

δ1.253
99.8
99.8
99.9

Table II: Kernel Parameters. Comparison of the interpolated depth D̃ wrt. different kernel parameters.

achieve better performance given more sparse measurements
while it is trained on 500 measurements. This is particularly
valuable in applications where the sparse measurements are
obtained from visual SLAM, in which case the number ofpoints might differ at every frame. We further compare to
CSPN [46] and NConv-CNN [49] given different number ofsparse measurements at inference time. Fig. 7 shows that our
method consistently outperforms the baselines.
Generalization wrt. Observation Pattern: We further validate the generalization ability of our method wrt. various

sparse measurement patterns such as laser scans with different number of beams. Specifically, we evaluate our guided
depth completion performance given 1/4/8 beams of laserscans and 500 random sparse measurements on KITTI in
Table IV (bottom). We retrain the network for each type ofmeasurements in this case due to the large variation of thesedifferent patterns. We first compare with [2] that uses 1 beamof laser range data as a baseline. Note that given the samesparse observation, our method outperforms [2] exploiting thelearned interpolation kernels. Furthermore, depth completion
performance can be improved by increasing the beams of the

GTFinal D̂Inter. D̃

D3 -R. [3]Image I

9

Figure 4: Qualitative Comparison on KITTI. From top-to-bottom: Guidance image I, depth completion results of D3 Random [3], our interpolated depth map D̃, our final depth map D̂ and the ground truth depth map (visually enhanced). White
circles highlight reflective surfaces and thin structures where our method provides more reliable depth estimates compared tothe baseline D3 -Random [3].

Image

t=0

t=1

t=2

t=3

t=4

t=5

[52]

Figure 5: Kernel Shape Evolution across Learning Iterations. t denotes the number of training iterations. The red dots inthe left column denote the locations of the visualized kernels. All parameters of the kernel parameter network are randomly
initialized, the kernel shapes are almost identical before training and change over iterations. We also show the heuristic kernels
proposed by Takeda et al. [52] in the right most column for comparison.

laser range scanner, validating the generalization ability of
our method wrt. different sensory information. Moreover, 500
random samples yield higher precision than 500 points locatedon a 1-beam laser scanline. This suggests that the observed
points from a single beam of laser range data are correlated
and thus not as informative as uniformly sampled points.

our full model, including both KernelNet and ResNet, requires
0.114 seconds for inference. Despite having less parameters,
KernelNet is more time-consuming due to the differentiable
kernel regression operation (w/o the kernel parameter network)
which takes 0.048 seconds for one forward pass.
V. C ONCLUSION

E. Computational Cost Analysis
Lastly, we evaluate the average inference time and thenumber of parameters of our proposed method in Table V. The
inference time is evaluated on a single NVIDIA 1080 Ti GPU
at an input resolution of 304 × 228 pixels. Table V shows that

This paper proposes a novel guided depth completion
method. By integrating differentiable kernel regression into theguided depth completion formulation, our method avoids theapplication of convolutions on extremely sparse depth maps
while still being end-to-end trainable. We conduct experiments

10

Image

Scaling Γ

Rotation Θ

Elongation Σ

Figure 6: Learned Kernel Parameters on NYUv2. Brighter colors denote larger values. Each kernel is represented with 3
parameters, including scaling Γ, rotation Θ and elongation Σ. The scale parameters are larger in smooth regions and smalleraround edges. The rotation parameters clearly distinguish vertical vs. horizontal edges. The elongation parameters are larger
around the edges and smaller in smooth regions.
Input

Loss

Network Architecture

RGB
RGB
RGB
RGB+Sparse
RGB+Sparse
RGB+Sparse
RGB+Sparse
RGB+Sparse
RGB+Sparse
RGB+Sparse
RGB+Sparse
RGB+Sparse

LdiracLdirac +Lmean
Ldirac +Lmean +Lgrad
LdiracLdirac +Lmean
Ldirac +Lmean +Lgrad
LdiracLdirac +Lmean
Ldirac +Lmean +Lgrad
LdiracLdirac +Lmean
Ldirac +Lmean +Lgrad

U-Net
U-Net
U-Net
U-Net
U-Net
U-Net
KernelNet + U-Net
KernelNet + U-Net
KernelNet + U-Net
KernelNet + ResNet
KernelNet + ResNet
KernelNet + ResNet

Error (↓)
rms
rel
0.804 0.220
0.819 0.223
0.791 0.219
0.281 0.039
0.198 0.027
0.189 0.024
0.176 0.034
0.147 0.024
0.134 0.022
0.122 0.016
0.115 0.015
0.111 0.015

δ1.02
7.29
7.31
7.51
57.6
69.9
71.6
58.5
71.5
72.7
84.1
84.6
84.8

δ1.05
17.6
17.6
18.2
81.0
87.9
90.4
81.6
90.0
90.5
93.4
94.0
94.1

Accuracy (↑)δ1.10 δ1.25
32.7 63.0
33.0 64.2
33.7 64.3
92.2 97.6
94.4 98.1
96.1 98.9
92.8 98.3
95.4 98.8
96.2 99.0
96.9 99.2
97.3 99.3
97.4 99.3

δ1.252 δ1.253
87.4 95.9
87.0 95.3
87.7 95.8
99.5 99.9
99.6 99.9
99.6 99.8
99.7 99.9
99.8 99.9
99.8 100.0
99.8 100.0
99.9 100.0
99.9 100.0

Table III: Ablation Study on NYUv2. We compare the results of our method with respect to different input modalities, loss
functions and network architectures. For the latter, we use the RGB image (with and without the sparse depth input) directlyfor our residual network without global skip connection (“U-Net”) as well as our full model (Fig. 2) without (“KernelNet +U-Net”) and with (“KernelNet + ResNet”) global skip connection.

on both indoor and outdoor datasets including NYUv2 and
KITTI. We experimentally show that our method is able toachieve superior performance compared to methods which
directly learn from sparse depth maps, as well as methods
that use hand-crafted interpolated depth maps as input. Our
ablation study demonstrates the advantages of our learned
steering kernel and analyzes the effectiveness of our architecture design. We also show the generalization ability of our
method with respect to various sparse measurement patterns,
including randomly sampled sparse measurements and sparse
LiDAR measurements. In future work, we plan to investigate
if the differentiable kernel regression module can be applied
to other image processing tasks such as image denoising.

R EFERENCES[1] F. Ma and S. Karaman, “Sparse-to-dense: Depth prediction from sparse
depth samples and a single image,” in Proc. IEEE International Conf.
on Robotics and Automation (ICRA), 2018, pp. 1–8.
[2] Y. Liao, L. Huang, Y. Wang, S. Kodagoda, Y. Yu, and Y. Liu, “Parse
geometry from a line: Monocular depth estimation with partial laser
observation,” in Proc. IEEE International Conf. on Robotics and Automation (ICRA), 2017, pp. 5059–5066.
[3] Z. Chen, V. Badrinarayanan, G. Drozdov, and A. Rabinovich, “Estimating depth from RGB and sparse sensing,” in Proc. of the European Conf.
on Computer Vision (ECCV), 2018, pp. 176–192.
[4] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentationand support inference from RGBD images,” in Proc. of the European
Conf. on Computer Vision (ECCV), 2012, pp. 746–760.
[5] Y. Liao, S. Kodagoda, Y. Wang, L. Shi, and Y. Liu, “Understand scene
categories by objects: A semantic regularized scene classifier using

11

Dataset

NYUv2

KITTI
KITTI

Input
300 points
400 points
500 points
600 points
700 points
1000 points
1500 points
2000 points
1-beam [2]1-beam4-beam8-beam
500 points

Error (↓)
rms
rel
0.149
0.020
0.123
0.016
0.111
0.015
0.106
0.014
0.101
0.013
0.092
0.012
0.083
0.010
0.078
0.010
4.500
0.113
3.686
0.094
3.593
0.085
3.105
0.048
2.708
0.037

δ1.02
79.2
83.0
84.8
85.9
86.8
88.6
90.1
91.0
–
29.9
33.8
68.1
76.4

δ1.05
91.2
93.3
94.1
94.6
95.0
95.7
96.4
96.8
–
52.0
61.6
84.0
88.0

Accuracy (↑)
δ1.10
δ1.25
95.9
98.8
97.0
99.2
97.4
99.3
97.6
99.4
97.8
99.5
98.1
99.6
98.4
99.6
98.6
99.7
–
87.4
72.7
90.1
78.5
91.1
90.4
95.2
92.7
96.3

δ1.252
99.7
99.8
99.9
99.9
99.9
99.9
99.9
100.0
96.0
96.3
96.9
97.9
98.3

δ1.253
99.9
100.0
100.0
100.0
100.0
100.0
100.0
100.0
98.4
98.3
98.6
99.0
99.2

Table IV: Generalization. Evaluation of our model in terms of generalization with respect to the number of points (top) anddifferent depth observation patterns (bottom). “1-beam [2]” denotes a baseline without learned kernel.

Figure 7: Generalization wrt. Number of Points on NYUv2.
All methods are trained on 500 sparse measurements andevaluated on different number of sparse measurements. Top
row: lower is better; bottom row: higher is better.

Average Inference Time (s)Number of Parameters (M)KernelNet
0.085
3.065

ResNet0.029
13.422

Table V: Computational Cost Analysis. Inference time evaluated on a single NVIDIA 1080 Ti GPU.

convolutional neural networks,” in Proc. IEEE International Conf. on
Robotics and Automation (ICRA), 2016, pp. 2318–2325.
[6] Y. Wang, S. Huang, R. Xiong, and J. Wu, “A framework for multisession RGBD SLAM in low dynamic workspace environment,” CAAITrans. on Intelligence Technology, vol. 1, no. 1, pp. 90–103, 2016.
[7] R. Mur-Artal and J. D. Tardós, “ORB-SLAM2: an open-source SLAM
system for monocular, stereo, and RGB-D cameras,” IEEE Trans. on
Robotics, vol. 33, no. 5, pp. 1255–1262, 2017.
[8] J. Uhrig, N. Schneider, L. Schneider, U. Franke, T. Brox, and A. Geiger,“Sparsity Invariant CNNs,” in Proc. of the International Conf. on 3D
Vision (3DV), 2017.
[9] A. Saxena, S. H. Chung, and A. Y. Ng, “Learning depth from single
monocular images,” in Advances in Neural Information Processing
Systems (NIPS), 2005, pp. 1161–1168.
[10] A. Saxena, M. Sun, and A. Y. Ng, “Make3d: Learning 3d scene structurefrom a single still image,” IEEE Trans. on Pattern Analysis and MachineIntelligence (PAMI), vol. 31, no. 5, pp. 824–840, 2009.
[11] D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from asingle image using a multi-scale deep network,” in Advances in Neural
Information Processing Systems (NIPS), 2014.
[12] D. Eigen and R. Fergus, “Predicting depth, surface normals and semanticlabels with a common multi-scale convolutional architecture,” in Proc.
of the IEEE International Conf. on Computer Vision (ICCV), 2015, pp.
2650–2658.
[13] F. Liu, C. Shen, G. Lin, and I. D. Reid, “Learning depth from single
monocular images using deep convolutional neural fields,” IEEE Trans.
on Pattern Analysis and Machine Intelligence (PAMI), vol. 38, no. 10,
pp. 2024–2039, 2016.
[14] M. Mancini, G. Costante, P. Valigi, and T. A. Ciarfuglia, “Fast robust
monocular depth estimation for obstacle detection with fully convolutional networks,” in Proc. IEEE International Conf. on Intelligent Robots
and Systems (IROS), 2016, pp. 4296–4303.
[15] I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab,“Deeper depth prediction with fully convolutional residual networks,”
in Proc. of the International Conf. on 3D Vision (3DV), 2016, pp. 239–
248.
[16] Y. Cao, Z. Wu, and C. Shen, “Estimating depth from monocular images
as classification using deep fully convolutional residual networks,” IEEE
Trans. Circuits Syst. Video Techn., vol. 28, no. 11, pp. 3174–3182, 2018.
[17] B. Li, Y. Dai, and M. He, “Monocular depth estimation with hierarchical fusion of dilated cnns and soft-weighted-sum inference,” Pattern
Recognition, vol. 83, pp. 328–339, 2018.
[18] K. Lasinger, R. Ranftl, K. Schindler, and V. Koltun, “Towards robust
monocular depth estimation: Mixing datasets for zero-shot cross-datasettransfer,” arXiv.org, vol. 1907.01341, 2019.
[19] C. Godard, O. Mac Aodha, and G. J. Brostow, “Unsupervised monoculardepth estimation with left-right consistency,” in Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6602–
6611.
[20] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsupervised
learning of depth and ego-motion from video,” in Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 6612–
6619.
[21] O. Mac Aodha, N. D. F. Campbell, A. Nair, and G. J. Brostow, “Patchbased synthesis for single depth image super-resolution,” in Proc. of theEuropean Conf. on Computer Vision (ECCV), 2012, pp. 71–84.
[22] M. Hornacek, C. Rhemann, M. Gelautz, and C. Rother, “Depth superresolution by rigid body self-similarity in 3d,” in Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2013, pp. 1123–
1130.
[23] Q. Yang, R. Yang, J. Davis, and D. Nistér, “Spatial-depth super resolution for range images,” in Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2007.
[24] D. Chan, H. Buisman, C. Theobalt, and S. Thrun, “A noise-aware filter12

for real-time depth upsampling,” in Workshop on Multi-camera andMulti-modal Sensor Fusion Algorithms and Applications, 2008.
[25] J. Dolson, J. Baek, C. Plagemann, and S. Thrun, “Upsampling range datain dynamic environments,” in Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2010, pp. 1141–1148.
[26] M. Liu, O. Tuzel, and Y. Taguchi, “Joint geodesic upsampling ofdepth images,” in Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2013, pp. 169–176.
[27] J. Diebel and S. Thrun, “An application of markov random fields torange sensing,” in Advances in Neural Information Processing Systems(NIPS), 2005, pp. 291–298.
[28] A. Harrison and P. Newman, “Image and sparse laser fusion for dense
scene reconstruction,” in Field and Service Robotics, 2009, pp. 219–228.
[29] J. Park, H. Kim, Y. Tai, M. S. Brown, and I. Kweon, “High quality depth
map upsampling for 3d-tof cameras,” in Proc. of the IEEE International
Conf. on Computer Vision (ICCV), 2011, pp. 1623–1630.
[30] D. Ferstl, C. Reinbacher, R. Ranftl, M. Rüther, and H. Bischof, “Image
guided depth upsampling using anisotropic total generalized variation,”in Proc. of the IEEE International Conf. on Computer Vision (ICCV),
2013, pp. 993–1000.
[31] P. Pinies, L. M. Paz, and P. Newman, “Too much TV is bad: Dense
reconstruction from sparse laser with non-convex regularisation,” inProc. IEEE International Conf. on Robotics and Automation (ICRA),
2015, pp. 135–142.
[32] G. Riegler, D. Ferstl, M. Rüther, and H. Bischof, “A deep primaldual network for guided depth super-resolution,” in Proc. of the British
Machine Vision Conf. (BMVC), 2016.
[33] V. Jampani, M. Kiefel, and P. V. Gehler, “Learning sparse high dimensional filters: Image filtering, dense crfs and bilateral neural networks,”
in Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2016, pp. 4452–4461.
[34] T. Hui, C. C. Loy, and X. Tang, “Depth map super-resolution by deep
multi-scale guidance,” in Proc. of the European Conf. on Computer
Vision (ECCV), 2016, pp. 353–369.
[35] Y. Wen, B. Sheng, P. Li, W. Lin, and D. D. Feng, “Deep color guided
coarse-to-fine convolutional network cascade for depth image superresolution,” IEEE Trans. on Image Processing (TIP), vol. 28, no. 2,
pp. 994–1006, 2019.
[36] D. Doria and R. J. Radke, “Filling large holes in lidar data by inpainting
depth gradients,” 2012, pp. 65–72.
[37] J. Liu, X. Gong, and J. Liu, “Guided inpainting and filtering for kinect
depth maps,” in Proc. of the International Conf. on Pattern Recognition(ICPR), 2012, pp. 2055–2058.
[38] D. H. C., J. Kannala, L. Ladicky, and J. Heikkilä, “Depth map inpainting
under a second-order smoothness prior,” in Proc. of the Scandinavian
Conference on Image Analysis, 2013, pp. 555–566.
[39] J. T. Barron and B. Poole, “The fast bilateral solver,” in Proc. of theEuropean Conf. on Computer Vision (ECCV), 2016, pp. 617–632.
[40] J. T. Barron and J. Malik, “Intrinsic scene properties from a single RGBD image,” IEEE Trans. on Pattern Analysis and Machine Intelligence(PAMI), vol. 38, no. 4, pp. 690–703, 2016.
[41] Y. Zhang and T. A. Funkhouser, “Deep depth completion of a single
RGB-D image,” in Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2018, pp. 175–185.
[42] Z. Huang, J. Fan, S. Cheng, S. Yi, X. Wang, and H. Li, “Hmsnet: Hierarchical multi-scale sparsity-invariant network for sparse depth
completion,” IEEE Trans. on Image Processing (TIP), vol. 29, pp. 3429–
3441, 2020.
[43] C. Cadena, A. R. Dick, and I. D. Reid, “Multi-modal auto-encoders as
joint estimators for robotics scene understanding,” in Proc. Robotics:
Science and Systems (RSS), 2016.
[44] J. Qiu, Z. Cui, Y. Zhang, X. Zhang, S. Liu, B. Zeng, and M. Pollefeys,“Deeplidar: Deep surface normal guided depth prediction for outdoor
scene from sparse lidar data and single color image,” in Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR), 2019, pp.
3313–3322.
[45] M. Jaritz, R. de Charette, É. Wirbel, X. Perrotton, and F. Nashashibi,“Sparse and dense data with cnns: Depth completion and semantic
segmentation,” in Proc. of the International Conf. on 3D Vision (3DV),2018, pp. 52–60.
[46] X. Cheng, P. Wang, and R. Yang, “Depth estimation via affinity
learned with convolutional spatial propagation network,” in Proc. of theEuropean Conf. on Computer Vision (ECCV), 2018, pp. 108–125.
[47] F. Ma, G. V. Cavalheiro, and S. Karaman, “Self-supervised sparseto-dense: Self-supervised depth completion from lidar and monocularcamera,” in Proc. IEEE International Conf. on Robotics and Automation(ICRA), 2019, pp. 3288–3295.

[48] S. S. Shivakumar, T. Nguyen, I. D. Miller, S. W. Chen, V. Kumar,and C. J. Taylor, “Dfusenet: Deep fusion of RGB and sparse depth
information for image guided dense depth completion,” in Proc. IEEE
Conf. on Intelligent Transportation Systems (ITSC), 2019, pp. 13–20.
[49] A. Eldesokey, M. Felsberg, and F. S. Khan, “Confidence propagation
through cnns for guided sparse depth regression,” IEEE Trans. on PatternAnalysis and Machine Intelligence (PAMI), vol. 42, no. 10, pp. 2423–
2436, 2020.
[50] E. A. Nadaraya, “On estimating regression,” Theory of Probability &Its Applications, vol. 9, no. 1, pp. 141–142, 1964.
[51] G. S. Watson, “Smooth regression analysis,” Sankhyā: The Indian
Journal of Statistics, Series A, pp. 359–372, 1964.
[52] H. Takeda, S. Farsiu, and P. Milanfar, “Kernel regression for image
processing and reconstruction,” IEEE Trans. on Image Processing (TIP),
vol. 16, no. 2, pp. 349–366, 2007.
[53] K. Q. Weinberger and G. Tesauro, “Metric learning for kernel regression,” in Conference on Artificial Intelligence and Statistics (AISTATS),
2007, pp. 612–619.
[54] Y.-K. Noh, M. Sugiyama, K.-E. Kim, F. Park, and D. D. Lee, “Generative
local metric learning for kernel regression,” in Advances in Neural
Information Processing Systems, 2017, pp. 2452–2462.
[55] S. Gershman and N. D. Goodman, “Amortized inference in probabilistic
reasoning,” in Proc. of the Annual Meeting of the Cognitive Science
Society, 2014.
[56] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks
for biomedical image segmentation,” in Medical Image Computing andComputer-Assisted Intervention (MICCAI), 2015.
[57] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning forimage recognition,” in Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2016.
[58] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” in Proc. of theInternational Conf. on Machine learning (ICML), 2015.
[59] J. Hu, M. Ozay, Y. Zhang, and T. Okatani, “Revisiting single image
depth estimation: Toward higher resolution maps with accurate object
boundaries,” in Proc. of the IEEE Winter Conference on Applications ofComputer Vision (WACV), 2019, pp. 1043–1051.
[60] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous
driving? the kitti vision benchmark suite,” in Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2012.

Lina Liu received her B.S. degree in automationfrom Zhejiang University in 2018. She is currently
a Ph.D. candidate of the Institute of Cyber Systems and Control, Department of Control Scienceand Engineering, Zhejiang University. Her researchinterests include computer vision and deep learning.

Yiyi Liao received her Ph.D. degree from the
Department of Control Science and Engineering,
Zhejiang University, China in 2018. She is currently
a PostDoctoral researcher at the Autonomous Vision
Group, Max Planck Institute for Intelligent Systems
and University of Tübingen, Germany. Her researchinterests include 3D vision and scene understanding.

13

Yue Wang received his Ph.D. degree from the
Department of Control Science and Engineering,
Zhejiang University, China in 2016. He is currently
an associate professor in the Department of Control
Science and Engineering, Zhejiang University. His
latest research interests include mobile robotics androbot perception.

Andreas Geiger received his Ph.D. degree in computer science from Karlsruhe Institute of Technology(KIT), Germany in 2013. He is currently a professorof computer science at the University of Tübingen,Germany and a group leader at the Max Planck Institute for Intelligent Systems in Tübingen, Germany.

Yong Liu received his B.S. degree in computerscience and engineering from Zhejiang University
in 2001, and the Ph.D. degree in computer sciencefrom Zhejiang University in 2007. He is currently
a professor in the Institute of Cyber Systems andControl, Department of Control Science and Engineering, Zhejiang University. He has publishedmore than 30 research papers in machine learning,
computer vision, information fusion, robotics. His
latest research interests include machine learning,
robotics vision, information processing and granular
computing.