DEPTH MAP UP-SAMPLING USING COST-VOLUME FILTERING
Ji-Ho Cho1 , Satoshi Ikehata2 , Hyunjin Yoo1 , Margrit Gelautz1 , and Kiyoharu Aizawa21Vienna University of Technology, Austria 2 The University of Tokyo, JapanABSTRACT

Depth maps captured by active sensors (e.g., ToF camerasand Kinect) typically suffer from poor spatial resolution, considerable amount of noise, and missing data. To overcome
these problems, we propose a novel depth map up-sampling
method which increases the resolution of the original depthmap while effectively suppressing aliasing artifacts. Assuming that a registered high-resolution texture image is available, the cost-volume filtering framework is applied to thisproblem. Our experiments show that cost-volume filtering
can generate the high-resolution depth map accurately andefficiently while preserving discontinuous object boundaries,
which is often a challenge when various state-of-the-art algorithms are applied.
Index Terms— Depth map super-resolution, cost-volume
filtering, up-sampling
1. INTRODUCTION
Depth map acquisition is an active research area in image
processing and computer vision. Various depth acquisition
methods have been proposed to date; these can generally becategorized into two approaches: passive and active. Passive
approaches generate a depth map of the scene by multiple
image correspondences and triangulation [1], whereas active
sensors, e.g., time-of-flight (ToF) and Kinect cameras, measure the distance from the camera to the objects directly using
active infrared illumination. While those active cameras arebecoming a popular alternative to passive approaches due totheir simplicity, depth maps acquired by active sensors often
contain considerable noise for objects with low reflectivity,and have very low-resolution.
So far, many works on depth map up-sampling haveemerged and they are mainly categorized into two classes.
One class is motivated by the image super-resolution literature, which explicitly considers the low-resolution image
formation process. Some algorithms in this class reconstruct a high-resolution depth map of a static scene by fusing
multiple low-resolution depth maps that were observed together [2, 3]. More recently, learning-based single image
super-resolution techniques were integrated into depth map
This work was supported in part by the Austrian Science Fund (FWF,M1383-N23) and JSPS Research Fellowships for Young Scientists.

super-resolution to handle dynamic scenes [4]. Though they
do not require a registered high-resolution texture image, atime-consuming dictionary learning for each magnification
factor is often required.
On the other hand, the second class determines depthvalues at interpolated coordinates of the input domain in themanner of multi-modal filtering [5–8] or Markov random
field (MRF) modeling [9, 10]. This class works well fornoisy low-resolution depth maps by leveraging a registeredhigh-resolution texture image assuming the co-occurrence ofdepth and texture structures. While general purpose multimodal filters (e.g., joint bilateral up-sampling [5] or guided
image up-sampling [6]) have been applied to up-sample a
low-resolution depth map, they sometimes give inaccurate
edges which result in considerable color bleeding artifacts.
To overcome this problem, Park et al. [10] integrated a nonlocal means filter into the smoothness term of an MRF-based
depth map up-sampling scheme, which reasonably preserves
object discontinuities. Moreover, some recent works quantize
the depth values into several discrete layers. Yang et al. [7]build a 3-D cost-volume using a low-resolution depth map
and then perform the joint bilateral filtering [11] for a 2-D
cost slice of each depth candidate. Min et al. [8] proposeda weighted mode filter which also finds the global mode ofa filtered cost-volume yet more efficiently than [7]. These
approaches use an original sparse depth map to construct a
cost function for the up-sampling process, which leads tocritical aliasing artifacts when the up-sampling ratio is high(e.g., more than 8×). To solve this problem, Yang et al. [7]iteratively perform the joint bilateral filtering. Min et al. [8]hierarchically iterate the up-sampling process from coarse tofine levels. While effective, these approaches sometimes giveover-smoothed edges and blurred details during iterations.
In this paper, we instead leverage the cost-volume filter
which relies on the guided image filter [12]. The cost-volume
filtering was originally proposed to alternate time-consuming
MRF-based solutions for optimizing labeling problems. Theauthors of [12] have shown that the framework can be applied
to many different applications such as stereo matching, optical flow, and binary segmentation and demonstrated that highquality results are obtained fast. The main differences between our method using cost-volume filtering and the method
proposed by Yang et al. are summarized as follows:
1) We apply the guided image filtering [6] when eachslice of cost-volume is filtered because the guided filteringhas more potential to achieve an efficient computation than
the joint bilateral filtering [11].
2) We design a new confidence measure which not only
reduces aliasing artifacts but also restores the missing depth
areas without time-consuming iterations.
As a result, the proposed method can up-scale the depth
map fast while preserving discontinuous object boundaries
and suppressing aliasing artifacts.
2. THE PROPOSED METHOD
The proposed method utilizes the cost-volume filtering framework to super-resolve the small resolution of an input image.
We assume that a well-aligned high-resolution texture (RGBimage) is available. The proposed method consists of threemain steps: 1) construction of a cost volume, 2) filtering of acost-volume, and 3) selection of the final label.
Cost-volume construction: the cost-volume C is constructed by using the absolute difference between the potential discrete depth label l ∈ {1, ..., N } and an initially upsampled depth dˆH from the input depth map dL with a weightvalue ω for each pixel p:
Cp,l = ωp ||l − dˆHp ||.

(1)

The initial high-resolution depth map dˆH is obtained bythe nearest neighbor approach. We apply this approach toavoid blurred pixels in the object boundaries because it computes new pixels as the value of the nearest pixel in the original image. However, the up-sampled result often suffers fromserious aliasing artifacts. To solve this problem, previousworks in [7, 8] iteratively perform the filtering process which
requires more computation time and produces blurry edges.
Instead, we measure the confidence ωp of dˆHp :(ωp =if dˆH
p < τ,0exp(−(||IpH −IpL ||)2
↓2σ 2(2)) otherwise.

where I H is a guidance high-resolution image, I L is thelow-resolution image obtained by the nearest neighbor downsampling from I H to align with the input low-resolution depth
map, p↓ is the corresponding location in the low-resolution
image (if p = (x, y), then p↓ = ([x/s], [y/s]), s is a scalingfactor), and σ = 0.1 in all experiments. τ is a threshold value
which determines the missing pixels caused by occlusions,shadows, and low-reflections.
The confidence is measured using the color difference ofpixels between coordinates p in I H at full-resolution and thecorresponding down-sampled coordinates p↓ in IL . It meansthat the up-sampled depth value has high confidence when its
corresponding down-sampled color is similar to the one fromthe original high-resolution image. Then, low-confidence
depth values are replaced by propagation from neighboring

Fig. 1. Our computed confidence map of the Moebius scene(8× up-scaling). Note that bright pixels mean high confidence.
pixels which have high confidence during the following filtering and selection steps. Fig. 1 shows the confidence map
containing the weights ω of the Moebius scene.
Cost-volume filtering: In this step, each piece of the costvolume is filtered by guided image filtering. The output of thefiltering at pixel p is a weighted average of pixels in the samelabel l:
X0
Cp,l=Wp,q (I H )Cq,l(3)q0where C is the filtered cost volume and p and q are the pixel
locations. The weight Wp,q , which is dependent on the highresolution color image I H , is as follows:

Wp,q =

1|wk |2X(1+(IpH −µk )T (Σk +U )−1 (IqH −µk )),(p,q)∈wk(4)where Σk is a 3 × 3 covariance matrix, µk is a 3 × 1 mean
vector of r, g, and b in each 3-D window wk with dimensions
wx × wy centered at pixel k, and U is a 3 × 3 identity matrix.
 is a user parameter and we set 0.04 for all our experiments.
More details about the parameter  can be found in [6].
Cost selection: Finally, for each pixel p, the final label fp
which has the minimum cost value is selected by:
0fp = arg min Cp,l.
l(5)3. EXPERIMENTAL RESULTS
This section describes the evaluation of our method on various
data sets. τ in Eq. (2) is 10 and the number of candidate depthvalues N is 256 (8 bits) for all experiments.
3.1. Middlebury stereo dataset
First, we compared our method with four different depth map
super-resolution methods: joint bilateral up-sampling [5],(a) JBU [5](b) MRFU [10](c) WMFU [8](d) Iterative JBU [7](e) our method(f) Ground-truth(g) JBU [5](h) MRFU [10](i) WMFU [8](j) Iterative JBU [7](k) our method(l) Ground-truth

Fig. 2. Experimental results of 8× image up-sampling. The upper row represents the results on the clean Moebius set and thelower one illustrates results on a noisy Art data set from [10].
MRF-based method [6], weighted mode filtering [8], and iterative joint bilateral up-sampling [7] using the Middlebury
Art and Moebius dataset. The original resolution of bothdatasets is 1376 ×1088.
In this experiment, the low-resolution depth image is upsampled with a guidance of the corresponding high-resolution
RGB image. As depicted in Fig. 2, our up-sampling method
can super-resolve the low-resolution depth map while preserving depth discontinuities. Table 1 shows that the proposed
method can generate lower error rates in terms of root means
squared error (RMSE). Table 2 shows the results on a noisy
dataset provided by [10]. The MRFU method produces thebest result in terms of RMSE in the noisy data up-samplingcase because it uses an MRF framework which is very robust
against noise. Our method also produces low error rates forthe noisy dataset, which indicates that our cost volume filtering approach is also able to cope with noise. Furthermore,
we have achieved a very fast computation time using a GPU
implementation. It takes about 0.5 seconds to up-scale thelow-resolution image into 1376×1088 size. The scaling factor does not affect the computation time.
3.2. Kinect dataset
Kinect captures a registered RGB image and depth map. The
maximum resolution of depth is 640×480 while the maximum resolution of RGB is 1280×960. The captured depth
often suffers from missing data caused by occlusion and low(a) Nearest neighbor up-sampling(b) our method

Fig. 3. Comparison of up-sampling results for the lowresolution depth from Kinect.
reflectivity as shown in Fig. 3. Since our weight for these
corrupted pixels is 0, their depth values are restored by highconfident neighboring pixels. As a result, our method cangenerate a high-resolution depth map while preserving depth
discontinuities.
4. CONCLUSIONSIn this paper, we have proposed a cost-volume filter based
depth map up-sampling. We have demonstrated that the proposed method can generate a high-resolution depth with discontinuous object boundaries being preserved without an iteration, while suppressing aliasing artifacts. A certain limitation of our method is that we need a well-aligned high-Table 1. Quantitative Evaluation of Depth Map Super-resolution (RMSE)
ArtMoebius

Method2×

4×8×16×2×4×8×16×

JBU [5]0.35380.62501.13272.03940.18860.42700.95461.6708MRFU [10]0.4306

0.67451.07342.21170.17950.29650.52180.8965WMFU [8]0.65210.90371.74603.29910.46720.64161.00441.7402Iterative JBU [7]0.57080.70021.50463.69030.38680.47600.6893

1.3660

Our method

0.36990.54080.83711.71010.14230.22520.41650.8107

Table 2. Quantitative Evaluation for Noisy Depth Map Super-resolution (RMSE)
Art

Moebius

Method2×

4×8×16×2×4×8×16×

JBU [5]1.50691.94842.9241

4.69261.70151.92062.34833.0232MRFU [10]1.24011.81592.70474.39401.0343

1.48942.12893.0910WMFU [8]1.97082.3191

3.38185.1306

1.86932.04302.56123.4895Iterative JBU [7]1.3592

1.93152.45354.51921.2506

1.63342.05593.2054

Our method

1.2903

1.96893.1132

4.69571.0633

1.53572.4692

3.0372resolution guidance image to achieve the best performance.

5. REFERENCES[1] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, andR. Szeliski, “A comparison and evaluation of multi-view
stereo reconstruction algorithms,” in Proc. of Computer
Vision and Pattern Recognition, 2006.
[2] C. Theobalt, J. Davis, and S. Thrun, “High-quality
scanning using time-of-flight depth superresolution,”in Proc. of Computer Vision and Pattern Recognition
Workshops, 2008.

in Proc. of European Conference on Computer Vision,2010.
[7] Q. Yang, R. Yang, J. Davis, and D. Nister, “Spatialdepth super resolution for range images,” in Proc. IEEE
Computer Vison and Pattern Recognition, 2007.
[8] D. Min, J. Lu, and M. N. Do, “Depth video enhancement based on weighted mode filtering,” IEEE Transactions on Image Processing, vol. 21, no. 3, pp. 1176–1190, 2012.
[9] J. Lu, D. Min, R. S. Pahwa, and M. N. Do, “A revisit
to mrf-based depth map super-resolution and enhancement,” in Proc. of International Conference on Acoustics, Speech and Signal Processing, 2011.

[3] S. Schuon, C. Theobalt, J. Davis, and S. Thrun, “Lidarboost: Depth superresolution for tof 3d shape scanning,” in Proc. of Computer Vison and Pattern Recognition, 2009.

[10] J. Park, H. Kim, Y-W. Tai, M. S. Brown, and I. Kweon,“High quality depth map upsampling for 3d-tof cameras,” in Proc. of International Conference on ComputerVision, 2011.

[4] O. M. Aodha, N. D. F. Campbell, A. Nair, and G. J.
Brostow, “Patch based synthesis for single depth image
super-resolution,” in Proc. of European Conference on
Computer Vision, 2012.

[11] G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen,
H. Hoppe, and K. Toyama, “Digital photography withflash and no-flash image pairs,” ACM Transactions onGraphics, vol. 23, no. 3, pp. 664–672, 2004.

[5] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele, “Joint bilateral upsampling,” ACM Transactions
on Graphics, vol. 26, no. 3, pp. 96, 2007.

[12] A. Hosni, C. Rhemann, M. Bleyer, C. Rother, andM. Gelautz, “Fast cost-volume filtering for visual correspondence and beyond,” IEEE Transactions on PatternAnalysis and Machine Intelligence, vol. 35, no. 2, pp.
504–511, 2013.

[6] K. He, J. Sun, and X. Tang, “Guided image filtering,”