Supplementary Material forConnecting the Dots:
Learning Representations for Active Monocular Depth Estimation

1

Gernot Riegler1,∗ Yiyi Liao2,∗ Simon Donne2 Vladlen Koltun1 Andreas Geiger22Intel Intelligent Systems Lab
Autonomous Vision Group, MPI-IS / University of Tübingen{firstname.lastname}@intel.com{firstname.lastname}@tue.mpg.de

Abstract
In this supplementary document, we first present details on the network architectures of our disparity decoder in Section 1.
Section 2 provides additional details on the edge decoder used in our work. In Section 3 we give additional ablation studies
for our method and the used baselines. Finally, we show additional results in Section 4.

1. Disparity Decoder
The disparity decoder is based on the DispNetS architecture presented in [5]. Table 1 shows the detailed disparity decoderarchitecture, where k denotes kernel size, s stride and p padding. Cin and Cout are the number of input and output channels.
up(·) denotes bilinear up-sampling. We use a ReLU layer after each convolutional layer and a scaled sigmoid after eachprediction layer. We do not show them in the table for clarity.

2. Edge Decoder
We exploit a shallow U-Net architecture for our edge decoder. It is also based on the DispNet architecture [5], but withfewer layers. Table 2 shows the detailed edge decoder architecture, with k denoting kernel size, s stride, and p padding. Cin
and Cout are the number of input and output channels. up(·) denotes bilinear up-sampling. There is a ReLU layer after eachconvolutional layer which we do not show in the table for clarity.
As mentioned in the main paper, we train an edge decoder to predict A0 = LCN (|∇A|), the local contrast normalized
gradient magnitude of the ambient image A. While the edge decoder requires the ambient image A as supervision, this iseasily obtained in practice (recording the scene with the laser projector turned on and off). In particular, our shallow U-Net
generalizes from few training samples: In the experiment, we assume that we have ambient images for 1, 024 short sequencesamong the full rendered training dataset of 8, 192 sequences.
Fig. 1 shows an example of an input image I from the test dataset, the edge image A0 and our prediction. We observe thatthe edge decoder clearly separates edge from non-edge regions.

3. Additional Ablation Studies
In this section we present additional ablation studies for our method and the utilized baselines. Section 3.1 demonstrates
the correlation between the photometric loss and some of the metrics that we evaluated on. In Section 3.2 we show an
ablation study of the disparity decoder. In Section 3.3 we evaluate our method and the baselines when applied with smooth
post-processing. Finally, Section 3.4 provides hyper-parameter tuning experiments and the influence of post-processing on
the HyperDepth [2] baseline used in our evaluations.

3.1. Correlation of Photometric Loss
Note that our self-supervised approach does not have access to ground truth geometry. We thus select the network parameters from all training epochs by minimizing the average photometric error on a validation set with 512 samples. In thislayer name
conv1
conv2
conv3
conv4
conv5
conv6
conv7
upconv7
concaticonv7upconv6
concaticonv6
upconv5
concaticonv5upconv4
concaticonv4predict disp4
upconv3
concaticonv3predict disp3
upconv2
concaticonv2predict disp2
upconv1
concaticonv1predict disp1

k × k, p, s

Cin

Cout

input nameoutput name7 × 7, 3, 2
7 × 7, 3, 1
5 × 5, 2, 2
5 × 5, 2, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 1

2
32
32
64
64
128
128
256
256
512
512
512
512
512
512
1024
512
1024
512
512
256
256
128
128
129
64
64
65
32
32
33
16

32
32
64
64
128
128
256
256
512
512
512
512
512
512
512
512
512
512
256
256
128
128
1
64
64
1
32
32
1
16
16
1

out conv1
out conv2
out conv3
out conv4
out conv5
out conv6
out conv7
out upconv7, out conv6
concat7out conv6
out upconv6, out conv5
concat6out conv5
out upconv5, out conv4
concat5out conv4
out upconv4, out conv3
concat4out iconv4
out conv3
out upconv3, out conv2, up(disp4)
concat3
out iconv3
out iconv3
out upconv2, out conv1, up(disp3)
concat2
out iconv2
out iconv2
out upconv1, out conv1, up(disp2)
concat1
out iconv1

out conv1
out conv2
out conv3
out conv4
out conv5
out conv6
out conv7
out upconv7
concat7out iconv7
out upconv6
concat6out iconv6
out upconv5
concat5
out iconv5
out upconv4
concat4out iconv4
disp4
out upconv3
concat3
out iconv3
disp3
out upconv2
concat2
out iconv2
disp2
out upconv1
concat1
out iconv1
disp1

Table 1: Architecture of Disparity Decoder. k denotes kernel size, s stride and p padding. Cin and Cout are the number of
input and output channels(a) Input I(b) Ground truth A0(c) Prediction

Figure 1: Qualitative Results of Edge Decoder.
section, we demonstrate that the photometric loss is well correlated to our evaluation metrics and is thus suitable as a proxy
metric. Fig. 2 illustrates the photometric loss and o(t) when training our model using only the photometric loss LP . Each dot

layer name
conv1
conv2
conv3
upconv3
concaticonv3predict edge3
upconv2
concaticonv2predict edge2
upconv1
concaticonv1predict edge1

k × k, p, s

Cin

Cout

input nameoutput name7 × 7, 3, 2
7 × 7, 3, 1
5 × 5, 2, 2
5 × 5, 2, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 1
3 × 3, 1, 2
3 × 3, 1, 1
3 × 3, 1, 1

2
32
32
64
64
128
128
128
64
64
65
32
32
33
16

32
32
64
64
128
128
64
64
1
32
32
1
16
16
1

out conv1
out conv2
out conv3
out upconv3, out conv2
concat3
out iconv3
out iconv3
out upconv2, out conv1, up(edge3)
concat2
out iconv2
out iconv2
out upconv1, out conv1, up(edge2)
concat1
out iconv1

out conv1
out conv2
out conv3
out upconv3
concat3
out iconv3
edge3
out upconv2
concat2
out iconv2
edge2
out upconv1
concat1
out iconv1
edge1

Table 2: Architecture of Edge Decoder. k denotes kernel size, s stride and p padding. Cin and Cout are the number of input
and output channels, respectively.

(a) o(0.1)(b) o(0.5)(c) o(1)(d) o(2)(e) o(5)Figure 2: Correlation of Photometric Loss and Evaluation Metrics. For each threshold t, we show the photometric loss
(x-axis) wrt. the corresponding metric (y-axis) across epochs as data points. We show the correlation coefficient above eachfigure, where 1 denotes maximal positive linear correlation.
represents the photometric loss (x-axis) and the corresponding metric (y-axis) at one particular epoch. We fit a line to thisdata using linear regression, and show the correlation coefficient above each sub-figure (here 1 represents maximal positive
linear correlation and 0 denotes no linear correlation). Fig. 2 suggests that the photometric loss is highly correlated with o(t)across different thresholds. Therefore, it is a viable criterion for model selection.

3.2. Ablation of Disparity Decoder
In the following evaluation we show that a large receptive field as present in our disparity decoder (Table 1) is needed
for accurate disparity estimation. For this evaluation we consecutively remove lower resolution parts of the network. We
train the network on a 100 rows crop of our train dataset and evaluate on the same 100 rows of the test dataset. The results
are summarized in Table 3. We can observe that the network performance gradually improves by adding layers at lower
resolutions that capture a larger receptive field. We also tried to train the network with CoordConvs [4] and BatchNorm [3],but did not observe any improvements on the test metrics.

3.3. Smooth Post-processing
We observe that some of our baselines produces a random noise in the prediction, where the results might be improved
with a smooth post-processing. For a fair comparison, we apply 5 × 5 median filtering to all methods in Tab. 3 of the main
paper. The results are shown in Table 4. Note that the smooth post-processing improves the performance of all comparedmethods, while our method still performs the best.

deptho(0.1)o(0.5)

o(1)

o(2)

o(5)

1
2
3
4
5
6
7

0.92310.89350.56460.5196
0.3636
0.41510.37450.67170.57680.27420.19990.11790.12070.10840.53030.44770.22990.13930.06630.06450.05750.40710.34680.19760.10640.04000.03610.03200.19890.16840.15020.08040.02240.01690.0154

Table 3: Ablation of U-Net architecture. A higher depth value indicates a larger receptive field of the network, with depth= 7 being the architecture presented in Table 1.

Block Matching
Block Matching + MF
FastMRFFastMRF + MF
HyperDepth
HyperDepth + MF
Ours
Ours + MFo(0.5)

o(1)

o(2)

o(5)

ou (1)ou (5)

7.84
6.78
12.07
12.00
15.01
9.09
6.77
6.28

7.20
6.15
8.36
8.34
12.63
7.41
3.88
3.62

7.06
5.98
6.71
6.69
11.83
6.68
2.57
2.46

6.83
5.71
5.14
5.12
11.49
6.20
1.63
1.58

4.44
3.57
5.25
5.23
7.39
3.97
1.75
1.64

4.23
3.32
3.57
3.57
6.73
3.31
0.70
0.69

Table 4: Quantitative Results on Synthetic Data with Median Filtering.

3.4. HyperDepth Hyper-Parameters
The most related work to our method is the random forest based HyperDepth [2]. Unfortunately, there is no implementation available and therefore, we had to implement this method ourselves. We tried to replicate the method as closely as
possible based on the original paper and also communicated with the authors regarding the details. For a fair comparison, wecross-validated the hyper-parameters on a validation set. In Table 5 we show results varying the total tree depth, the numberof random samples for split node optimization and the tree depth at which we switch from pixel to sub-pixel accuracy. Notethat in our experiments, we obtained better results with deeper trees.
Another set of hyper-parameters involve the post-processing in HyperDepth. In Fig. 3 we show quantitatively and qualitatively results for different settings. Note how with different hyper-parameters we can trade accuracy for completeness.
In the evaluation in our main paper we used the hyper-parameters that lead to the smallest harmonic mean of accuracy andcompleteness.

4. Additional Results
4.1. Rendered Data
In this section we show additional qualitative results on our synthetic dataset and the dataset provided by [1]. Fig. 4
depicts additional qualitative results as depth maps on our synthetic dataset for our method, block matching, FastMRF [1],and HyperDepth [2]. We also show results as 3D point-clouds from Fig. 5 to Fig. 7. For each method we show two point
clouds from different perspectives. Odd rows show the point cloud from the estimated depth map, with green indicating
accurate predictions, yellow are points with a distance of 1cm distance to the closest 3D ground-truth point, and red points
have a distance > 2cm to the nearest 3D ground-truth point. Even rows depict the ground-truth point cloud with the samecolor coding, but indicating the distance to the closest estimated 3D point. Hence, odd rows show the accuracy of a given
method, whereas even rows depict their completeness. Note that we don’t apply post-processing to HyperDepth on our
synthetic data, as the evaluation metrics – percentage of outliers remain the same regardless of the post-processing.

4.2. Real Data
Fig. 8 shows additional depth map results on the dataset provided by [1]. We compare our method to the same set ofbaselines as in the previous experiment. In addition we show results as 3D point-clouds in Fig. 9 for the model Angel, inFig. 10 for the model Arch, and in Fig. 11 for the model Gargoyle. For each method we show two point clouds from different
perspectives. Odd rows show the point cloud from the estimated depth map, with green indicating accurate predictions,o(0.1)o(0.5)

o(1)

o(2)

o(5)

depth=12, samples=1024, switch=6
depth=12, samples=1024, switch=8
depth=12, samples=1024, switch=10
depth=12, samples=4096, switch=6
depth=12, samples=4096, switch=8
depth=12, samples=4096, switch=10
depth=12, samples=16384, switch=6
depth=12, samples=16384, switch=8
depth=12, samples=16384, switch=100.78210.78170.78010.77500.76920.76940.76940.76740.76700.31230.3156
0.31300.31860.30280.3056
0.31120.31770.31520.22120.22560.22520.23500.22150.22900.23610.24330.24220.20170.20630.20610.21580.20420.21330.22010.22810.22730.19720.20150.20180.21130.20020.20940.21600.22370.2233depth=14, samples=1024, switch=8
depth=14, samples=1024, switch=10
depth=14, samples=1024, switch=12
depth=14, samples=4096, switch=8
depth=14, samples=4096, switch=10
depth=14, samples=4096, switch=12
depth=14, samples=16384, switch=8
depth=14, samples=16384, switch=10
depth=14, samples=16384, switch=120.74110.73010.74230.72990.72640.73090.72750.72780.73490.26850.25670.26270.25700.2546
0.25710.26150.26910.2746

0.19910.19150.19430.19210.19230.1956
0.19870.20880.21340.18430.17800.17900.17830.17940.18300.18500.19520.19970.18020.17410.17500.17480.17600.17970.18130.19200.1961depth=16, samples=1024, switch=10
depth=16, samples=1024, switch=12
depth=16, samples=1024, switch=14
depth=16, samples=4096, switch=10
depth=16, samples=4096, switch=12
depth=16, samples=4096, switch=14
depth=16, samples=16384, switch=10
depth=16, samples=16384, switch=12
depth=16, samples=16384, switch=140.72870.72750.73110.70780.71240.71970.70830.71160.71470.25190.24900.24480.22690.22930.23190.2346
0.23600.23420.18870.18520.17750.17040.17030.17160.17840.18100.17680.17500.17110.16320.15880.15790.1586
0.16650.16900.16440.17120.16740.1596
0.15540.15480.15530.16300.16560.1611

Table 5: Hyper-Parameter Tuning for HyperDepth [2]. We train a random forest with four trees on a single row of thesynthetic dataset. depth denotes the maximal tree depth, samples is the maximal number of training instances sampled to
optimize a given split node, and switch is the tree depth where we switch from integer accuracy to sub-pixel accuracy.
yellow are points with a distance of 5mm distance to the closest 3D ground-truth point, and red points have a distance> 1cm to the nearest 3D ground-truth point. Even rows depict the ground-truth point cloud with the same color coding, but
indicating the distance to the closest estimated 3D point. Hence, odd rows show the accuracy of a given method, whereas
even rows depict their completeness.

4.3. Real Data in Complex Real-World Scenarios
In addition to the dataset provided by [1], we further trained and evaluated our network in more complex real-world
scenarios with IR images collected by a Microsoft Kinect v1 (3, 191 for training and 623 for testing). Fig. 12 shows qualitative
3D results on the test set with a human in motion and an indoor scene respectively, demonstrating that our method generalizes
well to complex real-world scenarios.

(a) Input, GT(b) 127.170, 5.955, 11.378(c) 114.224, 5.955, 11.320(d) 113.926, 5.956, 11.320(e) 25.373, 6.794, 10.718(f) 22.722, 6.870, 10.550(g) 18.041, 7.045, 10.133(h) 17.505, 7.115, 10.118

Figure 3: Influence of HyperDepth Post-Processing. (a) Input IR image and projected ground-truth model. (b) No postprocessing, raw random forest output. (c) Mask out disparity values ∈/ [0, dmax ]. (d) Additionally mask out disparity values
with likelihood from forest < 0.1. (e) Additionally mask out disparity values, where best and second best predicted disparity
difference is > 10pixels. (f) Additionally mask out disparity values, where best and second best predicted disparity differenceis > 5pixels. (g) Additionally mask out disparity values, where best and second best predicted disparity difference is >2pixels. (h) Additionally mask out disparity values, where best and second best predicted disparity difference is > 1pixels.
The numbers in the sub-caption are accuracy, completeness and harmonic mean of those two numbers in mm.

(a) Input, GT(b) Block M.

(c) FastMRF [1](d) HyperD. [2](e) Ours LP(f) Ours +LD

Figure 4: Additional Qualitative Results on Synthetic Data.

(g) Ours +LG

accuracycompletenessaccuracy

completenessaccuracy

completenessaccuracy

Block Matching
FastMRF [1]HyperDepth [2]completeness

Ours
Figure 5: Additional Qualitative 3D Results on Synthetic Data. Event rows depict accuracy results: 3D point-cloud fromestimated depthmap. The color indicates the distance to the closest 3D point of the ground-truth model, from dark green= 0cm, over yellow = 1cm, to red ≥ 2cm. Odd rows show the completeness: 3D point-cloud of the ground-truth with thesame color coding indicating the distance to the closest estimated 3D point.

accuracycompletenessaccuracy

completenessaccuracy

completenessaccuracy

Block Matching
FastMRF [1]HyperDepth [2]completeness

Ours
Figure 6: Additional Qualitative 3D Results on Synthetic Data. Event rows depict accuracy results: 3D point-cloud fromestimated depthmap. The color indicates the distance to the closest 3D point of the ground-truth model, from dark green= 0cm, over yellow = 1cm, to red ≥ 2cm. Odd rows show the completeness: 3D point-cloud of the ground-truth with thesame color coding indicating the distance to the closest estimated 3D point.

accuracycompletenessaccuracy

completenessaccuracy

completenessaccuracy

Block Matching
FastMRF [1]HyperDepth [2]completeness

Ours
Figure 7: Additional Qualitative 3D Results on Synthetic Data. Event rows depict accuracy results: 3D point-cloud fromestimated depthmap. The color indicates the distance to the closest 3D point of the ground-truth model, from dark green= 0cm, over yellow = 1cm, to red ≥ 2cm. Odd rows show the completeness: 3D point-cloud of the ground-truth with thesame color coding indicating the distance to the closest estimated 3D point.

(a) Input, GT(b) Block M.

(c) FastMRF [1](d) HyperD. [2]Figure 8: Additional Qualitative Results on Real Data.

(e) Ours

accuracycompletenessaccuracy

completenessaccuracy

completenessaccuracy

Block Matching
FastMRF [1]HyperDepth [2]completeness

Ours
Figure 9: Additional Qualitative 3D Results on Real Data. Angel. Event rows depict accuracy results: 3D point-cloudfrom estimated depthmap. The color indicates the distance to the closest 3D point of the ground-truth model, from dark green= 0mm, over yellow = 5mm, to red ≥ 1cm. Odd rows show the completeness: 3D point-cloud of the ground-truth with thesame color coding indicating the distance to the closest estimated 3D point.

accuracycompletenessaccuracy

completenessaccuracy

completenessaccuracy

Block Matching
FastMRF [1]HyperDepth [2]completeness

Ours
Figure 10: Additional Qualitative 3D Results on Real Data. Arch. Event rows depict accuracy results: 3D point-cloudfrom estimated depthmap. The color indicates the distance to the closest 3D point of the ground-truth model, from dark green= 0mm, over yellow = 5mm, to red ≥ 1cm. Odd rows show the completeness: 3D point-cloud of the ground-truth with thesame color coding indicating the distance to the closest estimated 3D point. Arch.

accuracycompletenessaccuracy

completenessaccuracy

completenessaccuracy

Block Matching
FastMRF [1]HyperDepth [2]completeness

Ours
Figure 11: Additional Qualitative 3D Results on Real Data. Gargoyle. Event rows depict accuracy results: 3D point-cloudfrom estimated depthmap. The color indicates the distance to the closest 3D point of the ground-truth model, from dark green= 0mm, over yellow = 5mm, to red ≥ 1cm. Odd rows show the completeness: 3D point-cloud of the ground-truth with thesame color coding indicating the distance to the closest estimated 3D point.

(a) Input(b) Block Matching(c) Ours

Figure 12: Qualitative 3D Results on Real Data in Complex Real-World Scenarios.

References[1] Q. Chen and V. Koltun. Fast MRF optimization with application to depth reconstruction. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2014. 4, 5, 7, 8, 9, 10, 11, 12, 13, 14[2] S. R. Fanello, C. Rhemann, V. Tankovich, A. Kowdle, S. Orts-Escolano, D. Kim, and S. Izadi. Hyperdepth: Learning depth fromstructured light without matching. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 1, 4, 5, 7, 8, 9,
10, 11, 12, 13, 14[3] S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proc. of theInternational Conf. on Machine learning (ICML), 2015. 3[4] R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski. An intriguing failing of convolutional neural networks
and the coordconv solution. arXiv preprint arXiv:1807.03247, 2018. 3[5] N. Mayer, E. Ilg, P. Haeusser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for
disparity, optical flow, and scene flow estimation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 1