Supplementary Material for
Learning Non-volumetric Depth Fusion using Successive Reprojections
Simon Donné, Andreas Geiger
Autonomous Vision GroupMPI for Intelligent Systems and University of Tübingensimon.donne@tue.mpg.de, andreas.geiger@tue.mpg.deAbstract
In this supplementary document, we first give an exhaustive overview of the network architecture in Section 1, with a brief
discussion on runtime. More qualitative and quantitative results, both on depth maps and point clouds, are given in Section 2.
The supplementary video visualizes the concepts of reprojection and bounding/culling the reprojections, as well as the effectof the iterative refinement on the final point cloud.

1. Network architecture
We reprise the schematic overview of our network approach in Figure 1. As introduced in the main text, the depth fusion
step happens entirely in the image domain. In order to encode the information from neighbouring views, their depth maps
and image features are cast into space and reprojected onto the center view. After pooling the information from all observed
neighbours, our approach comprises several streams. The first residually refines the input depth values to improve nearlycorrect depth estimates, having only a limited spatial support. A second stream is intended to inpaint large unknown areas(with a much larger spatial support). A third network stream predicts the weighting between these two options to yield theoutput estimate. Finally, a confidence network predicts the confidence in this final estimate: the probability that the estimate iswithin a certain threshold of the ground truth. In order to preserve the absolute depth values, we do not use any normalization
layers in the refinement streams of the network.
center depthand featuresview 1 depth
and features

residualreﬁnement
reprojection

view 1 depth
and features
reprojected

poolingmean depth
and features
reprojected

max depth
and features
reprojectedview N depth
and features

view N depth
and features
reprojected

min residualand features
reprojected

head scoring
network

reﬁnedcenter depth

non-residual
inpainting

conﬁdenceclassiﬁcation

center depth
conﬁdence

Figure 1: Overview of our proposed fusion network. A difference in coloring represents a difference in vantage point, i.e.
the information is represented in different image planes. As outlined in Section 1 of the main paper, neighbouring views are
first reprojected, and then passed alongside the center depth estimate and the observed image. The output of the network is
an improved version of the input depth of the reference view as well as a confidence map for this output.

output

Cout

upsample x2
6F

2x2 maxpool

2F

upsample x2

2F

output2F

3x3 conv

3x3 conv

3x3 conv

input

F

3x3 conv

2x2 maxpoolFF

3x3 conv

3FF

3x3 conv

F

3x3 conv

3x3 conv

input

Cin

2F2F4F

Figure 2: Convolution blocks comprise a 3 × 3 convolution with corresponding reflection padding, a batch normalization
layer (if applicable) and a ReLU non-linearity. The number of internal channels is called F , and doubles at every level. Twoconvolutional groups are used instead of a UNet in the lowest resolution, while the last convolution on the highest resolution
has neither a normalization nor an activation.

1.1. Building block
The building block for our network is a UNet module with skip connections, as visualized in Figure 2. At each level, theinput data is processed with two convolution layers, and then downsampled with max-pooling to be processed on a lowerresolution. The output from the lower resolution is bilinearly upsampled and concatenated to the processed higher-resolution
features and processed by two more convolutions. On the lowest level, the down-sampled features are processed with two
convolutional blocks. We write an instantiation of this building block as UNet(Cin , F, Cout , depth).

1.2. Neighbour reprojection
First, we briefly recap the mathematical formulations for the reprojections as given in the main text. We consider aset of N images In (u) with corresponding camera matrices Pn = Kn [Rn |tn ], and estimated depth maps dn (u), whereTu = [u, v, 1] ∈ Ω is a pixel location. The 3D point corresponding with a given pixel’s estimate is then given by−1
xn (u) = RTn Kn (dn (u)u − Kn tn ) .

(1)

From either the bootstrap confidence network or the previous iteration’s network, each depth estimate dn (u) has a predictedconfidence. Those xn (u) for which the input confidence is larger than 0.5 are projected onto the center view 0. We callun→m (u) = Pm xn (u) the projection of xn (u) onto neighbour m, and zn→m (u) = [0 0 1] un→m (u) its depth. Thez-buffer in view 0 based on neighbour n’s estimate is then min zn→0 (un ), if Ωn (u) 6= Øzn (u) = un ∈Ωn (u)(2)
0,elsewhere
where Ωn (u) = {u ∈ Ω|P0 xn (un ) ∼ u} is the set of pixels in view n that reproject onto u in view 0. Finally, we callun (u) the pixel in view n for which zn (u) = zn→m (un (u)), i.e. the pixel responsible for the entry in the z-buffer.
At the same time, we calculate the lower bound gn (u) on the depth estimates in the center view as implied by neighbourn as the minimum depths for which the corresponding points x0 (u) are observed as empty space by neighbour n:gn (u) = min {d > 0 | dm (u0→n (u)) > z0→n (u)} .

(3)

We now construct the reprojected image as the reprojected image features for the closest points behind every pixel,assuming they conform to the calculated lower bound:(zn (u), if zn (u) ≤ gn (u)z̃n (u) =0,elsewhere((4)In (un (u)), if z̃n (u) > 0Ĩn (u) =0,elsewhere
Note that, here, counterintuively, zn (u) has to be smaller than or equal to the lower bound to be accepted. If it were larger
then it would be either the result of bleedthrough (and should be rejected) or it would be the backside of a surface (and shouldalso be rejected).

1.3. Neighbour pooling
The neighbour pooling modules take as input the reprojected image features {In (un (u))} and their corresponding depth
maps {z̃n (u)}, and return the mean, maximum, and minimum residual pooling (where I(·) is the indicator function):Pz̃n (u)z̃mean (u) = P nn?min (u) = argmin kz̃n (u) − d0 (u)k1
n?max (u) = argmax z̃n (u)I(z̃n (u) > 0)
nnn
P(7)
(5)
(6)
z̃min (u) = z̃n?min (u) (u)z̃max (u) = z̃n?max (u) (u)In (un (u))
Ĩmax (u) = In?max (u) (un (u))
Ĩmin (u) = In?min (u) (un (u))
Ĩmean (u) = PnI(z̃n (u) > 0)n1.4. Shared features
Prior to the other network heads, we firstcalculate a set of features that will be shared by all further streams.
 The input
to this shared feature module is Finput (u) = I0 (u), d0 (u), Ĩmean (u), z̃mean (u), Ĩmax (u), z̃max (u), Ĩmin (u), z̃min (u) , for a totalof 16 channels. We write the structure of this module as UNet(16, 8, 16, depth = 3), and call its output Fshared (u).

1.5. Residual refinement module
The residual refinement module acts on the original eight components plus the shared features: (Finput (u), Fshared (u)).
The spatial support of this network is limited to have it focus on the neighbour information: its structure is given by
UNet(32, 16, 1, depth = 1), and its output is added to d0 (u) to yield dˆrefined (u).

1.6. Inpainting module
To be able to fill in large areas with missing information, such as homogeneous areas or badly constrained areas, theinpainting module has a much larger spatial support. Similar to the residual refinement, the input is given by the 32 channelsof (Finput (u), Fshared (u)), but now the structure is given by UNet(32, 16, 1, depth = 5). Its output is called dˆinpainted (u).

1.7. Head scoring network
The headand inpainting results against one another. Its inputs are hence scoring network weights the residual refinement
ˆˆgiven as Finput (u), Fshared (u), drefined (u), dinpainted (u) . After the main UNet(34, 16, 2, depth = 3) block, we apply a spatial
ˆSoftmax layer, and use its output to weight both alternatives. The final estimate is called d(u),the output of our architecture.

1.8. Confidence classification
Finally, we also have the network return a prediction of its confidence. This module is intended to predict whether orˆ
not d(u) is within a certainthreshold from the ground truth depth estimate. To do so, a UNet(33, 16, 1, depth = 3) acts onˆFinput (u), Fshared (u), d(u). Its result is subsequently activated by a Sigmoid layer to yield a predicted likelihood.

1.9. Runtime
For a typical DTU scene, the timings are: 128s and 193s for the COLMAP and MVS-Net front-ends respectively, 13s periteration for our method, 128s for COLMAP geometric consistency+fusion, and 18s for MVSNet fusion (on an NVidia TitanXp). Therefore, our computational overhead is limited.

2. Additional results
We show more qualitative results for all elements in the test set of the DTU dataset [1] in Figure 3 and Figure 4, for
COLMAP and MVSNet inputs respectively. Table 1 contains more quantative results: both the total and per-view accuracy/completeness percentages, as well as the average accuracy and completeness values for those points that are considered
relevant (i.e. within a certain distance of the groundtruth). Table 2 contains results for the newly introduced synthetic datasets.
From Table 2 it becomes evident that the synthetic datasets are easier for COLMAP (its assumptions are now fulfilled, as
the evaluation does not have any sensor noise) than for MVSNet (which was trained on more realistic images). Applying
our proposed fusion steps significantly improves the results in nearly all cases. These evaluations run on quarter-scale input(480×270 rather than 1920×1080), but we use the original, high-resolution, point clouds as a reference for these comparison.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

Figure 3: Qualitative results for our approach on the COLMAP inputs, for all elements in the test set. For a given input
image (a) we show first the ground truth image (b), the input estimate (c) and its error (d). The subsequent columns show theoutput estimate, error and trust for three iterations. The error visualization works logarithmically: at error value 5, the colouris white. At infinite error, a pixel is shown as dark red and at zero error it is displayed a dark blue. While the network returns
nonsense values for the badly constrained areas (there is no information nor supervision available), these areas are correctly
filtered out with the trust estimate. Best viewed digitally.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

Figure 4: Qualitative results for our approach on the MVSNet inputs, for all elements in the test set. For a given input image(a) we show first the ground truth image (b), the input estimate (c) and its error (d). The subsequent columns show the output
estimate, error and trust for three iterations. The error visualization works logarithmically: at error value 5, the colour iswhite. At infinite error, a pixel is shown as dark red and at zero error it is displayed a dark blue. While the network returns
nonsense values for the badly constrained areas (there is no information nor supervision available), these areas are correctly
filtered out with the trust estimate. Best viewed digitally.

Table 1: Quantitative evaluation for the COLMAP [2] and MVSNet [3] front-ends followed by COLMAP fusion. Using 12neighbours, selected using a mixed near-far strategy, and three refinement iterations. We report the numbers in the paper bothfor τd = 1 and τd = 2 (all values reported here are expected to be lower for lower τd ). For the distances in mm, lower isbetter; for the percentages, higher is better.
completeness (mm)chamfer (mm)accuracy (%)

completeness (%)

mean (%)

accuracy (mm)completeness (mm)chamfer (mm)accuracy (%)

completeness (%)

mean (%)

Threshold τd = 1.0
COLMAP [2]
COLMAP [2] + ours
MVSNet [3]MVSNet [3] + ours

full cloud

accuracy (mm)per view

0.51
0.41
0.45
0.39

0.68
0.63
0.66
0.62

0.59
0.52
0.55
0.50

29.5
61.8
29.5
66.4

24.4
49.1
23.5
24.8

26.9
55.4
26.5
45.6

0.40
0.42
0.39
0.40

0.69
0.38
0.41
0.37

0.55
0.40
0.40
0.38

59.4
69.4
76.7
74.4

36.7
65.5
74.0
65.5

48.1
67.9
75.4
70.0

completeness (mm)chamfer (mm)accuracy (%)

completeness (%)

mean (%)

accuracy (mm)completeness (mm)chamfer (mm)accuracy (%)

completeness (%)

mean (%)

Threshold τd = 2.0
COLMAP [2]
COLMAP [2] + ours
MVSNet [3]MVSNet [3] + ours

full cloud

accuracy (mm)per view

0.96
0.60
0.74
0.55

1.17
0.94
1.06
0.91

1.06
0.77
0.90
0.73

62.5
83.9
76.0
92.2

51.5
65.9
34.8
34.0

56.0
75.9
55.9
63.6

0.59
0.71
0.57
0.65

1.03
0.57
0.55
0.56

0.81
0.64
0.56
0.60

72.6
81.288.3
86.3

72.0
72.4
66.6
65.5

72.3
76.8
77.4
75.9

Table 2: Quantitative evaluation on the two synthetic datasets. For UnrealDTU, we pre-train on FlyingThingsMVS. Thevalues for taud were chosen so as to make the percentages discriminative.
completeness (%)

mean (%)

accuracy (%)

completeness (%)

mean (%)

COLMAP [2]
COLMAP [2] + ours
MVSNet [3]MVSNet [3] + ours
COLMAP [2]
COLMAP [2] + ours
MVSNet [3]MVSNet [3] + ours

full cloud

accuracy (%)τd = 0.02τd = 0.4

UnrealDTU FT MVSper view

8791
55
77
83
93
38
82

65
7350
6175
80
1268

76
82
52
69
79
86
2575

77
86
79
71
69
86
53
70

78
94
62
83
60
97
77
95

78
90
70
77
64
92
65
82

References[1] R. R. Jensen, A. L. Dahl, G. Vogiatzis, E. Tola, and H. Aanæs. Large scale multi-view stereopsis evaluation. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2014. 3[2] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. In Proc. ofthe European Conf. on Computer Vision (ECCV), 2016. 6[3] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan. Mvsnet: Depth inference for unstructured multi-view stereo. arXiv.org, abs/1804.02505,
2018. 6