Supplementary Material for
Learning Priors for Semantic 3D Reconstruction
Ian Cherabier1,? Johannes L. Schönberger1,∗ Martin R. Oswald1
Marc Pollefeys1,2 Andreas Geiger1,3
1

ETH Zürich

2

Microsoft

3

MPI-IS and University of Tübingen

Abstract. In this supplementary document, we provide further insights
and details about our novel multi-scale optimization scheme. In particular, we prove the adjointness of the transformation matrices of the primaland dual update steps of the unrolled optimization. Moreover, we detailthe upsampling operators of the multi-scale scheme and show further
examples of learned shape priors across different scale levels. Finally, we
present additional experimental results which emphasize the fact that
our method with unrolled optimization performs well when trained onlittle data, while traditional fully convolutional networks fail to reachcomparable accuracy levels when trained with the same amount of data.

1

Multi-scale Optimization Scheme

This section provides additional details of our proposed multi-scale optimization
scheme.
Derivation and Intuition. As described in the paper, the main energy in asingle-scale setting can be numerically minimized using the first-order primaldual (PD) algorithm in [1] with the following update steps:
Xūt` − 1
3. ut+1 = Π[0,1] ut + τ (W ∗ ξ t+1 − f )1. ν t+1 = ν t + σ`
2. ξ t+1 = Πk·k≤1 ξ t + σW ūt
4. ūt+1 = 2ut+1 − ut(1)
Step 1 updates the Lagrangian dual variable which ensures the simplex constraint on the objective. Step 4 is an extrapolation step which speeds up the
optimization and which is necessary to ensure convergence. The important stepsof the algorithm are steps 2 and 3 which represent a (projected) gradient descentin the primal variable and a (projected) gradient ascent in the dual variable.
Remember, that W is a generalized learned gradient operator. Therefore, agood way to get intuition on the behaviour of W is to think in terms of classic
gradient ascent (resp. descent) in step 2. (resp. step 3), without projection. Thatis, in step 2, the original gradient ascent scheme, without projection, was definedasξ t+1 = ξ t + σ∇ūt(2)?

These authors share first authorship.

2

I. Cherabier, J.L. Schönberger, M.R. Oswald, M. Pollefeys, A. Geiger

In the multi-scale setting we have multiple versions of variables at different
resolution scales s. The idea is to combine the gradient updates of the current
resolution with the ones from the coarser scale with a linear combination:sξst+1 = ξst + σ ∇ss ūts + Us+1
(3)
∇ss+1 ūts+1
where ∇ss is the gradient operator on the current resolution, ∇ss+1 is the gradientsoperator on the coarser scale and Us+1
is the upsampling operator in order tocombine the two terms at the current scale level. This combined gradient ascentscheme then follows the deepest ascent directions of both resolution levels at
the same time. Now, we can go back and replace the gradient operator with our
proposed generalized matrix W and reintroduce the projection to arrive at theequation we presented in the paper:hissξst+1 = Πk·k≤1 ξst + σ Wss ūts + Us+1
Ws+1ūts+1(4)Here, it is important to note that we learn the matrices W implicitly also theirscaling which defines the relative contribution of the two ascent steps. Without
the learned weighting of these terms this gradient ascent scheme is unlikely towork. The derivation of the primal update step with the gradient descent isanalogous.
The proposed multi-scale optimization incorporates information from coarser
resolution levels (if available) in order to perform the update step. This leads toa weighted sum of gradient-based updates for the current and the next coarser
resolution. In addition, the learned weights also regulate the step size of individual gradient descent and ascent steps which are crucial for algorithmic efficiency
and to ensure convergence. An example of learned weights for different scale levels is given in Figure 1. Note that due to the learned weights, we cannot give any
convergence guarantees for our multi-scale optimization. A necessary condition
for the convergence of the primal-dual algorithm is however the adjointness ofthe primal and dual updates which we show in the following.
Adjointness of Update Steps in Primal-Dual Algorithm. In order toapply the primal-dual algorithm [1], the minimization problem over the voxel
labeling u is first transformed into a saddle point problem by introducing a dual
variable which results from replacing the norm in the TV-regularization by its
Legendre-Fenchel dual. In a discretized setting, we obtain
kW uk1 = max hW u, ξi .
kξk∞ ≤1(5)

The algorithm further uses the adjoint W ∗ of matrix W to transfer the differential operator from the primal to the dual variable:hW u, ξi = hu, W ∗ ξi .

(6)This leads to the property that the primal and dual gradient terms in the update equations of the algorithm must be adjoint. For our novel multi-resolution

Supplementary Material for Learning Priors for Semantic 3D Reconstruction

3

scheme, this property is not obvious and we therefore show the adjointness of
the update steps in the following. The update steps for the multi-resolution
optimization in the main paper are as follows:hissξst+1 = Πk·k≤1 ξst + σ Wss ūts + Us+1
Ws+1ūts+1hists∗ t+1s∗ t+1
t+1
ut+1
=
Π
u
+
τ
W
ξ
+
U
W
ξ
+
τ
ν
−
f
[0,1]
s
s
s s
s+1 s+1 s+1(8)

ūt+1 = 2ut+1 − ut

(9)

(7)

s
where Us+1
is the upsampling operator from scale s + 1 to scale s in the spacesof dual variables, and U s+1 is the upsampling operator in the space of primal
variables. A necessary condition for viability of the primal-dual algorithm istherefore the following proposition.

Proposition 1 (Adjointness of Multi-scale Primal-dual Updates). The
transformation matrices of the multi-resolution primal dual updatesss
Wss ūts + Us+1
Ws+1ūts+1ss∗ t+1
Wss∗ ξst+1 + U s+1 Ws+1
ξs+1

and

(10)

are adjoint operators.

Proof. We first rewrite the dual update step as a single matrix vector multiplication by stacking the matrices and vectors

 t t Tss
Ws+1
Wss Us+1ūs ūs+1,=

 s
 t t Tss
Is Us+1
Ws Ws+1ūs ūs+1,=

 t t Ts
Wss Ws+1ūs ūs+1,=

 t t Tsūs ūs+1
Wss Ws+1,=ūts ūts+1

T,

 t+1 t+1 Ts
Is Us+1
ξs+1
ξs
(11)
 t+1 t+1 Ts
Is Us+1 ξs
ξs+1
(12)
∗
 t+1 t+1 Tss
Is Us+1 Is Us+1 ξs
ξs+1
(13)
 t+1 t+1 T
Is Is+1 ξs
ξs+1
(14)


t+1 T
s∗
Wss∗ Ws+1
ξst+1 ξs+1
(15)

where Is is the identity at scale s. The transitions from (13) to (14) is due tothe fact that since our upsampling uses a copy operator to go from scale s + 1 toscale s, its adjoint is the corresponding downsampling operator. We can conclude

4

I. Cherabier, J.L. Schönberger, M.R. Oswald, M. Pollefeys, A. Geiger

with a similar reasoning: t+1 t+1 Ts∗Wss∗ Ws+1
ξs
ξs+1
(16)




T
t+1 T
s∗
ūts ūts+1 ,Is Is+1 Wss∗ Ws+1
ξst+1 ξs+1
(17)





∗
T
s
s
t+1 T
s∗
ūts ūts+1 , Is U s+1 Is U s+1 Wss∗ Ws+1
ξst+1 ξs+1
(18)
 t t T
 s∗


s
s
t+1 T
s∗
ξst+1 ξs+1
Is U s+1 ūs ūs+1 ,Is U s+1 Ws Ws+1
(19)
 t t T


s
s
t+1 T
s∗
ξst+1 ξs+1
Wss∗ U s+1 Ws+1
Is U s+1 ūs ūs+1 ,(20)
Tūts ūts+1 ,

=
=
=
=

t
u

This proves the proposition.
ssUpsampling Operators. Note that the upsampling operators Us+1
, U s+1 operate on different dimensions and spaces, respectively in the dual space andin the primal space. As shown above, they are also adjoint operators and, in
our setting, the adjointness corresponds to a matrix transpose. For upsampling
the primal and dual variables, we duplicate the values of coarser voxels on the
−s
−s
s
∈
finer resolution level. Let the weight matrix be Wss ∈ R2 M ×2 N and Ws+1−(s+1)−(s+1)−s−(s+1)
M ×2
N
s
M
R2
, then the upsampling matrix Us+1
∈ R2 M ×2
is binary and contains ones only when voxel indices j in the coarse grid correspondto voxel indices i in the finer grid:(1 if i = upSampleIdx(j)s(Us+1 )ij =(21)
0 otherwise

For a given voxel index of a coarser resolution level, the function upSampleIdx(·)
returns the corresponding index in the upsampled, next finer voxel grid.
s
The second upsampling operator U s+1 is similarly defined with the difference−s−(s+1)
sN
being that U s+1 ∈ R2 N ×2.

Supplementary Material for Learning Priors for Semantic 3D Reconstruction
W10

W11

W21

W22

groundbuilding

W00

5

Fig. 1: Learned multi-scale 3D shape priors for label transitions to free space. The
influence of the current and successive finer resolution to the update of the primal anddual variables has similar magnitude (e.g., similar magnitude of W00 and W10 contributing to update of ut+1
and ξst+1 ), while the overall magnitude of the weights betweensdifferent levels are increasing with higher scales to account for the different resolutions(e.g., W00 < W01 < W02 ). Note that the step sizes of the primal-dual algorithm andthe weighting between data cost and regularization are factored into W . In contrast
to manually choosing these hyper-parameters as in traditional approaches, our method
learns the balancing of these parameters automatically from data.

2

Additional 2D Segmentation Results

free spacegroundbuilding
vegetation
roof

Reconstruction Accuracy [%]In Figure 3 we present additional qualitative results of 100
the 2D experiments we used to study and assess the behavior of our model. In this series of examples we can80
see the various benefits that our approach to segmentation gains over traditional TV-L1 approaches. The
first is the capacity to propagate information through 60
large region with missing data (see for instance example 3). A second useful benefit is that our method is 40
able to adapt the segmentation to the labels. In traditional TV-L1, the regularizer looks for the minimal 20
TV-L1 (1K it.)boundary between different labels. This may lead toOurs (50 it.)unrealistic segmentations. Our approach on the other 0
hand incorporates shape priors in its regularizer, whichleads to more accurate segmentations (see,e.g., image2). Finally, our method learns sophisticated label interactions and ordering such as the fact that a roof can
Fig. 2: Per-label aconly exist above a building (see image 6).
Figure 2 depicts a per-label accuracy plot for the 2D curacies for the sesemantic segmentation experiments on synthetic data. mantic segmentations
It demonstrates that our learned semantic priors are on synthetic images.
much more useful for scene completion and denoisingthan the isotropic TV prior which does account for geometric nor semantic pixel
neighborhood relationships.

6

I. Cherabier, J.L. Schönberger, M.R. Oswald, M. Pollefeys, A. Geiger

1

2

3

4

5

6

7

8

9

10

Ground Truth

Noisy

TV-L1 (5K iter.) Ours (20 iter., 3 levels)

Fig. 3: 2D semantic segmentations on synthetic images. Our method is capableof filling-in missing regions for shape completion. In case of image 6, our method isable to hallucinate a building below the observation of a roof, even though all data
supporting the building is missing. This demonstrates the strong label interactions
that our method is able to learn (roof can only be located on top of building).

Supplementary Material for Learning Priors for Semantic 3D Reconstruction

3

Additional 3D Reconstruction Results

3.1

Quantitative Results

7

Occupied

Semantic

Input data
TV-L1 (50 it.)
TV-L1 (500 it.)C2F (50 it.)Ours-5 (0 it.)Ours-5 (50 it.)Ours-300 (0 it.)Ours-300 (50 it.)Freespace

Methods

Overall

We present additional quantitative results for 3D reconstruction in Tab. 1. Morespecifically, we introduce some other baseline against which we compare ourmethod. First of all, we present accuracy results for a solution extracted fromthe input datacost (entry Input data in Tab. 1. We then trained our method
without unrolled optimization. This corresponds to a classic fully convolutional
network with alternating convolution layers and ReLU activation layers. We
trained this network on 300 scenes, but also on only 5 scenes which were selectedby choosing the 5 training scenes with the largest variety of present semanticlabels. The results are presented at the entry Ours-5 (0 it.) and Ours-300 (0it.) in Tab. 1. We can see that this fully convolutional approach obtains goodresults when trained on 300 scenes, with an overall accuracy of 97.3%, though
lower than our approach trained on 300 scenes. What is more interesting, isthat the resutls are considerably degraded when trained on only 5 scenes, withoverall accuracy of 65.1%, where our approach still reaches an overall accuracyof 96.7%. This illustrates how our method with unrolled optimization can learn
strong models from little data.

59.8
92.8
95.8
21.0
65.1
96.7
97.3
98.7

39.1
71.0
86.4
26.7
98.0
95.8
97.6
98.6

99.7
91.4
92.3
99.9
83.5
93.9
92.3
94.4

68.4
87.8
88.5
31.4
76.2
86.4
90.2
91.5

Table 1: 3D Reconstruction accuracy for ScanNet [2]. The table reproduces thetable from Fig. 7 in the paper, with the additional results for “Ours-5 (0 it.)” corresponding to our method without unrolled optimization trained on 5 scenes only. These
results show that the variational unrolling significantly helps the network to generalize
from very small amounts of training data and generally improves the overall performance.

3.2

Qualitative Results

In Figures 4, 5, 6, and 7 we show additional 3D reconstruction results for the
ScanNet [2] dataset and outdoor scenes used in [3]. The reconstructions of our
method are often more complete than the ground truth scene as our model learns

I. Cherabier, J.L. Schönberger, M.R. Oswald, M. Pollefeys, A. Geiger

Ours-5 (0 it.) Ours-5 (50 it.) Ours-300 (0 it.) Ours-300 (50 it.) TV-L1 (500 it.) Ground Truth

8

Fig. 4: 3D reconstruction results for ScanNet [2] for different types of scenes andmethods.

how to interpolate missing data, resulting in plausible and visually pleasing
semantic 3D reconstruction results compared to the incomplete scans provided
as ground truth. While classic fully convolutional network (our method without
unrolled optimization) work well when trained on all 300 scenes, a significant
loss in performance can be observed when trained on only 5 scenes. In Figure 7,
we compare results of our approach with and without unrolled optimization, andsee that the later helps with removing strong artifacts.
3.3

Application on SUNCG Data

We tested our method on the more challenging task of single view reconstruction
by applying it to the synthetic indoor dataset SUNCG [4]. We wish first to stressthat [4] solve a different problem than ours: they use a non-semantic TSDF as
input and tune their approach for single view reconstruction. Our approach on

9Ours-5 (0 it.)Ours-5 (50 it.)Ours-300 (0 it.) Ours (50 iters.) TV-L1 (500 iters.) Ground Truth

Supplementary Material for Learning Priors for Semantic 3D Reconstruction

Fig. 5: 3D reconstruction results for ScanNet [2] for different types of scenes andmethods.

the other hand requires a multi-label input datacost, and is inherently adapted
to a multi-view setting as it works by denoising and propagating data, rather
than hallucinating new structures. Since we have neither tuned nor designed our
method for such applications, this explains that the results are worse. We trained
our multi-scale approach for 50 iterations and 3 scales on 500 views from SUNCG
for 1770 epochs, and tested on 74 views. Results are presented in table 2.
The results support our intuition that our method is better suited for multiview 3D reconstruction. We can see especially that it is more difficult to obtain
good semantic accuracies.

I. Cherabier, J.L. Schönberger, M.R. Oswald, M. Pollefeys, A. Geiger

Ours-5 (0 it.)Ours-5 (50 it.)Ours-300 (0 it.) Ours (50 iters.)

TV-L1 (500 iters.) Ground Truth

10

Fig. 6: 3D reconstruction results for ScanNet [2] for different types of scenes andmethods.

Table 2: Accuracies of our network on SUNCG data [4]MethodOurs-500 (50it - 3 level)

Overall Freespace Occupied Semantic
82.2

84.0

90.1

70.5

11

Ours (50 it.)Ours (0 it.)

TV-L1 (1000 it.)

Input data cost

Ground Truth

Supplementary Material for Learning Priors for Semantic 3D Reconstruction

Fig. 7: 3D reconstruction results for Outdoor scenes. We use the results from [3]as a groundtruth, and learn how to reproduce them. The results for TV-L1 with 1000
iterations correspond to convergence case. These results show that the absence of shapepriors lead to incorrect reconstruction, such as the absence of ground. We can see thatthe results of our method without unrolled optimization can achieve good denoising,but with strong remaining artifacts for the left and right scene, or mislabeling in thecase of the middle scene. Our approach with unrolled optimization overcomes these
difficulties.

12

I. Cherabier, J.L. Schönberger, M.R. Oswald, M. Pollefeys, A. Geiger

References
1. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems
with applications to imaging. Journal of Mathematical Imaging and Vision (2011)
2. Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proc. Conference on
Computer Vision and Pattern Recognition (CVPR) (2017)
3. Häne, C., Zach, C., Cohen, A., Angst, R., Pollefeys, M.: Joint 3d scene reconstruction and class segmentation. In: Proc. Conference on Computer Vision and Pattern
Recognition (CVPR) (2013)
4. Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.: Semantic scene
completion from a single depth image. Proc. Conference on Computer Vision and
Pattern Recognition (CVPR) (2017)