Noname manuscript No.
(will be inserted by the editor)

Learning 3D Shape Completion under Weak Supervision
David Stutz · Andreas Geiger

Received: date / Accepted: date

Abstract We address the problem of 3D shape completion from sparse and noisy point clouds, a fundamental problem in computer vision and robotics. Recent approaches are either data-driven or learning-based: Datadriven approaches rely on a shape model whose parameters are optimized to fit the observations; Learningbased approaches, in contrast, avoid the expensive optimization step by learning to directly predict complete shapes from incomplete observations in a fullysupervised setting. However, full supervision is often
not available in practice. In this work, we propose aweakly-supervised learning-based approach to 3D shapecompletion which neither requires slow optimization nor
direct supervision. While we also learn a shape prior on
synthetic data, we amortize, i.e., learn, maximum likelihood fitting using deep neural networks resulting in
efficient shape completion without sacrificing accuracy.
On synthetic benchmarks based on ShapeNet (Changet al, 2015) and ModelNet (Wu et al, 2015) as well as
on real robotics data from KITTI (Geiger et al, 2012)
and Kinect (Yang et al, 2018), we demonstrate thatthe proposed amortized maximum likelihood approachis able to compete with the fully supervised baseline of
Dai et al (2017) and outperforms the data-driven approach of Engelmann et al (2016), while requiring lesssupervision and being significantly faster.

(a) ShapeNet (Synthetic)(b) KITTI (Real)(c) ModelNet (Synthetic)(d) Kinect (Real)

Fig. 1: 3D Shape Completion. Results for cars on
ShapeNet (Chang et al, 2015) and KITTI (Geiger et al,2012) and for chairs and tables on ModelNet (Wuet al, 2015) and Kinect (Yang et al, 2018). Learning
shape completion on real-world data is challenging due
to sparse and noisy observations and missing groundtruth. Occupancy grids (bottom) or meshes from signed
distance functions (SDFs, top) at various resolutions inbeige and point cloud observations in red.
Keywords 3D shape completion · 3D reconstruction ·weakly-supervised learning · amortized inference ·benchmark1 Introduction
David Stutz
Max Planck Institute for Informatics
Campus E1 4, 66123 Saarbrücken, Germany
Tel.: +49 681 9325 2021
E-mail: david.stutz@mpi-inf.mpg.deAndreas Geiger
Max Planck Institute for Intelligent Systems and Universityof Tübingen
Max-Planck-Ring 4, 72076 Tübingen, Germany

3D shape perception is a long-standing and fundamental problem both in human and computer vision (Pizlo, 2007, 2010; Furukawa and Hernandez, 2013) withmany applications to robotics. A large body of work
focuses on 3D reconstruction, e.g., reconstructing objects or scenes from one or multiple views, which is an
inherently ill-posed inverse problem where many configurations of shape, color, texture and lighting may

2David Stutz, Andreas Geiger(1) Shape Prior (Section 3.2)

(2) Shape Inference (Section 3.3)

Synthetic Reference Shapes

Real Observations w/o Targets
no correspondence needed

retain fixed decoder

encoderz

newencoder

decoder

Shape yRec. Shape ỹReconstruction Loss

z

fixed
decoder

Prop. Shape ỹObservation x
Maximum Likelihood Loss

Fig. 2: Amortized Maximum Likelihood (AML) for 3D Shape Completion on KITTI. (1) We train adenoising variational auto-encoder (DVAE) (Kingma and Welling, 2014; Im et al, 2017) as shape prior on ShapeNet
using occupancy grids and signed distance functions (SDFs) to represent shapes. (2) The fixed generative model,i.e., decoder, then allows to learn shape completion using an unsupervised maximum likelihood (ML) loss bytraining a new recognition model, i.e., encoder. The retained generative model constraints the space of possible
shapes while the ML loss aligns the predicted shape with the observations.
result in the very same image. While the primary goalof human vision is to understand how the human visualsystem accomplishes such tasks, research in computer
vision and robotics is focused on the task of devising
3D reconstruction systems. Generally, work by Pizlo(2010) suggests that the constraints and priors used for3D perception are innate and not learned. Similarly, incomputer vision, cues and priors are commonly built
into 3D reconstruction pipelines through explicit assumptions. Recently, however – leveraging the successof deep learning – researchers started to learn shapemodels from large collections of data, as for example
ShapeNet (Chang et al, 2015). Predominantly generative models have been used to learn how to generate,
manipulate and reason about 3D shapes (Girdhar et al,
2016; Brock et al, 2016; Sharma et al, 2016; Wu et al,
2016b, 2015).
In this paper, we focus on the specific problem of inferring and completing 3D shapes based on sparse andnoisy 3D point observations as illustrated in Fig. 1. This
problem occurs when only a single view of an individual object is provided or large parts of the object are
occluded as common in robotic applications. For example, autonomous vehicles are commonly equipped with
LiDAR scanners providing a 360 degree point cloud of
the surrounding environment in real-time. This point
cloud is inherently incomplete: back and bottom of objects are typically occluded and – depending on material properties – the observations are sparse and noisy,
see Fig. 1 (top-right) for an illustration. Similarly, indoor robots are generally equipped with low-cost, realtime RGB-D sensors providing noisy point clouds ofthe observed scene. In order to make informed decisions(e.g., for path planning and navigation), it is of utmost
importance to efficiently establish a representation of
the environment which is as complete as possible.
Existing approaches to 3D shape completion can be
categorized into data-driven and learning-based methods. The former usually rely on learned shape priors
and formulate shape completion as an optimization problem over the corresponding (lower-dimensional) latent
space (Rock et al, 2015; Haene et al, 2014; Li et al,2015; Engelmann et al, 2016; Nan et al, 2012; Bao et al,2013; Dame et al, 2013; Nguyen et al, 2016). These approaches have demonstrated good performance on realdata, e.g., on KITTI (Geiger et al, 2012), but are often
slow in practice.
Learning-based approaches, in contrast, assume afully supervised setting in order to directly learn shapecompletion on synthetic data (Riegler et al, 2017a; Smith
and Meger, 2017; Dai et al, 2017; Sharma et al, 2016;
Fan et al, 2017; Rezende et al, 2016; Yang et al, 2018;
Wang et al, 2017; Varley et al, 2017; Han et al, 2017).
They offer advantages in terms of efficiency as prediction can be performed in a single forward pass, however,require full supervision during training. Unfortunately,even multiple, aggregated observations (e.g., from multiple views) will not be fully complete due to occlusion,sparse sampling of views and noise, see Fig. 14 (rightcolumn) for an example.
In this paper, we propose an amortized maximum
likelihood approach for 3D shape completion (cf. Fig. 2)
avoiding the slow optimization problem of data-driven
approaches and the required supervision of learning-Learning 3D Shape Completion under Weak Supervision

based approaches. Specifically, we first learn a shapeprior on synthetic shapes using a (denoising) variational
auto-encoder (Im et al, 2017; Kingma and Welling, 2014).
Subsequently, 3D shape completion can be formulated
as a maximum likelihood problem. However, insteadof maximizing the likelihood independently for distinct
observations, we follow the idea of amortized inference(Gershman and Goodman, 2014) and learn to predictthe maximum likelihood solutions directly. Towards thisgoal, we train a new encoder which embeds the observations in the same latent space using an unsupervised maximum likelihood loss. This allows us to learn
3D shape completion in challenging real-world situations, e.g., on KITTI, and obtain sub-voxel accurate
results using signed distance functions at resolutions
up to 643 voxels. For experimental evaluation, we introduce two novel, synthetic shape completion benchmarks based on ShapeNet and ModelNet (Wu et al,2015). We compare our approach to the data-driven
approach by Engelmann et al (2016), a baseline inspired by Gupta et al (2015) and the fully-supervised
learning-based approach by Dai et al (2017); we additionally present experiments on real data from KITTI
and Kinect (Yang et al, 2018). Experiments show that
our approach outperforms data-driven techniques andrivals learning-based techniques while significantly reducing inference time and using only a fraction of supervision.
A preliminary version of this work has been published at CVPR’18 (Stutz and Geiger, 2018). However,
we improved the proposed shape completion method,
the constructed datasets and present more extensive
experiments. In particular, we extended our weaklysupervised amortized maximum likelihood approach toenforce more variety and increase visual quality significantly. On ShapeNet and ModelNet, we use volumetric fusion to obtain more detailed, watertight meshes
and manually selected – per object-category – 220 highquality models to synthesize challenging observations.
We additionally increased the spatial resolution andconsider two additional baselines (Dai et al, 2017; Gupta
et al, 2015). Our code and datasets will be made publicly available1 .
The paper is structured as follows: We discuss related work in Section 2. In Section 3 we introduce theweakly-supervised shape completion problem and describe the proposed amortized maximum likelihood approach. Subsequently, we introduce our synthetic shapecompletion benchmarks and discuss the data preparation for KITTI and Kinect in Section 4.1. Next, we discuss evaluation in Section 4.2, our training procedure in1

https://avg.is.tuebingen.mpg.de/research_
projects/3d-shape-completion.

3

Section 4.3, and the evaluated baselines in Section 4.4.
Finally, we present experimental results in Section 4.5
and conclude in Section 5.

2 Related Work
2.1 3D Shape Completion and Single-View 3D
Reconstruction
In general, 3D shape completion is a special case of
single-view 3D reconstruction where we assume point
cloud observations to be available, e.g. from laser-based
sensors as on KITTI (Geiger et al, 2012).
3D Shape Completion: Following Sung et al (2015),
classical shape completion approaches can roughly be
categorized into symmetry-based methods and datadriven methods. The former leverage observed symmetry to complete shapes; representative works include(Thrun and Wegbreit, 2005; Pauly et al, 2008; Zheng
et al, 2010; Kroemer et al, 2012; Law and Aliaga, 2011).
Data-driven approaches, in contrast, as pioneered by
Pauly et al (2005), pose shape completion as retrieval
and alignment problem. While Pauly et al (2005) allow
shape deformations, Gupta et al (2015), use the iterative closest point (ICP) algorithm (Besl and McKay,
1992) for fitting rigid shapes. Subsequent work usually avoids explicit shape retrieval by learning a latent
space of shapes (Rock et al, 2015; Haene et al, 2014;
Li et al, 2015; Engelmann et al, 2016; Nan et al, 2012;
Bao et al, 2013; Dame et al, 2013; Nguyen et al, 2016).
Alignment is then formulated as optimization problem
over the learned, low-dimensional latent space. For example, Bao et al (2013) parameterize the shape priorthrough anchor points with respect to a mean shape,
while Engelmann et al (2016) and Dame et al (2013) directly learn the latent space using principal component
analysis and Gaussian process latent variable models(Prisacariu and Reid, 2011), respectively. In these cases,
shapes are usually represented by signed distance functions (SDFs). Nguyen et al (2016) use 3DShapeNets(Wu et al, 2015), a deep belief network trained on occupancy grids, as shape prior. In general, data-driven
approaches are applicable to real data assuming knowledge about the object category. However, inference involves a possibly complex optimization problem, which
we avoid by amortizing, i.e., learning, the inference procedure. Additionally, we also consider multiple object
categories.
With the recent success of deep learning, several
learning-based approaches have been proposed (Firman
et al, 2016; Smith and Meger, 2017; Dai et al, 2017;
Sharma et al, 2016; Rezende et al, 2016; Fan et al, 2017;

4

Riegler et al, 2017a; Han et al, 2017; Yang et al, 2017,2018). Strictly speaking, these are data-driven, as well;
however, shape retrieval and fitting are both avoided bydirectly learning shape completion end-to-end, underfull supervision – usually on synthetic data from ShapeNet (Chang et al, 2015) or ModelNet (Wu et al, 2015).
Riegler et al (2017a) additionally leverage octrees topredict higher-resolution shapes; most other approaches
use low resolution occupancy grids (e.g., 323 voxels). Instead, Han et al (2017) use a patch-based approach toobtain high-resolution results. In practice, however, fullsupervision is often not available; thus, existing models
are primarily evaluated on synthetic datasets. In orderto learn shape completion without full supervision, we
utilize a learned shape prior to constrain the space ofpossible shapes. In addition, we use SDFs to obtain subvoxel accuracy at higher resolutions (up to 48×108×48
or 643 voxels) without using patch-based refinement oroctrees. We also consider significantly sparser observations.

Single-View 3D Reconstruction: Single-view 3D
reconstruction has received considerable attention overthe last years; we refer to (Oswald et al, 2013) for an
overview and focus on recent deep learning approaches,
instead. Following Tulsiani et al (2018), these can be
categorized by the level of supervision. For example,(Girdhar et al, 2016; Choy et al, 2016; Wu et al, 2016b;
Häne et al, 2017) require full supervision, i.e., pairs ofimages and ground truth 3D shapes. These are generally derived synthetically. More recent work (Yan et al,
2016; Tulsiani et al, 2017, 2018; Kato et al, 2017; Linet al, 2017; Fan et al, 2017; Tatarchenko et al, 2017; Wu
et al, 2016a), in contrast, self-supervise the problem by
enforcing consistency across multiple input views. Tulsiani et al (2018), for example, use a differentiable ray
consistency loss; and in (Yan et al, 2016; Kato et al,
2017; Lin et al, 2017), differentiable rendering allows
to define reconstruction losses on the images directly.
While most of these approaches utilize occupancy grids,
Fan et al (2017) and Lin et al (2017) predict point
clouds instead. Tatarchenko et al (2017) use octrees topredict higher-resolution shapes. Instead of employing
multiple views as weak supervision, however, we do notassume any additional views in our approach. Instead,knowledge about the object category is sufficient. In
this context, concurrent work by Gwak et al (2017) is
more related to ours: a set of reference shapes implicitly
defines a prior of shapes which is enforced using an adversarial loss. In contrast, we use a denoising variational
auto-encoder (DVAE) (Kingma and Welling, 2014; Im
et al, 2017) to explicitly learn a prior for 3D shapes.

David Stutz, Andreas Geiger

2.2 Shape Models
Shape models and priors found application in a wide variety of different tasks. In 3D reconstruction, in general,shape priors are commonly used to resolve ambiguities
or specularities (Dame et al, 2013; Gney and Geiger,
2015; Kar et al, 2015). Furthermore, pose estimation(Sandhu et al, 2011, 2009; Prisacariu et al, 2013; Aubryet al, 2014), tracking (Ma and Sibley, 2014; Leotta andMundy, 2009), segmentation (Sandhu et al, 2011, 2009;
Prisacariu et al, 2013), object detection (Zia et al, 2013,2014; Pepik et al, 2015; Song and Xiao, 2014; Zheng
et al, 2015) or recognition (Lin et al, 2014) – to namejust a few – have been shown to benefit from shape models. While most of these works use hand-crafted shapemodels, for example based on anchor points or part
annotations (Zia et al, 2013, 2014; Pepik et al, 2015;
Lin et al, 2014), recent work (Liu et al, 2017; Sharma
et al, 2016; Girdhar et al, 2016; Wu et al, 2016b, 2015;
Smith and Meger, 2017; Nash and Williams, 2017; Liuet al, 2017) has shown that generative models such as
VAEs (Kingma and Welling, 2014) or generative adversarial networks (GANs) (Goodfellow et al, 2014) allow
to efficiently generate, manipulate and reason about 3D
shapes. We use these more expressive models to obtainhigh-quality shape priors for various object categories.

2.3 Amortized Inference
To the best of our knowledge, the notion of amortized
inference was introduced by Gershman and Goodman(2014) and picked up repeatedly in different contexts(Rezende and Mohamed, 2015; Wang et al, 2016; Ritchie
et al, 2016). Generally, it describes the idea of learning
to infer (or learning to sample). We refer to (Wanget al, 2016) for a broader discussion of related work.
In our context, a VAE can be seen as specific exampleof learned variational inference (Kingma and Welling,
2014; Rezende and Mohamed, 2015). Besides using a
VAE as shape prior, we also amortize the maximum
likelihood problem corresponding to our 3D shape completion task.

3 Method
In the following, we introduce the mathematical formulation of the weakly-supervised 3D shape completion problem. Subsequently, we briefly discuss denoising variational auto-encoders (DVAEs) (Kingma andWelling, 2014; Im et al, 2017) which we use to learna strong shape prior that embeds a set of referenceLearning 3D Shape Completion under Weak Supervision(a) Reference Shapes Y(b) Observation xn∗(c) Ground Truth yn

Fig. 3: Weakly-Supervised Shape Completion.
Given reference shapes Y and incomplete observations
X , we want to learn a mapping xn 7→ ỹ(xn ) such thatỹ(xn ) matches the unknown ground truth shape yn∗ asclose as possible. The observations xn are split into freespace (i.e., xn,i = 0, right) and point observations (i.e.,
xn,i = 1, left). Shapes are shown in beige and observations in red.
shapes in a low-dimensional latent space. Then, we formally derive our proposed amortized maximum likelihood (AML) approach. Here, we use maximum likelihood to learn an embedding of the observations within
the same latent space – thereby allowing to perform
shape completion. The overall approach is also illustrated in Fig. 2.

5On real data, e.g., KITTI (Geiger et al, 2012), supervised learning is often not possible as obtaining groundtruth annotations is labor intensive, cf. (Menze andGeiger, 2015; Xie et al, 2016). Therefore, we target aweakly-supervised variant of the problem instead: Givenobservations X and reference shapes Y = {ym }M
m=1 ⊆RR both of the same, known object category, learn amapping xn 7→ ỹ(xn ) such that the predicted shapeỹ(xn ) matches the unknown ground truth shape yn∗ asclose as possible – or, in practice, the sparse observation
xn while being plausible considering the set of reference
shapes, cf. Fig. 3. Here, supervision is provided in theform of the known object category. Alternatively, thereference shapes Y can also include multiple object categories resulting in an even weaker notion of supervision
as the correspondence between observations and object
categories is unknown. Except for the object categories,
however, the set of reference shapes Y, and its size M ,is completely independent of the set of observations X ,and its size N , as also highlighted in Fig. 2. On realdata, e.g., KITTI, we additionally assume the object
locations to be given in the form of 3D bounding boxes
in order to extract the corresponding observations X .
In practice, the reference shapes Y are derived from watertight, triangular meshes, e.g., from ShapeNet (Changet al, 2015) or ModelNet (Wu et al, 2015).

3.2 Shape Prior
3.1 Problem Formulation
In a supervised setting, the task of 3D shape completion can be described as follows: Given a set of incomR
plete observations X = {xn }N
n=1 ⊆ R and correspond∗∗ N
ing ground truth shapes Y = {yn }n=1 ⊆ RR , learn
a mapping xn 7→ yn∗ that is able to generalize to previously unseen observations and possibly across object
categories. We assume RR to be a suitable representation of observations and shapes; in practice, we resort tooccupancy grids and signed distance functions (SDFs)
defined on regular grids, i.e., xn , yn∗ ∈ RH×W ×D ' RR .
Specifically, occupancy grids indicate occupied space,∗i.e., voxel yn,i
= 1 if and only if the voxel lies on or inside the shape’s surface. To represent shapes with subvoxel accuracy, SDFs hold the distance of each voxel’s
center to the surface; for voxels inside the shape’s surface, we use negative sign. Finally, for the (incomplete)observations, we write xn ∈ {0, 1, ⊥}R to make missing information explicit; in particular, xn,i = ⊥ corresponds to unobserved voxels, while xn,i = 1 and xn,i =0 correspond to occupied and unoccupied voxels, respectively.

We approach the weakly-supervised shape completion
problem by first learning a shape prior using a denoising variational auto-encoder (DVAE). Later, this priorconstrains shape inference (see Section 3.3) to predict
reasonable shapes. In the following, we briefly discuss
the standard variational auto-encoder (VAE), as introduced by Kingma and Welling (2014), as well as its
denoising extension, as proposed by Im et al (2017).
Variational Auto-Encoder (VAE): We propose to
use the provided reference shapes Y to learn a generative model of possible 3D shapes over a low-dimensional
latent space Z = RQ , i.e., Q  R. In the framework of
VAEs, the joint distribution p(y, z) of shapes y and latent codes z decomposes into p(y|z)p(z) with p(z) beinga unit Gaussian, i.e., N (z; 0, IQ ) and IQ ∈ RR×R being
the identity matrix. This decomposition allows to sample z ∼ p(z) and y ∼ p(y|z) to generate random shapes.
For training, however, we additionally need to approximate the posterior p(z|y). To this end, the so-calledrecognition model q(z|y) ≈ p(z|y) takes the formq(z|y) = N (z; µ(y), diag(σ 2 (y)))

(1)

6

David Stutz, Andreas Geiger

where µ(y), σ 2 (y) ∈ RQ are predicted using the encoder neural network. The generative model p(y|z) decomposes over voxels yi ; the corresponding probabilitiesp(yi |z) are represented using Bernoulli distributions foroccupancy grids or Gaussian distributions for SDFs:
p(yi |z) = Ber(yi ; θi (z))

or2p(yi |z) = N (yi ; µi (z), σ ).

(2)

In both cases, the parameters, i.e., θi (z) or µi (z), arepredicted using the decoder neural network. For SDFs,
we explicitly set σ 2 to be constant (see Section 4.3).
Then, σ 2 merely scales the corresponding loss, thereby
implicitly defining the importance of accurate SDFs relative to occupancy grids as described below.
In the framework of variational inference, the parameters of the encoder and the decoder neural networks are found by maximizing the likelihood p(y). In
practice, the likelihood is usually intractable and the evidence lower bound is maximized instead, see (Kingma
and Welling, 2014; Blei et al, 2016). This results in thefollowing loss to be minimized:
LVAE (w) = −Eq(z|y) [ln p(y|z)] + KL(q(z|y)|p(z)).

(3)

Here, w are the weights of the encoder and decoder hidden in the recognition model q(z|y) and the generativemodel p(y|z), respectively. The Kullback-Leibler divergence KL can be computed analytically as described in
the appendix of (Kingma and Welling, 2014). The negative log-likelihood − ln p(y|z) corresponds to a binary
cross-entropy error for occupancy grids and a scaled
sum-of-squared error for SDFs. The loss LVAE is minimized using stochastic gradient descent (SGD) by approximating the expectation using samples:−Eq(z|y) [ln p(y|z)] ≈ −LX

ln p(y|z (l) )

(4)

l=1

The required samples z (l) ∼ q(z|y) are computed using
the so-called reparameterization trick,z (l) = µ(y) + (l) σ(y)with (l) ∼ N (; 0, IQ ),(5)

in order to make LVAE , specifically the sampling process, differentiable. In practice, we found L = 1 samples to be sufficient – which conforms with results by
Kingma and Welling (2014). At test time, the sampling
process z ∼ q(z|y) is replaced by the predicted meanµ(y). Overall, the standard VAE allows us to embed
the reference shapes in a low-dimensional latent space.
In practice, however, the learned prior might still include unreasonable shapes.
Denoising VAE (DVAE): In order to avoid inappropriate shapes to be included in our shape prior, we consider a denoising variant of the VAE allowing to obtaina tighter bound on the likelihood p(y). More specifically, a corruption process y 0 ∼ p(y 0 |y) is considered
and the corresponding evidence lower bound results inthe following loss:
LDVAE (w) = − Eq(z|y0 ) [ln p(y|z)]+ KL(q(z|y 0 )|p(z)).

(6)

Note that the reconstruction error − ln p(y|z) is still
computed with respect to the uncorrupted shape y whilez, in contrast to Eq. (3), is sampled conditioned on thecorrupted shape y 0 . In practice, the corruption processp(y 0 |y) is modeled using Bernoulli noise for occupancy
grids and Gaussian noise for SDFs. In experiments,
we found DVAEs to learn more robust latent spaces– meaning the prior is less likely to contain unreasonable shapes. In the following, we always use DVAEs as
shape priors.

3.3 Shape Inference
After learning the shape prior, defining the joint distribution p(y, z) of shapes y and latent codes z as product of generative model p(y|z) and prior p(z), shapecompletion can be formulated as a maximum likelihood(ML) problem for p(y, z) over the lower-dimensional latent space Z = RQ . The corresponding negative loglikelihood − ln p(y, z) to be minimized can be writtenasX
LML (z) = −ln p(yi = xi |z) − ln p(z).
(7)
xi 6=⊥As the prior p(z) is Gaussian, the negative log-probability− ln p(z) is proportional to kzk22 and constrains the problem to likely, i.e., reasonable, shapes with respect tothe shape prior. As before, the generative model p(y|z)
decomposes over voxels; here, we can only consider actually observed voxels xi 6= ⊥. We assume that thelearned shape prior can complete the remaining, unobserved voxels xi = ⊥. Instead of solving Eq. (7) for eachobservation x ∈ X independently, however, we follow
the idea of amortized inference (Gershman and Goodman, 2014) and train a new encoder z(x; w) to learn
ML. To this end, we keep the generative model p(y|z)
fixed and train only the weights w of the new encoderz(x; w) using the ML objective as loss:
X
LdAML (w) = −ln p(yi = xi |z(x; w))xi 6=⊥(8)− λ ln p(z(x; w)).
Here, λ controls the importance of the shape prior. The
exact form of the probabilities p(yi = xi |z) depends on

Learning 3D Shape Completion under Weak Supervision

7

the used shape representation. For occupancy grids, thisterm results in a cross-entropy error as both the predicted voxels yi and the observations xi are, for xi 6= ⊥,binary. For SDFs, however, the term is not well-definedas p(yi |z) is modeled with a continuous Gaussian distribution, while the observations xi are binary. As solution, we could compute (signed) distance values along
the rays corresponding to observed points (e.g., following (Steinbrucker et al, 2013)) in order to obtain continuous observations xi ∈ R for xi 6= ⊥. However, as
illustrated in Fig. 4, noisy observations cause the distance values along the whole ray to be invalid. This canpartly be avoided when relying only on occupancy torepresent the observations; in this case, free space (cf.
Fig. 3) observations are partly correct even though observed points may lie within the corresponding shapes.
For making SDFs tractable (i.e., to predict subvoxel accurate, visually smooth and appealing shapes,
see Section 4.5) while using binary observations, we propose to define p(yi = xi |z) through a simple transformation. In particular, as p(yi |z) is modeled using
a Gaussian distribution N (yi ; µi (z), σ 2 ) where µi (z) is
predicted using the fixed decoder (σ 2 is constant), andxi is binary (for xi 6= ⊥), we introduce a mapping
θi (µi (z)) transforming the predicted mean SDF value
to an occupancy probability θi (µi (z)):
p(yi = xi |z) = Ber(yi = xi ; θi (µi (z)))(9)

As, by construction (see Section 3.1), occupied voxels
have negative sign or value zero in the SDF, we canderive the occupancy probability θi (µi (z)) as the probability of a non-positive distance:
θi (µi (z)) = N (yi ≤ 0; µi (z), σ 2 )1−µi (z)√=.
1 + erf
2
σ π

(10)
(11)

Here, erf is the error function which, in practice, can beapproximated following (Abramowitz, 1974). Eq. (11)is illustrated in Fig. 4 where the occupancy probability
θi (µi (z)) is computed as the area under the Gaussianbell curve for yi ≤ 0. This per-voxel transformation
can easily be implemented as non-linear layer and its
derivative wrt. µi (z) is, by construction, a Gaussian.
Note that the transformation is correct, not approximate, based on our model assumptions and the definitions in Section 3.1. Overall, this transformation allows
us to easily minimize Eq. (8) for both occupancy grids
and SDFs using binary observations. The obtained encoder embeds the observations in the latent shape spaceto perform shape completion.

occ
sdf
1
−3.8
1
−2.8
1
−1.8
1
−0.8
0
0.2
0
1.2
0
2.2

sdf

occ

⊥

⊥

1

0

2

0

p(yi )p(y 0 ≤ yi )ip(yi ≤ 0)
1

3

0

0.8

4

0

0.6

5

0

6

0

surface 7
ray
(a)
(b)

θi (µi (z))

0.4µi (z)

0.2
0

0

(c)

−3 −2 −1yi01Fig. 4: Left: Problem with SDF Observations. Illustration of a ray (red line) correctly hitting a surface (blue line) causing the (signed) distance values
and occupancy values computed for voxels along the
ray to be correct (cf. (a)). A noisy ray, however, causes
all voxels along the ray to be assigned incorrect distance values (marked red ) wrt. to the true surface(blue line) because the ray ends far behind the actual
surface (cf. (b)). When using occupancy only, in contrast, only the voxels behind the surface are assigned
invalid occupancy states (marked red ); the remaining
voxels are labeled correctly (marked green ; cf. (c)).
Right: Proposed Gaussian-to-Bernoulli Transformation. For p(yi ) := p(yi |z) = N (yi ; µi (z), σ 2 )(blue), we illustrate the transformation discussed in
Section 3.3 allowing to use the binary observations xi(for xi 6= ⊥) to supervise the SDF predictions. This isachieved by transforming the predicted Gaussian distribution to a Bernoulli distribution with occupancy probability θi (µi (z)) = p(yi ≤ 0) (blue area).
3.4 Practical Considerations
Encouraging Variety: So far, our AML formulation
assumes a deterministic encoder z(x, w) which predicts,
given the observation x, a single code z corresponding
to a completed shape. A closer look at Eq. (8), however,reveals an unwanted problem: the data term scales with
the number of observations, i.e., |{xi 6= ⊥}|, while theregularization term stays constant – with less observations, the regularizer gains in importance leading tolimited variety in the predicted shapes because z(x; w)
tends towards zero.
In order to encourage variety, we draw inspirationfrom the VAE shape prior. Specifically, we use a probabilistic recognition modelq(z|x) = N (z; µ(x), diag(σ 2 (x)))(12)(cf. see Eq. (1)) and replace the negative log-likelihood− ln p(z) with the corresponding Kullback-Leibler divergence KL(q(z|x)|p(z)) with p(z) = N (z; 0, IQ ). Intuitively, this makes sure that the encoder’s predictions

8David Stutz, Andreas Geiger(a) Original(b) TSDF Fusion, 2563

(c) Simplification, 5k Faces(d) Reconstruction, 24×54×24/323(e) Observations(f) Voxelization, 24×54×24/323

Fig. 5: ShapeNet and ModelNet Data Generation Pipeline. On ShapeNet and ModelNet we illustrate:(a) samples from the original datasets; (b) fused watertight meshes from TSDF fusion at 2563 voxels resolution
using (Riegler et al, 2017a); (c) simplified meshes (5k faces); (d) marching cubes (Lorensen and Cline, 1987)
reconstructions from the SDFs computed from (c) (resolutions 24×54×24 and 323 voxels; note that steps (b)and (c) are necessary to derive exact SDFs); (e) observations obtained by projection into a single view; and (f)voxelized observations and shapes. Shapes (meshes and occupancy grids) in beige and observations in red.
“cover” the prior distribution – thereby enforcing variety. Mathematically, the resulting loss, i.e.,X
ln p(yi = xi |z)LAML (w) = − Eq(z|x) (13)
xi 6=⊥+ λKL(q(z|x)p(z)),
can be interpreted as the result of maximizing the evidence lower bound of a model with observation processp(x|y) (analogously to the corruption process p(y 0 |y)for DVAEs in (Im et al, 2017) and Section 3.2). The expectation is approximated using samples (following thereparameterization trick in Eq. (5)) and, during testing, the sampling process z ∼ q(z|x) is replaced by themean prediction µ(x). In practice, we find that Eq. (13)
improves visual quality of the completed shapes. We
compare this AML model to its deterministic variant
dAML in Section 4.5.
Handling Noise: Another problem of our AML formulation concerns noise. On KITTI, for example, specular or transparent surfaces cause invalid observations– laser rays traversing through these surfaces cause observations to lie within shapes or not get reflected. However, our AML framework assumes deterministic, i.e.,trustworthy, observations – as can be seen in the reconstruction error in Eq. (13). Therefore, we introduce pervoxel weights κi computed using the reference shapes
Y = {ym }M
m=1 :
!
M
1 X
κi = 1 −ym,i ∈ [0, 1]
(14)
M m=1
where ym,i = 1 if and only if the corresponding voxel
is occupied. Applied to observations xi = 0, these aretrusted less if they are unlikely under the shape prior.

Note that for point observations, i.e., xi = 1, this is notnecessary as we explicitly consider “filled” shapes (seeSection 4.1). This can also be interpreted as imposing
an additional mean shape prior on the predicted shapeswith respect to the observed free space. In addition, we
use a corruption process p(x0 |x) consisting of Bernoulli
and Gaussian noise during training (analogously to the
DVAE shape prior).

4 Experiments
4.1 Data
We briefly introduce our synthetic shape completion
benchmarks, derived from ShapeNet (Chang et al, 2015)
and ModelNet (Wu et al, 2015) (cf. Fig. 5), and our datapreparation for KITTI (Geiger et al, 2012) and Kinect(Yang et al, 2018) (cf. Fig. 6); Table 1 summarizes key
statistics including the level of supervision computed
as the fraction of observed voxels, i.e. |{xn,i 6=⊥}|/HW D,
averaged over observations xn .
ShapeNet: We utilize the truncated SDF (TSDF) fusion approach of Riegler et al (2017a) to obtain watertight versions of the provided car shapes allowing toreliably and efficiently compute occupancy grids andSDFs. Specifically, we use 100 depth maps of 640×640
pixels resolution, distributed uniformly on the spherearound the shape, and perform TSDF fusion at a resolution of 2563 voxels. Detailed watertight meshes, without
inner structures, can then be extracted using marching cubes (Lorensen and Cline, 1987) and simplified to
5k faces using MeshLab’s quadratic simplification algorithm (Cignoni et al, 2008), see Fig. 5a to c. Finally, we
manually selected 220 shapes from this collection, re-Learning 3D Shape Completion under Weak Supervision

9
Synthetic
SN-clean/-noisy ModelNet

Real
KITTI
KinectTraining/Test Sets
#Shapes for Shape Prior, #Views for Shape Inference#Shapes#Views500/100
5000/1000

1000/200––
10000/2000 8442/9194 30/10

Observed Voxels in % (< 5%) & Resolutions
Low = 24×54×24/323 ; Medium = 32×72×32/483 ; High = 48×108×48/643(a) KITTI, Point Clouds(b) Kinect, Occupancy Grids

Fig. 6: Extracted KITTI and Kinect Data. For
KITTI, we show observed points in red and the accumulated, partial ground truth in green. Note that for
the first example ground truth is not available due tomissing past/future observations. For Kinect, we show
observations in red and ElasticFusion (Whelan et al,2015) ground truth in beige. Note that the objects arerotated and not aligned as in ModelNet (cf. Fig. 5).
moving exotic cars, unwanted configurations, or shapeswith large holes (e.g., missing floors or open windows).
The shapes are splitted into |Y| = 100 reference
shapes, |Y ∗ | = 100 shapes for training the inference
model, and 20 test shapes. We randomly perturb rotation and scaling to obtain 5 variants of each shape, voxelize them using triangle-voxel intersections and subsequently “fill” the obtained volumes using a connectedcomponents algorithm (Jones et al, 2001). For computing SDFs we use SDFGen2 . We use three different
resolutions: H×W ×D = 24×54×24, 32×72×32 and48×108×48 voxels. Examples are shown in Fig. 5d to f.
Finally, we use the OpenGL renderer of Gney andGeiger (2015) to obtain 10 depth maps per shape. The
incomplete observations X are obtained by re-projecting
them into 3D and marking voxels with at least one point
as occupied and voxels between occupied voxels and thecamera center as free space. We obtain more dense point
clouds at 48×64 pixels resolution and sparser point
clouds using depth maps of 24×32 pixels resolution. For
the latter, more challenging case we also add exponentially distributed noise (with rate parameter 70) to thedepth values, or randomly (with probability 0.075) set
them to the maximum depth to simulate the deficiencies of point clouds captured with real sensors, e.g., on
KITTI. These two variants are denoted SN-clean andSN-noisy. The obtained observations are illustrated inFig. 5e.
KITTI: We extract observations from KITTI’s Velodyne point clouds using the provided ground truth 3D
bounding boxes to avoid the inaccuracies of 3D object
detectors (train/test split by Chen et al (2016)). As the
3D bounding boxes in KITTI fit very tightly, we first2

https://github.com/christopherbatty/SDFGen.

Low
MediumHigh

7.66/3.86
6.1/2.13
2.78/0.93

9.71
8.74
8.28

6.79
5.24
3.44

0.87
–
–

Table 1: Dataset Statistics. We report the number of(rotated and scaled) meshes, used as reference shapes,
and the resulting number of observations (i.e., views,
10 per shape). We also report the average fraction ofobserved voxels, i.e., |{xi 6=⊥}|/HW D. For ModelNet, we
exemplarily report statistics for chairs; and for Kinect,
we report statistics for tables.

padded them by factor 0.25 on all sides; afterwards, theobserved points are voxelized into voxel grids of size
H×W ×D = 24×54×24, 32×72×32 and 48×108×48
voxels. To avoid taking points from the street, nearby
walls, vegetation or other objects into account, we only
consider those points lying within the original (i.e., notpadded) bounding box. Finally, free space is computed
using ray tracing as described above. We filter all observations to ensure that each observation contains aminimum of 50 observations. For the bounding boxes inthe test set, we additionally generated partial groundtruth by accumulating the 3D point clouds of 10 futureand 10 past frames around each observation. Examples
are shown in Fig. 6.
ModelNet: We use ModelNet10, comprising 10 popular object categories (bathtub, bed, chair, desk, dresser,monitor, night stand, table, toilet) and select, for eachcategory, the first 200 and 20 shapes from the provided
training and test sets. Then, we follow the pipeline outlined in Fig. 5, as on ShapeNet, using 10 random variants per shape. Due to thin structures, however, SDF
computation does not work well (especially for low resolution, e.g., 323 voxels). Therefore, we approximate the
SDFs using a 3D distance transform on the occupancy
grids. Our experiments are conducted at a resolutionof H×W ×D = 323 , 483 and 643 voxels. Given the increased difficulty, we use a resolution of 642 , 962 and1282 pixels for the observation generating depth maps.
In our experiments, we consider bathtubs, chairs, desks
and tables individually, as well as all 10 categories together (resulting in 100k views overall). For Kinect, we
additionally used a dataset of rotated chairs and tablesaligned with Kinect’s ground plane.

643 (4)

(8)
conv

GT

DVAE,
Low

DVAE,
High

DVAE,
Low

DVAE,
High

nnup+conv

GT,
High

(a) Reconstructions, Low and High Resolution (cf. Table 1)(4)323 (16)

(32)
conv

nnup+conv(8) 48 × 108 × 24

(16)

163 (32)

(64)
conv

nnup+conv(32) 24 × 54 × 24

(64)

(128)

83 (64)nnup+conv(64) 12 × 18 × 12

43 (128)

conv
(128)

23 (128)
reshapennup+conv

fc(128) 6 × 6 × 6zfc(128) 2 × 2 × 2

10
fc

10

(64)

(128)
conv+pool

reshape(128) 2 × 2 × 2

(128)

conv

conv+pool(64)

ShapeNet, KITTI(128) 6 × 6 × 6

83 (64)(32)
conv+poolconv(64) 12 × 18 × 12

(32)

(16)

323 (8)163 (32)
conv(16) 24 × 54 × 24

conv+pool(8)

conv(4) 48 × 108 × 24

conv+pool

643 (4)ModelNet, Kinect

23 (128)David Stutz, Andreas Geiger
43 (128)

10

Fig. 7: Network Architectures. We use different resolutions for ShapeNet and KITTI as well as ModelNet and Kinect (bottom and top, respectively). In bothcases, architectures for higher resolutions employ one
additional stage in the en- and decoder (in gray). Eachconvolutional layer is followed by ReLU activations andbatch normalization (Ioffe and Szegedy, 2015); the window sizes for max pooling and nearest-neighbor upsampling can be derived from the context; the number ofchannels are given in parentheses.
Kinect: Yang et al. provide Kinect scans of various
chairs and tables. They provide both single-view observations as well as ground truth from ElasticFusion(Whelan et al, 2015) as occupancy grids. However, theground truth is not fully accurate, and only 40 views
are provided per object category. Still, the objects havebeen segmented to remove clutter and are appropriate
for experiments in conjunction with ModelNet10. Unfortunately, Yang et al. do not provide SDFs; again,
we use 3D distance transforms as approximation. Additionally, the observations do not indicate free space
and we were required to guess an appropriate groundplane. For our experiments, we use 30 views for training
and 10 views for testing, see Fig. 6 for examples.

4.2 Evaluation
For occupancy grids, we use Hamming distance (Ham)and intersection-over-union (IoU) between the (thresholded) predictions and the ground truth; note that lower
Ham is better, while lower IoU is worse. For SDFs, weconsider a mesh-to-mesh distance on ShapeNet and amesh-to-point distance on KITTI. We follow (Jensen
et al, 2014) and consider accuracy (Acc) and completeness (Comp). To measure Acc, we uniformly sample
roughly 10k points on the reconstructed mesh and average their distance to the target mesh. Analogously,Comp is the distance from the target mesh (or the
ground truth points on KITTI) to the reconstructed
mesh. Note that for both Acc and Comp, lower is bet-

Low

Low

Low

High

High

High

(b) Random Samples, Low and High Resolution (cf. Table 1)

Fig. 8: DVAE Shape Prior. Reconstructions and random samples on ShapeNet and ModelNet at multiple
resolutions (cf. Table 1); false negative and false positive voxels in green and red. Our DVAE shape priorprovides high-quality reconstructions and meaningful
random samples across resolutions.
ter. On ShapeNet and ModelNet, we report both Acc
and Comp in voxels, i.e., in multiples of the voxel edge
length (i.e., in [vx], as we do not know the absolute scaleof the models); on KITTI, we report Comp in meters(i.e., in [m]).
4.3 Architectures and Training
As depicted in Fig. 7, our network architectures are
kept simple and shallow. Considering a resolution of
24×54×24 voxels on ShapeNet and KITTI, the encoder
comprises three stages, each consisting of two convolutional layers (followed by ReLU activations and batchnormalization (Ioffe and Szegedy, 2015)) and max pooling; the decoder mirrors the encoder, replacing max
pooling by nearest neighbor upsampling. We consistently use 33 convolutional kernels. We use a latent
space of size Q = 10 and predict occupancy using Sigmoid activations.
We found that the shape representation has a significant impact on training. Specifically, learning both
occupancy grids and SDFs works better compared totraining on SDFs only. Additionally, following prior art
in single image depth prediction (Eigen and Fergus,
2015; Eigen et al, 2014; Laina et al, 2016), we consider
log-transformed, truncated SDFs (logTSDFs) for training: given a signed distance yi , we compute sign(yi ) log(1+min(5, |yi |)) as the corresponding log-transformed, trun-

11

AML

dAML

Learning 3D Shape Completion under Weak Supervision

Fig. 9: Comparison of AML and dAML. Our deterministic variant, dAML, suffers from inferior results.
Predicted shapes in beige and observations in red at
low resolution (24×54×24 voxels).

cated signed distance. TSDFs are commonly used in theliterature (Newcombe et al, 2011; Riegler et al, 2017a;
Dai et al, 2017; Engelmann et al, 2016; Curless andLevoy, 1996) and the logarithmic transformation additionally increases the relative importance of values
around the surfaces (i.e., around the zero crossing).
For training, we combine occupancy grids and logTSDFs in separate feature channels and randomly translate both by up to 3 voxels per axis. Additionally, we
use Bernoulli noise (probability 0.1) and Gaussian noise(variance 0.05). We use Adam (Kingma and Ba, 2015),
a batch size of 16 and the initialization scheme by Glorot and Bengio (2010). The shape prior is trained for3000 to 4000 epochs with an initial learning rate of
10−4 which is decayed by 0.925 every 215 iterations
until a minimum of 10−16 has been reached. In addition, weight decay (10−4 ) is applied. For shape inference, training takes 30 to 50 epochs, and an initial
learning rate of 10−4 is decayed by 0.9 every 215 iterations. For our learning-based baselines (see Section 4.4)
we require between 300 and 400 epochs using the same
training procedure as for the shape prior. On the Kinect
dataset, where only 30 training examples are available,
we used 5000 epochs. We use log σ 2 = −2 as an empirically found trade-off between accuracy of the reconstructed SDFs and ease of training – significantly lower
log σ 2 may lead to difficulties during training, including divergence. On ShapeNet, ModelNet and Kinect,
the weight λ of the Kullback-Leibler divergence KL (forboth DVAE and (d)AML) was empirically determined
to be λ = 2, 2.5, 3 for low, medium and high resolution,
respectively. On KITTI, we use λ = 1 for all resolutions.
In practice, λ controls the trade-off between diversity(low λ) and quality (high λ) of the completed shapes.
In addition, we reduce the weight in free space areas toone fourth on SN-noisy and KITTI to balance between
occupied and free space. We implemented our networks
in Torch (Collobert et al, 2011).

(a) DVAE t-SNE(b) DVAE Projection(c) AML t-SNE(d) AML Projection

Fig. 10: Learned Latent Spaces. In (a) and (b), weshow a t-SNE (van der Maaten and Hinton, 2008) visualization and a two-dimensional projection of the DVAE
latent space on ModelNet10. The plots illustrate thatthe DVAE is able to separate the ten object categories.
In (c) and (d), we show a t-SNE visualization anda projection of the latent space corresponding to ourlearned AML model on SN-clean. We randomly picked
10 ground truth shapes, “x”, and the corresponding
observations (10 per shape), points (gray pixels indicate remaining shapes/observations). The plots illustrate that AML is able to associate observations withthe corresponding ground truth shapes under weak supervision.

4.4 Baselines
Data-Driven Approaches: We consider the works
by Engelmann et al (2016) and Gupta et al (2015) as
data-driven baselines. Additionally, we consider regular maximum likelihood (ML). Engelmann et al (2016)– referred to as Eng16 – use a principal component
analysis shape prior trained on a manually selected setof car models3 . Shape completion is posed as optimization problem considering both shape and pose. The pretrained shape prior provided by Engelmann et al. assumes a ground plane which is, according to KITTI’s
LiDAR data, fixed at 1m height. Thus, we don’t needto optimize pose on KITTI as we use the ground truth
bounding boxes; on ShapeNet, in contrast, we need to
optimize both pose and shape to deal with the random
rotations in SN-clean and SN-noisy.
3

https://github.com/VisualComputingInstitute/
ShapePriors_GCPR1612David Stutz, Andreas Geiger

Supervisionin %MethodSN-cleanSN-noisy
KITTI
Ham↓ IoU↑ Acc [vx] ↓ Comp [vx] ↓ Ham↓ IoU↑ Acc [vx] ↓ Comp [vx] ↓ Comp [m] ↓Low Resolution: 24 × 54 × 24 voxels; * independent of resolution(shape prior) DVAE
100

< 7.7

Dai et al (2017) (Dai17)Sup
Naı̈veMean
ML*Gupta et al (2015) (ICP)*Engelmann et al (2016) (Eng16)
dAML
AML

0.019 0.885

0.283

0.527

0.021 0.872
0.026 0.841
0.067 0.596
0.052 0.697
0.04 0.756(mesh only)(mesh only)
0.034 0.784
0.034 0.779

0.321
0.409
0.999
0.79
0.637
0.534
1.235
0.532
0.549

0.564
0.607
1.335
0.938
0.8
0.503
1.237
0.741
0.753

(same shape prior as on SN-clean)
0.027 0.836
0.028 0.833
0.064 0.609
0.052 0.696
0.041 0.755(mesh only)(mesh only)
0.036 0.772
0.036 0.771

0.391
0.407
0.941
0.79
0.625
7.551
1.974
0.557
0.57

0.633
0.637
1.29
0.938
0.829
6.372
1.312
0.76
0.761

0.278
0.331
0.52
0.493
0.523

0.491
0.573
0.725
0.67
0.7

0.128
0.091
–
–
(too slow)(too slow)
0.13(see AML)
0.12

Low Resolution: 24 × 54 × 24 voxels; Multiple, k > 1 Fused Views
100
< 16
< 24
< 40

Dai et al (2017) (Dai17), k = 5
Sup, k = 5
AML, k = 2
AML, k = 3
AML, k = 5

0.012 0.924
0.022 0.866
0.032 0.794
0.031 0.809
0.031 0.804

0.214
0.336
0.489
0.471
0.502

0.436
0.566
0.695
0.667
0.686

0.018 0.887
0.024 0.86
0.034 0.79
0.031 0.81
0.035 0.799

n/a
n/a

Medium Resolution: 32 × 72 × 32 voxels(shape prior) DVAE
100
≤ 6.1

Dai et al (2017) (Dai17)SupAML

0.019 0.877

0.24

0.47

0.02 0.869
0.027 0.834
0.031 0.788

0.399
0.498
0.415

0.674
0.789
0.584

(same shape prior as on SN-clean)
0.026 0.83
0.029 0.815
0.036 0.766

0.51
0.571
0.721

0.767
0.843
0.953

0.074
0.09
0.083

High Resolution: 48 × 108 × 48 voxels(shape prior) DVAE
100
< 3.5

Dai17
Sup
AML

0.87

0.272

0.434

0.017 0.88
0.023 0.843
0.028 0.796

0.018

0.517
0.677
0.433

0.827
1.032
0.579

(same shape prior as on SN-clean)
0.054 0.664
0.052 0.674
0.045 0.659

1.559
1.52
1.4

2.067
1.981
1.957

0.066
0.091
0.078

Table 2: Quantitative Results on ShapeNet and KITTI. We consider Hamming distance (Ham) and intersection over union (IoU) for occupancy grids as well as accuracy (Acc) and completeness (Comp) for meshes on
SN-clean, SN-noisy and KITTI. For Ham, Acc and Comp, lower is better; for IoU, higher is better. The unit of
Acc and Comp is voxels (voxel length at 24×54×48 voxels) or meters. Note that the DVAE shape prior (in gray)is only reported as reference (i.e., bound on (d)AML). We indicate the level of supervision in percentage, relative
to the corresponding resolution (see Table 1) and mark the best results under full supervision in red and under
weak supervision in green.
Inspired by the work by Gupta et al (2015) we also
consider a shape retrieval and fitting baseline. Specifically, we perform iterative closest point (ICP) (Besland McKay, 1992) fitting on all training shapes andsubsequently select the best-fitting one. To this end, we
uniformly sample 1Mio points on the training shapes,
4
and perform point-to-point   ICP for a maximum of 100
iterations using R t = I3 0 as initialization. On the
training set, we verified that this approach is always
able to retrieve the perfect shape.
Finally, we consider a simple ML baseline iteratively
minimizing Eq. (7) using stochastic gradient descent(SGD). This baseline is similar to the work by Engelmann et al., however, like ours it is bound to the voxel
grid. Per example, we allow a maximum of 5000 iterations, starting with latent code z = 0, learning rate 0.05
and momentum 0.5 (decayed every 50 iterations at rate
0.85 and 1.0 until 10−5 and 0.9 have been reached).
Learning-Based Approaches: Learning-based approaches usually employ an encoder-decoder architecture to directly learn a mapping from observations xn
4

http://www.cvlibs.net/software/libicp/.

to ground truth shapes yn∗ in a fully supervised setting(Wang et al, 2017; Varley et al, 2017; Yang et al, 2018,2017; Dai et al, 2017). While existing architectures differ slightly, they usually rely on a U-net architecture(Ronneberger et al, 2015; zgn Cicek et al, 2016). In thispaper, we use the approach of Dai et al (2017)5 – referred to as Dai17 – as a representative baseline for thisclass of approaches. In addition, we consider a custom
learning-based baseline which uses the architecture of
our DVAE shape prior, cf. Fig. 7. In contrast to (Daiet al, 2017), this baseline is also limited by the lowdimensional (Q = 10) bottleneck as it does not use
skip connections.
4.5 Experimental Evaluation
Quantitative results are summarized in Table 2 (ShapeNet and KITTI) and 3 (ModelNet). Qualitative results
5
We use https://github.com/angeladai/cnncomplete.
On ModelNet we added one convolutional stage in the enand decoder for larger resolutions; on ShapeNet and KITTI,
we needed to adapt the convolutional strides to fit the corresponding resolutions.

Learning 3D Shape Completion under Weak Supervision
Obs

Dai17

Obs

Dai17

Dai17

Eng16

ML

13
ML

AML

AML

GT

GT

GT

GT

(a) SN-clean (Top) and SN-noisy (Bottom), Low Resolution (24×54×24)
Dai17

ICP

ML

ML

AML

AML

(b) ModelNet Bathtubs, Chairs, Desks and Tables, Low Resolution (323 )Fig. 11: Qualitative Results on ShapeNet and ModelNet. Results for AML, Dai17, Eng16, ICP and ML on
SN-clean, SN-noisy and ModelNet’s bathtubs, chairs, desks and tables. AML outperforms data-driven approaches(ML, Eng16, ICP) and rivals Dai17 while requiring significantly less supervision. Occupancy grids and meshes inbeige, observations in red.
for the shape prior are shown in Fig. 9 and 10; shapecompletion results are shown in Fig. 11 (ShapeNet andModelNet) and 14 (KITTI and Kinect).
Latent Space Dimensionality: Regarding our DVAE
shape prior, we found the dimensionality Q to be of crucial importance as it defines the trade-off between reconstruction accuracy and random sample quality (i.e.,the quality of the generative model). A higher-dimensional latent space usually results in higher-quality reconstructions but also imposes the difficulty of randomly
generating meaningful shapes. Across all datasets, wefound Q = 10 to be suitable – which is significantly
smaller compared to related work: 35 in (Liu et al,2017), 6912 in (Sharma et al, 2016), 200 for (Wu et al,

2016b; Smith and Meger, 2017) or 64 in (Girdhar et al,2016). Still, we are able to obtain visually appealing
results. Finally, in Fig. 9 we show qualitative results,
illustrating good reconstruction performance and reasonable random samples across resolutions.
Fig. 10 shows a t-SNE (van der Maaten and Hinton,2008) visualization as well as a projection of the Q = 10
dimensional latent space, color coding the 10 object categories of ModelNet10. The DVAE clusters the object
categories within the support region of the unit Gaussian. In the t-SNE visualization, we additionally see
ambiguities arising in ModelNet10, e.g., night stands
and dressers often look indistinguishable while monitors are very dissimilar to all other categories. Overall,

14David Stutz, Andreas Geiger

Supervisionin %MethodbathtubHam↓ IoU↑

chair
desk
Ham↓ IoU↑ Acc [vx]↓ Comp [vx]↓ Ham↓ IoU↑

table
Ham↓ IoU↑

ModelNet10
Ham↓ IoU↑

0.015 0.699

0.025 0.517

0.884

0.72

0.028 0.555

011

0.608

0.023 0.714

0.022 0.59 0.019 0.61
0.023 0.618 0.03 0.478(mesh only) (mesh only)
0.028 0.503 0.033 0.414
0.026 0.503 0.033 0.373

0.663
0.873
1.483
1.489
1.088

0.671
0.813
0.89
1.065
0.785

0.027 0.568 0.011 0.648
0.036 0.458 0.017 0.497(mesh only) (mesh only)
0.048 0.323 0.029 0.318
0.041 0.389 0.018 0.423

0.03 0.646
0.038 0.589(mesh only)(too slow)
0.04 0.509

Low Resolution: 323 voxels; * independent of resolution(shape prior) DVAE
100
< 10

Dai et al (2017) (Dai17)
Sup* Gupta et al (2015) (ICP)ML
AML

Medium Resolution: 483 voxels(shape prior) DVAE

0.014 0.671

0.748

0.697

0.025 0.525

100<90.018 0.609 0.016 0.576 0.513
0.024 0.459 0.029 0.347 1.025
High Resolution: 643 voxels

0.508
0.805

0.023 0.532 0.008 0.65
0.034 0.361 0.015 0.384

0.014 0.644
0.018 0.54
0.023 0.46

0.705
0.53
0.852

0.024 0.506 0.009 0.548
0.021 0.525 0.007 0.673
0.042 0.31 0.012 0.407

Dai et al (2017) (Dai17)
AML(shape prior) DVAE
100
Dai et al (2017) (Dai17)<9AML0.021 0.491

0.02 0.474
0.016 0.548
0.026 0.333

0.702
0.47
0.893

0.01

0.548

Table 3: Quantitative Results on ModelNet. Results for bathtubs, chairs, desks, tables and all ten categories
combined (ModelNet10). As the ground truth SDFs are merely approximations (cf. Section 4.1), we concentrateon Hamming distance (Ham; lower is better) and intersection-over-union (IoU; higher is better). Only for chairs,
we report accuracy Acc and completeness Comp in voxels (voxel length at 323 voxels). We also indicate the level ofsupervision (see Table 1). Again, we report the DVAE shape prior as reference and color the best weakly-supervised
approach using green and the best fully-supervised approach in red.

Ablation Study: In Table 2, we show quantitative results of our model on SN-clean and SN-noisy. First, wereport the reconstruction quality of the DVAE shapeprior as reference. Then, we consider the DVAE shapeprior (Naı̈ve), and its mean prediction (Mean) as simple baselines. The poor performance of both illustrates
the difficulty of the benchmark. For AML, we also consider its deterministic variant, dAML (see Section 3).
Quantitatively, there is essentially no difference; however, Fig. 9 demonstrates that AML is able to predict
more detailed shapes. We also found that using both
occupancy and SDFs is necessary to obtain good performance – as is using both point observations and freespace.

Comparison to Baselines on Synthetic Data: For
ShapeNet, Table 2 demonstrates that AML outperforms
data-driven approaches such as Eng16, ICP and ML
and is able to compete with fully-supervised approaches,
Dai17 and Sup, while using only 8% or less supervision. We also note that AML outperforms ML, illustrating that amortized inference is beneficial. Furthermore, Dai17 outperforms Sup, illustrating the advantage of propagating low-level information (through skip
connections) without bottleneck. Most importantly, theperformance gap between AML and Dai17 is rather
small considering the difference in supervision (morethan 92%) and on SN-noisy, the drop in performancefor Dai17 and Sup is larger than for AML suggestingthat AML handles noise and sparsity more robustly.
Fig. 11 shows that these conclusions also apply visually
where AML performs en par with Dai17.

Considering Fig. 10, we additionally demonstratethat the embedding learned by AML, i.e., the embedding of incomplete observations within the latent shapespace, is able to associate observations with corresponding shapes even under weak supervision. In particular,
we show a t-SNE visualization and a projection of thelatent space for AML trained on SN-clean. We colorcode 10 randomly chosen ground truth shapes, resulting
in 100 observations (10 views per shape). AML is usually able to embed observations near the corresponding
ground truth shapes, without explicit supervision (e.g.,
for violet, pink, blue or teal, the observations – points– are close to the corresponding ground truth shapes –“x”). Additionally, AML also matches the unit Gaussian prior distribution reasonably well.

For ModelNet, in Table 3, we mostly focus on occupancy grids (as the derived SDFs are approximate,cf. Section 4.1) and show that chairs, desks or tables
are more difficult. However, AML is still able to predict high-quality shapes, outperforming data-driven approaches. Additionally, in comparison to ShapeNet, thegap between AML and fully-supervised approaches (Dai17
and Sup) is surprisingly small – not reflecting the difference in supervision. This means that even under fullsupervision, these object categories are difficult to complete. In terms of accuracy (Acc) and completeness (Comp),
e.g., for chairs, AML outperforms ICP and ML; Dai17
and Sup, on the other hand, outperform AML. Still,
considering Fig. 11, AML predicts visually appealing
meshes although the reference shape SDFs on Model-

these findings support our decision to use a DVAE with
Q = 10 as shape prior.

Learning 3D Shape Completion under Weak Supervision
k=3

AML

GT

k=5

AML

GT

15
Dai17

AML

GT

Dai17

AML

GT

(a) SN-clean and -noisy, k Views, Low Resolution (24×54×24)
Dai17

AML

GT

Dai17

AML

GT

(b) SN-clean and -noisy, Medium (32×72×32) and High
(48×108×48) Resolution
Dai17

AML

GT

Dai17

AML

GT

(c) ModelNet desks and chairs, Medium (483 ) and High (643 )Resolution

Fig. 12: Multi-View and Higher-Resolution Results on ShapeNet and ModelNet. While AML is
designed for especially sparse observations, it also performs well in a multi-view setting. Additionally, higher
resolutions allow to predict more detailed shapes.
Shapes, occupancy grids or meshes, in beige and observations in red.

Net are merely approximate. Qualitatively, AML also
outperforms its data-driven rivals; only Dai17 predicts
shapes slightly closer to the ground truth.
Multiple Views and Higher Resolutions: In Table 2, we consider multiple, k ∈ {2, 3, 5}, randomly
fused observations (from the 10 views per shape). Generally, additional observations are beneficial (also cf.
Fig. 12); however, fully-supervised approaches such as
Dai17 benefit more significantly than AML. Intuitively,
especially on SN-noisy, k = 5 noisy observations seemto impose contradictory constraints that cannot be resolved under weak supervision. We also show that higher
resolution allows both AML and Dai17 to predict moredetailed shapes, see Fig. 12; for AML this is significant
as, e.g., on SN-noisy, the level of supervision reduces
to less than 1%. Also note that AML is able to handle the slightly asymmetric desks in Fig. 12 due to theFig. 13: Category-Agnostic Results on ModelNet10. AML is able to recover detailed shapes of thecorrect object category even without category supervision (as provided to Dai17). Shapes (occupancy grids
and meshes) in beige and observations in red at low
resolution (323 voxels).
strong shape prior which itself includes symmetric andless symmetric shapes.
Multiple Object Categories: We also investigate
the category-agnostic case, considering all ten ModelNet10 object categories; here, we train a single DVAE
shape prior (as well as a single model for Dai17 andSup) across all ten object categories. As can be seenin Table 3, the gap between AML and fully-supervised
approaches, Dai17 and Sup, further shrinks; even fullysupervised methods have difficulties distinguishing object categories based on sparse observations. Fig. 12shows that AML is able to not only predict reasonableshapes, but also identify the correct object category. In
contrast to Dai17, which predicts slightly more detailed
shapes, this is significant as AML does not have accessto object category information during training.
Comparison on Real Data: On KITTI, considering
Fig. 14, we illustrate that AML consistently predicts detailed shapes regardless of the noise and sparsity in the
inputs. Our qualitative results suggest that AML is able
to predict more detailed shapes compared to Dai17 andEng16; additionally, Eng16 is distracted by sparse andnoisy observations. Quantitatively, instead, Dai17 andSup outperform AML. However, this is mainly due to
two factors: first, the ground truth collected on KITTI
does rarely cover the full car; and second, we put significant effort into faithfully modeling KITTI’s noise statistics in SN-noisy, allowing Dai17 and Sup to generalize
very well. The latter effort, especially, can be avoided
by using our weakly-supervised approach, AML.

16

David Stutz, Andreas Geiger
Obs

Dai17

Eng16

AML

AML

GT

AML

GT

AML

GT

AML

Dai17

Dai17

(a) Difficulties with Exotic Shapes and Fine Structures
Dai17

AML

GT

Dai17

AML

GT

(b) Difficulties with Multiple Object Categories

AML

Obs

(a) KITTI, Medium Resolution (32×72×32)(b) Kinect, Low Resolution (323 )Fig. 14: Qualitative Results on KITTI and
Kinect. On KITTI, AML visually outperforms both
Dai17 and Eng16 while being faster and requiring lesssupervision. On Kinect, AML demonstrates that it isable to generalize from as few as 30 training samples.
Predicted shapes (occupancy grids or meshes) in beige
and observations in red; additionally, partial groundtruth in green.

On Kinect, also considering Fig. 14, only 30 observations are available for training. It can be seen thatAML predicts reasonable shapes for tables. We find it
interesting that AML is able to generalize from only 30
training examples. In this sense, AML functions similar to ML, in that the objective is trained to overfit to
few samples. This, however, cannot work in all cases,
as demonstrated by the chairs where AML tries to predict a suitable chair, but does not fit the observations
as well. Another problem witnessed on Kinect, is thatthe shape prior training samples need to be aligned tothe observations (with respect to the viewing angles).
For the chairs, we were not able to guess the viewing
trajectory correctly (cf. (Yang et al, 2018)).
Failure Cases: AML and Dai17 often face similar
problems, as illustrated in Fig. 15, suggesting that these
problems are inherent to the used shape representationsor the learning approach independent of the level ofsupervision. For example, both AML and Dai17 have

Fig. 15: Failures Cases. On the top, we show thatAML has difficulties with exotic shapes, not represented
in the latent space; and both AML and Dai17 havedifficulties with fine details. The bottom row demonstrates that it is difficult to infer the correct object category from sparse observations, even under full supervision as required by Dai17. Shapes (occupancy grids andmehses) in beige and observations in red from various
resolutions.
problems with fine, thin structures that are hard toreconstruct properly at any resolution. Furthermore,
identifying the correct object category on ModelNet10
from sparse observations is difficult for both AML andSup. Finally, AML additionally has difficulties with exotic objects that are not well represented in the latentshape space as, e.g., designed chairs.
Runtime: At low resolution, AML as well as the fullysupervised approaches Dai17 and Sup, are particular
fast, requiring up to 2ms on a NVIDIATM GeForce R
GTX TITAN using Torch (Collobert et al, 2011). Datadriven approaches (e.g., Eng16, ICP and ML), on theother hand, take considerably longer. Eng16, for instance requires 168ms on average for completing theshape of a sparse LIDAR observation from KITTI using an Intel R Xeon R E5-2690 @2.6Ghz and the multithreaded Ceres solver (Agarwal et al, 2012). ICP and
ML take longest, requiring up to 38s and 75s (not taking into account the point sampling process for the
shapes), respectively. Except for Eng16 and ICP, allapproaches scale with the used resolution and the employed architecture.

5 Conclusion
In this paper, we presented a novel, weakly-supervised
learning-based approach to 3D shape completion fromsparse and noisy point cloud observations. We used a(denoising) variational auto-encoder (Im et al, 2017;
Kingma and Welling, 2014) to learn a latent space of
shapes for one or multiple object categories using syn-

Learning 3D Shape Completion under Weak Supervision

thetic data from ShapeNet (Chang et al, 2015) or ModelNet (Wu et al, 2015). Based on the learned generativemodel, i.e., decoder, we formulated 3D shape completion as a maximum likelihood problem. In a second
step, we then fixed the learned generative model andtrained a new recognition model, i.e. encoder, to amortize, i.e. learn, the maximum likelihood problem. Thus,
our Amortized Maximum Likelihood (AML) approach to 3D shape completion can be trained in a
weakly-supervised fashion. Compared to related datadriven approaches, e.g., (Rock et al, 2015; Haene et al,2014; Li et al, 2015; Engelmann et al, 2016, 2017; Nanet al, 2012; Bao et al, 2013; Dame et al, 2013; Nguyen
et al, 2016), our approach offers fast inference at test
time; in contrast to other learning-based approaches,e.g., (Riegler et al, 2017a; Smith and Meger, 2017; Daiet al, 2017; Sharma et al, 2016; Fan et al, 2017; Rezende
et al, 2016; Yang et al, 2018; Wang et al, 2017; Varley
et al, 2017; Han et al, 2017), we do not require full supervision during training. Both characteristics render
our approach useful for robotic scenarios where full supervision is often not available such as in autonomous
driving, e.g., on KITTI (Geiger et al, 2012), or indoor
robotics, e.g., on Kinect (Yang et al, 2018).
On two newly created synthetic shape completion
benchmarks, derived from ShapeNet’s cars and ModelNet10, as well as on real data from KITTI and, we
demonstrated that AML outperforms related data-driven
approaches (Engelmann et al, 2016; Gupta et al, 2015)
while being significantly faster. We further showed thatAML is able to compete with fully-supervised approaches(Dai et al, 2017), both quantitatively and qualitatively,
while using only 3 − 10% supervision or less. In contrast to (Rock et al, 2015; Haene et al, 2014; Li et al,2015; Engelmann et al, 2016, 2017; Nan et al, 2012;
Bao et al, 2013; Dame et al, 2013), we additionally
showed that AML is able to generalize across object
categories without category supervision during training. On Kinect, we also demonstrated that our AML
approach is able to generalize from very few training
examples. In contrast to (Girdhar et al, 2016; Liu et al,
2017; Sharma et al, 2016; Wu et al, 2015; Dai et al,
2017; Firman et al, 2016; Han et al, 2017; Fan et al,2017), we considered resolutions up to 48×108×48 and643 voxels as well as significantly sparser observations.
Overall, our experiments demonstrate two key advantages of the proposed approach: significantly reduced
runtime and increased performance compared to datadriven approaches showing that amortizing inference ishighly effective.
In future work, we would like to address several
aspects of our AML approach. First, the shape prioris essential for weakly-supervised shape completion, as

17also noted by Gwak et al (2017). However, training expressive generative models in 3D is still difficult. Second, larger resolutions imply significantly longer training times; alternative shape representations and data
structures such as point clouds (Qi et al, 2017a,b; Fanet al, 2017) or octrees (Riegler et al, 2017b,a; Häne et al,2017) might be beneficial. Finally, jointly tackling pose
estimation and shape completion seems promising (Engelmann et al, 2016).

References
Abramowitz M (1974) Handbook of Mathematical Functions,With Formulas, Graphs, and Mathematical Tables. Dover
Publications
Agarwal S, Mierle K, Others (2012) Ceres solver. http://
ceres-solver.org
Aubry M, Maturana D, Efros A, Russell B, Sivic J (2014)
Seeing 3D chairs: exemplar part-based 2D-3D alignment
using a large dataset of CAD models. In: Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR)
Bao S, Chandraker M, Lin Y, Savarese S (2013) Dense object
reconstruction with semantic priors. In: Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR)
Besl P, McKay H (1992) A method for registration of 3d
shapes. IEEE Trans on Pattern Analysis and Machine Intelligence (PAMI) 14:239–256
Blei DM, Kucukelbir A, McAuliffe JD (2016) Variational inference: A review for statisticians. arXivorg 1601.00670
Brock A, Lim T, Ritchie JM, Weston N (2016) Generativeand discriminative voxel modeling with convolutional neural networks. arXivorg 1608.04236
Chang AX, Funkhouser TA, Guibas LJ, Hanrahan P, Huang
Q, Li Z, Savarese S, Savva M, Song S, Su H, Xiao J, Yi
L, Yu F (2015) Shapenet: An information-rich 3d model
repository. arXivorg 1512.03012
Chen X, Kundu K, Zhu Y, Ma H, Fidler S, Urtasun R (2016)
3d object proposals using stereo imagery for accurate object class detection. arXivorg 1608.07711
Choy CB, Xu D, Gwak J, Chen K, Savarese S (2016) 3d-r2n2:
A unified approach for single and multi-view 3d object reconstruction. In: Proc. of the European Conf. on Computer
Vision (ECCV)
zgn Cicek, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger
O (2016) 3d u-net: Learning dense volumetric segmentationfrom sparse annotation. arXivorg 1606.06650
Cignoni P, Callieri M, Corsini M, Dellepiane M, Ganovelli
F, Ranzuglia G (2008) Meshlab: an open-source mesh processing tool. In: Eurographics Italian Chapter Conference
Collobert R, Kavukcuoglu K, Farabet C (2011) Torch7: A
matlab-like environment for machine learning. In: Advances in Neural Information Processing Systems (NIPS)
Workshops
Curless B, Levoy M (1996) A volumetric method for building
complex models from range images. In: ACM Trans. on
Graphics
Dai A, Qi CR, Nießner M (2017) Shape completion using
3d-encoder-predictor cnns and shape synthesis. In: Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR)
Dame A, Prisacariu V, Ren C, Reid I (2013) Dense reconstruction using 3D object shape priors. In: Proc.

18
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR)
Eigen D, Fergus R (2015) Predicting depth, surface normals
and semantic labels with a common multi-scale convolutional architecture. In: Proc. of the IEEE International
Conf. on Computer Vision (ICCV)
Eigen D, Puhrsch C, Fergus R (2014) Depth map prediction from a single image using a multi-scale deep network.
In: Advances in Neural Information Processing Systems(NIPS)Engelmann F, Stückler J, Leibe B (2016) Joint object pose
estimation and shape reconstruction in urban street scenes
using 3D shape priors. In: Proc. of the German Conference
on Pattern Recognition (GCPR)
Engelmann F, Stückler J, Leibe B (2017) SAMP: shape andmotion priors for 4d vehicle reconstruction. In: Proc. of
the IEEE Winter Conference on Applications of Computer
Vision (WACV), pp 400–408
Fan H, Su H, Guibas LJ (2017) A point set generation network for 3d object reconstruction from a single image. Proc
IEEE Conf on Computer Vision and Pattern Recognition(CVPR)
Firman M, Mac Aodha O, Julier S, Brostow GJ (2016) Structured prediction of unobserved voxels from a single depth
image. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
Furukawa Y, Hernandez C (2013) Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphics andVision 9(1-2):1–148
Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? The KITTI vision benchmark suite. In:
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
Gershman S, Goodman ND (2014) Amortized inference inprobabilistic reasoning. In: Proc. of the Annual Meeting of
the Cognitive Science Society
Girdhar R, Fouhey DF, Rodriguez M, Gupta A (2016) Learning a predictable and generative vector representation for
objects. In: Proc. of the European Conf. on Computer Vision (ECCV)
Glorot X, Bengio Y (2010) Understanding the difficulty of
training deep feedforward neural networks. In: Conferenceon Artificial Intelligence and Statistics (AISTATS)
Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, WardeFarley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets. In: Advances in Neural Information
Processing Systems (NIPS)
Gupta S, Arbeláez PA, Girshick RB, Malik J (2015) Aligning
3D models to RGB-D images of cluttered scenes. In: Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR)
Gwak J, Choy CB, Garg A, Chandraker M, Savarese S (2017)
Weakly supervised generative adversarial networks for 3dreconstruction. arXivorg 1705.10904
Gney F, Geiger A (2015) Displets: Resolving stereo ambiguities using object knowledge. In: Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR)
Haene C, Savinov N, Pollefeys M (2014) Class specific
3d object shape priors using surface normals. In: Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR)Han X, Li Z, Huang H, Kalogerakis E, Yu Y (2017) Highresolution shape completion using deep neural networks for
global structure and local geometry inference. In: Proc. of
the IEEE International Conf. on Computer Vision (ICCV),
pp 85–93

David Stutz, Andreas GeigerHäne C, Tulsiani S, Malik J (2017) Hierarchical surface prediction for 3d object reconstruction. arXivorg 1704.00710
Im DJ, Ahn S, Memisevic R, Bengio Y (2017) Denoising criterion for variational auto-encoding framework. In: Proc. of
the Conf. on Artificial Intelligence (AAAI), pp 2059–2065
Ioffe S, Szegedy C (2015) Batch normalization: Accelerating
deep network training by reducing internal covariate shift.
In: Proc. of the International Conf. on Machine learning(ICML)
Jensen RR, Dahl AL, Vogiatzis G, Tola E, Aanæs H (2014)
Large scale multi-view stereopsis evaluation. In: Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR)
Jones E, Oliphant T, Peterson P, et al (2001) SciPy: Open
source scientific tools for Python. URL http://www.scipy.
org/
Kar A, Tulsiani S, Carreira J, Malik J (2015) Categoryspecific object reconstruction from a single image. In: Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR)
Kato H, Ushiku Y, Harada T (2017) Neural 3d mesh renderer.
arXivorg 1711.07566
Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Proc. of the International Conf. on Learning
Representations (ICLR)
Kingma DP, Welling M (2014) Auto-encoding variational
bayes. Proc of the International Conf on Learning Representations (ICLR)
Kroemer O, Amor HB, Ewerton M, Peters J (2012) Point
cloud completion using extrusions. In: IEEE-RAS International Conference on Humanoid Robots (Humanoids)
Laina I, Rupprecht C, Belagiannis V, Tombari F, Navab N(2016) Deeper depth prediction with fully convolutional
residual networks. In: Proc. of the International Conf. on
3D Vision (3DV)
Law AJ, Aliaga DG (2011) Single viewpoint model completion of symmetric objects for digital inspection. Computer
Vision and Image Understanding (CVIU) 115(5):603–610
Leotta MJ, Mundy JL (2009) Predicting high resolution image edges with a generic, adaptive, 3-d vehicle model. In:
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
Li Y, Dai A, Guibas L, Nießner M (2015) Database-assisted
object retrieval for real-time 3d reconstruction. In: Computer Graphics Forum
Lin C, Kong C, Lucey S (2017) Learning efficient point cloud
generation for dense 3d object reconstruction. arXivorg
1706.07036
Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D,
Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: Proc. of the European Conf. on Computer Vision (ECCV)
Liu S, II AGO, Giles CL (2017) Learning a hierarchical latentvariable model of voxelized 3d shapes. arXivorg 1705.05994
Lorensen WE, Cline HE (1987) Marching cubes: A high resolution 3d surface construction algorithm. In: ACM Trans.
on Graphics
Ma L, Sibley G (2014) Unsupervised dense object discovery,
detection, tracking and reconstruction. In: Proc. of the European Conf. on Computer Vision (ECCV)van der Maaten L, Hinton G (2008) Visualizing highdimensional data using t-sne. Journal of Machine Learning
Research (JMLR) 9:25792605
Menze M, Geiger A (2015) Object scene flow for autonomousvehicles. In: Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR)

Learning 3D Shape Completion under Weak Supervision
Nan L, Xie K, Sharf A (2012) A search-classify approach for cluttered indoor scene understanding. ACM TG
31(6):137:1–137:10
Nash C, Williams CKI (2017) The shape variational autoencoder: A deep generative model of part-segmented 3d objects. Eurographics Symposium on Geometry Processing(SGP) 36(5):1–12
Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D,
Davison AJ, Kohli P, Shotton J, Hodges S, Fitzgibbon
A (2011) Kinectfusion: Real-time dense surface mapping
and tracking. In: Proc. of the International Symposium on
Mixed and Augmented Reality (ISMAR)
Nguyen DT, Hua B, Tran M, Pham Q, Yeung S (2016) A
field model for repairing 3d shapes. In: Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR)
Oswald MR, Töppe E, Nieuwenhuis C, Cremers D (2013) A
review of geometry recovery from a single image focusing
on curved object reconstruction. In: Innovations for ShapeAnalysis, Models and Algorithms, pp 343–378
Pauly M, Mitra NJ, Giesen J, Gross MH, Guibas LJ (2005)
Example-based 3d scan completion. In: Eurographics Symposium on Geometry Processing (SGP)
Pauly M, Mitra NJ, Wallner J, Pottmann H, Guibas LJ(2008) Discovering structural regularity in 3d geometry.
ACM Trans on Graphics 27(3):43:1–43:11
Pepik B, Stark M, Gehler PV, Ritschel T, Schiele B (2015)
3d object class detection in the wild. In: Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), pp
1–10
Pizlo Z (2007) Human perception of 3d shapes. In: Proc. of
the International Conf. on Computer Analysis of Images
and Patterns (CAIP)
Pizlo Z (2010) 3D shape: Its unique place in visual perception.
MIT Press
Prisacariu V, Segal A, Reid I (2013) Simultaneous monocular
2d segmentation, 3d pose recovery and 3d reconstruction.
In: Proc. of the Asian Conf. on Computer Vision (ACCV)
Prisacariu VA, Reid I (2011) Nonlinear shape manifolds as
shape priors in level set segmentation and tracking. In:
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
Qi CR, Su H, Mo K, Guibas LJ (2017a) Pointnet: Deep learning on point sets for 3d classification and segmentation. In:
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
Qi CR, Yi L, Su H, Guibas LJ (2017b) Pointnet++: Deephierarchical feature learning on point sets in a metric space.
In: Advances in Neural Information Processing Systems(NIPS)
Rezende DJ, Mohamed S (2015) Variational inference withnormalizing flows. In: Proc. of the International Conf. on
Machine learning (ICML)
Rezende DJ, Eslami SMA, Mohamed S, Battaglia P, Jaderberg M, Heess N (2016) Unsupervised learning of 3d structure from images. arXivorg 1607.00662
Riegler G, Ulusoy AO, Bischof H, Geiger A (2017a) OctNetFusion: Learning depth fusion from data. In: Proc. of theInternational Conf. on 3D Vision (3DV)
Riegler G, Ulusoy AO, Geiger A (2017b) Octnet: Learning deep 3d representations at high resolutions. In: Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR)
Ritchie D, Horsfall P, Goodman ND (2016) Deep amortized
inference for probabilistic programs. arXivorg 1610.05735
Rock J, Gupta T, Thorsen J, Gwak J, Shin D, Hoiem D(2015) Completing 3d object shape from one depth image.

19
In: Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR)
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In:
Medical Image Computing and Computer-Assisted Intervention (MICCAI)
Sandhu R, Dambreville S, Yezzi AJ, Tannenbaum A (2009)
Non-rigid 2d-3d pose estimation and 2d image segmentation. In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
Sandhu R, Dambreville S, Yezzi AJ, Tannenbaum A (2011) A
nonrigid kernel-based framework for 2d-3d pose estimationand 2d image segmentation. IEEE Trans on Pattern Analysis and Machine Intelligence (PAMI) 33(6):1098–1115
Sharma A, Grau O, Fritz M (2016) Vconv-dae: Deep volumetric shape learning without object labels. arXivorg
1604.03755
Smith E, Meger D (2017) Improved adversarial systems for 3d
object generation and reconstruction. arXivorg 1707.09557
Song S, Xiao J (2014) Sliding shapes for 3D object detection in depth images. In: Proc. of the European Conf. on
Computer Vision (ECCV)Steinbrucker F, Kerl C, Cremers D (2013) Large-scale multiresolution surface reconstruction from rgb-d sequences. In:
Proc. of the IEEE International Conf. on Computer Vision(ICCV)
Stutz D, Geiger A (2018) Learning 3d shape completionfrom laser scan data with weak supervision. In: Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR)
Sung M, Kim VG, Angst R, Guibas LJ (2015) Data-driven
structural priors for shape completion. ACM Trans on
Graphics 34(6):175:1–175:11
Tatarchenko M, Dosovitskiy A, Brox T (2017) Octree generating networks: Efficient convolutional architectures forhigh-resolution 3d outputs. In: Proc. of the IEEE International Conf. on Computer Vision (ICCV)
Thrun S, Wegbreit B (2005) Shape from symmetry. In:
Proc. of the IEEE International Conf. on Computer Vision (ICCV), pp 1824–1831
Tulsiani S, Zhou T, Efros AA, Malik J (2017) Multi-view
supervision for single-view reconstruction via differentiable
ray consistency. In: Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR)
Tulsiani S, Efros AA, Malik J (2018) Multi-view consistency
as supervisory signal for learning shape and pose prediction. arXivorg 1801.03910
Varley J, DeChant C, Richardson A, Ruales J, Allen PK(2017) Shape completion enabled robotic grasping. In:
Proc. IEEE International Conf. on Intelligent Robots andSystems (IROS)
Wang S, Bai M, Mattyus G, Chu H, Luo W, Yang B, Liang J,
Cheverie J, Fidler S, Urtasun R (2016) Torontocity: Seeing
the world with a million eyes. arXivorg 1612.00423
Wang W, Huang Q, You S, Yang C, Neumann U (2017) Shapeinpainting using 3d generative adversarial network and recurrent convolutional networks. In: Proc. of the IEEE International Conf. on Computer Vision (ICCV)
Whelan T, Leutenegger S, Salas-Moreno RF, Glocker B,
Davison AJ (2015) Elasticfusion: Dense SLAM without A
pose graph. In: Proc. Robotics: Science and Systems (RSS)Wu J, Xue T, Lim JJ, Tian Y, Tenenbaum JB, Torralba
A, Freeman WT (2016a) Single image 3d interpreter network. In: Proc. of the European Conf. on Computer Vision(ECCV)

20
Wu J, Zhang C, Xue T, Freeman B, Tenenbaum J (2016b)
Learning a probabilistic latent space of object shapes via
3d generative-adversarial modeling. In: Advances in Neural
Information Processing Systems (NIPS)
Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J(2015) 3d shapenets: A deep representation for volumetric shapes. In: Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR)
Xie J, Kiefel M, Sun MT, Geiger A (2016) Semantic instanceannotation of street scenes by 3d to 2d label transfer. In:
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
Yan X, Yang J, Yumer E, Guo Y, Lee H (2016) Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In: Advances in Neural Information Processing Systems (NIPS)
Yang B, Wen H, Wang S, Clark R, Markham A, Trigoni N(2017) 3d object reconstruction from a single depth view
with adversarial learning. arXivorg 1708.07969
Yang B, Rosa S, Markham A, Trigoni N, Wen H (2018) 3d
object dense reconstruction from a single depth view. arXivorg abs/1802.00411
Zheng Q, Sharf A, Wan G, Li Y, Mitra NJ, Cohen-Or D,
Chen B (2010) Non-local scan consolidation for 3d urban
scenes. ACM Trans on Graphics 29(4):94:1–94:9
Zheng S, Prisacariu VA, Averkiou M, Cheng MM, Mitra
NJ, Shotton J, Torr PHS, Rother C (2015) Object proposal estimation in depth images using compact 3d shape
manifolds. In: Proc. of the German Conference on Pattern
Recognition (GCPR)
Zia M, Stark M, Schiele B, Schindler K (2013) Detailed
3D representations for object recognition and modeling.
IEEE Trans on Pattern Analysis and Machine Intelligence(PAMI) 35(11):2608–2623
Zia MZ, Stark M, Schindler K (2014) Are cars just 3d boxes?
jointly estimating the 3d shape of multiple objects. In:
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp 3678–3685

David Stutz, Andreas Geiger