Occupancy Networks: Learning 3D Reconstruction in Function Space
Lars Mescheder1 Michael Oechsle1,2 Michael Niemeyer1 Sebastian Nowozin3† Andreas Geiger1
1
Autonomous Vision Group, MPI for Intelligent Systems and University of Tübingen
2
ETAS GmbH, Stuttgart
3
Google AI Berlin{firstname.lastname}@tue.mpg.de

nowozin@gmail.com

Abstract
With the advent of deep neural networks, learning-based
approaches for 3D reconstruction have gained popularity.
However, unlike for images, in 3D there is no canonical representation which is both computationally and memory efficient yet allows for representing high-resolution geometryof arbitrary topology. Many of the state-of-the-art learningbased 3D reconstruction approaches can hence only represent very coarse 3D geometry or are limited to a restricteddomain. In this paper, we propose Occupancy Networks,
a new representation for learning-based 3D reconstruction
methods. Occupancy networks implicitly represent the 3D
surface as the continuous decision boundary of a deep neural network classifier. In contrast to existing approaches,
our representation encodes a description of the 3D output
at infinite resolution without excessive memory footprint.
We validate that our representation can efficiently encode
3D structure and can be inferred from various kinds of input. Our experiments demonstrate competitive results, both
qualitatively and quantitatively, for the challenging tasks of
3D reconstruction from single images, noisy point clouds
and coarse discrete voxel grids. We believe that occupancy
networks will become a useful tool in a wide variety oflearning-based 3D tasks.

1. Introduction
Recently, learning-based approaches for 3D reconstruction have gained popularity [4, 9, 23, 58, 75, 77]. In contrast
to traditional multi-view stereo algorithms, learned models
are able to encode rich prior information about the space of
3D shapes which helps to resolve ambiguities in the input.
While generative models have recently achieved remarkable successes in generating realistic high resolution images [36, 47, 72], this success has not yet been replicatedin the 3D domain. In contrast to the 2D domain, the com† Part of this work was done while at MSR Cambridge.

(a) Voxel(b) Point(c) Mesh(d) Ours

Figure 1: Overview: Existing 3D representations discretize
the output space differently: (a) spatially in voxel representations, (b) in terms of predicted points, and (c) in terms ofvertices for mesh representations. In contrast, (d) we propose to consider the continuous decision boundary of a classifier fθ (e.g., a deep neural network) as a 3D surface whichallows to extract 3D meshes at any resolution.
munity has not yet agreed on a 3D output representationthat is both memory efficient and can be efficiently inferredfrom data. Existing representations can be broadly categorized into three categories: voxel-based representations[4, 19, 43, 58, 64, 69, 75] , point-based representations [1, 17]
and mesh representations [34, 57, 70], see Fig. 1.
Voxel representations are a straightforward generalization of pixels to the 3D case. Unfortunately, however, thememory footprint of voxel representations grows cubicallywith resolution, hence limiting naı̈ve implementations to
323 or 643 voxels. While it is possible to reduce the memory
footprint by using data adaptive representations such as octrees [61, 67], this approach leads to complex implementations and existing data-adaptive algorithms are still limited
to relatively small 2563 voxel grids. Point clouds [1,17] andmeshes [34, 57, 70] have been introduced as alternative representations for deep learning, using appropriate loss functions. However, point clouds lack the connectivity structureof the underlying mesh and hence require additional postprocessing steps to extract 3D geometry from the model.

Existing mesh representations are typically based on deforming a template mesh and hence do not allow arbitrary
topologies. Moreover, both approaches are limited in thenumber of points/vertices which can be reliably predictedusing a standard feed-forward network.
In this paper1 , we propose a novel approach to 3Dreconstruction based on directly learning the continuous
3D occupancy function (Fig. 1d). Instead of predicting avoxelized representation at a fixed resolution, we predictthe complete occupancy function with a neural network fθ
which can be evaluated at arbitrary resolution. This drastically reduces the memory footprint during training. At
inference time, we extract the mesh from the learned modelusing a simple multi-resolution isosurface extraction algorithm which trivially parallelizes over 3D locations.
In summary, our contributions are as follows:• We introduce a new representation for 3D geometry
based on learning a continuous 3D mapping.
• We show how this representation can be used for reconstructing 3D geometry from various input types.
• We experimentally validate that our approach is able
to generate high-quality meshes and demonstrate thatit compares favorably to the state-of-the-art.

2. Related Work
Existing work on learning-based 3D reconstruction canbe broadly categorized by the output representation they
produce as either voxel-based, point-based or mesh-based.
Voxel Representations: Due to their simplicity, voxels are
the most commonly used representation for discriminative[45, 55, 63] and generative [9, 23, 58, 64, 75, 77] 3D tasks.
Early works have considered the problem of reconstructing 3D geometry from a single image using 3D convolutional neural networks which operate on voxel grids[9, 68, 77]. Due to memory requirements, however, theseapproaches were limited to relatively small 323 voxel grids.
While recent works [74, 76, 79] have applied 3D convolutional neural networks to resolutions up to 1283 , this is only
possible with shallow architectures and small batch sizes,
which leads to slow training.
The problem of reconstructing 3D geometry from multiple input views has been considered in [31, 35, 52]. Ji
et al. [31] and Kar et al. [35] encode the camera parameters together with the input images in a 3D voxel representation and apply 3D convolutions to reconstruct 3D scenes
from multiple views. Paschalidou et al. [52] introduced anarchitecture that predicts voxel occupancies from multiple
images, exploiting multi-view geometry constraints [69].
Other works applied voxel representations to learn generative models of 3D shapes. Most of these methods are
1 Also see [8, 48, 51] for concurrent work that proposes similar ideas.

either based on variational auto-encoders [39, 59] or generative adversarial networks [25]. These two approaches were
pursued in [4, 58] and [75], respectively.
Due to the high memory requirements of voxel representations, recent works have proposed to reconstruct 3D
objects in a multi-resolution fashion [28, 67]. However, theresulting methods are often complicated to implement andrequire multiple passes over the input to generate the final
3D model. Furthermore, they are still limited to comparably
small 2563 voxel grids. For achieving sub-voxel precision,
several works [12,42,60] have proposed to predict truncatedsigned distance fields (TSDF) [11] where each point in a
3D grid stores the truncated signed distance to the closest
3D surface point. However, this representation is usuallymuch harder to learn compared to occupancy representations as the network must reason about distance functions
in 3D space instead of merely classifying a voxel as occupied or not. Moreover, this representation is still limited by
the resolution of the underlying 3D grid.
Point Representations: An interesting alternative representation of 3D geometry is given by 3D point clouds which
are widely used both in the robotics and in the computer
graphics communities. Qi et al. [54, 56] pioneered point
clouds as a representation for discriminative deep learning
tasks. They achieved permutation invariance by applying afully connected neural network to each point independently
followed by a global pooling operation. Fan et al. [17] introduced point clouds as an output representation for 3D reconstruction. However, unlike other representations, this approach requires additional non-trivial post-processing steps[3, 6, 37, 38] to generate the final 3D mesh.
Mesh Representations: Meshes have first been considered for discriminative 3D classification or segmentation
tasks by applying convolutions on the graph spanned by themesh’s vertices and edges [5, 27, 71].
More recently, meshes have also been considered as output representation for 3D reconstruction [26,33,41,70]. Unfortunately, most of these approaches are prone to generating self-intersecting meshes. Moreover, they are only able
to generate meshes with simple topology [70], require areference template from the same object class [33, 41, 57]
or cannot guarantee closed surfaces [26]. Liao et al. [43]proposed an end-to-end learnable version of the marching
cubes algorithm [44]. However, their approach is still limited by the memory requirements of the underlying 3D gridand hence also restricted to 323 voxel resolution.
In contrast to the aforementioned approaches, our approach leads to high resolution closed surfaces without selfintersections and does not require template meshes from thesame object class as input. This idea is related to classicallevel set [10, 14, 50] approaches to multi-view 3D reconstruction [18,24,32,40,53,78]. However, instead of solving

a differential equation, our approach uses deep learning toobtain a more expressive representation which can be naturally integrated into an end-to-end learning pipeline.

3. Method
In this section, we first introduce Occupancy Networks as
a representation of 3D geometry. We then describe how we
can learn a model that infers this representation from various forms of input such as point clouds, single images andlow-resolution voxel representations. Lastly, we describe atechnique for extracting high-quality 3D meshes from ourmodel at test time.

3.1. Occupancy Networks
Ideally, we would like to reason about the occupancy not
only at fixed discrete 3D locations (as in voxel respresentations) but at every possible 3D point p ∈ R3 . We call the
resulting functiono : R3 → {0, 1}(1)

the occupancy function of the 3D object. Our key insightis that we can approximate this 3D function with a neural network that assigns to every location p ∈ R3 an occupancy probability between 0 and 1. Note that this network is
equivalent to a neural network for binary classification, except that we are interested in the decision boundary which
implicitly represents the object’s surface.
When using such a network for 3D reconstruction of an
object based on observations of that object (e.g., image,point cloud, etc.), we must condition it on the input. Fortunately, we can make use of the following simple functional
equivalence: a function that takes an observation x ∈ X
as input and has a function from p ∈ R3 to R as output
can be equivalently described by a function that takes a pair(p, x) ∈ R3 × X as input and outputs a real number. The
latter representation can be simply parameterized by a neural network fθ that takes a pair (p, x) as input and outputs areal number which represents the probability of occupancy:fθ : R3 × X → [0, 1](2)

We call this network the Occupancy Network.

3.2. Training
To learn the parameters θ of the neural network fθ (p, x),
we randomly sample points in the 3D bounding volume ofthe object under consideration: for the i-th sample in a training batch we sample K points pij ∈ R3 , j = 1, . . . , K. We
then evaluate the mini-batch loss LB at those locations:|B|

LB (θ) =K

1 XX
L(fθ (pij , xi ), oij )|B| i=1 j=1Here, xi is the i’th observation of batch B, oij ≡ o(pij ) denotes the true occupancy at point pij , and L(·, ·) is a crossentropy classification loss.
The performance of our method depends on the sampling scheme that we employ for drawing the locations
pij that are used for training. In Section 4.6 we perform a detailed ablation study comparing different sampling
schemes. In practice, we found that sampling uniformly inside the bounding box of the object with an additional small
padding yields the best results.
Our 3D representation can also be used for learning probabilistic latent variable models. Towards thisgoal, we introduce an encoder network gψ (·) that takes
locations pij and occupancies oij as input and predicts mean µψ and standard deviation σψ of a Gaussian distribution qψ (z|(pij , oij )j=1:K ) on latent z ∈ RL
as output. We optimize a lower bound [21, 39, 59]
to the negative log-likelihood of the generative modelp((oij )j=1:K |(pij )j=1:K ):

(3)

|B|

Lgen
B (θ, ψ) =

K

1 XhX
L(fθ (pij , zi ), oij )|B| i=1 j=1i+ KL (qψ (z|(pij , oij )j=1:K ) k p0 (z))(4)

where KL denotes the KL-divergence, p0 (z) is a prior distribution on the latent variable zi (typically Gaussian) andzi is sampled according to qψ (zi |(pij , oij )j=1:K ).

3.3. Inference
For extracting the isosurface corresponding to a new observation given a trained occupancy network, we introduce
Multiresolution IsoSurface Extraction (MISE), a hierarchical isosurface extraction algorithm (Fig. 2). By incrementally building an octree [30, 46, 66, 73], MISE enables us to
extract high resolution meshes from the occupancy network
without densely evaluating all points of a high-dimensional
occupancy grid.
We first discretize the volumetric space at an initial resolution and evaluate the occupancy network fθ (p, x) for all pin this grid. We mark all grid points p as occupied for whichfθ (p, x) is bigger or equal to some threshold2 τ . Next, wemark all voxels as active for which at least two adjacentgrid points have differing occupancy predictions. These are
the voxels which would intersect the mesh if we applied
the marching cubes algorithm at the current resolution. We
subdivide all active voxels into 8 subvoxels and evaluate all
new grid points which are introduced to the occupancy gridthrough this subdivision. We repeat these steps until thedesired final resolution is reached. At this final resolution,
2 The threshold τ is the only hyperparameter of our occupancy network.
It determines the “thickness” of the extracted 3D surface. In our experiments we cross-validate this threshold on a validation set.

in (6) uses second order gradient information and can be efficiently implemented using Double-Backpropagation [15].
Note that this last step removes the discretization artifacts of the Marching Cubes approximation and would not
be possible if we had directly predicted a voxel-based representation. In addition, our approach also allows to efficiently extract normals for all vertices of our output meshby simply backpropagating through the occupancy network.
In total, our inference algorithm requires 3s per mesh.

3.4. Implementation Details

Figure 2: Multiresolution IsoSurface Extraction: We firstmark all points at a given resolution which have alreadybeen evaluated as either occupied (red circles) or unoccupied (cyan diamonds). We then determine all voxels thathave both occupied and unoccupied corners and mark themas active (light red) and subdivide them into 8 subvoxels
each. Next, we evaluate all new grid points (empty circles)that have been introduced by the subdivision. The previous
two steps are repeated until the desired output resolution isreached. Finally we extract the mesh using the marching
cubes algorithm [44], simplify and refine the output meshusing first and second order gradient information.
we apply the Marching Cubes algorithm [44] to extract anapproximate isosurface{p ∈ R3 | fθ (p, x) = τ }.

(5)

Our algorithm converges to the correct mesh if the occupancy grid at the initial resolution contains points from every connected component of both the interior and the exterior of the mesh. It is hence important to take an initialresolution which is high enough to satisfy this condition.
In practice, we found that an initial resolution of 323 was
sufficient in almost all cases.
The initial mesh extracted by the Marching Cubes algorithm can be further refined. In a first step, we simplify
the mesh using the Fast-Quadric-Mesh-Simplification algorithm3 [20]. Finally, we refine the output mesh using firstand second order (i.e., gradient) information. Towards thisgoal, we sample random points pk from each face of theoutput mesh and minimize the loss
KX
k=1

(fθ (pk , x) − τ )2 + λ∇p fθ (pk , x)
− n(pk )k∇p fθ (pk , x)k2(6)

where n(pk ) denotes the normal vector of the mesh at pk . In
practice, we set λ = 0.01. Minimization of the second term
3 https://github.com/sp4cerat/Fast-Quadric-Mesh-Simplification

We implemented our occupancy network using a fullyconnected neural network with 5 ResNet blocks [29] andcondition it on the input using conditional batch normalization [13, 16]. We exploit different encoder architectures
depending on the type of input. For single view 3D reconstruction, we use a ResNet18 architecture [29]. For point
clouds we use the PointNet encoder [54]. For voxelized inputs, we use a 3D convolutional neural network [45]. For
unconditional mesh generation, we use a PointNet [54] forthe encoder network gψ . More details are provided in the
supplementary material.

4. Experiments
We conduct three types of experiments to validate theproposed occupancy networks. First, we analyze the representation power of occupancy networks by examining how
well the network can reconstruct complex 3D shapes from alearned latent embedding. This gives us an upper bound on
the results we can achieve when conditioning our representation on additional input. Second, we condition our occupancy networks on images, noisy point clouds and low resolution voxel representations, and compare the performanceof our method to several state-of-the-art baselines. Finally,
we examine the generative capabilities of occupancy networks by adding an encoder to our model and generating
unconditional samples from this model.4
Baselines: For the single image 3D reconstruction task, we
compare our approach against several state-of-the-art baselines which leverage various 3D representations: we evaluate against 3D-R2N2 [9] as a voxel-based method, Point
Set Generating Networks (PSGN) [17] as a point-based
technique and Pixel2Mesh [70] as well as AtlasNet [26] asmesh-based approaches. For point cloud inputs, we adapted
3D-R2N2 and PSGN by changing the encoder. As meshbased baseline, we use Deep Marching Cubes (DMC) [43]which has recently reported state-of-the-art results on thistask. For the voxel super-resolution task we assess the improvements wrt. the input.
4 The code to reproduce our experiments is available under https://github.com/LMescheder/Occupancy-Networks.

163

323

643

1283

ours

Figure 3: Discrete vs. Continuous. Qualitative comparisonof our continuous representation (right) to voxelizations at
various resolutions (left). Note how our representation encodes details which are lost in voxel-based representations.
Dataset: For all of our experiments we use the ShapeNet[7] subset of Choy et al. [9]. We also use the same voxelization, image renderings and train/test split as Choy et al.
Moreover, we subdivide the training set into a training anda validation set on which we track the loss of our method
and the baselines to determine when to stop training.
In order to generate watertight meshes and to determine
if a point lies in the interior of a mesh (e.g., for measuring
IoU) we use the code provided by Stutz et al. [64]. For a faircomparison, we sample points from the surface of the watertight mesh instead of the original model as ground truth
for PSGN [17], Pixel2Mesh [70] and DMC [43]. All of ourevaluations are conducted wrt. these watertight meshes.
Metrics: For evaluation we use the volumetric IoU, theChamfer-L1 distance and a normal consistency score.
Volumetric IoU is defined as the quotient of the volumeof the two meshes’ union and the volume of their intersection. We obtain unbiased estimates of the volume of the intersection and the union by randomly sampling 100k points
from the bounding volume and determining if the points lie
inside our outside the ground truth / predicted mesh.
The Chamfer-L1 distance is defined as the mean of an
accuracy and and a completeness metric. The accuracy metric is defined as the mean distance of points on the output
mesh to their nearest neighbors on the ground truth mesh.
The completeness metric is defined similarly, but in opposite direction. We estimate both distances efficiently by randomly sampling 100k points from both meshes and usinga KD-tree to estimate the corresponding distances. Like
Fan et al. [17] we use 1/10 times the maximal edge lengthof the current object’s bounding box as unit 1.
Finally, to measure how well the methods can capture
higher order information, we define a normal consistency
score as the mean absolute dot product of the normals in one
mesh and the normals at the corresponding nearest neighbors in the other mesh.

4.1. Representation PowerIn our first experiment, we investigate how well occupancy networks represent 3D geometry, independent of theinaccuracies of the input encoding. The question we try
to answer in this experiment is whether our network can

Figure 4: IoU vs. Resolution. This plot shows the IoU
of a voxelization to the ground truth mesh (solid blue line)
in comparison to our continuous representation (solid orange line) as well as the number of parameters per model
needed for the two representations (dashed lines). Note how
our representation leads to larger IoU wrt. the ground truth
mesh compared to a low-resolution voxel representation. At
the same time, the number of parameters of a voxel representation grows cubically with the resolution, whereas thenumber of parameters of occupancy networks is independent of the resolution.
learn a memory efficient representation of 3D shapes whileat the same time preserving as many details as possible.
This gives us an estimate of the representational capacityof our model and an upper bound on the performance we
may expect when conditioning our model on additional input. Similarly to [67], we embed each training sample in a
512 dimensional latent space and train our neural network
to reconstruct the 3D shape from this embedding.
We apply our method to the training split of the “chair”
category of the ShapeNet dataset. This subset is challenging to represent as it is highly varied and many models contain high-frequency details. Since we are only interested
in reconstructing the training data, we do not use separate
validation and test sets for this experiment.
For evaluation, we measure the volumetric IoU to theground truth mesh. Quantitative results and a comparison
to voxel representations at various resolutions are shownin Fig. 4. We see that the Occupancy Network (ONet) isable to faithfully represent the entire dataset with a highmean IoU of 0.89 while a low-resolution voxel representation is not able to represent the meshes accurately. At thesame time, the occupancy network is able to encode all 4746
training samples with as little as 6M parameters, independently of the resolution. In contrast, the memory requirements of a voxel representation grow cubically with resolution. Qualitative results are shown in Fig. 3. We observethat the occupancy network enables us to represent detailsof the 3D geometry which are lost in a low-resolution voxelization.

Input3D-R2N2PSGN

Pix2Mesh AtlasNet

Ours

Figure 5: Single Image 3D Reconstruction. The input image is shown in the first column, the other columns show
the results for our method compared to various baselines.

4.2. Single Image 3D Reconstruction
In our second experiment, we condition the occupancy
network on an additional view of the object from a random
camera location. The goal of this experiment is to evaluate how well occupancy functions can be inferred fromcomplex input. While we train and test our method on the
ShapeNet dataset, we also present qualitative results for the
KITTI [22] and the Online Products dataset [49].
ShapeNet: In this experiment, we use a ResNet-18 image
encoder, which was pretrained on the ImageNet dataset. Fora fair comparison, we use the same image encoder for both
3D-R2N2 and PSGN5 . For PSGN we use a fully connecteddecoder with 4 layers and 512 hidden units in each layer.
The last layer projects the hidden representation to a 3072
dimensional vector which we reshape into 1024 3D points.
As we use only a single input view, we remove the recurrent network in 3D-R2N2. We reimplemented the methodof [70] in PyTorch, closely following the Tensorflow implementation provided by the authors. For the method of [26],
we use the code and pretrained model from the authors6 .

For all methods, we track the loss and other metrics onthe validation set and stop training as soon as the target metric reaches its optimum. For 3D-R2N2 and our method
we use the IoU to the ground truth mesh as target metric,
for PSGN and Pixel2Mesh we use the Chamfer distance tothe ground truth mesh as target metric. To extract the final
mesh, we use a threshold of 0.4 for 3D-R2N2 as suggested
in the original publication [9]. To choose the threshold parameter τ for our method, we perform grid search on thevalidation set (see supplementary) and found that τ = 0.2
yields a good trade-off between accuracy and completeness.
Qualitative results from our model and the baselines areshown in Fig. 5. We observe that all methods are able tocapture the 3D geometry of the input image. However,
3D-R2N2 produces a very coarse representation and hencelacks details. In contrast, PSGN produces a high-fidelity
output, but lacks connectivity. As a result, PSGN requires
additional lossy post-processing steps to produce a final
mesh7 . Pixel2Mesh is able to create compelling meshes,
but often misses holes in the presence of more complicatedtopologies. Such topologies are frequent, for example, for
the “chairs“ category in the ShapeNet dataset. Similarly,
AtlasNet captures the geometry well, but produces artifacts
in form of self-intersections and overlapping patches.
In contrast, our method is able to capture complex
topologies, produces closed meshes and preserves most of
the details. Please see the supplementary material for additional high resolution results and failure cases.
Quantitative results are shown in Table 1. We observethat our method achieves the highest IoU and normal consistency to the ground truth mesh. Surprisingly, while nottrained wrt. Chamfer distance as PSGN, Pixel2Mesh or AtlasNet, our method also achieves good results for this metric. Note that it is not possible to evaluate the IoU for PSGNor AtlasNet, as they do not yield watertight meshes.
Real Data: To test how well our model generalizes to realdata, we apply our network to the KITTI [22] and Online
Products datasets [49]. To capture the variety in viewpointsof KITTI and Online Products, we rerendered all ShapeNet
objects with random camera locations and retrained our network for this task.
For the KITTI dataset, we additionally use the instancemasks provided in [2] to mask and crop car regions. We
then feed these images into our neural network to predict
the occupancy function. Some selected qualitative results
are shown in Fig. 6a. Despite only trained on synthetic data,
we observe that our method is also able to generate realisticreconstructions in this challenging setting.
For the Online Products dataset, we apply the same pretrained model. Several qualitative results are shown in
Fig. 6b. Again, we observe that our method generalizes rea-

5 See supplementary for a comparison to the original architectures.
6 https://github.com/ThibaultGROUEIX/AtlasNet

7 See supplementary material for meshing results.

3D-R2N2PSGN

IoU
Pix2Mesh

AtlasNetONet

3D-R2N2

PSGN

Chamfer-L1
Pix2Mesh

AtlasNetONet

3D-R2N2

Normal Consistency
PSGN Pix2Mesh AtlasNet

airplanebenchcabinetcar
chairdisplay
lamp
loudspeaker
rifle
sofa
table
telephonevessel

0.426
0.373
0.667
0.661
0.439
0.440
0.281
0.611
0.375
0.626
0.420
0.611
0.482

-

0.420
0.323
0.664
0.552
0.396
0.490
0.323
0.599
0.402
0.613
0.395
0.661
0.397

-

0.571
0.485
0.733
0.737
0.501
0.471
0.371
0.647
0.474
0.680
0.506
0.720
0.530

0.227
0.194
0.217
0.213
0.270
0.314
0.778
0.318
0.183
0.229
0.239
0.195
0.238

0.137
0.181
0.215
0.169
0.247
0.284
0.314
0.316
0.134
0.224
0.222
0.161
0.188

0.187
0.201
0.196
0.180
0.265
0.239
0.308
0.285
0.164
0.212
0.218
0.149
0.212

0.104
0.138
0.175
0.141
0.209
0.198
0.305
0.245
0.115
0.177
0.190
0.128
0.151

0.147
0.155
0.167
0.159
0.228
0.278
0.479
0.300
0.141
0.194
0.189
0.140
0.218

0.629
0.678
0.782
0.714
0.663
0.720
0.560
0.711
0.670
0.731
0.732
0.817
0.629

-

0.759
0.732
0.834
0.756
0.746
0.830
0.666
0.782
0.718
0.820
0.784
0.907
0.699

0.836
0.779
0.850
0.836
0.791
0.858
0.694
0.825
0.725
0.840
0.832
0.923
0.756

0.840
0.813
0.879
0.852
0.823
0.854
0.731
0.832
0.766
0.863
0.858
0.935
0.794

mean

0.493

-

0.480

-

0.571

0.278

0.215

0.216

0.175

0.215

0.695

-

0.772

0.811

0.834

ONet

category

Table 1: Single Image 3D Reconstruction. This table shows a numerical comparison of our approach and the baselines for
single image 3D reconstruction on the ShapeNet dataset. We measure the IoU, Chamfer-L1 distance and Normal Consistency
for various methods wrt. the ground truth mesh. Note that in contrast to prior work, we compute the IoU wrt. the highresolution mesh and not a coarse voxel representation. All methods apart from AtlasNet [26] are evaluated on the test split by
Choy et al. [9]. Since AtlasNet uses a pretrained model, we evaluate it on the intersection of the test splits from [9] and [26].
Input

Reconstruction

Input

Reconstruction
3D-R2N2
PSGN
DMC
ONet

IoU

Chamfer-L1 †Normal Consistency

0.565
0.674
0.778

0.169
0.144
0.117
0.079

0.719
0.848
0.895

Table 2: 3D Reconstruction from Point Clouds. This table shows a numerical comparison of our approach wrt. thebaselines for 3D reconstruction from point clouds on the
ShapeNet dataset. We measure IoU, Chamfer-L1 distanceand Normal Consistency wrt. the ground truth mesh.

(a) KITTI(b) Online Products

Figure 6: Qualitative results for real data. We applied ourtrained model to the KITTI and Online Products datasets.
Despite only trained on synthetic data, our model generalizes reasonably well to real data.
sonably well to real images despite being trained solely on
synthetic data. An additional quantitative evaluation on the
Pix3D dataset [65] can be found in the supplementary.

4.3. Point Cloud Completion
As a second conditional task, we apply our method tothe problem of reconstructing the mesh from noisy point
clouds. Towards this goal, we subsample 300 points fromthe surface of each of the (watertight) ShapeNet models andapply noise using a Gaussian distribution with zero meanand standard deviation 0.05 to the point clouds.
Again, we measure both the IoU and Chamfer-L1 distance wrt. the ground truth mesh. The results are shown in
Table 2. We observe that our method achieves the highest

IoU and normal consistency as well as the lowest ChamferL1 distance. Note that all numbers are significantly better
than for the single image 3D reconstruction task. This canbe explained by the fact that this task is much easier for therecognition model, as there is less ambiguity and the model
only has to fill in the gaps.

4.4. Voxel Super-Resolution
As a final conditional task, we apply occupancy networks to 3D super-resolution [62]. Here, the task is to reconstruct a high-resolution mesh from a coarse 323 voxelization of this mesh.
The results are shown in Table 3. We observe that ourmodel considerably improves IoU, Chamfer-L1 distanceand normal consistency compared to the coarse input mesh.
Please see the supplementary for qualitative results.

4.5. Unconditional Mesh Generation
Finally, we apply our occupancy network to unconditional mesh generation, training it separately on four categories of the ShapeNet dataset in an unsupervised fashion.
† Result for PSGN was corrected after CVPR camera-ready version.

IoU
Input
ONet

Chamfer-L1

0.631
0.703

0.136
0.109

Normal Consistency
0.810
0.879

Table 3: Voxel Super-Resolution. This table shows a numerical comparison of the output of our approach in comparison to the input on the ShapeNet dataset.

Uniform
Uniform (64)
Equal
Surface

IoU

Chamfer-L1

Normal Consistency

0.571
0.554
0.475
0.536

0.215
0.256
0.291
0.254

0.834
0.829
0.835
0.822

(a) Influence of Sampling Strategy

Full model
No ResNet
No CBNIoU

Chamfer-L1

Normal Consistency

0.571
0.559
0.522

0.215
0.243
0.301

0.834
0.831
0.806

(b) Influence of Occupancy Network Architecture

Table 4: Ablation Study. When we vary the sampling strategy, we observe that uniform sampling in the bounding volume performs best. Similarly, when we vary the architecture, we find that our ResNet architecture with conditional
batch normalization yields the best results.
Figure 7: Unconditional 3D Samples. Random samplesof our unsupervised models trained on the categories “car“,“airplane“, “sofa“ and “chair“ of the ShapeNet dataset. We
see that our models are able to capture the distribution of
3D objects and produce compelling new samples.
Our goal is to explore how well our model can represent
the latent space of 3D models. Some samples are shownin Figure 7. Indeed, we find that our model can generatecompelling new models. In the supplementary material weshow interpolations in latent space for our model.

4.6. Ablation StudyIn this section, we test how the various components ofour model affect its performance on the single-image 3Dreconstruction task.
Effect of sampling strategy First, we examine how thesampling strategy affects the performance of our final
model. We try three different sampling strategies: (i) sampling 2048 points uniformly in the bounding volume of theground truth mesh (uniform sampling), (ii) sampling 1024
points inside and 1024 points outside mesh (equal sampling) and (iii) sampling 1024 points uniformly and 1024
points on the surface of the mesh plus some Gaussian noisewith standard deviation 0.1 (surface sampling). We also examine the effect of the number of sampling points by decreasing this number from 2048 to 64.
The results are shown in Table 4a. To our surprise, wefind that uniform, the simplest sampling strategy, works
best. We explain this by the fact that other sampling strategies introduce bias to the model: for example, when sampling an equal number of points inside and outside the mesh,
we implicitly tell the model that every object has a volumeof 0.5. Indeed, when using this sampling strategy, we observe thickening artifacts in the model’s output. Moreover,
we find that reducing the number of sampling points from2048 to 64 still leads to good performance, although themodel does not perform as well as a model trained with
2048 sampling points.
Effect of architecture To test the effect of the various
components of our architecture, we test two variations: (i)we remove the conditional batch normalization and replace
it with a linear layer in the beginning of the network thatprojects the encoding of the input to the required hidden
dimension and (ii) we remove all ResNet blocks in the decoder and replace them with linear blocks. The results arepresented in Table 4b. We find that both components arehelpful to achieve good performance.

5. Conclusion
In this paper, we introduced occupancy networks, a new
representation for 3D geometry. In contrast to existing representations, occupancy networks are not constrained by thediscretization of the 3D space and can hence be used to represent realistic high-resolution meshes.
Our experiments demonstrate that occupancy networks
are very expressive and can be used effectively both for supervised and unsupervised learning. We hence believe thatoccupancy networks are a useful tool which can be applied
to a wide variety of 3D tasks.

Acknowledgements
This work was supported by the Intel Network on Intelligent Systems and by Microsoft Research through its PhD
Scholarship Programme.

References[1] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. J. Guibas.
Learning representations and generative models for 3D point
clouds. In Proc. of the International Conf. on Machine learning (ICML), 2018. 1[2] H. A. Alhaija, S. K. Mustikovela, L. Mescheder, A. Geiger,
and C. Rother. Augmented reality meets deep learning for
car instance segmentation in urban scenes. In Proc. of the
British Machine Vision Conf. (BMVC), 2017. 6[3] F. Bernardini, J. Mittleman, H. Rushmeier, C. Silva, and
G. Taubin. The ball-pivoting algorithm for surface reconstruction. IEEE Trans. on Visualization and Computer
Graphics (VCG), 5(4):349–359, 1999. 2[4] A. Brock, T. Lim, J. M. Ritchie, and N. Weston. Generativeand discriminative voxel modeling with convolutional neuralnetworks. arXiv.org, 1608.04236, 2016. 1, 2
[5] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst. Geometric deep learning: Going beyond euclidean data. Signal Processing Magazine, 34(4):18–42,
2017. 2[6] F. Calakli and G. Taubin.
SSD: smooth signed distance surface reconstruction. Computer Graphics Forum,30(7):1993–2002, 2011. 2[7] A. X. Chang, T. A. Funkhouser, L. J. Guibas, P. Hanrahan,
Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su,
J. Xiao, L. Yi, and F. Yu. ShapeNet: An information-rich 3Dmodel repository. arXiv.org, 1512.03012, 2015. 5[8] Z. Chen and H. Zhang. Learning implicit fields for generative
shape modeling. Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2019. 2[9] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3DR2N2: A unified approach for single and multi-view 3D object reconstruction. In Proc. of the European Conf. on Computer Vision (ECCV), 2016. 1, 2, 4, 5, 6, 7
[10] D. Cremers, M. Rousson, and R. Deriche. A review of statistical approaches to level set segmentation: integrating color,
texture, motion and shape. International journal of computer
vision, 72(2):195–215, 2007. 2[11] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In ACM Trans. on
Graphics (SIGGRAPH), 1996. 2[12] A. Dai, C. R. Qi, and M. Nießner. Shape completion using
3D-encoder-predictor CNNs and shape synthesis. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2017. 2[13] H. de Vries, F. Strub, J. Mary, H. Larochelle, O. Pietquin, and
A. C. Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems (NIPS), 2017. 4[14] A. Dervieux and F. Thomasset. A finite element method forthe simulation of a rayleigh-taylor instability. In Approximation methods for Navier-Stokes problems, pages 145–158.
Springer, 1980. 2[15] H. Drucker and Y. Le Cun. Improving generalization performance using double backpropagation. IEEE Trans. on Neural Networks, 3(6):991–997, 1992. 4[16] V. Dumoulin, I. Belghazi, B. Poole, O. Mastropietro,
A. Lamb, M. Arjovsky, and A. Courville. Adversariallylearned inference. In Proc. of the International Conf. on
Learning Representations (ICLR), 2017. 4[17] H. Fan, H. Su, and L. J. Guibas. A point set generation network for 3D object reconstruction from a single image. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 1, 2, 4, 5[18] O. Faugeras and R. Keriven. Level set methods and the stereo
problem. In International Conference on Scale-Space Theories in Computer Vision, 1997. 2[19] M. Gadelha, S. Maji, and R. Wang. 3D shape induction from
2d views of multiple objects. In Proc. of the International
Conf. on 3D Vision (3DV), 2017. 1[20] M. Garland and P. S. Heckbert. Simplifying surfaces withcolor and texture using quadric error metrics. In Visualization’98. Proceedings, pages 263–269. IEEE, 1998. 4[21] M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J.
Rezende, S. M. A. Eslami, and Y. W. Teh. Neural processes.
arXiv.org, 1807.01622, 2018. 3[22] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun. Vision
meets robotics: The KITTI dataset. International Journalof Robotics Research (IJRR), 32(11):1231–1237, 2013. 6[23] R. Girdhar, D. F. Fouhey, M. Rodriguez, and A. Gupta.
Learning a predictable and generative vector representationfor objects. In Proc. of the European Conf. on Computer
Vision (ECCV), 2016. 1, 2
[24] B. Goldluecke and M. Magnor. Space-time isosurface evolution for temporally coherent 3d reconstruction. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2004. 2[25] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu,
D. Warde-Farley, S. Ozair, A. C. Courville, and Y. Bengio.
Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2014. 2[26] T. Groueix, M. Fisher, V. G. Kim, B. Russell, and M. Aubry.
AtlasNet: A papier-mâché approach to learning 3d surface
generation. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2018. 2, 4, 6, 7
[27] K. Guo, D. Zou, and X. Chen. 3D mesh labeling via deep
convolutional neural networks. In ACM Trans. on Graphics(SIGGRAPH), 2015. 2[28] C. Häne, S. Tulsiani, and J. Malik. Hierarchical surface prediction for 3D object reconstruction. In Proc. of the International Conf. on 3D Vision (3DV), 2017. 2[29] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learningfor image recognition. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2016. 4[30] C. L. Jackins and S. L. Tanimoto. Oct-trees and their use in
representing three-dimensional objects. Computer Graphics
and Image Processing, 14(3):249–270, 1980. 3[31] M. Ji, J. Gall, H. Zheng, Y. Liu, and L. Fang. SurfaceNet:an end-to-end 3D neural network for multiview stereopsis.
In Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2017. 2[32] H. Jin, D. Cremers, A. J. Yezzi, and S. Soatto. Shedding
light on stereoscopic segmentation. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2004. 2

[33] A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik. Endto-end recovery of human shape and pose. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2018. 2[34] A. Kanazawa, S. Tulsiani, A. A. Efros, and J. Malik. Learning category-specific mesh reconstruction from image collections. In Proc. of the European Conf. on Computer Vision(ECCV), 2018. 1[35] A. Kar, C. Häne, and J. Malik. Learning a multi-view stereo
machine. In Advances in Neural Information Processing Systems (NIPS), 2017. 2[36] T. Karras, T. Aila, S. Laine, and J. Lehtinen. Progressive
growing of GANs for improved quality, stability, and variation. In Proc. of the International Conf. on Learning Representations (ICLR), 2018. 1[37] M. M. Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface
reconstruction. In Eurographics Symposium on Geometry
Processing (SGP), 2006. 2[38] M. M. Kazhdan and H. Hoppe. Screened poisson surface
reconstruction. ACM Trans. on Graphics (SIGGRAPH),
32(3):29, 2013. 2[39] D. P. Kingma and M. Welling. Auto-encoding variational
bayes. In Proc. of the International Conf. on Learning Representations (ICLR), 2014. 2, 3[40] K. Kolev, T. Brox, and D. Cremers. Robust variational segmentation of 3d objects from multiple views. In Joint Pattern
Recognition Symposium, 2006. 2[41] C. Kong, C.-H. Lin, and S. Lucey. Using locally corresponding CAD models for dense 3D reconstructions from a single
image. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2017. 2[42] L. Ladicky, O. Saurer, S. Jeong, F. Maninchedda, and
M. Pollefeys. From point clouds to mesh using regression.
In Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2017. 2[43] Y. Liao, S. Donne, and A. Geiger. Deep marching cubes:
Learning explicit surface representations. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2018. 1, 2, 4, 5[44] W. E. Lorensen and H. E. Cline. Marching cubes: A highresolution 3D surface construction algorithm. In ACM Trans.
on Graphics (SIGGRAPH), 1987. 2, 4[45] D. Maturana and S. Scherer. Voxnet: A 3D convolutional
neural network for real-time object recognition. In Proc.
IEEE International Conf. on Intelligent Robots and Systems(IROS), 2015. 2, 4
[46] D. Meagher. Geometric modeling using octree encoding.
Computer graphics and image processing, 19(2):129–147,
1982. 3[47] L. Mescheder, A. Geiger, and S. Nowozin. Which training
methods for GANs do actually converge? In Proc. of theInternational Conf. on Machine learning (ICML), 2018. 1[48] M. Michalkiewicz, J. K. Pontes, D. Jack, M. Baktashmotlagh, and A. Eriksson. Deep level sets: Implicit surface representations for 3d shape inference. arXiv.org, 2019. 2[49] H. Oh Song, Y. Xiang, S. Jegelka, and S. Savarese. Deep
metric learning via lifted structured feature embedding. In

Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 6[50] S. Osher and J. A. Sethian. Fronts propagating withcurvature-dependent speed: algorithms based on hamiltonjacobi formulations. Journal of computational physics,
79(1):12–49, 1988. 2[51] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. arXiv.org, 2019. 2[52] D. Paschalidou, A. O. Ulusoy, C. Schmitt, L. van Gool, and
A. Geiger. Raynet: Learning volumetric 3D reconstruction
with ray potentials. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2018. 2[53] J.-P. Pons, R. Keriven, and O. Faugeras. Modelling dynamic
scenes by registering multi-view image sequences. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2005. 2[54] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deeplearning on point sets for 3D classification and segmentation.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 2, 4[55] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. Guibas.
Volumetric and multi-view CNNs for object classification on
3D data. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 2[56] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. PointNet++:Deep hierarchical feature learning on point sets in a metricspace. In Advances in Neural Information Processing Systems (NIPS), 2017. 2[57] A. Ranjan, T. Bolkart, S. Sanyal, and M. J. Black. Generating
3D faces using convolutional mesh autoencoders. In Proc. of
the European Conf. on Computer Vision (ECCV), 2018. 1, 2
[58] D. J. Rezende, S. M. A. Eslami, S. Mohamed, P. Battaglia,
M. Jaderberg, and N. Heess. Unsupervised learning of 3D
structure from images. In Advances in Neural Information
Processing Systems (NIPS), 2016. 1, 2[59] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic
backpropagation and approximate inference in deep generative models. In Proc. of the International Conf. on Machinelearning (ICML), 2014. 2, 3[60] G. Riegler, A. O. Ulusoy, H. Bischof, and A. Geiger. OctNetFusion: Learning depth fusion from data. In Proc. of theInternational Conf. on 3D Vision (3DV), 2017. 2[61] G. Riegler, A. O. Ulusoy, and A. Geiger. OctNet: Learning
deep 3D representations at high resolutions. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2017. 1[62] E. Smith, S. Fujimoto, and D. Meger. Multi-view silhouetteand depth decomposition for high resolution 3d object representation. In Advances in Neural Information Processing
Systems (NIPS). 2018. 7[63] S. Song and J. Xiao. Deep sliding shapes for amodal 3D
object detection in RGB-D images. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), June
2016. 2[64] D. Stutz and A. Geiger. Learning 3D shape completion fromlaser scan data with weak supervision. In Proc. IEEE Conf.

on Computer Vision and Pattern Recognition (CVPR), 2018.
1, 2, 5[65] X. Sun, J. Wu, X. Zhang, Z. Zhang, C. Zhang, T. Xue, J. B.
Tenenbaum, and W. T. Freeman. Pix3d: Dataset and methods
for single-image 3d shape modeling. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2018. 7[66] R. Szeliski. Rapid octree construction from image sequences. CVGIP: Image understanding, 58(1):23–32, 1993.
3[67] M. Tatarchenko, A. Dosovitskiy, and T. Brox. Octree generating networks: Efficient convolutional architectures forhigh-resolution 3D outputs. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017. 1, 2, 5[68] S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik. Multi-view
supervision for single-view reconstruction via differentiable
ray consistency. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2017. 2[69] A. O. Ulusoy, A. Geiger, and M. J. Black. Towards probabilistic volumetric reconstruction using ray potentials. In
Proc. of the International Conf. on 3D Vision (3DV), 2015.
1, 2[70] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang.
Pixel2Mesh: Generating 3D mesh models from single RGB
images. In Proc. of the European Conf. on Computer Vision(ECCV), 2018. 1, 2, 4, 5, 6[71] P. Wang, Y. Gan, Y. Zhang, and P. Shui. 3D shape segmentation via shape fully convolutional networks. Computers &Graphics, 1702.08675, 2017. 2[72] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and
B. Catanzaro. High-resolution image synthesis and semantic
manipulation with conditional GANs. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2018.
1[73] K.-Y. Wong and R. Cipolla. Structure and motion from silhouettes. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2001. 3[74] J. Wu, Y. Wang, T. Xue, X. Sun, B. Freeman, and J. Tenenbaum. MarrNet: 3D shape reconstruction via 2.5D sketches.
In Advances in Neural Information Processing Systems(NIPS), 2017. 2[75] J. Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum.
Learning a probabilistic latent space of object shapes via 3D
generative-adversarial modeling. In Advances in Neural Information Processing Systems (NIPS), 2016. 1, 2[76] J. Wu, C. Zhang, X. Zhang, Z. Zhang, W. T. Freeman, andJ. B. Tenenbaum. Learning shape priors for single-view 3Dcompletion and reconstruction. In Proc. of the European
Conf. on Computer Vision (ECCV), 2018. 2[77] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, andJ. Xiao. 3D ShapeNets: A deep representation for volumetricshapes. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2015. 1, 2[78] A. Yezzi and S. Soatto. Stereoscopic segmentation. International Journal of Computer Vision, 53(1):31–43, 2003. 2[79] X. Zhang, Z. Zhang, C. Zhang, J. B. Tenenbaum, W. T. Freeman, and J. Wu. Learning to reconstruct shapes from unseenclasses. In Advances in Neural Information Processing Systems (NIPS), 2018. 2