Occupancy Networks:
Learning 3D Reconstruction in Function Space
Andreas Geiger
Autonomous Vision Group
University of Tübingen / MPI for Intelligent Systems

University of Tübingen
MPI for Intelligent Systems
Autonomous Vision GroupOccupancy Networks:
Learning 3D Reconstruction in Function Space[Mescheder, Oechsle, Niemeyer, Nowozing & Geiger, CVPR 2019]

What is a good output representation?

3D Representations

Voxels:I Discretization of 3D space into grid
I Easy to process with neural networks
I Cubic memory O(n3 ) ⇒ limited resolution
I Manhattan world bias
[Maturana et al., IROS 2015]

4

3D Representations

Points:I Discretization of surface into 3D points
I Does not model connectivity / topology
I Limited number of points
I Global shape description[Fan et al., CVPR 2017]

4

3D Representations

Meshes:I Discretization into vertices and faces
I Limited number of vertices / granularity
I Requires class-speciﬁc template – or –I Leads to self-intersections[Groueix et al., CVPR 2018]4

3D Representations

This work:
I Implicit representation ⇒ No discretization
I Arbitrary topology & resolution
I Low memory footprint
I Not restricted to speciﬁc class4

Occupancy Networks
Key Idea:
I Do not represent 3D shape explicitly
I Instead, consider surface implicitly
as decision boundary of a non-linear classiﬁer:

3DLocation

Condition(eg, Image)Occupancy
Probability

Concurrent work:
I DeepSDF [Park et al., CVPR 2019]
I IM-NET [Chen et al., CVPR 2019]
Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

5

Training Objective

Occupancy Network:Variational Occupancy Encoder:L(θ, ψ) =KXBCE(fθ (pij , zi ), oij ) + KL [qψ (z|(pij , oij )j=1:K ) k p0 (z)]j=1

I K: Randomly sampled 3D points (K = 2048)
I BCE: Cross-entropy lossI qψ : Encoder

Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

6

Results

Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

7

Can we also learn about object appearance?

Texture Fields3D Model

Textured 3D Model
Texture Field
2D Image

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

9

Texture Fields

Image Encoder

2D Image

Texture Field

Rendering

Unprojection

Depth Map

3D Shape

Color
3D Point

Predicted Image

Recon.
Loss

True Image

Sampling

Color Legend:
Conditional Model
GAN Model
VAE Model

Shape Encoder

Point Cloud

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

10Texture Fields

Texture Field

Rendering

Unprojection

Depth Map

3D Shape

Color
3D Point

Predicted Image

True Image

Sampling

Color Legend:
Conditional Model
GAN Model
VAE Model

Shape Encoder

GAN Discriminator

Adversarial
Loss

Point Cloud

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

10

Texture Fields
KL Divergence

VAE Encoder

Texture Field

Rendering

Unprojection

Depth Map

3D Shape

Color
3D Point

Predicted Image

Recon.
Loss

True Image

Sampl

ing

Color Legend:
Conditional Model
GAN ModelVAE Model

Shape Encoder

Point Cloud

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

10

Representation PowerI Ground truth vs. Texture Field vs. Voxelization

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

11

Results

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

12

What about object motion?

Occupancy FlowI Extending Occupancy Networks to 4D is hard (curse of dimensionality)
I Represent shape at t = 0 using a 3D Occupancy NetworkI Represent motion by temporally and spatially continuous vector ﬁeld
I Relationship between 3D trajectory s and velocity v given by (differentiable) ODE:∂s(t)= v(s(t), t)∂t
Niemeyer, Mescheder, Oechsle and Geiger: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. ICCV, 2019.

14

Occupancy Flow

Niemeyer, Mescheder, Oechsle and Geiger: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. ICCV, 2019.

15

Results

I No correspondences needed ⇒ implicitly established by our model!
Niemeyer, Mescheder, Oechsle and Geiger: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. ICCV, 2019.

16

Summary17

Thank you!
http://autonomousvision.github.io