Learning 3D Reconstruction in Function Space
Andreas Geiger
Autonomous Vision GroupMPI for Intelligent Systems and University of Tübingen

April 27, 2020

University of Tübingen
MPI for Intelligent Systems
Autonomous Vision GroupOur goal is to make intelligent systems
more autonomous, robust and safe

Intelligent systems interact with a 3D world

3D reconstruction is a hard problem

1963: Blocks World

Larry Roberts: Machine Perception of Three-Dimensional Solids. PhD Thesis, MIT, 1963.

5

Traditional 3D Reconstruction Pipeline

Input Images

Camera Poses

Dense Correspondences

3D Reconstruction

Depth Map Fusion

Depth Maps
6

Humans recognize 3D from a single 2D image

Can we learn to infer 3D from a 2D image?

3D Reconstruction from a 2D Image

Input Images

Neural Network

3D Reconstruction

10What is a good output representation?

3D Representations

Voxels:I Discretization of 3D space into grid
I Easy to process with neural networks
I Cubic memory O(n3 ) ⇒ limited resolution
I Manhattan world bias
[Maturana et al., IROS 2015]12

3D Representations

Points:I Discretization of surface into 3D points
I Does not model connectivity / topologyI Limited number of points
I Global shape description[Fan et al., CVPR 2017]12

3D Representations

Meshes:I Discretization into vertices and faces
I Limited number of vertices / granularity
I Requires class-speciﬁc template – or –I Leads to self-intersections[Groueix et al., CVPR 2018]12

3D Representations

This work:
I Implicit representation ⇒ No discretization
I Arbitrary topology & resolution
I Low memory footprint
I Not restricted to speciﬁc class12Occupancy Networks
Key Idea:
I Do not represent 3D shape explicitly
I Instead, consider surface implicitly
as decision boundary of a non-linear classiﬁer:3DLocation

Condition(eg, Image)Occupancy
Probability

Concurrent work:
I DeepSDF [Park et al., CVPR 2019]
I IM-NET [Chen et al., CVPR 2019]
Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

13

Network Architecture+Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

14

Training Objective

Occupancy Network:Variational Occupancy Encoder:L(θ, ψ) =K
X

BCE(fθ (pij , zi ), oij ) + KL [qψ (z|(pij , oij )j=1:K ) k p0 (z)]j=1

I K: Randomly sampled 3D points (K = 2048)
I BCE: Cross-entropy loss
I qψ : Encoder

Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

15

Occupancy Networks
N times

evaluate grid points

mark voxels

subdivide voxels

evaluate grid points

marching cubesMultiresolution IsoSurface Extraction (MISE):I Build octree by incrementally querying the occupancy network
I Extract triangular mesh using marching cubes algorithm (1-3 seconds in total)

Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

16

Results

Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

17

Can we also learn about object appearance?

Texture Fields

3D ModelTextured 3D Model
Texture Field
2D ImageOechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

19

Texture Fields

Image Encoder

2D Image

Texture Field

Rendering

Unprojection

Depth Map

3D Shape

Color
3D Point

Predicted Image

Recon.
Loss

True Image

Sampl

ing

Color Legend:
Conditional Model
GAN Model
VAE Model

Shape Encoder

Point Cloud

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

20

Texture Fields

Texture Field

Rendering

Unprojection

Depth Map

3D ShapeColor
3D Point

Predicted Image

True Image

Sampl

ing

Color Legend:
Conditional Model
GAN Model
VAE Model

Shape Encoder

GAN Discriminator

Adversarial
Loss

Point Cloud

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

20

Texture Fields
KL Divergence

VAE Encoder

Texture Field

Rendering

Unprojection

Depth Map

3D Shape

Color
3D Point

Predicted Image

Recon.
Loss

True Image

Sampl

ing

Color Legend:
Conditional Model
GAN Model
VAE Model

Shape Encoder

Point Cloud

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

20

Representation PowerI Ground truth vs. Texture Field vs. Voxelization

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

21

Representation Power (Fit to 10 Models)

Oechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

22

ResultsOechsle, Mescheder, Niemeyer, Strauss and Geiger: Texture Fields: Learning Texture Representations in Function Space. ICCV, 2019.

23

What about object motion?

Occupancy Flow

I Extending Occupancy Networks to 4D is hard (curse of dimensionality)
I Represent shape at t = 0 using a 3D Occupancy Network
I Represent motion by temporally and spatially continuous vector ﬁeld
I Relationship between 3D trajectory s and velocity v given by (differentiable) ODE:∂s(t)= v(s(t), t)∂t
Niemeyer, Mescheder, Oechsle and Geiger: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. ICCV, 2019.

25

Temporal Encoder

Niemeyer, Mescheder, Oechsle and Geiger: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. ICCV, 2019.

26

Occupancy Flow

Niemeyer, Mescheder, Oechsle and Geiger: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. ICCV, 2019.

27

Loss Functions
Reconstruction Loss: 1
Lrecon θ, θ̂ =|B|X

BCE(ôθ,θ̂ (p, τ, x), o)(p,τ,x,o)∈B

Correspondence Loss: 1Lcorr θ̂ =|B|XkΦxθ̂ (s(0), τ ) − s(τ )k2(s,τ,x)∈B

Neat feat:I The correspondence loss is optional
I Correspondences are implicitly established by our model!
Niemeyer, Mescheder, Oechsle and Geiger: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. ICCV, 2019.

28

Results

I No correspondences needed ⇒ implicitly established by our model!
Niemeyer, Mescheder, Oechsle and Geiger: Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics. ICCV, 2019.

29

Can we learn implicit representations from images?

Architecture++Occupancy
Probability

Niemeyer, Mescheder, Oechsle and Geiger: Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. CVPR, 2020.

31

Forward Pass(Rendering)Differentiable Volumetric Rendering

Forward Pass:
I For all pixels u
I Find surface point p̂ along ray w
via ray marching and root ﬁnding
I Evaluate texture ﬁeld tθ (p̂) at p̂
I Insert color tθ (p̂) at pixel u

Niemeyer, Mescheder, Oechsle and Geiger: Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. CVPR, 2020.

33

Backward Pass(Differentiation)Differentiable Volumetric Rendering
Backward Pass:
I Image Observation IPI Loss L(Î, I) = u kÎu − Iu k
I Gradient of loss function:∂L∂θ=∂ Îu∂θ=X ∂Lu∂ Îu·∂ Îu∂θ∂tθ (p̂) ∂tθ (p̂) ∂ p̂
+·∂θ
∂ p̂
∂θ

I Differentiation of fθ (p̂) = τ yields:
∂ p̂
= −w
∂θ



∂fθ (p̂)·w∂ p̂

−1∂fθ (p̂)∂θ⇒ Analytic solution and no need forstoring intermediate results

Niemeyer, Mescheder, Oechsle and Geiger: Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. CVPR, 2020.

35

Results

Niemeyer, Mescheder, Oechsle and Geiger: Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. CVPR, 2020.

36

Very Recent Results

Universal Differentiable Renderer for Implicit Neural Representations

Yariv, Atzmon and Lipman: Universal Differentiable Renderer for Implicit Neural Representations. Arxiv, 2020.

38

Learning Implicit Surface Light Fields
Image Encoding

Image
Image Encoder
EncoderLight setting

Input Image

Conditional ImplicitSurface Light Field (cSLF)
3D Point

Predicted Image

Appearance Field

Lighting Model

Input Shape
Loss

View direction

Geometry
Shape Encoder
Encoder

Shape Encoding

Oechsle, Niemeyer, Mescheder, Strauss and Geiger: Learning Implicit Surface Light Fields. Arxiv, 2020.

Target Image

39

Convolutional Occupancy Networks

Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. Arxiv, 2020.

40

PIFu: Pixel-Aligned Implicit Function

Saito, Huang, Natsume, Morishima, Kanazawa and Li: PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. ICCV, 2019.

41

Deep Structured Implicit Functions

Genova, Cole, Sud, Sarna and Funkhouser: Deep Structured Implicit Functions. CVPR, 2020.

42

NeRF: Representing Scenes as Neural Radiance FieldsZ tf

 Z tf
C(r) =exp −σ(r(s))dsσ(r(t)) c(r(t), d) dt| {z } | {z }tntn|{z} Density
Color
TransmittanceMidenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng: NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Arxiv, 2020.

43

PointRend: Image Segmentation as Rendering

Kirillov, Wu, He and Girshick: PointRend: Image Segmentation as Rendering. CVPR, 2020.

44

Take-Home Messages

Take-Home Messages
Neural Implicit Models:I Effective output representation for shape, appearance, material, motion, etc.
I No discretization, model arbitrary topology
I Can be learned from images via differentiable rendering
I Many applications: reconstruction, view synthesis, segmentation, etc.
However:
I Geometry must be extracted in post-processing step (3 sec for ONet)I Extension to 4D not straightforward (curse of dimensionality)
I Fully connected architecture and global condition lead to oversmooth results
I Promising: Local features (ConvONet, PiFU), Better input encoding (NeRF)
46

Thank you!
http://autonomousvision.github.io