Implicit Neural Representations: From Objects to 3D Scenes
Andreas Geiger
Autonomous Vision Group
University of Tübingen / MPI for Intelligent Systems Tübingen

June 19, 2020

University of Tübingen
MPI for Intelligent Systems
Autonomous Vision GroupCollaborators

SongyouPeng

Michael
OechsleCarolinSchmitt

MichaelNiemeyer

Lars
Mescheder

Simon
Donne

GernotRiegler

Vladlen
KoltunMarc
Pollefeys

AndreasGeiger
23D Representations

I Traditional Explicit Representations ⇒ Discrete
I Implicit Neural Representation ⇒ Continuous

Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

3

Limitations
Structure of implicit neural representations:

3D Location
Occupancy
ProbabilityFeatures
Input1DEncoder
Fully Connected
NetworkI Global latent code ⇒ no local information, overly smooth geometry

I Fully connected architecture ⇒ does not exploit translation equivariance
Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

4

Limitations
Implicit models work well for simple objects but poorly on complex scenes:

Mescheder, Oechsle, Niemeyer, Nowozin and Geiger: Occupancy Networks: Learning 3D Reconstruction in Function Space. CVPR, 2019.

4

How to reconstruct large-scale 3D scenes withimplicit neural representations?
Convolutional Occupancy Networks

Convolutional Occupancy Networks
3D Location

Input

2D U-Net
PointNet
Encoder

2D U-Net

Features
BilinearInterpolation
Occupancy
Network

2D U-Net
2D Feature Plane

Occupancy
Probability

2D Feature Planes

I 2D Plane Encoder: Local PointNet processes input, project onto canonical plane
I 2D Plane Decoder: Processed by U-Net, query features via bilinear interpolation
I Occupancy Readout: Shallow occupancy network fθ (·)
Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

6

Convolutional Occupancy NetworksTrilinear
Interpolation

Input

3D Location

Occupancy
ProbabilityFeatures
PointNet
Encoder

3D U-Net

3D Feature Volume

Occupancy
Network
3D Feature Volume

I 3D Volume Encoder: Local PointNet processes input, volumetric feature encoding
I 3D Volume Decoder: Processed by 3D U-Net, query features via trilinear interp.
I Occupancy Readout: Shallow occupancy network fθ (·)
Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

6

Comparison
3D Location

Occupancy(NetworksOccupancy
Probability
FeaturesInput

1DEncoder
Fully Connected
Network

Convolutional(Occupancy(Networks

TrilinearInterpolation

3D Location
Occupancy
Probability
Features

Input

3D Encoder+ 3D Conv

Fully Connected
Network
3D Feature Volume

Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

Voxel Centers

7

ResultsObject-Level Reconstruction

Input

ONetOurs

Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

GT9

Training Speed

Validation IoU

0.80.7
ONet
PointConv
Ours-2D (642)0.6

Ours-2D (3 × 642)
Ours-3D (323)1611162126
Training Iterations (×10K)

Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

3110

Training Speed

Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

10

Scene-Level Reconstruction

Input

ONetSPSR

OursGTI Trained and evaluated on synthetic rooms
Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

11

Scene-Level Reconstruction

InputONetSPSROurs

I Trained on synthetic rooms, evaluated on ScanNet
Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

12

Large-Scale Reconstruction

Results on Matterport3D

Room-Level
Reconstruction

I Fully convolutional model
I Trained on synthetic crops
I Sliding window evaluation
I Scales to any scene size

Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

13

Key Insights

Key Insights:
I Convolutional models allow for scaling implicit models to larger scenes
I Convolutional models train faster than fully implicit models
I Convolutional models allow for incorporating local feature information
I For objects, the 3-plane model has the best accuracy/memory trade-off
I For scenes, the volumetric representation performs best
I Models transfer from synthetic to real scenes

Peng, Niemeyer, Mescheder, Pollefeys and Geiger: Convolutional Occupancy Networks. arXiv, 2020.

14

How to capturing the visual appearance of objects?
Conditional Surface Light Fields

Problem Deﬁnition
Manipulating the Illumination

Input Image

NeuralNetwork
Changing the Viewpoint

3D Geometry

Oechsle, Niemeyer, Mescheder, Strauss and Geiger: Learning Implicit Surface Light Fields. arXiv, 2020.

16

Existing Representation

Texture Fields
I 3D consistent
I Generalize across objects
I View-point independent
I Do not model lighting[Oechsle et al., ICCV 2019]

Oechsle, Niemeyer, Mescheder, Strauss and Geiger: Learning Implicit Surface Light Fields. arXiv, 2020.

17

Conditional Surface Light Field
Rendering equation:
ZL(p, v, l, n) =ΩsvBRDF(p, r, v) · l(r) · (nT r) dr

Conditional surface light ﬁeld:
LcSLF (p, v, l) : R3 × R3 × RM → R3
Light setting

View direction

Neural Network
3D Point

Oechsle, Niemeyer, Mescheder, Strauss and Geiger: Learning Implicit Surface Light Fields. arXiv, 2020.

18

Overﬁtting to Single Objects
Light setting

Conditional Implicit
Surface Light Field (cSLF)
3D Point

Appearance Field

Predicted Image

Lighting Model

Input ShapeLossView direction

Target ImageOechsle, Niemeyer, Mescheder, Strauss and Geiger: Learning Implicit Surface Light Fields. arXiv, 2020.

19

Single-Image Appearance Prediction
Image Encoding

Image
Image Encoder
EncoderLight setting

Input Image

Conditional Implicit
Surface Light Field (cSLF)
3D Point

Predicted Image

Appearance Field

Lighting Model

Input Shape
Loss

View direction

Geometry
Shape Encoder
Encoder

Shape Encoding

Oechsle, Niemeyer, Mescheder, Strauss and Geiger: Learning Implicit Surface Light Fields. arXiv, 2020.

Target Image

20

Generative Model
KL Divergence

VAE
Encoder

Image Encoder

Light setting

Target Image
Loss

Conditional Implicit
Surface Light Field (cSLF)
3D Point

Predicted Image

Appearance Field

Lighting Model

Input Shape
View direction

Geometry
Shape Encoder
Encoder

Shape Encoding

Oechsle, Niemeyer, Mescheder, Strauss and Geiger: Learning Implicit Surface Light Fields. arXiv, 2020.

21

How to obtain training data with materials?
Joint Estimation of Pose, Geometry and svBRDF

Joint Estimation of Pose, Geometry and svBRDF
Goal: Dataset of 3D indoor scenes
captured with high accuracy
from a handheld mobile sensor.
Custom built sensor rig:
I Custom IR depth sensor
similar to Microsoft Kinect
I Active illumination + RGB camera
for material estimation

Schmitt, Donné, Riegler, Koltun and Geiger: On Joint Estimation of Pose, Geometry and svBRDF From a Handheld Scanner. CVPR, 2020.

23

Joint Estimation of Pose, Geometry and svBRDF

Schmitt, Donné, Riegler, Koltun and Geiger: On Joint Estimation of Pose, Geometry and svBRDF From a Handheld Scanner. CVPR, 2020.

23

Materials ←→ Geometry−→Accurate geometry reconstruction requires known appearance properties←−Accurate appearance estimation requires very well known geometry←→ Joint estimation requires only a rough initialization for bothSchmitt, Donné, Riegler, Koltun and Geiger: On Joint Estimation of Pose, Geometry and svBRDF From a Handheld Scanner. CVPR, 2020.

24

Joint Estimation of Pose, Geometry and svBRDF
Contributions:
I Joint formulation
I Single objective function
minimized using off-the-shelf
gradient-based solvers
I Meaningful segmentationdifferentiably part of the optimization
I Accurate geometry
with very ﬁne details

X ∗ = argmin L(X )X

Schmitt, Donné, Riegler, Koltun and Geiger: On Joint Estimation of Pose, Geometry and svBRDF From a Handheld Scanner. CVPR, 2020.

25

Joint Estimation of Pose, Geometry and svBRDF
Contributions:
I Joint formulation
I Single objective function
minimized using off-the-shelf
gradient-based solvers
I Meaningful segmentationdifferentiably part of the optimization
I Accurate geometry
with very ﬁne details
Reconstruction

Segmentation

Schmitt, Donné, Riegler, Koltun and Geiger: On Joint Estimation of Pose, Geometry and svBRDF From a Handheld Scanner. CVPR, 2020.

25

Joint Estimation of Pose, Geometry and svBRDF
Contributions:
I Joint formulation
I Single objective function
minimized using off-the-shelf
gradient-based solvers
I Meaningful segmentationdifferentiably part of the optimization
I Accurate geometry
with very ﬁne details

Geometric error

Schmitt, Donné, Riegler, Koltun and Geiger: On Joint Estimation of Pose, Geometry and svBRDF From a Handheld Scanner. CVPR, 2020.

25

Qualitative Results
Conclusion:
I Joint estimation helps
I This is only a ﬁrst stepI Object-level reconstruction
remains challenging withlimited observations
I Scaling to larger scenes
I Scaling to scenes withexternal illumination
Relighting

Novel Viewpoint

Schmitt, Donné, Riegler, Koltun and Geiger: On Joint Estimation of Pose, Geometry and svBRDF From a Handheld Scanner. CVPR, 2020.

26

How to obtain training data with semantic labels?
KITTI-360

KITTI-360

Xie, Kiefel, Sun, Geiger: Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer. CVPR, 2016

28

KITTI-360
Sensors:
I Front-facing stereo cameraI 360◦ ﬁsheye cameras
I Velodyne HDL 64 laser scanner
I SICK pushbroom laser scanner
I IMU/GPS localization system
Features:
I Driving distance: 73.7 km

Frames: 4 × 83,000

I All frames accurately geolocalized (⇒ OpenStreetMap)
I Semantic label deﬁnition consistent with Cityscapes, 19 classes for evaluation
I Each instance assigned with a consistent instance ID across all frames
Xie, Kiefel, Sun, Geiger: Semantic Instance Annotation of Street Scenes by 3D to 2D Label Transfer. CVPR, 201629

Sensors
All heights wrt. road surface

Wheel axis
(0.30m)1.60 mCamerainclination:~5° (down)
Cam 2z0.60 mCam 1z

Perspectivecameras(1.55 m)xy

Fisheyecameras Cam 4(1.95 m) z

y x
0.08 m0.92 mVelodyne xx (1.73 m)

y0.05 m

SICKz(1.69 m)y0.40 mGPS/IMU xy(0.9 m)x0.79 mz 0.81 m

Cam 3yz

0.32 m

0.48 m2.71 m30

360◦ 2D Sensors

31

360◦ 3D Sensors

32

3D Annotations

RGB

Bounding Box

Semantic

Instance
33

2D Annotations

Semantic

Instance

Conﬁdence

Bounding Box

34

Thank you!
http://autonomousvision.github.io