Generative Neural Scene Representationsfor 3D-Aware Image Synthesis
Andreas Geiger
Autonomous Vision Group
University of Tübingen and MPI for Intelligent Systems

Covered Papers
GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
Katja Schwarz and Yiyi Liao and Michael Niemeyer and Andreas Geiger
NeurIPS 2020
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields
Michael Niemeyer, Andreas Geiger
CVPR 2021
CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields
Michael Niemeyer, Andreas Geiger
3DV 20212

Collaborators

Michael Niemeyer

Katja Schwarz

Yiyi Liao

3

Generative models are fantastic!

Generative Models
Sample a latent code from the prior distribution.

Latent Code5

Generative Models
Pass latent code to trained generator Gθ .

Latent Code

Generator Gθ

5

Generative Models
The generator outputs a synthesized image.

Latent Code∗

Generator Gθ

Generated Image∗The generated images are samples from StyleGAN2.

5

Generative Models
Sample more latent codes to get different generated images.

Latent Code∗

Generator Gθ

Generated Image∗The generated images are samples from StyleGAN2.

5

Generative Models
Sample more latent codes to get different generated images.

Latent Code∗

Generator Gθ

Generated Image∗The generated images are samples from StyleGAN2.

5

Is generating photorealistic images all we need?

Generative Models
For many applications, we require control over the image generation process:7Generative Models
For many applications, we require control over the image generation process:Animation Movies

Video Source: Disney’s Toy Story 4 Trailer
7

Generative Models
For many applications, we require control over the image generation process:Video Games

Video Source: Gran Turismo 7 Trailer
7

Generative Models
For many applications, we require control over the image generation process:
Virtual Reality

Video Source: Oculus Rift Trailer
7

Generative Models
Goal: A generative model for 3D-aware image synthesis which allows us to:8

Generative Models
Goal: A generative model for 3D-aware image synthesis which allows us to:
I Generate photorealistic images

8Generative Models
Goal: A generative model for 3D-aware image synthesis which allows us to:
I Generate photorealistic images
I Control individual objects wrt. their appearance, pose and size in 3D8

Generative Models
Goal: A generative model for 3D-aware image synthesis which allows us to:
I Generate photorealistic images
I Control individual objects wrt. their appearance, pose and size in 3D
I Control the camera viewpoint in 3D8

Generative Models
Goal: A generative model for 3D-aware image synthesis which allows us to:
I Generate photorealistic images
I Control individual objects wrt. their appearance, pose and size in 3D
I Control the camera viewpoint in 3D
I Train from unstructured and unposed images

8

What representation should we use for
3D-aware image synthesis?

3D Representations
Voxel-based 3D Shape with Volumetric Rendering

SampleRendering

COLMAP

3D Object

PlatonicGAN [Henzler et al., ICCV 2019]

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

10

3D Representations
Voxel-based 3D Shape with Volumetric Rendering

SampleRendering

COLMAP

3D Object

PlatonicGAN [Henzler et al., ICCV 2019]

Multi-view consistent

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

10

3D Representations
Voxel-based 3D Shape with Volumetric Rendering

SampleRendering

COLMAP

3D Object

PlatonicGAN [Henzler et al., ICCV 2019]

Multi-view consistent
Low image ﬁdelity, high memory consumption

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

10

3D Representations
Voxel-based 3D Latent Feature with Learnable Projection

NeuralSampleRendering

COLMAP

3D feature

HoloGAN [Nguyen-Phuoc et al., ICCV 2019]

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

11

3D Representations
Voxel-based 3D Latent Feature with Learnable Projection

NeuralSampleRendering

COLMAP

3D feature

HoloGAN [Nguyen-Phuoc et al., ICCV 2019]

High image ﬁdelity

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

11

3D Representations
Voxel-based 3D Latent Feature with Learnable Projection

NeuralSampleRendering

COLMAP

3D feature

HoloGAN [Nguyen-Phuoc et al., ICCV 2019]

High image ﬁdelity
Object shape and identity vary with viewpoint due to learnable projection

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

11

3D Representations
Generative Radiance Fields

SampleRendering

COLMAP

Radiance Field

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

12

3D Representations
Generative Radiance Fields

SampleRendering

COLMAP

Radiance Field

Continuous representation, multi-view consistent

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

12

3D Representations
Generative Radiance Fields

SampleRendering

COLMAP

Radiance Field

Continuous representation, multi-view consistentHigh image ﬁdelity, low memory consumption

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

12

GRAF: Generative Radiance Fields
Sample camera matrix K, camera pose ξ ∼ pξ , and patch sampling pattern ν ∼ pν .

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

13

GRAF: Generative Radiance Fields
Pass K, ξ, and ν to generator Gθ which samples corresponding pixels / rays.

Ray
Sampling

Patch Generator

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

13

GRAF: Generative Radiance Fields
For each ray, get viewing direction dr and sample 3D points xir along ray.

Ray
Ray
Sampling
3D Point
Sampling
Ray

Patch Generator

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

13

GRAF: Generative Radiance Fields
Push dr and xir through positional encoding γ as input to conditional radiance ﬁeld gθ .

Ray
Ray
Sampling
3D Point
Sampling
Ray

Conditional Radiance Field

3D Point

Patch Generator

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

13

GRAF: Generative Radiance Fields
Sample latent shape and appearance codes zs , za and pass them as condition to gθ .

Ray
Ray
Sampling
3D Point
Sampling
Ray

Conditional Radiance Field

3D Point

Patch Generator

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

13

GRAF: Generative Radiance Fields
Perform volume-rendering for each ray, resulting in predicted patch P0 .

Ray
Ray
Sampling

Conditional Radiance Field
Volume Rendering

3D Point
Sampling
Ray

Predicted Patch

3D Point

Patch Generator

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

13

GRAF: Generative Radiance Fields
Sample patch P from real image I drawn from the data distribution pD with pattern ν.

Ray
Ray
Sampling

Conditional Radiance Field
Volume Rendering

3D Point
Sampling
Ray

Predicted Patch

3D Point

Pixel
Sampling

Real
Patch

Patch Generator

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

13

GRAF: Generative Radiance Fields
Pass fake and real patch P0 , P to discriminator Dφ and train with adversarial loss.
Patch
Discriminator
Ray
Ray
Sampling

Conditional Radiance Field
Volume Rendering

3D Point
Sampling
Ray

Predicted Patch

3D Point

Pixel
Sampling

Real
Patch

Patch Generator

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

13

GRAF: Generative Radiance Fields
Patch
Discriminator
Ray
Ray
Sampling

Conditional Radiance Field
Volume Rendering

3D Point
Sampling
Ray

Predicted Patch

3D Point

Pixel
Sampling

Real
Patch

Patch Generator

I Key idea: Generator/discriminator for image patches of size 32 × 32 pixels
I Patches sampled at random locations and with random scales (using dilation)
Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

14

Conditional Radiance Fields

EncodingccColor Head

Density Head

Conditional Radiance Field Architecture:
I Fully-connected MLP gθ with ReLU activations
I Appearance code za and view direction inﬂuence only the output color
I Inductive biases allow the model to disentangle shape and appearance
Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

15

Results

Generative Radiance Fields
Results on synthetic Carla dataset at 2562 pixels:

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

17

Generative Radiance Fields
Results on real CelebA-HQ dataset at 2562 pixels:

Schwarz, Liao, Niemeyer, Geiger: GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. NeurIPS, 2020.

18

How can we scale this idea tomore complex, multi-object scenes?

GIRAFFE: Compositional Generative Neural Feature Fields
GRAF:
I Incorporate a 3D representation into the generative model

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

20

GIRAFFE: Compositional Generative Neural Feature Fields
GRAF:
I Incorporate a 3D representation into the generative model
GIRAFFE:
I Incorporate a compositional 3D scene representation into the generative model

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

20

GIRAFFE: Compositional Generative Neural Feature Fields
GRAF:
I Incorporate a 3D representation into the generative model
GIRAFFE:
I Incorporate a compositional 3D scene representation into the generative model
I Incorporate a neural renderer to yield fast and high-quality inference

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

20

GIRAFFE
Sample N shape and appearance codes.

Shape andAppearance

Shape andAppearance

Shape andAppearance

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

21

GIRAFFE
Get N feature ﬁelds. Note: We show features in RGB color for clarity.

Shape andAppearance

Shape andAppearance

Shape andAppearance

SampledFeature Fields

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

21

GIRAFFE
Sample size and pose for each feature ﬁeld.

Shape andAppearance

Pose

Shape andAppearance

Pose

Shape andAppearance

Pose

SampledFeature Fields

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

21

GIRAFFE
Get posed feature ﬁelds.

Shape andAppearance

Pose

Shape andAppearance

Pose

Shape andAppearance

Pose

SampledFeature Fields

Posed
Feature Fields

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

21

GIRAFFE
Composite all feature feature ﬁelds into a single 3D scene representation.

Implicit 3D Scene
Representation

Shape andAppearance

Pose

Shape andAppearance

Pose

Shape andAppearance

Pose

SampledFeature Fields

Posed
Feature Fields

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

21

GIRAFFE
Sample a camera pose.

Implicit 3D Scene
Representation

Shape andAppearance

Pose

Shape andAppearance

Pose

Pose
Shape andAppearance

CameraPose

SampledFeature Fields

Posed
Feature Fields

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

21

GIRAFFE
Perform volume rendering and get low-resolution feature image.

Implicit 3D Scene
Representation

Shape andAppearance

Pose

Shape andAppearance

Pose

Feature Image

Pose
Shape andAppearance

CameraPose

SampledFeature Fields

Posed
Feature Fields

Volume Renderingof Feature Image

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

21

GIRAFFE
Pass feature image to neural renderer to obtain ﬁnal high-resolution image.
Output Image
Implicit 3D Scene
Representation

Shape andAppearance

Pose

Shape andAppearance

Decoder
2D CNN

Pose

Feature Image

Pose
Shape andAppearance

CameraPose

SampledFeature Fields

Posed
Feature Fields

Volume Renderingof Feature Image

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

Neural Renderingof Output Image

21

GIRAFFE
At test time, we sample individual codes to control shapes, appearances and poses.
Output Image
Implicit 3D Scene
Representation

Shape andAppearance

Pose

Decoder
2D CNN

Shape andAppearance

Pose

Feature Image

Pose
Shape andAppearance

CameraPose

SampledFeature Fields

Posed
Feature Fields

Volume Renderingof Feature Image

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

Neural Renderingof Output Image

21

GIRAFFE
Output Image
Implicit 3D Scene
Representation

Shape andAppearance

Pose

Shape andAppearance

Decoder
2D CNN

Pose

Feature Image

Pose
Shape andAppearance

CameraPose

SampledFeature Fields

Posed
Feature Fields

Volume Renderingof Feature Image

Neural Renderingof Output Image

I We volume-render the feature image at 162 pixels and upsample via the decoder
I Unlike GRAF, we can directly train with an adversarial loss at full resolution
Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

22

GIRAFFE
We use GRAF’s architecture, but replace the color head with a feature head:

EncodingccColor Head

Density Head

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

23

GIRAFFE
We use GRAF’s architecture, but replace the color head with a feature head:

EncodingcFeature Headc
Density Head

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

23

GIRAFFE
Scene Composition
We have N feature ﬁelds
hi (x, d) = (σi , fi )which predict a density σi and a feature vector fi for any (x, d).
We combine densities by summation and features by density weighted averaging:
σ=

N
X
i=1

N

σi

f=

1X
σi fi
σ
i=1

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

24

Results

GIRAFFE
We compare object translation for a 2D-based GAN (left) and our method (right):

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

26

GIRAFFE
We can perform more complex operations like circular translations

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

27

GIRAFFE
We can add more objects at test time (trained on two-object)

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

28

GIRAFFE
We can rotate the object

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

29

GIRAFFE
We can translate the object

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

30

GIRAFFE
We can change the object shape

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

31

GIRAFFE
We can change the object appearance

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

32

GIRAFFE
We can generate out-of-distribution samples

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

33

GIRAFFE
Rendering Time

GRAF
GIRAFFE

64 × 64

256 × 256110.1ms
4.8ms

1595.0ms
5.9ms

I We volume-render the feature image at 16 × 16 pixels.
I CNN-based neural rendering and upsampling yields fast inference.

Niemeyer, Geiger: GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. CVPR, 2021.

34

Can we learn the camera distribution?

CAMPARI: Camera-Aware Generative Neural Radiance Fields
GRAF, GIRAFFE:I Learn a 3D-aware image generator from unposed image collections
I Requires careful tuning of camera pose distributions

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

36

CAMPARI: Camera-Aware Generative Neural Radiance Fields
GRAF, GIRAFFE:I Learn a 3D-aware image generator from unposed image collections
I Requires careful tuning of camera pose distributions
CAMPARI:
I Learn a 3D aware image generator and a camera generator jointly.

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

36

CAMPARISample prior camera ξ prior ∼ pξ . Parameters: intrinsics fx , fy and extrinsics r, αr , αe .

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

37

CAMPARI
pred
Pass ξ prior to camera generator GC.
θ and obtain predicted camera ξ

Camera Generator

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

37

CAMPARI
Pass ξ pred and sampled FG / BG latent codes to 3D-aware image generator.

VolumeRendering

Camera Generator

3D-Aware Image Generator

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

37

CAMPARI
Train entire method with adversarial objective similar to GRAF, GIRAFFE.

VolumeRendering

Camera Generator

3D-Aware Image Generator

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

Discriminator

37

Results

CAMPARICAMPARI learns to match the GT distribution for synthetic datasets.

Prior
GT
Predicted

80

60

40

20

0

20

Elevation Angle (Degrees)

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

40

60

80

39

CAMPARICAMPARI learns to match the GT distribution for synthetic datasets.

Prior
GT
Predicted

80

60

40

20

0

20

Elevation Angle (Degrees)

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

40

60

80

39

CAMPARI
This leads to signiﬁcantly improved performance compared to untuned models.

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

40

CAMPARI
This leads to signiﬁcantly improved performance compared to untuned models.

Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

40

Generative Neural Scene Representations
Summary
I We propose novel methods for 3D controllable image synthesis
I We demonstrate training from raw, unposed image collections
I We incorporate compositional 3D scene structure into the generative model
I We gain explicit control over individual objects during synthesis
Future Research
I Scale to larger and more complex multi-object scenes
I Enhance photorealism to match 2D GANs (e.g., StyleGAN2)I Disentangle lighting, materials, etc.
Niemeyer, Geiger: CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields. 3DV, 2021.

41

Thank you!
http://autonomousvision.github.io