Neural Rendering
Chuan Li
Lambda Labs
Collaborators: Thu Nguyen-Phuoc, Bing Xu, Yongliang Yang, Stephen Balaban, Lucas Theis, Christian Richardt, Junfei
Zhang, Rui Wang, Kun Xu, Rui TangForward (Computer Graphics)

ModelPictures

Forward (Computer Graphics)

Model

PicturesInverse (Computer Vision)

Integral of the incident radians

BRDF

32K SPP Ray Tracing (90 mins 12 CPU Cores)
The Tungsten Renderer

P0
P1

P0
P1

P0
P1
R01 | T01

Inverse (Computer Vision)

P0P1
R01 | T01

Inverse (Computer Vision)

P0
P2

P1
R01 | T01R 12 |T 12

Building Rome in a Day
Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard SzeliskiSub-moduleEnd-2-EndModelPictures

Diﬀerentiable Rendering

1 SPP

2048 SPPSub-modules

Mastering the game of Go with deep neural networks and tree search
David Silver et al.

Sub-modules

Value Network

Mastering the game of Go with deep neural networks and tree search
David Silver et al.

Sub-modules

Policy Network

Value Network

Mastering the game of Go with deep neural networks and tree search
David Silver et al.

Value Networks
Denoising
4 SPP

2^15 SPP

Value Networks
Denoising4 SPP

2^15 SPP

Policy Networks
Same SPP

Value Networks
Denoising
4 SPP

2^15 SPP

Policy Networks
Same SPP

4 SPP

Denoised1 sec 2080 Ti

32K SPP Ray Tracing
90 mins 12 cores CPU

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation
B Xu et al. Siggraph Asia 2019

InputEncoderx

Decoder

Outputloss
Ref

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation
B Xu et al. Siggraph Asia 2019

L1 VGG Loss

L1 VGG Loss + GAN

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation
B Xu et al. Siggraph Asia 2019

Diffuse
Input

Specular
Input

Encoder

Encoderxx

Decoder

Decoder

Diffuse
OutputSpecular
OutputOutputloss
Ref

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation
B Xu et al. Siggraph Asia 2019

Diffuse
Input

Specular
Input

Encoder

Encoderxx

Decoder

Decoder

Diffuse
OutputSpecular
OutputOutputloss
Ref

Auxiliary
Albedo, normal, depth

xElement-wise
Biasing
Conv
LeakyReLU
Conv

AuxiliaryxElement-wise
ScalingElement-wise
Biasing
Conv

Conv

LeakyReLU

LeakyReLU

Conv

Conv

AuxiliaryxElement-wise
Scaling (AND)Element-wise
Biasing (OR)
Conv

Conv

LeakyReLU

LeakyReLU

Conv

Conv

AuxiliaryDenoise comparison 4 SPP

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation
B Xu et al. Siggraph Asia 2019

Value Networks
Denoising
4 SPP

2^15 SPP

Policy Networks
Same SPPNeural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019

incidence radiance map
Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019

Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019

Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019

Sub-moduleEnd-2-EndModelPictures

Diﬀerentiable RenderingRay Tracing
Image Centric

Rasterization
Object Centric

Visibility
Ray Tracing
Image CentricRasterization
Object CentricShading
Ray Tracing
Image Centric

Rasterization
Object CentricDepth Map

Voxel

Point Cloud

MeshMemory

GoodVery Poor

Poor

Very Good

NN friendly

GreatYes

NoEnemyDepth Map

Voxel

Point Cloud

MeshMemory

Good

Very Poor

Poor

Very Good

NN friendly

GreatYes

NoEnemyDepth Map

Voxel

Point Cloud

MeshMemory

GoodVery Poor

Poor

Very GoodNN friendly

GreatYes

NoEnemyDepth Map

Voxel

Point Cloud

MeshMemoryGoodVery Poor

Poor

Very Good

NN friendly

GreatYes

NoEnemyRenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels
3DEncoder

NeuralVoxels
32 x 32 x 32 x 16

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels
3D
Encoder

Visibility

NeuralVoxels

3D-2D

32 x 32 x 32 x 16

NeuralPixels
32 x 32 x 512

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels
3D
Encoder

Visibility

NeuralVoxels

3D-2D

32 x 32 x 32 x 16

NeuralPixels
32 x 32 x 512

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

Neural Voxels
3D
Encoder

Visibility

NeuralVoxels

3D-2D

32 x 32 x 32 x 16

Shading
NeuralPixels

2D
Decoder

32 x 32 x 512

MSE pixel loss

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

ContourToon

Ambient Occlusion
RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

3D
Encoder

NeuralVoxels

3D-2D

NeuralPixels

2D
Decoder64 x 64 x 64 x 1

Channel-wise Concatenation

orTexture
NetworkNeuralTexture
Voxels64 x 64 x 64 x 4

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

Same shape, diﬀerent textures

Same texture, diﬀerent shapes
RenderNet: A deep convolutional network for diﬀerentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018

Depth Map

VoxelPoint Cloud

MeshMemoryGood

Very Poor

Poor

Very Good

NN friendly

GreatYesNoEnemy

Rasterization a RGB point cloud

Neural Point-Based Graphics
KA Aliev et al, arxiv 2019

Rasterization a neural point cloud(First three PCA dimensions of the neural descriptor)

Neural Point-Based Graphics
KA Aliev et al, arxiv 2019

Rasterization a neural point cloud(First three PCA dimensions of the neural descriptor)

Neural Point-Based Graphics
KA Aliev et al, arxiv 2019

RBG rasterization
Neural rasterization

Neural Point-Based Graphics
KA Aliev et al, arxiv 2019

Deferred Neural Rendering:Image Synthesis using Neural Textures
J Thies et al, Siggraph 2019

Neural 3D Mesh Renderer
H Kato et al, CVPR 2018

Sub-moduleEnd-2-End

Model

Pictures?

Approximation

TargetApproximation

Rendered
Approximation

TargetRendered
Approximation

Approximation

Loss

Back-propagate

TargetUpdated
Approximation

Rendered
Approximation

Loss

Back-propagate

TargetUpdated
Approximation

Rendered
Approximation

Loss

Back-propagate
For FreeTargetUpdated
Approximation

Rendered
Approximation

Expensive
Back-propagate

Loss

TargetRendered
Approximation

Encoder

Decoder

Reconstruction

Rendering

Loss

TargetInductive Bias: Separate Appearance from Pose

Human perception imposes coordinate frame on objects

Learning 3D representation from natural images without 3D supervision

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

Conditional GANs

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

Conditional GANs

Info GANs
HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator

RenderNet

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator

RenderNet

3D StyleGAN

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator

RenderNet

3D StyleGAN

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator

RenderNet

3D StyleGANHoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator

RenderNet

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator

RenderNet

Real/Fake

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

3D Generator

RenderNet

A representation that is unbreakable
under 3D rigid-body transformations

HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

z1z2

Shape Controller

Texture Controller

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

HoloGAN: Unsupervised learning of 3D representations from natural images
Thu Nguyen-Phuoc et al, ICCV 2019

Forward (Computer Graphics)

Model

PicturesInverse (Computer Vision)

Sub-module for Ray Tracing (Value / Policy Networks)End-2-End Rasterization (Depthmap, Voxel, Point Cloud, Mesh)

Model

PicturesDiﬀerentiable Rendering (Representation Learning)Thu Nguyen-Phuoc
Lucas Theis

Bing Xu

Christian Richardt

Yongliang Yang
Junfei Zhang

Rui Wang

Stephen Balaban
Kun XuRui Tang