Learning Neural Light Transport
Paul Sanzenbacher
Autonomous Vision Group
University of Tübingen, Germany

Lars MeschederAmazonTübingen, Germany

Andreas Geiger
Autonomous Vision Group
University of Tübingen, Germany

paul.sanzenbacher@tue.mpg.delamesch@amazon.de

andreas.geiger@tue.mpg.de

Abstract

times of up to multiple hours for a single image. This makes
it difficult to use in the context of training neural networks.
On the other hand, recent advances in deep generativemodels (e.g., GANs [22]) enabled the generation of highly
realistic images [50, 34, 35] in milliseconds on commercial GPUs. However, most existing approaches make useof rather abstract latent representations which do not allow for precise control over the 3D content. Moreover, thelack of a holistic 3D scene description limits neural rendering approaches in their ability to render images that areconsistent across viewpoints or time. While several recent
works [78, 3, 52, 61, 53] have shown that neural networks
can produce consistent images for a given scene, these approaches usually do not explicitly reason about light transport. Consequently, they are not able to handle fine-grained
geometric scene manipulation and do not explicitly integrate illumination into the 3D representation. This motivates us to seek a more explicit neural representation thatreasons about how light propagates through 3D space andthat is able to capture dynamic global illumination effects.
We summarize the contributions of this work as follows.

In this work, we investigate the importance of 3D reasoning for photorealistic and controllable neural rendering. Towards this goal, we develop an approach which explicitly formulates light transport in static and dynamic 3D
scenes using a neural network. In contrast to existing approaches that operate primarily in the 2D image domain,
our approach reasons in both 3D and 2D space, thus enabling both global illumination effects and manipulation of
3D scene geometry. Our differentiable model can be trained
jointly on multiple scenes from noisy renderings and is able
to produce photorealistic renderings with accurate lighting,capturing shadows, reflections and refractions. Moreover, it
compares favorably to baselines which combine path tracing and image denoising at the same computational budget.

1. Introduction
Photorealistic rendering is a core problem in graphics
and vision. Algorithms which are able to reason about direct and indirect illumination of a scene (i.e., global illumination) have become an essential building block for awide range of applications such as gaming, virtual reality,
movies and others. With the advent of deep learning, synthetic data generation emerged as another important application [2, 18, 77, 14, 85] with the potential to satisfy the notorious data hunger of modern deep learning systems. However, as modern deep neural networks require large amountsof data, most existing approaches rely on approximate rendering techniques to accelerate training [18, 77, 14, 85].
Training embodied agents (i.e., using reinforcement learning) poses even stronger demands wrt. simulation time [17].
Historically, photorealistic image synthesis is achieved
using sampling-based rendering techniques [64, 86] where
the physics of light transport [33] is exploited to transforma geometric description of a scene into a realistic image.
However, while physically based rendering yields photorealistic results, it is also notoriously slow with rendering

Contributions: We investigate the importance of 3D vs.
2D reasoning for efficient learning-based photorealistic rendering. Towards this goal, we present a learning-based
approach (see Fig. 1 for a high-level overview) which allows for inferring photorealistic images from a point-cloud
based scene representation in real time. In contrast to existing approaches, our method performs reasoning both in
3D and 2D space which allows for learning the physicallight transport in a scene. This enables our method to handle scene modifications such as object translations, object
removal and lighting changes. At the same time, our framework allows for learning useful heuristics (e.g., shadowsthat are not affected by moving objects) from the training data, enabling fast rendering without sacrificing quality.
We introduce two variants of our approach: (1) a PointNetbased [66] model and (2) an extension of this model using
photon sampling which improves the quality of shadowsand specular reflections. We show, both theoretically andempirically, that our model can be trained without bias us1

Light Transport Layer

Image Synthesis Layer

Neural Image

Figure 1. Motivation. We learn photorealistic rendering using a 3D Light Transport Layer in combination with a 2D Image Synthesis
Layer. We demonstrate that our hybrid 3D-2D approach is able to synthesize realistic images with global illumination effects in real-time.

ing noisy renderings from a physically-based renderer. We
will release our code and data upon publication.

[1] and Nalbach et al. [55] describe methods for generating
renderings from multiple image buffers such as depth andmaterials. While this allows for rendering realistic images,
a major limitation is that operations are performed in image
space, making it hard to model global illumination.

2. Related Work
Rendering: Physically-based rendering (PBR) is a wellstudied field [86, 64] where much of the research in recent
years focuses on optimizing different parts of the rendering pipeline [54, 87, 70, 68] or denoising of noisy PBR
renderings [7, 48, 8]. Moreover, there is a trend of making rendering algorithms differentiable in order to estimate
scene properties [4, 47, 21, 20, 56, 31, 11] or to use them fortraining deep neural networks [84, 60, 43, 36]. While recent
approaches strive to achieve real-time photorealistic rendering [75, 74], they often require additional assumptions suchas temporal smoothness and are limited by temporal accumulation of information in screen space. In this paper, weprobe the suitability of neural networks for learning lighttransport end-to-end, with the goal of differentiable rendering of dynamic scenes with complex lighting.

Scene Representations: [3, 65, 52, 24] propose several
deep models for rendering novel views from point clouds[19, 67, 93, 27]. However, existing approaches are limited
to single objects or small static scenes. Moreover, they do
not explicitly model light transport or consider only diffuse,homogeneous materials.
Recently, several alternative scene representations havebeen considered [79, 83, 51, 62, 59]. Sitzmann et al. [78]propose DeepVoxels, where a single static object-centric
scene is encoded in a voxel grid of learned features. Rematas et al. [69] present Neural Voxel Renderer, a neural rendering framework that maps a voxelized scene to an
RGB image. While both methods allow for rendering a sequence of coherent images, they are limited to comparably
small scenes with a single object due to the high memory requirements of voxel-based representations. In contrast, ourapproach exploits a point-based scene representation andtherefore scales to larger scenes with multiple objects. Furthermore, neither of them reasons explicitly about materials,
light transport or global illumination.
Deferred Neural Rendering [83] proposes a neural texture representation for novel view synthesis of single objects with fixed lighting. Deep Appearance Models [45] encode the facial geometry and texture of a particular person.
Neural Volumes [46] encode multiple images of an object
into a neural volume representation which is rendered using
ray marching. While producing impressive results, theseworks are trained for a single object or scene and assume
static lighting. Moreover, they only allow for limited editing as their main focus is on view synthesis. In contrast, thefocus of our work is on representing multiple scenes witha single model, where both objects and lighting can be dynamically rearranged in real time during inference.

Generative Models: Recently, deep generative models
such as variational autoencoders (VAEs) [26, 9, 38, 28]or (conditional) generative adversarial networks (GANs)[50, 34, 35, 6, 29] have demonstrated that neural networks
are capable of generating photorealistic synthetic imagery.
While some methods have addressed the problem of 3D
controllable image synthesis [44, 57, 76, 58], these methods are currently restricted to a small number of comparably
simple objects and do not explicitly reason about materials
or global light transport.
Novel View Synthesis: Novel view synthesis approaches[63, 94, 15, 82, 16, 40, 12, 90, 91, 71, 10, 88, 16, 53, 92, 49,
72] focus on generating novel views of a single densely captured scene. However, since these methods lack an explicit
scene representation which captures geometry, material andlighting, it is hard to gain precise control over their output.
In contrast, we learn to render images in a differentiable
manner from a holistic scene representation. Alhaija et al.
2

Processed
Features

Projection

Occlusion Filter

Point Cloudwith Features

View Transform

Light
Transport
Layer

Image
Synthesis
Layer
Prediction
Projected
Features

Scene Geometry
and Appearance

Depth Map

Normal Map

Albedo Map

View Ray Map

Evaluation

Loss

Ground Truth

Supervision

Figure 2. Model Overview. Given an input 3D mesh, we sample a uniform point cloud and associate each point with additional properties(albedo, emitted light color). These features are processed using the Light Transport Layer which learns to approximate the light transport
in the scene. The resulting features are projected into the 2D image domain and occluded points are removed. The final image is synthesizedusing the Image Synthesis Layer that takes the projected features as well as additional low-level 2D image space information as input.

3. Methodmain parts as illustrated in Fig. 2: a Light Transport Layer,
a 3D-to-2D projection step and an Image Synthesis Layer.
The Light Transport Layer models global illumination effects that cannot be modeled in image space: consider for
example a movable lamp which is present in the scene but
not visible from the current point of view. While the position, color and intensity of the lamp strongly affect the overall illumination of the scene, a purely image-based method,by definition, will fail to reason about these effects. We
therefore propose to reason in both 3D and 2D space.

Our goal is to train a deep neural network to render aphotorealistic scene specified in terms of a 3D model in realtime while accurately modeling light transport including reflection, refraction and global illumination. In this section,
we first discuss our scene representation. Next, we describe
our neural rendering architecture which is able to learn complex illumination effects by exploiting both 3D and 2D information. Finally, we describe how we train our modelusing noisy renderings for supervision and show that undermoderate assumptions our gradient estimates are unbiased.
An overview of our approach is given in Fig. 2.

Our Light Transport Layer takes a set of Nsurf randomly
sampled 3D surface points {pj } and associated attributes
for each point {aj } as input. These attributes comprisethe surface albedo and the light intensity/color emitted by
the point if the point is located on a light-emitting surface.
Our goal is to define an architecture that is able to modelor approximate light transport in a scene sufficiently wellsuch that illumination effects like reflections and shadowsare predicted correctly.

Scene Representation: How should a 3D scene be represented for efficient and photorealistic rendering? Traditionally, 3D geometry is often represented in the form of textured 3D meshes. However, while meshes and texture atlases are compact and encode useful geometric properties,
they are inconvenient for neural networks due to their irregular structures. In contrast, voxel-based representations canbe processed conveniently using 3D convolutions, yet they
are limited by their cubic memory requirements. In thiswork, we therefore opt for a hybrid 2D-3D representationconsisting of both image-space buffers such as albedo, normal and depth maps as well as 3D information. We represent 3D information in form of an unstructured point cloud
sampled from the scene’s surface with learned feature embeddings enriched by albedo and light intensity/color.

Towards this goal, we first predict a feature embedding fj
for each point pj using a PointNet-based architecture [66].
While we found that such a global representation is able toreason about global illumination to some extent, we additionally propose a more explicit model for light transport
to model illumination effects more accurately. Inspired byphoton mapping [32], we sample additional Nphot photon
points {qk } from all light sources in the scene. Photons
are randomly cast into the scene and their (first) intersection with the scene geometry {q0k } is computed. For each

Architecture: Our neural rendering model comprises three3

Figure 3. Visualization of a Single Data Sample. (1) Noisy supervision (used during training). (2) Ground truth rendering (for reference
only, not used during training). (3-6) Image-space information A: depth, normal, albedo and view ray maps. (7) Point cloud {pj }. (8)Point cloud {pj } after occlusion masking. (9) Per-point albedo which serves as an input to the light transport layer. (10) Emitting surface
points {qj } colored in emitter spectrum. (11) Emitting photon points {qk }. (12) Intersections of emitted photons with scene {q0k }.

photon intersection q0k we process the position, color anddirection of the initial photon point qk with a fully connected neural network, resulting in a feature vector fk at
q0k . The photon network thus encodes information about
the light color, intensity and direction which is necessary
for photorealistic shading.
Next, we remove occluded points in the 3D scene using
the depth map D and project the remaining point features fj
and photon features fk onto the image plane using perspective projection φ(j) = [KTpj ] where pj denotes the point
location, K is the camera matrix and T the rigid world-toview transformation matrix. The resulting 2D feature map
F is obtained by pooling the features of all points projecting
onto the same pixel. Formally, we obtain Fu at pixel u as(P1
fj if φ−1 (u) 6= ∅
−1
−1
(1)
Fu = |φ (u)| j∈φ (u)0,otherwise

Training: We train our model using a dataset D ={(Xi , Îi )}N
i=1 which comprises pairs of 3D scene representations Xi and cheap noisy renderings Îi that are obtained
from a physically-based renderer which we run for few iterations. The input Xi = (Pi , Ti , Ai ) consists of a scene
represented by a point cloud Pi , a view represented by aworld-to-view transform Ti and additional image-space information Ai . Let ϕθ (·) denote our model with θ the parameters of the light transport and image synthesis layer.
Our objective is to find a parameter vector θ∗ which minimizes the mean squared error (MSE) between the image
predicted by our model ϕθ (Xi ) and the noisy rendering Îi :θ∗ = arg min
θNX

kÎi − ϕθ (Xi )k2

(2)

i=1

Since obtaining clean renderings is very time-consuming,
we propose the use of noisy renderings from a physicallybased renderer [42, 39]. One of our key insights and contributions is to demonstrate that we can exploit the unbiasedness of rendering algorithms like bidirectional path tracing[41] to obtain unbiased gradient estimates:

where φ−1 (u) := {j : φ(j) = u} denotes the inverse projection. We concatenate the resulting feature map F withadditional image-space information A (i.e., a depth map,a normal map and an albedo map) which we obtain using
OpenGL shaders. Note that this additional image-space information can be computed cheaply and complements theglobal scene representation F with high-frequency albedo,
normal and depth information. Additionally, we create aview ray map that encodes for each pixel a normalized vector pointing from the camera center to the pixel center inworld coordinates. This information is necessary for learning specular reflection and refraction effects. The final image synthesis is performed using the Image Synthesis Layer
which we implement using a conventional 2D U-Net architecture [73]. Fig. 3 provides an illustration of the various
inputs and features for a bathroom scene.

Lemma 1. Let X be an input representation of a scene, ϕθ
our rendering network and Î a noisy rendering of X following a distribution p(Î|X) which depends on the chosen
sampling-based rendering algorithm. Assume that the true
(noise-free) rendering is given by I(X). Further assume
that the rendering algorithm is unbiased, i.e., EÎ|X [Î] =I(X). In this case, the following equality holds, i.e., thegradient estimates are also unbiased:hi
EÎ|X ∇θ kϕθ (X) − Îk2 = ∇θ kϕθ (X) − I(X)k2 (3)Proof. See supplementary material.
4Implementation Details: For the Light Transport Layer,
we use a PointNet-based architecture [66] with ResNetblocks [23] of depth two. For the Image Synthesis Layer weuse a UNet [73] with four downsampling and four upsamling blocks. The network architecture used for photon feature creation is a fully-connected ResNet [23] architecturewith two residual blocks consisting of two fully-connected
layers each. The input, hidden and output dimension is thesame as for the PointNet architecture. For training, we use
the Adam optimizer [37] with a learning rate of 5 · 10−4
and a batch size of 128 for static scenes and 32 for dynamic
scenes (see Section 4). The learning rate is decayed exponentially by multiplying it by a factor of 0.99 after every
epoch. More details are provided in the supplementary.

black. For fair comparison, we use the same light-weight
2D convolutional architecture for all baselines and our image synthesis layer. We remark that the primary focus of
our experiments is not on improving the state-of-the-art inimage denoising, but to demonstrate the efficacy of joint
2D and 3D reasoning for learning-based neural rendering.
Both, our model as well as the denoising baselines, would
benefit from more powerful (but slower) backbones.
Metrics: For quantitative comparison, we evaluate meansquared error (MSE) and mean structural similarity index(MSSIM) [89] with a window size of 7 × 7 pixels. MSE
and MSSIM measure mostly low-level similarity. To alsomeasure perceptual similarity, we compute the FID [25] anda Feature-L1 distance [62] between predicted and groundtruth images. For both the FID and Feature-L1 distance,
we use the features of the final average pooling layer of an
Inception v3 network [80, 81] trained on ImageNet [13].

4. Experiments
In our experiments, we investigate the importance of 3D
reasoning for learning photorealistic rendering from noisy
observations. We conduct two types of experiments: In our
first set of experiments, we analyze the importance of 3D information and the influence of the different components of
the Image Synthesis Layer. To analyze these properties independently of light transport, we first run our approach ona static scene observed from varying viewpoints. Our second set of experiments addresses dynamic scenes (moving
objects and light sources) using our complete pipeline including the Light Transport Layer.

4.1. Ablation Study on Static Scene
For our ablation study, we first conduct experiments ona single static scene that does not contain moving objects orlight sources. Our primary goal is to investigate the influence of the different elements of the Image Synthesis Layeras well as the importance of 3D information.
We compare the performance of our model without Light
Transport Layer for different input modalities. Fig. 4 shows
the different configurations which are evaluated against
each other. We choose a subset of 6 (out of 24 = 16) representative configurations to highlight the importance of each
input. While configuration 1, 2 and 3 use only 3D information (but no image space information), configuration 4 and5 rely solely on image space information. Finally, configuration 6 combines both 3D and image space information.

Datasets: For our experiments on static scenes, we evaluate
our approach on a simple static indoor scene containing a table, two light sources and a glass egg [86]. Our experimentson dynamic scenes are based on four realistic indoor scenes
from [5]. We use Mitsuba [30] for both rendering and point
sampling. Renderings are created using bidirectional pathtracing, a modification of path tracing that is unbiased andconverges faster [86]. For each scene, we create a training
set of 100,000 images at a resolution of 256 × 256 pixels,
varying the camera pose for each training sample. We sample 10,000 surface points for each scene. For our experiments on dynamic scenes, we randomly translate or remove
objects in addition to varying the camera pose.

Results: Configurations 1, 2, 3 and 5 show similar performance in terms of MSE, while configuration 5, which does
not receive any projected point cloud information as input,
clearly outperforms the other three configurations in termsof MSSIM and Fréchet inception distance (FID). However,surface normal information only yields good results if supplemented by viewpoint information, as becomes evident
when comparing configurations 4 and 5. The most important insight is that all inputs in combination (configuration6), outperform the other configurations for all metrics by alarge margin. This result supports our initial hypothesis thatreasoning in both 3D and 2D is crucial for this task. Fig. 4(top) shows qualitative results. While configurations 1, 2
and 3 achieve reasonable qualitative results, they also contain several artifacts (e.g., the table) which do not occur inconfiguration 6. Configurations 4 and 5 do not exploit 3D
information, thus severely degrading visual fidelity. Thishighlights the importance of 3D information for learningbased rendering. Configuration 6 which uses both 2D asBaselines: For our main experiment on dynamic scenes we
use three baseline methods: (1) a 2D CNN baseline which
predicts images from the image-space input Ai alone, (2) asimple denoising approach similar to the model of Lehtinen
et al. [42], which learns to predict smooth renderings using
noisy renderings as input and (3) a simple feature projection
approach similar to Aliev et al. [3] without Light Transport
Layer. For the denoising approach we trade off accuracy
with run-time by adapting the number of pixels for which
we run the bidirectional path tracer. We report results for
1/1, 1/4, 1/16 and 1/64 of the total number of image pixels with four samples per pixel, setting all other pixels to5Supervisionconfig
1
2
3
4
5
6

1

2

3

4

5

GroundTruth

6position

point features

normal map

ray direction map

MSE

MSSIM

FID

yes
no
yes
no
no
yes

no
yes
yes
no
no
yes

no
no
no
yes
yes
yes

no
no
no
no
yes
yes

0.01060.01070.01080.01610.01070.0084

0.81
0.80
0.81
0.78
0.83
0.88

154.6
158.6
149.4
138.1
124.0
86.1

Figure 4. Ablation Study on Static Scene. Comparing different input configurations for a static scene. The metrics are evaluated on a
separate held-out validation set comprising 2048 samples. All networks were trained for 200,000 iterations with a batch size of 128. Note
how the full model (6) is able to predict images that are significantly less noisy than the supervision signal used for training.

Results: Fig. 5 shows qualitative and quantitative resultsfor our approach and the baselines. We clearly see that ourfull model which uses both the Light Transport and the Image Synthesis Layers outperforms the other real-time approaches (lower section of the table), both qualitatively andquantitatively in terms of MSE, MSSIM, FID and FeatureL1 distance. While the non-real-time denoising approach“Denoising (1/1)” achieves the best results, the real-time
denoising approach that uses much fewer samples performs
the worst. We further analyze this behavior by plotting the
MSSIM as a function of rendering time in Fig. 6. While denoising approaches are able to achieve compelling results,
the proposed neural rendering approach provides a better
accuracy/runtime trade-off while being fully differentiable.

well as 3D information yields the best results. Moreover, it
is remarkable that our model predicts images that are significantly less noisy than the images used for training.

4.2. Results on Dynamic Scenes
To investigate the utility of 3D reasoning, we now turn
our attention to dynamic scenes where objects (and lightsources) are modified.
4.2.1

Dynamic Objects and Fixed Lights

We first train our network on a set of four scenes where
objects are randomly removed or translated in the scene,
but keep all light sources fixed.
6Supervision

Denoising(1/1)

Denoising(1/64)

FeatureProjection

CNN only

Ours w/o
Photons

Ours w/
Photons

GroundTruth

Architecture

time / frame

MSE (↓)MSSIM (↑)

FID (↓)

Feature L1 (↓)

Denoising (1/1)

1.5059s

0.0005

0.880

26.4

0.163

Denoising (1/64)
CNN only
Feature Projection
Ours (w/o Photons)
Ours (w/ Photons)

0.0283s0.0191s
0.0210s
0.0243s
0.0459s

0.00290.00430.00370.00440.0028

0.781
0.835
0.841
0.841
0.849

94.0
36.1
32.5
31.4
30.6

0.281
0.195
0.185
0.184
0.182

Figure 5. Dynamic Objects and Fixed Lights. Results on dynamic scenes where objects are modified but light sources kept fixed. We
show the non-real-time denoising baseline “Denoising (1/1)” for reference. Additional results are provided in the supplementary material.

baseline is unable to acquire an understanding of illumination effects in the presence of movable light sources that are
not present in the current view. To see this effect, we augment the dataset from the previous experiment by turning
all static light sources off and replacing them with a rectangular area light at the ceiling, which we move randomly.
Results: Results from our method with photons, our approach without photons and the feature projection baseline
are shown in Fig. 7. We observe that the feature projection
baseline produces considerable artifacts while our approachwith photons leads to much sharper shadows and more consistent global illumination. This is also evident from theerror maps in Fig. 7. The last row shows a failure case of
our approach which does not accurately recover the mirror
reflection due to the limited number of samples. We provide
a full quantitative evaluation in the supplementary material.

Figure 6. Dynamic Objects and Fixed Lights. Quantitative comparison of our approach to the denoising baseline, varying the sample density. Reconstruction accuracy in terms of MSSIM and FID
over inference time. Numbers refer to the ratio of dropped pixels.

As evident from Fig. 5, our simple feature projection
baseline performs only slightly weaker than our variant
without photon mapping. We attribute this to the fact thatmost of the light field in the scene can be encoded in local
features and only dynamic parts like sharp shadows have tobe learned. This highlights the capability of neural rendering approaches to learn useful heuristics from the training
data. As in the previous experiment, our full architecturewith photon mapping (which reasons more explicitly aboutlight transport) achieves the best quantitative results.
4.2.2

5. Conclusion
In this work, we conducted a systematic investigation of
the importance of 3D vs. 2D reasoning for learning-based
photorealistic neural rendering. Our experiments demonstrated that neural rendering benefits from joint 3D-2D reasoning. This also confirms our initial hypothesis that reasoning in 3D is indeed helpful in the presence of moving
objects and light sources. In contrast to denoising methods which rely on outputs from a sampling-based renderer,
the presented approach is fully differentiable and can beused for training deep models using differentiable rendering while faithfully capturing global illumination effects.

Dynamic Objects and Dynamic Lights

In the previous experiment, both the feature projection andour approach without photons were able to handle shadows
and other illumination effects well. The reason for this isthat the light sources were assumed static, making it possible to encode viewpoint-dependent light properties into thepoint features. However, by design the feature projection
7

Feature
Projection

Error

Ours w/o
Photons

Error

Ours w/
Photons

Error

Ground Truth

Figure 7. Dynamic Objects and Dynamic Lights. We show the output of the feature projection baseline and our network’s predictions
with and without photons alongside the corresponding error maps for moving light sources. The last row shows a failure case of ourapproach which in this case is not able to accurately recover the mirror reflection due to the limited number of reflected samples.

8References

ral networks. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2015. 2[16] S. M. Ali Eslami, Danilo Jimenez Rezende, Frédéric Besse,
Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor,David P. Reichert, Lars Buesing, Theophane Weber, Oriol
Vinyals, Dan Rosenbaum, Neil C. Rabinowitz, Helen King,
Chloe Hillier, Matt M. Botvinick, Daan Wierstra, Koray
Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 360:1204–1210, 2018. 2[17] Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, VladFiroiu, Tim Harley, Iain Dunning, Shane Legg, and Koray
Kavukcuoglu. IMPALA: scalable distributed deep-rl with
importance weighted actor-learner architectures. In Jennifer G. Dy and Andreas Krause, editors, Proc. of the International Conf. on Machine learning (ICML), 2018. 1[18] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora
Vig. Virtual worlds as proxy for multi-object tracking analysis. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2016. 1[19] Alberto Garcia-Garcia,
Francisco Gomez-Donoso,José Garcı́a Rodrı́guez, Sergio Orts-Escolano, MiguelCazorla, and Jorge Azorı́n López. Pointnet: A 3d convolutional neural network for real-time object class recognition.
In International Joint Conference on Neural Networks(IJCNN), 2016. 2[20] Ioannis Gkioulekas, Anat Levin, and Todd E. Zickler. An
evaluation of computational imaging techniques for heterogeneous inverse scattering. In Proc. of the European Conf.
on Computer Vision (ECCV), 2016. 2[21] Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd E.
Zickler, and Anat Levin. Inverse volume rendering with material dictionaries. ACM Trans. on Graphics, 2013. 2[22] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing
Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville,
and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS),
2014. 1[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2016. 5[24] P. Hermosilla, S. Maisch, T. Ritschel, and T. Ropinski. Deeplearning the latent space of light transport. Computer Graphics Forum, 2019. 2[25] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner,
Bernhard Nessler, and Sepp Hochreiter. Gans trained by a
two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems(NIPS), 2017. 5[26] Geoffrey E. Hinton and Ruslan Salakhutdinov. Reducing
the dimensionality of data with neural networks. Science,
313:504–507, 2006. 2[27] Binh-Son Hua, Minh-Khoi Tran, and Sai-Kit Yeung. Pointwise convolutional neural networks. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2018. 2[1] Hassan Alhaija, Siva Mustikovela, Andreas Geiger, andCarsten Rother. Geometric image synthesis. In Proc. of the
Asian Conf. on Computer Vision (ACCV), 2018. 2[2] Hassan Alhaija, Siva Mustikovela, Lars Mescheder, Andreas
Geiger, and Carsten Rother. Augmented reality meets computer vision: Efficient data generation for urban driving
scenes. International Journal of Computer Vision (IJCV),
126(9):961–972, 2018. 1[3] Kara-Ali Aliev, Dmitry Ulyanov, and Victor S. Lempitsky.
Neural point-based graphics. arXiv.org, 1906.08240, 2019.
1, 2, 5[4] Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, andMatthias Nießner. Inverse path tracing for joint material andlighting estimation. In Proc. Computer Vision and Pattern
Recognition (CVPR), IEEE, 2019. 2[5] Benedikt Bitterli.
Rendering resources, 2016.
https://benedikt-bitterli.me/resources/. 5[6] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large
scale GAN training for high fidelity natural image synthesis.
In Proc. of the International Conf. on Learning Representations (ICLR), 2019. 2[7] A. Buades, B. Coll, and J. . Morel. A non-local algorithm forimage denoising. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2005. 2[8] Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan,
Christoph Schied, Marco Salvi, Aaron E. Lefohn, Derek
Nowrouzezahrai, and Timo Aila. Interactive reconstructionof monte carlo image sequences using a recurrent denoising
autoencoder. ACM Trans. on Graphics, 36(4):98:1–98:12,
2017. 2[9] Tony F. Chan and Jianhong Shen. Variational image inpainting. 58:579–619, 2005. 2[10] Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In Proc. of theIEEE International Conf. on Computer Vision (ICCV), 2017.
2[11] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaako
Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation-based differentiable renderer. In Advances in Neural Information Processing Systems (NIPS), 2019. 2[12] Xi Chen, Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing
generative adversarial nets. In Advances in Neural Information Processing Systems (NIPS), 2016. 2[13] Jia Deng, Wei Dong, Richard Socher, Li jia Li, Kai Li, and Li
Fei-fei. Imagenet: A large-scale hierarchical image database.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2009. 5[14] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio
Lopez, and Vladlen Koltun. CARLA: An open urban driving
simulator. In Proc. Conf. on Robot Learning (CoRL), 2017.
1[15] Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas
Brox. Learning to generate chairs with convolutional neu-

9for 3d controllable image synthesis. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2020. 2[45] Stephen Lombardi, Jason M. Saragih, Tomas Simon, andYaser Sheikh. Deep appearance models for face rendering.
ACM Trans. on Graphics, 37(4):68:1–68:13, 2018. 2[46] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel
Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images.
In ACM Trans. on Graphics, 2019. 2[47] Matthew M. Loper and Michael J. Black. Opendr: An approximate differentiable renderer. In Proc. of the European
Conf. on Computer Vision (ECCV), 2014. 2[48] Michael Mara, Morgan McGuire, Benedikt Bitterli, and Wojciech Jarosz. An efficient denoising algorithm for global
illumination. In High-Performance Graphics (HPG), 2017.
2[49] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi,
Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. arXiv.org, 2008.02268, 2020. 2[50] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin.
Which training methods for gans do actually converge?
In Proc. of the International Conf. on Machine learning(ICML), 2018. 1, 2[51] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks:
Learning 3d reconstruction in function space. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2019. 2[52] Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues
Hoppe, Rohit Pandey, Noah Snavely, and Ricardo MartinBrualla. Neural rerendering in the wild. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2019.
1, 2[53] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik,
Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF:
Representing scenes as neural radiance fields for view synthesis. In Proc. of the European Conf. on Computer Vision(ECCV), 2020. 1, 2[54] Thomas Müller, Brian Mcwilliams, Fabrice Rousselle,
Markus Gross, and Jan Novák. Neural importance sampling.
ACM Trans. on Graphics, 2019. 2[55] Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta,
Hans-Peter Seidel, and Tobias Ritschel. Deep shading: Convolutional neural networks for screen space shading. Computer Graphics Forum, 36(4):65–78, 2017. 2[56] Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, and YongLiang Yang. Rendernet: A deep convolutional network for
differentiable rendering from 3d shapes. In Advances in Neural Information Processing Systems (NIPS), 2018. 2[57] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, ChristianRichardt, and Yong-Liang Yang. Hologan: Unsupervised
learning of 3d representations from natural images. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2019. 2[58] Thu Nguyen-Phuoc, Christian Richardt, Long Mai, YongLiang Yang, and Niloy Mitra. Blockgan: Learning 3d[28] Huaibo Huang, Zhihang Li, Ran He, Zhenan Sun, and Tieniu
Tan. Introspective variational autoencoders for photographic
image synthesis. In Advances in Neural Information Processing Systems (NIPS), 2018. 2[29] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A.
Efros. Image-to-image translation with conditional adversarial networks. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2017. 2[30] Wenzel Jakob. Mitsuba renderer, 2010. http://www.mitsubarenderer.org. 5[31] Krishna Murthy Jatavallabhula, Edward J. Smith, JeanFrancois Lafleche, Clement Fuji Tsang, Artem Rozantsev,
Wenzheng Chen, Tommy Xiang, Rev Lebaredian, and Sanja
Fidler. Kaolin: A pytorch library for accelerating 3d deeplearning research. arXiv.org, 1911.05063, 2019. 2[32] Henrik Wann Jensen. Global illumination using photon
maps. In Proceedings of the Eurographics Workshop on Rendering Techniques, 1996. 3[33] James T. Kajiya. The rendering equation. In ACM Trans. on
Graphics, 1986. 1[34] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
Progressive growing of GANs for improved quality, stability,and variation. In Proc. of the International Conf. on Learning
Representations (ICLR), 2018. 1, 2[35] Tero Karras, Samuli Laine, and Timo Aila. A style-based
generator architecture for generative adversarial networks. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1, 2[36] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 2[37] Diederik P. Kingma and Jimmy Ba. Adam: A method forstochastic optimization. In Proc. of the International Conf.
on Learning Representations (ICLR), 2015. 5[38] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. Proc. of the International Conf. on Learning
Representations (ICLR), 2014. 2[39] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug.
Noise2void - learning denoising from single noisy images.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 4[40] Tejas D. Kulkarni, William F. Whitney, Pushmeet Kohli, andJoshua B. Tenenbaum. Deep convolutional inverse graphics network. In Advances in Neural Information Processing
Systems (NIPS), 2015. 2[41] Eric P Lafortune and Yves D Willems. Bi-directional pathtracing. 1993. 4[42] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli
Laine, Tero Karras, Miika Aittala, and Timo Aila.
Noise2noise: Learning image restoration without clean data.
In Proc. of the International Conf. on Machine learning(ICML), 2018. 4, 5[43] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge
sampling. ACM Trans. on Graphics, 2018. 2[44] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas
Geiger. Towards unsupervised learning of generative models

10In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015. 4, 5[74] Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul
Patney, Chakravarty R. Alla Chaitanya, John Burgess, Shiqiu
Liu, Carsten Dachsbacher, Aaron E. Lefohn, and MarcoSalvi. Spatiotemporal variance-guided filtering: real-timereconstruction for path-traced global illumination. In HighPerformance Graphics (HPG), 2017. 2[75] Christoph Schied, Christoph Peters, and Carsten Dachsbacher. Gradient estimation for real-time adaptive temporal
filtering. PACMCGIT, 2018. 2[76] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas
Geiger. Graf: Generative radiance fields for 3d-aware image
synthesis. In Advances in Neural Information Processing
Systems (NeurIPS), 2020. 2[77] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua
Susskind, Wenda Wang, and Russell Webb. Learningfrom simulated and unsupervised images through adversarial training. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2017. 1[78] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias
Nießner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2019. 1, 2[79] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3dstructure-aware neural scene representations. In Advancesin Neural Information Processing Systems (NIPS), 2019. 2[80] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,
Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent
Vanhoucke, and Andrew Rabinovich. Going deeper withconvolutions. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2015. 5[81] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe,
Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2016. 5[82] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox.
Multi-view 3d models from single images with a convolutional network. In Proc. of the European Conf. on Computer
Vision (ECCV), 2016. 2[83] Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: image synthesis using neural textures. In ACM Trans. on Graphics, 2019. 2[84] Julien Valentin, Cem Keskin, Pavel Pidlypenskyi, Ameesh
Makadia, Avneesh Sud, and Sofien Bouaziz. Tensorflow
graphics: Computer graphics meets deep learning. 2019. 2[85] Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid.
Learning from synthetic humans. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2017. 1[86] Eric Veach. Robust Monte Carlo Methods for Light Transport Simulation. PhD thesis, Stanford University, Stanford,
CA, USA, 1998. 1, 2, 5

object-aware scene representations from unlabelled images.
arXiv.org, 2020. 2[59] Michael Niemeyer, Lars Mescheder, Michael Oechsle, andAndreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 2[60] Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: a retargetable forward and inverse renderer. 38(6):203, 2019. 2[61] Atsuhiro Noguchi and Tatsuya Harada. RGBD-GAN: unsupervised 3d representation learning from natural image
datasets via RGBD image synthesis. arXiv.org, 1909.12573,
2019. 1[62] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo
Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2019. 2, 5[63] Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan,
and Alexander C. Berg. Transformation-grounded image
generation network for novel 3d view synthesis. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2017. 2[64] Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physically
based rendering: From theory to implementation. Morgan
Kaufmann, 2016. 1, 2
[65] Francesco Pittaluga, Sanjeev J. Koppal, Sing Bing Kang, andSudipta N. Sinha. Revealing scenes by inverting structurefrom motion reconstructions. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 2[66] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.
Pointnet: Deep learning on point sets for 3d classificationand segmentation. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2017. 1, 3, 5[67] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a
metric space. In Advances in Neural Information Processing
Systems (NIPS), 2017. 2[68] Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim
Weyrich. Neural btf compression and interpolation. Computer Graphics Forum, 2019. 2[69] Konstantinos Rematas and Vittorio Ferrari. Neural voxel renderer: Learning an accurate and controllable rendering tool.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 2[70] Peiran Ren, Jiaping Wang, Minmin Gong, Stephen Lin, Xin
Tong, and Baining Guo. Global illumination with radiance
regression functions. ACM Trans. on Graphics, 2013. 2[71] Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware representation for 3d human pose
estimation. In Proc. of the European Conf. on Computer Vision (ECCV), 2018. 2[72] Gernot Riegler and Vladlen Koltun. Free view synthesis. In
Proc. of the European Conf. on Computer Vision (ECCV),
2020. 2[73] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomedical image segmentation.

11[87] Delio Vicini, Vladlen Koltun, and Wenzel Jakob. A learned
shape-adaptive subsurface scattering model. ACM Trans. on
Graphics, 2019. 2[88] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao,Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 2[89] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P.
Simoncelli. Image quality assessment: from error visibility
to structural similarity. IEEE Trans. on Image Processing(TIP), 13(4):600–612, 2004. 5[90] Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Interpretable transformations
with encoder-decoder networks. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2017. 2[91] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, andHonglak Lee. Perspective transformer nets: Learning singleview 3d object reconstruction without 3d supervision. In
Advances in Neural Information Processing Systems (NIPS),
2016. 2[92] Xiuming Zhang, Sean Ryan Fanello, Yun-Ta Tsai, TianchengSun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano,
Philip L. Davidson, Christoph Rhemann, Paul E. Debevec,
Jonathan T. Barron, Ravi Ramamoorthi, and William T. Freeman. Neural light transport for relighting and view synthesis.
arXiv.org, 2008.03806, 2020. 2[93] Zhiyuan Zhang, Binh-Son Hua, and Sai-Kit Yeung. Shellnet: Efficient point cloud convolutional neural networks using concentric shells statistics. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 2[94] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A. Efros. View synthesis by appearance flow.
In Proc. of the European Conf. on Computer Vision (ECCV),
2016. 212