Supplementary Material for
Learning Neural Light Transport
Paul Sanzenbacher
Autonomous Vision Group
University of Tübingen, Germany

Lars MeschederAmazonTübingen, Germany

Andreas Geiger
Autonomous Vision Group
University of Tübingen, Germany

paul.sanzenbacher@tue.mpg.delamesch@amazon.de

andreas.geiger@tue.mpg.de

Abstract
This supplementary document provides additional information on our approach and more experimental results. First,
we provide detailed information on the Light Transport and Image Synthesis Layers in Sec. 1. We then describe the data
generation pipeline in more detail in Sec. 2. Afterwards, we provide more information on the training procedure in Sec. 3,
including a proof showing that we can train our models using noisy, unbiased renderings as supervision signal. Finally, we
provide additional qualitative and quantitative results in Sec. 4.

1. Architectures
1.1. Light Transport Layer
The core of the Light Transport Layer is a PointNet-based architecture [9] with fully-connected ResNet blocks [2], which
is illustrated in Fig. 1. While the PointNet architecture can have arbitrary depth (number of ResNet blocks), we use a depthof two for all the experiments in the paper.
Since we train our model on dynamic scenes with a variable number number of visible objects, the input point clouds havedifferent sizes for different training samples. In theory this is not a big problem, as PointNets can handle arbitrary point cloudsizes. However, since we are using mini batches for training, having the same number of points for each training sample isdesirable. Therefore, our model always operates on the maximum point cloud size, and invisible objects are masked in thearchitecture using per-point visibility flags.

1.2. 3D-to-2D Projection Step
In the 3D-to-2D projection step, the 3D point features are projected to image space, where the point locations are discretized. Points that are occluded by the scene’s geometry are masked out, which is determined by performing an occlusion
check using a rendered depth map. To make sure that we do not accidentally remove points on the scene’s surface, we use
a tolerance of ε = 10−3 in the occlusion check. If multiple features are projected to the same pixel, we compute the mean
feature vector for all points projecting to that pixel. If a pixel has no points projecting to it, its feature vector is defined aszero.

1.3. Image Synthesis Layer
The input to the Image Synthesis Layer are the projected features from the projection step and additional informationin image space, which can be computed cheaply using OpenGL shaders These image space buffers contain information
about the geometry and material information observed from the current view. They include depth map, albedo (diffuse
reflectance), normal map in world coordinates as well as a view ray map, which contains for each pixel the ray direction inworld coordinates going from the camera center through the respective pixel center. The intention behind using these image
space layers is to leverage the image formation process in multiple ways. The normal and view direction information canbe used by the network to infer shading in image space. The albedo layer supports texture synthesis where point projections
are sparse. In addition, by providing this information in image space, the light transport layer can solely focus on the task1

of modeling the illumination in the scene. However, the image space layers do not contain useful information for reasoning
about light transport in the scene. A detailed visualization and description of the Image Synthesis Layer is provided in Fig. 2.

2. Datasets
2.1. Data Generation and Sampling
The datasets used in our experiments comprise a single scene for each static scene dataset, and four scenes for dynamic
experiments [1]. Since the data generation procedure for static scenes is a simplification of the dynamic case, we only
describe the dynamic case in this section. Since we use learnable feature descriptors in our model, we must ensure that there
are point correspondences between different training samples of the same scene. To this end, we sample an initial, static
point cloud for each scene. This point cloud is then modified according to the scene modifications in the training sample. If
an object is removed from the scene, the points are removed from the initial point cloud. If an object is translated, the points
sampled from its surface are translated accordingly. For each scene in the dataset, we first sample a static point cloud, whichis then modified for each sample in the dataset. A positive side effect of this is that we only have to store scene modification
information for each sample, saving memory.

2.2. View Sampling
For each scene, we would like to cover the space of possible viewing locations and directions as accuractely as possible. At
the same time we want to have a high number of views where a lot of scene details are visible to have an effective supervision
signal for training. We observe that most of the objects in a scene are arranged along the walls or the floor. Therefore, we
sample a viewing location uniformly from a bounding box that is slightly smaller than the scene’s bounding box. Note that
this means that a few of the sampled locations might lie inside an object. However, we found that these “outliers” do notpose a problem to our method in practice as long as we observe a sufficiently large number of views outside of objects during
training. Next, we sample a viewing direction by sampling a look-at location uniformly from a bounding box that is half thesize of the location bounding box. As a result, the distance between the camera and scene objects is far enough to renderviews with rich image content.

2.3. Point Cloud Sampling
We define a scene by a set of shapes S, where each shape Si ∈ S is itself a set of triangles. Each shape is assigned asampling importance w(Si ) corresponding to its surface area, which is the sum of triangle areas for that shape. Given thesampling importances and a point cloud of size N , we first sample N shapes according to a distribution where the probabilityof sampling a shape is proportional to its sampling importance. This can be achieved by using discrete inverse transform
sampling, where a discrete cumulative distribution is calculated for the sequence of shapes (S1 , . . . , Sn ):Pi

j=1 w(Sj )cdf(i) = Pn

j=1 w(Sj )(1)Using a uniform sample s ∼ U(0, 1), a shape index i can be sampled according toi = arg max {k : cdf(k) < u} ,(2)kwhich can be implemented efficiently using bisection.
For each shape sampled from the distribution, our goal is to obtain a point sampled uniformly from the shape’s surface.
Since we work with a mesh scene representation, all the shapes are represented by a set of triangles. Therefore, for eachpoint we first sample the triangle with the same technique we used for shape sampling, using the triangle area as sampling
importance. Then, we sample a point location uniformly from the triangle. This way, uniformly distributed samples from theshapes’ surfaces can be obtained.

2.4. Scene Modification Sampling
To train our model on all possible scene configurations (each object or light source could be located anywhere or not bepresent in the scene at all), we must cover this distribution well in the dataset. To this end, we manually define for each
dynamic object an axis-aligned bounding box from which we sample a position for each training sample. The boundingboxes can also be limited to one or two dimensions, e.g. if an object can only be translated along a wall. Although we do2not always get realistic object arrangements using this sampling strategy, this is not a limitation, as it makes our model moregeneral (i.e. our model is trained for both realistic and non-realistic object arrangements). In addition to object translations,
we randomly remove objects with a probability of 0.2 per object from the scene.

3. Training
3.1. Hyperparameters
We train all models using the Adam optimizer [4] with a learning rate of λ = 5 × 10−4 , which we decay by a factor of
0.99 after each epoch. These hyperparameters are the result of a hyperparameter optimization using grid search, where wetested different learning rates and decay rates for Adam and RMSprop for 100,000 iterations. For the static scene experimentin Section 4.1 of the main paper we use a batch size of 128. For the dynamic scene experiments in Section 4.2 of the main
paper we use a batch size of 32, as more GPU memory is required for the Light Transport Layer implementation. All models
were created and trained using PyTorch 1.01 [8].

3.2. Supervised Learning with Noisy Renderings
Since rendering a large set of photorealistic renderings for training would require a lot of time, we use noisy renderings
from a physically based renderer as supervision. More specifially, we use the bidirectional path tracing implementationin Mitsuba [3]. Similar techniques have recently been used to learn image denoising [7, 5]. Our key insight is that we can
exploit the unbiasedness of state-of-the-art rendering algorithms like bidirectional path tracing [6] to obtain unbiased gradientestimates.
To this end, we describe the input to our network by random variable X, which comprises a point cloud P, a view
represented by a world-to-view transform T and additional image-space information A as described in Chapter 3 of themain paper. As supervision signal, we render a noisy image Î that is an unbiased estimate of the ground truth rendering
I(X). When we train our network using the mean squared error (MSE) and stochastic gradient descent, our gradients will be
unbiased when using these noisy supervision renderings from such an unbiased rendering algorithm. Formally, this can be
expressed as follows:
Lemma 1. Let X be an input representation of a scene, ϕθ our rendering network and Î a noisy rendering of X following
a distribution p(Î|X) which depends on the chosen sampling-based rendering algorithm. Assume that the true (noise-free)rendering is given by I(X). Further assume that the rendering algorithm is unbiased, i.e., EÎ|X [Î] = I(X). In this case, thefollowing equality holds, i.e., the gradient estimates are unbiased:hi(3)
EÎ|X ∇θ kϕθ (X) − Îk2 = ∇θ kϕθ (X) − I(X)k2
Proof. Since the expectation does not depend on the parameters θ, the gradient can be pulled out of the expectation. The left
side of Eq. (3) becomeshihi
EÎ|X ∇θ kϕθ (X) − Îk2 = ∇θ EÎ|X kϕθ (X) − Îk2

(4)

h i
By applying the binomial theorem and the property of the estimator Î being unbiased, which means that EÎ|X Î = I(X),the expectation term can be further expanded tohihi
EÎ|X kϕθ (X) − Îk2 = EÎ|X kϕθ (X)k2 − 2hϕθ (X), Îi + kÎk2
h
i
h
i


= EÎ|X kϕθ (X)k2 − 2 EÎ|X hϕθ (X), Îi + EÎ|X kÎk2
hi= kϕθ (X)k2 − 2 hϕθ (X), I(X)i + EÎ|X kÎk2

(5)
(6)
(7)

Taking the gradient with respect to θ in Eq. (7) allows for removing or adding terms that are constant with respect to θ. Thus,
1 https://pytorch.org

3hi
we can replace EÎ|X kÎk2 with kI(X)k2 :
ii
i
h
h
h
∇θ EÎ|X kϕθ (X) − Îk2 = ∇θ kϕθ (X)k2 − 2 hϕθ (X), I(X)i + EÎ|X kÎk2= ∇θ kϕθ (X)k2 − 2 hϕθ (X), I(X)i + kI(X)k2= ∇θ kϕθ (X) − I(X)k

2

(8)
(9)
(10)

Inserting this into Eq. (4) results in Eq. (3), concluding the proof.

4. Additional Results
We tested our model on two additional challenging static scenes, with results shown in Fig. 3 and Fig. 4, respectively. For
this experiment, we used a realistic bathroom scene and a realistic kitchen scene [1] at an image resolution of 256 × 256
pixels. Both scenes were trained with a batch size of 128 for 150,000 iterations. Although there is no light transport to belearned in these static scene experiments, we find that our model is able to encode realistic static scenes well, and renders
novel views accurately.
For dynamic scenes we also conducted two experiments: one where we compared our approach to a set of baselines
in Section 4.2.1 of the main paper, using a dataset with dynamic objects and fixed lights, where we translated and removed
objects randomly. And another experiment where we highlight the importance of the Light Transport Layer and the additional
photon architecture in Section 4.2.2, on a dataset with dynamic objects and dynamic lights, where we additionally translate
rectangular light sources randomly along the ceiling. Tab. 1 shows the full quantitative evaluation of the experiments for
dynamic objects and fixed lights. For the experiment with dynamic objects and dynamic lights we provide a full quantitativeevaluation in Tab. 2.
Fig. 6 shows additional visual results for the baseline comparison for dynamic objects and fixed lights, complementing
Fig. 5 of the main paper. In Fig. 7 we show examples where our method does not predict the illumination accurately. Theseerror images also show that for the denoising approaches errors occur mostly in image regions with high frequency components, i.e. edges and textures. For our approach, errors sometimes also occur in larger image regions when the prediction is
inaccurate or sparse. This also explains that while our approach performs best for most of the metrics in Tab. 1 and Tab. 2,
the MSE is lower for the denoising approaches.
We show some additional failure cases for dynamic objects and dynamic lights in Fig. 8.
In addition to the quantitative comparison for dynamic objects and fixed lights, we show a more comprehensive quantitative comparison for dynamic objects and dynamic lights in Fig. 5. In addition to MSSIM and FID, we compare L1 feature
losses from different stages of the Inception v3 network [11, 12], showing that our approach clearly outperforms the denoisingbaselines on different levels of image abstraction.

4

fc

act

fc

fc

act

fc

act

fc

preprocess+maximum

fc

...

act

...

fc

sharedweights

fc

...

act

sharedweights

fc

...

act

sharedweights...

fc

concatenation

sharedweightsfc

residual block

act

fc

act

fc

+

maximum

fc

...

fc
sharedweights

fc

act
...

act

fc
sharedweights

fc

act
...

act

PointNet block

...

fc

...

shortcutconcatenation

shortcutsharedweightsfc

fc

residual block

PointNet block

Figure 1. Light Transport Layer. In the first stage of the Light Transport Layer all points and supplementary point information pi areprocessed in a preprocessing layer, whose purpose is to align its output feature dimension with the input feature dimension of the PointNet
block. The point features are then processed in two consecutive PointNet blocks. A PointNet block comprises a residual block, where local
features are computed for each point. The fully connected layers (hidden, output and shortcut layers) consist of 32 output neurons each,where the weights within one layer are shared between the input points. The input dimension to the first fully connected layer in a residual
block is aligned with the output dimension of a PointNet block (64). Therefore, a fully connected shortcut layer is required for matching
the feature dimensions at the end of a residual block. Following the residual block within a PointNet block, point features are concatenatedwith a global feature, which is computed as the maximum feature vector of all local features. The output features of the second PointNet
block are denoted by fi . We denote fully connected layers by fc and ReLU activation functions by act.

5

2d

4d

8d

4d

h

h/2

h/4

h/8

h/4

h/2

h

skip connections

2d

d

d
convolutional ResNet block

max-poolingupsampling

convolutional layer3

concatenation

Figure 2. Image Synthesis Layer. For final image synthesis we use a UNet [10] architecture where the resolution is reduced in threesteps and expanded again. To this end, each level comprises a convolutional ResNet [2] block consisting of two 3 × 3 convolutional layers
and ReLu activations. We use the same feature dimension for the input, hidden and output layers in a convolutional ResNet block. The
convolutional ResNet blocks are followed by a downsampling step, which is implemented using max-pooling layers. The feature dimensionof the convolutional layers depend on the level, starting at a dimension of d = 64, which is then doubled after each downsampling step.
The features of the lowest level are then upsampled again using bilinear interpolation, concatenated with the convolutional ResNet block
output from the respective downsampling layer through a skip connection and processed in another convolutional ResNet block. Afterthe last upsampling layer an additional convolutional layer is used to render an image with three channels. The numbers below the layers
correspond to the number of feature maps in each layer. The numbers inside the layers correspond to the layer’s resolution, starting at asquare resolution of h × h. We use h = 128 in our static scene ablation study and h = 256 for the other experiments.

Teaching Input

Prediction

Ground Truth

Teaching Input

Prediction

Figure 3. Bathroom Scene. Results of our model on a realistic static bathroom scene.

6

Ground Truth

Teaching Input

Prediction

Ground Truth

Teaching Input

Prediction

Figure 4. Kitchen Scene. Results of our model on a realistic static kitchen scene.

7

Ground Truth

Architecture

time / frame

MSE

MSSIM

FID

Feature L1

Denoising (1/1)
Denoising (1/4)
Denoising (1/16)
Denoising (1/32)
Denoising (1/64)

1.5059s
0.3800s
0.0986s
0.0532s
0.0283s0.00050.00070.00120.00180.0029

0.880
0.867
0.835
0.813
0.781

26.4
28.1
38.7
54.1
94.0

0.163
0.172
0.203
0.233
0.281

CNN only
Feature projection
Ours (w/o photons)
Ours (w/ photons)

0.0191s
0.0210s
0.0243s
0.0459s

0.00430.00370.00440.0028

0.835
0.841
0.841
0.849

36.1
32.5
31.4
30.6

0.195
0.185
0.184
0.182

Table 1. Dynamic Objects and Fixed Lights. Quantitative evaluation for our experiment on dynamic objects and fixed lights.

Architecture

time / frame

MSE

MSSIM

FID

Feature L1

Denoising (1/1)
Denoising (1/4)
Denoising (1/16)
Denoising (1/32)
Denoising (1/64)

1.5059s
0.3801s
0.0988s
0.0518s
0.0283s0.00020.00020.00050.00080.0016

0.930
0.923
0.896
0.874
0.839

17.1
17.6
23.5
38.6
84.1

0.137
0.143
0.172
0.207
0.269

CNN only
Feature projection
Ours (w/o photons)
Ours (w/ photons)

0.0190s
0.0208s
0.0243s
0.0468s0.01000.00980.00290.0014

0.827
0.827
0.871
0.887

33.7
32.9
30.0
25.1

0.199
0.197
0.184
0.172

Table 2. Dynamic Objects and Dynamic Lights. Quantitative evaluation for our experiment on dynamic objects and dynamic lights.

8

Figure 5. Dynamic Objects and Dynamic Lights. This plot shows a quantitative comparison of our approach with the denoising baseline
for different sample densities. We plot reconstruction accuracy over inference time for our experiment on dynamic objects and dynamic
lights. The denoising labels refer to the ratio of pixels that are dropped. The layer indices (0–3) for the Feature L1 losses refer to outputsof the four major layers in the Inception v3 network.

9

Teaching
Input

Denoising(1/1)

Denoising(1/64)

FeatureProjection

CNN only

Ours w/o
Photons

Ours w/
Photons

GroundTruth

Figure 6. Dynamic Objects and Fixed Lights. Additional results for our method as well as for the baselines for dynamic objects and fixed
lights, complementing Fig. 5 of the main paper.

10

Denoising (1/4)

Denoising (1/16)

Denoising (1/32)

Denoising (1/64)

Ours w/o photons

Ours w/ photons

Figure 7. Dynamic Objects and Fixed Lights. Predictions and error images with respect to ground truth for different denoising approaches
and our approach for dynamic objects and fixed lights. Error plots are shown below the respective prediction.

11

Feature
Projection

Error

Ours w/o
Photons

Error

Ours w/
Photons

Error

Ground Truth

Figure 8. Dynamic Objects and Dynamic Lights. Example scenarios that are challenging for our approach with dynamic objects anddynamic lights. We observe failure cases for specular materials and mirrors, when objects are close to the camera and in the presence offine shadows.

12References[1] Benedikt Bitterli. Rendering resources, 2016. https://benedikt-bitterli.me/resources/. 2, 4
[2] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 1, 6
[3] Wenzel Jakob. Mitsuba renderer, 2010. http://www.mitsuba-renderer.org. 3[4] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of the International Conf.
on Learning Representations (ICLR), 2015. 3[5] Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void - learning denoising from single noisy images. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 3[6] Eric P Lafortune and Yves D Willems. Bi-directional path tracing. 1993. 3[7] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila.
Noise2noise: Learning image restoration without clean data. In Proc. of the International Conf. on Machine learning (ICML), 2018. 3[8] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban
Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017. 3[9] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classificationand segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 1[10] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015. 6[11] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent
Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2015. 4[12] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception
architecture for computer vision. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016. 4

13