Supplementary Material for
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

Abstract
In this supplementary document, we first clarify implementation specifics in Section 1. We then give more detailed results
for the experiments introduced in the main paper in Section 2. Additional experimental results to study the impact of theemployed regularization techniques follow in Section 3. Finally, we showcase the challenges that NeRF-like techniques facewith generalization to out-of-distribution camera poses in Section 4. The supplementary video shows qualitative comparisons
between KiloNeRF and the NeRF baseline.

1. Implementation Details
For KiloNeRF, we normalize position inputs such that each network receives positions in [−1, 1]3 . We initialize KiloNeRF
such that the parameters of the individual networks are identical to each other. For the NeRF baseline, we use the PyTorch
implementation from Lin Yen-Chen [3]. This implementation was verified to be numerically identical to the original TensorFlow implementation of NeRF. For NSVF, we use the implementation provided by the authors. Except for the numberof training iterations for NeRF, both baselines were run with the hyperparameters advocated in the respective papers. The
quantitative results of NSVF on all scenes and of NeRF on the Synthetic-NeRF dataset were taken from the respective papers.
To render novel views with NSVF on our test machine equipped with a GTX 1080 Ti (11 GB VRAM), it was required todecrease the batch/chunk size for some scenes.

1.1. KiloNeRF Optimizations
A suitable implementation of KiloNeRF is crucial for achieving a good speed-up. Please note that our optimizations aremostly specific to the computation required in KiloNeRF (querying many tiny MLPs) and could not be used to accelerate therendering of vanilla NeRFs. On top of PyTorch, we make use of custom CUDA kernels and the HPC library MAGMA since
the computational primitives exposed by PyTorch are not adequate for efficiently querying many tiny MLPs.
1.1.1

Network Evaluation

In a vanilla MLP, for each linear layer, a matrix-matrix multiplication is calculated that processes a batch of B inputs inparallel. More specifically, a B × I matrix is multiplied with a I × O, where I refers to the number of input features and O
refers to the number of output features of the associated layer. KiloNeRF consists of many independent MLPs and therefore
during evaluation for each network with index i a separate matrix-matrix multiplication needs to be calculated. In otherwords, for each network i a Bi × I matrix needs to be multiplied with a I × O matrix. Note how the batch size Bi depends on
the network i. This is necessary since individual networks are queried for a different number of input points Bi . For instance,
networks that are further away from the camera are queried for fewer points than close-by networks since the distance between
points on adjacent rays increases the further away the sample points are from the camera. A straightforward possibility to
implement linear layer evaluation directly in PyTorch is by calling torch.matmul in a loop for each network. As aconsequence, a separate CUDA kernel is launched for each matrix multiplication associated with the individual networks.
By default, individual CUDA kernels are executed sequentially, which would imply here that only a small fraction of theGPU’s SMs are used. To increase parallelism execution of the individual CUDA kernels can be overlapped with help of
CUDA streams. Nevertheless, this solution is suboptimal since there still have to be launched as many kernels as there
are networks (thousands), which leads to considerable kernel launch overhead and therefore bad performance. The batchedmatrix multiplication routine exposed by PyTorch (torch.bmm) appears like a better option since it theoretically allows
that only a single CUDA kernel is launched that handles the matrix multiplications associated with all networks in parallel.
However, the routine exposed by PyTorch requires that all input matrices have the same shape. Since different networks
are in general queried for different numbers of points the matrices associated with different networks have different shapes(Bi is not constant). Therefore torch.bmm cannot be used. In contrast, MAGMA indeed provides a routine that can also
handle input matrices of variable shapes and is therefore well suited for the more general case present in KiloNeRF. The
inner workings of the employed MAGMA routine are described in [1]. For inference only we use a self-developed routinethat fuses the entire network evaluation (consisting of Fourier feature calculation, linear layers, and activation functions)
into a single CUDA kernel. This has the advantage that intermediate results (e.g. network activations) do not have to bewritten and read from global memory, which is especially important for our case because the ratio of arithmetic operations tomemory transfers is low for multiplications of small matrices. Since intermediate results are necessary for backpropagation
this strategy can only be used for inference. In our fused kernel each CUDA block is responsible for a different networkand each thread is responsible for handling a single network input. The intermediate results can be fully stored in per-thread
registers, which is only possible due to the small network width of 32. The kernel uses 96 registers, which implies thata maximum of 640 threads per SM can be active and that occupancy on a GPU with Compute Capability 6.1 amounts to0.3125. Since there should be enough arithmetic operations to hide the small memory latency incurred by reading a network
input and writing the final RGBσ output, a higher occupancy is not likely to lead to performance gains. Due to the small sizeof the individual MLPs of approximately 25 KB all parameters of a particular MLP can be loaded into shared memory, which
implies that network parameters only have to be loaded once per frame from global memory into on-chip memory. Network
evaluation could be further sped up significantly by making use of Tensor Cores, which is likely to enable real-timerendering of full HD images.
1.1.2

Input Reordering

Both the MAGMA routine for matrix multiplication and the self-developed CUDA kernel require that input points are sorted
according to the index i of the network that is responsible for that point. As mentioned above for each network a Bi × I
matrix Xi is multiplied with a weight matrix Wi with shape I × O. Here it is required that the matrix Xi is stored densely.
However,
PB the unprocessed result from the preceding sampling point generation step is the unordered B × 6 matrix X̃, whereB = i=1 Bi and I = 6 because of the 3-dimensional position and direction inputs. With each input X̃j (j = 1, . . . , B)
there is associated a network with index ij that is responsible for that input. We use thrust to perform a key-value sort where
the keys are i1 , . . . , iB and the values are the row vectors of X̃. The result consists of the dense matrices Xi . To further
decrease the non-negligible overhead introduced by this sorting step, we do not directly reorder the input matrix where
each row consists of 6 floats. Memory traffic can be decreased by reordering instead a list of integers, where each element
corresponds to an index from which position and direction can be calculated.

2. Detailed Results
Quantitative results for each scene are given in Tables 2, 3, 4 and 5. Qualitative results of all scenes that were not shownin the main paper are given in Fig. 3, Fig. 4, Fig. 5 and Fig. 6. KiloNeRF matches the quality of the baselines, while being
significantly faster.

3. Impact of Distillation and L2 Regularization
We performed a large-scale ablation study on all 25 scenes to determine which impact distillation and L2 regularization
have on quality. In Table 1, we can see that both techniques lead to an improvement in terms of PSNR, SSIM and LPIPS.
Note that due to distillation mainly artifacts in free space are reduced. These artifacts are striking to the human eye as can beseen in Fig. 1, but do not have a strong influence on quality metrics, since only a small amount of pixels are affected.
MethodKiloNeRF w/o Distillation and L2 Regularization
KiloNeRF w/o Distillation
KiloNeRF

PSNR ↑SSIM ↑LPIPS ↓

29.54
30.44
30.66

0.932
0.940
0.945

0.051
0.048
0.043

Table 1: Large-scale Ablation Study. Both distillation and L2 Regularization have a positive effect on quality metrics.

Figure 1: Importance of Distillation. Without distillation cloud-like artifacts in empty space emerge (top row). The proposedtraining strategy mitigates this issue (bottom row)

4. Out-of-distribution Generalization
In general, since NeRF models are trained on a per-scene basis, rendering a novel view that requires querying for view
directions that are not part of the training distribution yields poor results. More concretely, envision an object of which only
photographs on a circle around that object are available. Synthesizing a novel view with NeRF from any point on this circle
works well since it is ensured that the model is only queried for viewing directions that are inside the training distribution. It
is also possible to zoom in or out as long as the camera stays on the plane on which the circle lies. However, if we leave thatplane e.g. by increasing camera elevation, results are getting significantly worse as the model is queried for view directions
outside the training distribution. Since a NeRF model is trained on a per-scene basis, there is no knowledge about material
properties available and conclusively colors for out-of-distribution view directions cannot be inferred. With that in mind let
us take a look at the novel views synthesized by NeRF respectively KiloNeRF from Fig. 2. For that scene, all training images
lie approximately on a circle. In particular, the training set does not contain any images from high viewpoints. The images
shown in Fig. 2 are rendered from such an out-of-distribution viewpoint. As a result, both NeRF and KiloNeRF predict colors
poorly. In KiloNeRF’s case, an aggravating factor is that networks are independent and hence each network is biased in adifferent way. A possible solution is thus to combine KiloNeRF with a method that learns inductive biases from multiple
scenes like for instance [2].

(a) Out-of-distribution camera pose(b) NeRF(c) KiloNeRF

Figure 2: Out-of-distribution generalization. Novel views are synthesized with NeRF (b) and KiloNeRF (c) from anout-of-distribution camera pose that is visualized in (a), where gray cameras correspond to the training images and the redcamera corresponds to the rendered pose. Both methods have problems dealing with this situation. In KiloNeRF’s case, theindependence of the networks additionally might cause that each network is biased towards its local training distribution,
which results in block-like artifacts.

Ground truth

NeRF

NSVF

KiloNeRF

Figure 3: Qualitative results for the scenes Character (BlendedMVS), Jade (BlendedMVS), Chair (Synthetic NeRF), Drums(Synthetic NeRF) and Ficus (Synthetic NeRF)

Ground truth

NeRF

NSVF

KiloNeRF

Figure 4: Qualitative results for the scenes Hotdog (Synthetic NeRF), Materials (Synthetic NeRF), Mic (Synthetic NeRF),
Bike (Synthetic NSVF) and Lifestyle (Synthetic NSVF)

Ground truth

NeRF

NSVF

KiloNeRF

Figure 5: Qualitative results for the scenes Palace (Synthetic NSVF), Robot (Synthetic NSVF), Spaceship (Synthetic NSVF),
Steamtrain (Synthetic NSVF) and Toad (Synthetic NSVF)

Ground truth

NeRF

NSVF

KiloNeRF

Figure 6: Qualitative results for the scenes Wineholder (Synthetic NSVF), Barn (Tanks & Temples), Caterpillar (Tanks &Temples), Ignatius (Tanks & Temples) and Truck (Tanks & Temples)768 × 576

Character

Fountain

Jade

Statues

PSNR ↑NeRF
NSVF
KiloNeRF

29.43
27.95
29.44

28.04
27.73
28.50

26.52
26.96
27.14

25.17
24.97
24.49

SSIM ↑NeRF
NSVF
KiloNeRF

0.95
0.92
0.95

0.91
0.91
0.93

0.89
0.90
0.91

0.87
0.86
0.88

LPIPS ↓NeRF
NSVF
KiloNeRF

0.03
0.07
0.04

0.07
0.11
0.06

0.08
0.09
0.06

0.09
0.170.08

Render timein ms ↓NeRF
NSVF
KiloNeRF

37266
2291
17

37266
5863
3837266
5633
42

37266
3806
22

Speedupover NeRF ↑NSVF
KiloNeRF

162150

6989

7892

10
1715

Table 2: BlendedMVS

800 × 800

Chair

Drums

Ficus

HotdogLego

Materials

Mic

Ship

PSNR ↑NeRF
NSVF
KiloNeRF

33.00
33.19
32.91

25.01
25.18
25.25

30.13
31.23
29.76

36.18
37.14
35.56

32.54
32.29
33.02

29.62
32.68
29.20

32.91
34.27
33.06

28.65
27.93
29.23

SSIM ↑NeRF
NSVF
KiloNeRF

0.97
0.97
0.97

0.93
0.93
0.93

0.96
0.97
0.97

0.97
0.98
0.98

0.96
0.96
0.97

0.95
0.97
0.95

0.98
0.99
0.98

0.86
0.85
0.88

LPIPS ↓NeRF
NSVF
KiloNeRF

0.05
0.04
0.02

0.09
0.07
0.05

0.04
0.02
0.02

0.12
0.03
0.02

0.05
0.03
0.02

0.06
0.02
0.02

0.03
0.01
0.01

0.21
0.16
0.08

Render timein ms ↓NeRF
NSVF
KiloNeRF

56185
2492
18561854130
2556185
3990
19561855599
29561854231
22561855521
29561852060
22561856730
43

Speedupover NeRF ↑NSVF
KiloNeRF

23318114222114295410
192613254810
195427252381295

Table 3: Synthetic NeRF

800 × 800

Bike

Lifestyle

PalaceRobot

Spaceship

Steamtrain

Toad

Wineholder

PSNR ↑NeRF
NSVF
KiloNeRF

25.17
37.75
35.49

32.26
34.60
33.15

33.57
34.05
34.42

33.57
35.24
32.93

34.66
39.00
36.48

33.42
35.13
33.36

30.77
33.25
31.41

28.97
32.04
29.72

SSIM ↑NeRF
NSVF
KiloNeRF

0.87
0.99
0.99

0.95
0.97
0.97

0.95
0.97
0.96

0.95
0.99
0.98

0.98
0.99
0.99

0.98
0.99
0.98

0.94
0.97
0.95

0.94
0.97
0.95

LPIPS ↓NeRF
NSVF
KiloNeRF

0.09
0.00
0.01

0.03
0.02
0.02

0.02
0.02
0.02

0.02
0.01
0.02

0.02
0.01
0.01

0.05
0.01
0.01

0.05
0.03
0.04

0.05
0.02
0.03

Render timein ms ↓NeRF
NSVF
KiloNeRF

56185
2520
235618523102
355618516929
31561852482
21561859023
265618510278
21561853264
2156185
16381
29

Speedupover NeRF ↑NSVF
KiloNeRF

222438

21586

31804

232739

62149

52670

172625

31958

Table 4: Synthetic NSVF

1920 × 1080

Barn

Caterpillar

FamilyIgnatius

Truck

PSNR ↑NeRF
NSVF
KiloNeRF

27.71
27.16
27.81

25.87
26.44
25.61

33.33
33.58
33.65

27.79
27.91
27.92

26.92
26.92
27.04

SSIM ↑NeRF
NSVF
KiloNeRF

0.85
0.82
0.85

0.89
0.90
0.90

0.95
0.95
0.96

0.94
0.93
0.94

0.89
0.90
0.90

LPIPS ↓NeRF
NSVF
KiloNeRF

0.19
0.31
0.16

0.12
0.14
0.10

0.05
0.06
0.04

0.08
0.11
0.06

0.11
0.15
0.10

Render timein ms ↓NeRF
NSVF
KiloNeRF

18267125142136

18267117244
80

1826719314
701826718072
56182671
18715
114

Speedupover NeRF ↑NSVF
KiloNeRF

71341

112275202598233288101606

Table 5: Tanks & Temples

References[1] Ahmad Abdelfattah, Azzam Haidar, Stanimire Tomov, and Jack Dongarra. Novel hpc techniques to batch execution of many variable
size blas computations on gpus. In International Conference on Supercomputing, 2017. 2[2] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2021. 4[3] Lin Yen-Chen. PyTorchNeRF: a PyTorch implementation of NeRF, 2020. 1