Differentiable Rendering with Reparameterized Volume Sampling
Authors: Kirill Struminsky
Oleg Desheulin
Venue: NeurIPS 2022 Submitted (2022)
Keywords: neural radiance fields
differentiable rendering
importance sampling
reparameterization trick

Abstract: We propose an alternative rendering algorithm for neural radiance fields based on importance sampling. In view synthesis, a neural radiance field approximates underlying density and radiance fields based on a sparse set of views of a scene. To generate a pixel of a novel view, it marches a ray through the pixel and computes a weighted sum of radiance emitted from a dense set of ray points. This rendering algorithm is fully differentiable and facilitates gradient-based optimization of the fields. However, in practice, only a tiny opaque portion of the ray contributes most of the radiance to the sum. Therefore, we can avoid computing radiance in the rest part. In this work, we use importance sampling to pick non-transparent points on the ray. Specifically, we generate samples according to the probability distribution induced by the density field. Our main contribution is the reparameterization of the sampling algorithm. It allows end-to-end learning with gradient descent as in the original rendering algorithm. With our approach, we can optimize a neural radiance field with just a few radiance field evaluations per ray. As a result, we alleviate the costs associated with the color component of the neural radiance field at the additional cost of the density sampling algorithm.

Differentiable Rendering with Reparameterized Volume Sampling
Anonymous Author(s)
Affiliation Address
email
Abstract
We propose an alternative rendering algorithm for neural radiance fields based on importance sampling. In view synthesis, neural radiance field approximates underlying density and radiance fields based on sparse set of scene views. To generate pixel of a novel view, marches ray through the pixel and computes a weighted sum of radiance emitted from a dense set of ray points. This rendering algorithm fully differentiable and facilitates gradient- masden optimization of the fields. However. in practice, only tiny opaque portion of the ray contributes most of the radiance to the sum. Therefore, we can avoid computing radiance in the rest part. In this work, we use importance sampling to pick non-transparent points on the ray. Specifically, we generate samples according to the probability distribution induced by the density field. Our main contribution is the reparameterization of the sampling algorithm. It allows end-to-end learning with gradient descent as the original rendering algorithm. With our approach, we radiance field with just a few radiance field evaluations per ray. As a result, we alleviate the costs associated with the color component of the neural radiance field at the additional cost of the density sampling algorithm.
10
11
12
optimize neural
14
16
Introduction
18 We propose a volume rendering algorithm for learning 3D scenes and generating novel views. Recently, learning- based approaches led to significant progress in this area. As early instance, 8 proposed to represent a scene via a density field and radiance (color) field parameterized with MIP. They run differentiable volume rendering algorithm with the MI.P-based fields and minimize the discrepancy between the produced images and a set of reference images to learn a scene representation. The algorithm we propose is drop-in replacement for the volume rendering algorithm used in NeRF [8] and follow-ups.
20
21
22
24
The underlying model in NeRF generates an image point in the following way. It casts ray from a camera through the point and defines the point color as a weighted sum along the ray. The sum aggregates the radiance of each ray point with weights induced by the density field. Each summand involves a costly neural network query, and model has a trade-off between rendering quality and computational load. NeRF obtained a better off with two-stage sampling algorithm used get 30 ray points with higher weights. The algorithm is reminiscent of importance sampling, yet requires training auxiliary model.
26
28
31
32 In this work we propose rendering algorithm based on importance sampling. Our algorithm also acts in two stages. In the first stage, it marches through the ray estimate density. In the second 34 stage, constructs Monte-Carlo color approximation using the density to pick points along the ray. The resulting estimate is dullydiifferentiabee and does not require any auxiliary models. Besides that, 36 we only need a few samples to construct precise color approximation. An intuitive explanation is that
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37 we only need to compute the radiance of the point where ray hits solid surface. In the experiments,
38 we query radiance for 16 fewer ray points during training compared to baseline. Nevertheless, we manage to obtain competitive model and rendering quality.
that distinct models to parameterize radiance and density. Specifically, the first stage only queries the density field, whereas 42 the second stage only queries the radiance field. Compared the standard rendering algorithm, the
As result, algorithm is more suitable for recent solutions
TO
13 12
41
43 second stage of our algorithm avoids redundant radiance queries and reduces the memory required
44
for rendering.
Neural Radiance Fields
45
Neural radiance fields represent 3D scenes with scalar density field - R and vector radiance field RR R R The scalar field represents volume density each spatial location æ, and c( d) returns the light emitted from spatial location in directionn d represented as a normalized three dimensional vector.
46
47
49
50 For novel view synthesis, they adapt a volume rendering technique that computes a pixel color 51 C(o, d) (denoted with capital letter). particular, the expected color along ray o + id going
52 from the camera through the pixel is
pr(t)c(o+ td, d)dt, for pr ao + td) exp (-/ o tm
odo+s sd)ds) (I)
C(o,d)
53 Here, Pr (t) is a probability density function random variable T a ray r. Intuitively, T the 54 location on the ray where portion of light coming into the point o was emitted.
One way approximate the integral would be off the integral depth ( and then grid 56 In t0 < 41 < Im = by to compute the integral with a Riemann sum
C Riemann (o, d) (t. t, t.-1)pr, :0 10+ t,d, d),
(2)
1=1
= Pr,; where Pr,j
= t,d) exp
- -t, -t--)oo- +jd) j=1
(3)
57
Importantly, iq fully differentiable and can be used part of gradient- based learning pipeline.
58 While such approximation works in practice, a fauithfull approximation requires a dense grid and
59 multiple evaluations of and Besides that, common situation is when ray intesects solid surface at some point E [l ly]. In this case, probability density pr (!) will concentrate its mass near and will be close to zero other parts the ray. As a result, most of the summands Eq. 2 will make negligible contribution to the sum.
60
61
82
Monte Carlo methods give another way apporximate the color. Given i.i.d. samples f,, ....... Pr (t), the color estimate is gatherd by
64
1 m m Σ clo (,
Časclo, d)
65 Due to the importance sampling with distribution pr- (t), each term in Eq contributes equally to the sum as the samples come from regions with non-negligible density. Unlike the grid estimate in Eq, 2 the Monte-Caue estimate for the parameters of For instance, NeRF adresses the issue via a hierarchical sampling
67
estimate depends on the scene density implicitly and requires a custom gradient
69 scheme. trains a coarse model with a grid approximation to generate importance-weighiee ray locations for separate fine-grained model.
70
71 In the next section, we propose propose a novel principled approach to training neural radiance
fields with weportedce-weighted color approximation in Eq.

---

73 3
Learning with Stochastic Color Estimates
74 In this section, we will discuss stochastic approximations to the expected color C( d) in detail.
75 Recall that C(o, d) = Erc(o + Td, d), where T is a random variable with density specified in Eq. [ Even though density (t( involves an integral we cannot compute in closed form, below we first assume that we have an algorithm to compute fr pr(s)ds (s)s used in Pr (t).
76
77
78
Given a groundtruth expected color Cgt. optimization objective in NeRF captures the difference L(C(o, d), Cyt) between Cgt and the estimated color Č(o, d). To reconstruct a scene NeRF runs gradient based optimizer to minimize the objective averaged across multiple rays and multiple viewpoints. Such approach works for grid estimate Č(o, d) Ö Ciimann (o, d) that depends on density o, explicitly, but Monte-Caue estimate Cmc(o, d) of the expectation depends on a implicitly and naive automatic differentiation algorithm will return zero gradients.
79
80
81
82
83
84 In the rest of the section, we first introduce algortihm to compute Čmc d) and derive gradient estimate for the algorithm. Then, we conclude with a discussion our implementation of the estimate. To ease the notation, we will also introduce or (1) a(o + Id) and e, (1) = c(o + Id, d) to denote fields restricted to a ray r = o td.
85
87
88
3.1
Estimate Reparameterization
To make the dependence of Č( d) on or explicit, we change the variables in the expectation ETC, (T). For F(t) exp (- St, (s)ddss) and y = F(t) we write
89
90
(T) [ (())p(())d f U5 cr(FF
31 Function F()) acts as cumulative distribution function of the variable T with a single exception that, in general, y/ lim,- F(t) 1. In volume rendering. F()) is called the opacity function
88 with yy heing equal to pixel opaqueness. Bounds of integration are where yn = Ft.... = 0 and
94 ys lim Ft)). For simplicity, below we replace y, with Ftyy) where ty is the maximum ray
95 depth.
In the right-hand side of Eq. 5 integration boundaries depend on the opacity F and, thus, on the volume density or. We further simplify the integral by changing the integration boundaries to [o, 1]
97
I Ms c, cF (v))
L. (us (wy yn Un)cr (F- + (vs yn )
(6)
98 With this, we arrive the following reparameterized Monte-Carde estimate of the expected color um:
39 obtained with i.i.d U10, 1] samples
1 Lws - Un Vn) (F- (yn + (us Un))
C (o d) = m
100
In the above estimate sampling does not depend on volume density or color cy. Essentially, this is reparameterized Monte-Carlo estimate that generates samples from pt( using the inverse cumulative distribution function F-1(yn + (ws yn Ju).
102
We further improve the estimate using stratified sampling. To do this, replace the uniform samples U1..... 14 with uniform independent samples within regular grid bins o, ~ U[ m 1....,m and derive reparameterized (R) stratified (S) Monte Carlo estimate
104
105
C
Σ (ur (F (vs -yy--) )n)))))
(8)
106 is easy to show that both 7 are unbiased estimates of
107 Next, we will discuss algorithms used to compute the inverse opacity function F-1(y) and compute the gradients of the function with automatic differentiation.
3

---

109
3.2
Implementation of Inverse Opacity for Volume Sampling
110 To compute the estimates in Eqs. eqs. 7 and (8). we need to compute the inverse opacity F (v) along with its gradient. In practice, we start with a black-box density field (z) and compute the induced density Pr((t) and opacity F(t) on a ray r via approximations. Assuming we have an algorithm to compute _ (r(s)ss, below we show how to implement the inverse opacity F-!
112
113
exp We invert F(I) = =1 1 exp (-Sh or(s)ds) with binary search. Note that F(() is monotonic
function and for y (V]: uf) (F(),,, () the inverse lies in (4n, (y). To compute F-1 (y), we start with boundaries 1 = In and I, = b and gradually decrease the gap between the boundaries based on the comparison of P with Importantly, such procedure is easy to parallelize across multiple inputs and multiple rays.
116
117
119 However, we cannot backpropagate through the binary search iterations and need workarond to compute the gradient 39 of t(0) F- (y. 8). To do this, compute differentials of the right and the left hand side of equation y(0) = Fl,,0)
121
dy do
OF Ot
OF
d0 +
d0.
00
0t 00
00
122
By the definition of F(1,0) 0) have
aF
(1 F(t, F(t,0))o,(tt,)
(10)
ōt
of (1 F(t, 0)) a (f (r))
(11)
00
We solve Eq. 9 for ät ao and substitute the partial derivatives using Eqs. eqs. 10) and 11
to obtain the
124 final expression for the gradient
Ōt
(1 -1- F 9 Sr or 70
36
(12)
00
(1 - F(t, (t,
In our implementation, use automatic differentiation to compute 0y/00 and a Sr, os))s to combine the results as in Eq. 12
127
3.3
Computing Opacity in Practice
To describe the sampling procedure, we assumed that we have an oracle for computing Jr, (s)ds along with its gradient. The integral is required compute opacity F(t). In this work, consider an arbitrary volumetric density (s) and approximate with a linear spline on a ray r = 0 td to sample take grid to
130
< f'm and compute (to), ..... : ar (f,) to construct the spline (s) (Fig. D For the piecewise linear function (z) we can compute the integral f", (s)ds in a closed form. Additionally, approximation compute the gradients of knots a, (to)), (1m)) Thus, we obtain a differentiable rendering algorithm for an arbitrary density field a. Besides that, some recent works parameterize
the points the ray. Specifically,
132
can backpropagate the gradients through the
134
Density Field
Opacity
1.0
ott) Spine ö(t)
Opacity F(t)
0.5
Approximation F(t)
F-lly)
En
4n
tr
Figure 1: Illustration of opacity inversion. We approximate arbitrary density field with linear
spline(left). Then the spline to approximate opacity F(t) and compute f-1(z) (right).
135
196
density fields thruogh voxel grids. For a voxel grid, when o, is trilinear interpolation of the grid values, we can compute the integral in closed form.

---

138 4
Related Work
Neural Radiance Fields & Efficient Sampling Even in the orginial work on neural radiance fields 8 the authors aimed to find an efficient sampling algorithm for volume rendering. Our importance sampling approach is reminscent of their hierarchical sampling solution. On the first stage, they use an auxilliary model on a sparse grid. Then they use the predicted densities to generate dense grid with a improtance sampling-like algorithm. As opposed NeRF, we compute density on dense grid at the first stage and then use a sparse set of samples to evaluate radiance on the second stage. Our algorithm also allows training without auxilliary models.
140
142
143
144
145
146
Several recent fo works also aimed to improve NeRF rendering time and overall efficiency. Most of these works consider trainable encoding 0 and utilize some efficient data structure to make each evaluation of multi-layered perceptron fast or avoid evaluating MLP at all. One of the earliest work in this direction was NSVF 77]. The authors proposed embeddings and then estimate query point embedding with a trilinear interpolation and positional encoding. During training, the octree gradually increased resolutionn in the regions of interest and pruned the empty areas. However, this method still requires the time-consuming training of MsPs Voxel-based embedding structure was further studied in recent works and it was shown that positional encoding doesn't I affect model convergence - the network can be trained with fully trainable embedding without any encoding. And also, what is more important, such a structure allows for
148
octree to store point-based
150
152
153
154
155
158
making neural network (MLP) shallower and consequently faster. Following this idea, Vox
157
proposes to avoid MLP at all in density computation, while Instant Neural Graphic Primitives uses it to solve hash collisions. When density field is a piecewise linear we can compute opacity closed-form.
158
159
160
Reparameterization Trick & Implicit Differentiation Our solution is inspired by the literature on deep latent variable models [6] [T] and approximate inference. In this area, models often contain an internal sampling algorithm with parameters we need to optimize. The now-common approach for continuous random variables is the reparameterization trick, which we apply in our setup. The authors of (9) give a comprehensive overview of the area state.
161
162
163
164
A closely related work in the context of deep variable models is [.. They were first to apply implicit diffcrentiation to estimate gradients for the reparameterization trick. While we use the implicit differentiation to compute the gradient of binary search output, the same approach applies to other iterative algorithms. The examples include ODE solves (3) fixed-point iterators [ and optimization algorithms.
186
168
5 Experiments
171
5.1
Importance Sampling for a Single Ray
Foggy Density Field
Glass and Wall Density Field
Wall Density Field
Opecio
Opacity
Opacity
Radiance
Radiance
Radiance
Estimate Variance
Estimate Variance
Estimate Variance
10
10
10
10
10"
10
Ult.. 6]]
10x
10-3
with statifed sampling
(2) (ouis)
with snatified sampling
J0
100
125
10"
Number of Samples
Number Samples
Number of Samples
Figure 2: Color estimate variance compared for a varying number of samples. The upper plot illustrates underlying opacity function on a ray; the lower graph depicts variance in logarithmic scale. Our importance sampling approach (solid green) has significantly lower deviation than a stratified baseline (solid red) typically used in volume rendering.
5

---

172
We begin with an evaluation of color estimates in one- ene-dimensional setting. Our experiment models light propagation on a single ray in three typical situations. The upper row of Fig. ddefines scalar radiance field (orange) e(t) and opacity functions (blue) F(t) for
173
174
175
"Foggy" density field. It models a semi-transparent volume. Similar fields occur after model initialization during density field training;
176
177
"Glass and wall" density field. Models light passing through nearly transparent volumes such as glass. The light is emitted at three points: the inner and outer surface of the transparent volume and an opaque volume near the end of the ray;
178
179
180
"Wall" density field. Light is emitted from a single point on a ray. Such fields are most common in applications.
For the three fields we estimated the expected radiance C = C= S. c(t)dF()). We considered two baseline methods (both in red in Fi the first was Monte Carlo estimate of C obtained with
182
183
U ty] samples, the second was stochastic modification of Eq. Jusing grid in = to < ....
185
mm ly:
dF
C= = E(t, (t: (((1))(T))
with independent T; ~ Ult, - 1, ty].
(13)
dt l=
186
In other terms, the second baseline uses stratified sampling to reduce the baseline Monte Carlo estimate variance. [3 is an instance of vanilla volume rendering algorithm one may encounter in
187
188
practice. We compared the baseline against estimate from Eq. eq. (7) and its stratified counterpart from Eq. 8 All estimates are unbiased. Therefore, we only compared the estimates variances for varying number of samples m.
189
190
In all setups, our stratified estimate uniformly outperformed the baselines. For the most challenging "Toggy" field, approximately m = 32 samples we required to match the baseline performance for m 128. We matched the baseline with only a m = 4 samples for other fields. Importance sampling requires only a few points for degenerate distributions. In further experiments, we take m 8, 32 to obtain a precise color estimate even when model did not converge degenerate distribution.
192
193
194
195
196
5.2
Scene Reconstruction with Reparameterized Volume Sampling
Next, apply our algorithm to 3D scene reconstruction based on set of image projections. As benchmark, we use the common Lego dataset. The primary goal of the experiment is to demonstrate computational advantages of our algorithm compared basic volume rendering baseline.
198
199
200
As starting point, we took the original NeRF's MLP g with eight layers used to compute density and radiance. Then modified the architecture
only three first layers to compute the density field. When the density field is queried, we only compute the first three layers, while for the radiance compute the whole network. Even though such modification may have put additional limitations on the density model, it illustrates the benefit of using fewer radiance queries. For density, we used softplus activation ensure its positivity, while for the radiance we used sigmoid activation to ensure that the output will be a valid RGB image.
202
204
206
In our experiment, we did not reproduce the expensive hierarchical sampling used in NeRF and trained a single model in all experiments. Our baseline calculated color using Eq. ??. We took a 208
dense grid of m = 128 points along each ray and trained the model using Huber loss with the ground truth colors and the predict colors. We additionally perturbed the grid to egularize the model. We used Adam (5) optimizer for training and decayed the learning rate during 100 epochs of training from 3e-4 to 3e-7 following MIP-NeRF's scheduler [2] with image batch size equal 8 and each epoch consisting of 8000 batches. To form training batch, for each image in an image batch we selected 375 pixels and cal culated loss over them.
210
212
214
215
We evaluated the importance sampling based rendering algorithm with the same architecture and hyperparameters as with the baseline model. We used the same algorithm to sample a dense grid of
216
m = 128 points to query the density field and construct an approximating spline. Then we calculated color approximation with Eq. with m' {8, 32) samples from the inverse cumulative density function approximated by the spline.
218

---

Model
PSNR (1)
SSIM (1)
LPIPS (alex) 14 [4]
Baseline
27.247
0.904
0.1138
Splines, #pts in estimation 8
Training
Validation
8 pts
1 pts
23.377
0.822
0.1819
8 pts
2 pts
25.193
0,858
0.1449
8 pts
4 pts
26.210
0.883
0.1215
8 pts
8 pts
26,502
0.892
0.1243
8 pts
16 pts
26.570
0.894
0.1333
8 pts
32 pts
26,585
0.895
0.1369
32 pts
I pts
22.519
0.805
0,2050
32 pts
2 pts
24.902
0.846
0.1523
32 pts
4 pts
26.523
0.881
0,1181
32 pts
8 pts
27.100
0.897
0.1083
32 pts
16 pts
27.252
0.902
0.1167
32 pts
32 pts
27.286
0.904
0.1223
Table Ablation study and comparison with the baseline. Metries are calculated over test views for Lego scene
8
220
First, we compared the rendering quality of our algorithm against the baseline. Tab. contains the quantitative results and figs. 3 and contain qualitative results. From the rendering quality viewpoint 1 with m' 32 samples, our model works on par with the baseline, while with m' 8 samples has slightly worse performance. Though we did not aim to reproduce the tate-off-the-rt results, we speculate that better density model could improve the results even further. Fig. ??, we compared
222
224
225
the rend ering performance of importance sampling for varying m'. Our algorithm produced sensible
renders even for mí' = 1, however noise artifacts only disappeared for m' 32. Fig. 4 shows
226
stratified estimate renders (Eq. 8) along with Monte Carlo renders (Eq, 7 for yr' 32. With
228
the same rendering complexity, the variance reduction obtained via stratified sampling purges the
rendering artifacts that naive Monte Carlo estimate has.
Model
Iter/sec (1)
Mem Usage (L)
Baseline
3.90
8.5 Gb
Splines 1 pts
4.89
1,8 Gb
Splines 2 pts
5.06
1.8 Gb
Splines 4 pts
4.88
2.1 Gb
Splines 8 pts
4.53
2.2 Gb
Splines 16 pts
3.81
2.5 Gb
Splines 32 pts
2.98
2.8 Gb
Table 2: Speed & memory estimates. Iteration time is measured during training on GTX 1080 ti, memory usage is measured during inference with batch size equal 1024
230 Besides the rendering quality, we estimated the training speed and memory footprint of our algorithm in Tab. For m' = training iterations were on average x 1.2 faster, while for m' 32 training iterations took 1.3 more time. The difference occurred due to a varying number radiance queries. For memory footprint viewpoint, our algorithm used x3.0 and x3.9 less memory for m' = 32 and 234 m correspondingly. With this, important sampling leaves room for further optimization as allows to work with bigger batches with moderate variability in rendering speed and quality.
232

---

points
Target
m
mous
Figure 3: Rendering results with a different number of samples in the stratified estimate. From left to right and from top to down: 1, 2, 32 points estimates, Baseline and Target for reference.
mir
Figure 4: Comparation of rendering results from different viewing angles with Monte- Carlo estimate (top row) and stratified Monte-Care Carlo estimate (bottom row), both with 32 points along each ray

---

Conclusion
236
We proposed alternative classic volume rendering algorithms used in 3D scene reconstruction. For synthetic experiment and in full-scale reconstruction task we achieve better estimation results in terms of variance with significantly smaller computation footprint. In particular, our algorithm allows for significant memory reductions and even increased inference time. At the same time, we demonstrate competitive rendering quality. We believe that our approach is promising altenative to standard volume rendering techniques.
238
240
241
242
243
6.1
Broader Impact
244
We hypothesize that models like NeRFs may be used in online stores for better user experience. Then people will choose more suitable products. We are not aware of any possibilities to use this in a negative way. Furthermore, are sure that the efficient sampling proposed for 3D rendering may reduce computation costs and therefore environmental damage.
245
247
References
248
[1] Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
250
[2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
252
253
254
255
[3] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
257
[4] Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients.
258
Advances in Neural Information Processing Systems, 31, 2018
259
[5] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412.6441..69800 2014.
260
[6] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 131.666 14, 2013.
262
263
[7] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33:15651-15663, 2020.
264
285
[8] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-421 Springer. 2020.
287
268
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. J. Mach. Learn. Res,, 21(132):1-62, 2020.
289
270
[10] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. arXiv preprint arXiv:2201.05986 2022.
272
[1I] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
273
274
275
[12] Cheng Sun, Min Sun, and Hwann-Tzone Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. arXiv preprint 1.. 2021.
276
277
[13] Alex Yu, Sara Fridovich- Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:2112.05/33 2021.
278
279
280
[14] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018
281

---

Checklist
282
283
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
285
contributions and scope? [Yes] See Section T
286
(b) Did you descrihe the limitations of your work? [Yes] See Section D
287
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
208
289
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
292
(a) Did you state the full set of assumptions of all theoretical results? [Yes] We listed our assumptions in section
293
294
(b) Did you include complete proofs of all theoretical results? [Yes] Proofs in section are complete and rely on basic math.
296
3. If you ran experiments...
297
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] But we plan to release code soon, within supplementary materials
299
300
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section [2.2
301
302
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] We have reruned our experiments several times and so that results are pretty similar. We plan to continue experiments and made more run with other MLP architectures well as voxel-based models.
304
306
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5.2
307
308
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
309
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes, we cite Pytorch, pytorch3d and NeRF's s creators as well as many other relevant. See Section 5.2
310
(b) Did you mention the license of the assets? [No] We use opensource assets and citc/share links to them.
312
313
(c) Did you include any new assets either in the supplemental material or as a URL? [No] But we plan to release code soon, within supplementary materials
314
315
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Not applicable, data is artificially generated
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Not applicable, data is artificially generated
318
319
5. If you used crowdsourcing or conducted research with human subjects...
320
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] Not applicable
321
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] Not applicable
323
324
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] Not applicable
325
10


Reviews:
[{"review_id": "vXRy-6KAie", "reviewer": "Reviewer_jLZr", "summary": "This submission proposes a differentiable importance sampling to estimate the rendering equation used in NeRF based methods. The method is derived using the re-parameterization method. The gradient descent is derived based on the re-parameterization method. Compared with the previous sampling method used in NeRF based methods, the proposed method leads to much lower sampling variance under the same computational cost. \n", "questions": "For spline construction (L128-L130), what if the underlying density field is not piecewise linear, or have sharp spikes along a ray (eg. thin structure)? Will the spline approximation still be accurate enough to get opacity estimation?\n\nL131-133: how to get the sampled grid t_0<...<t_m in the first place? Are they predefined? If so, how to guarantee that they don’t miss any non-empty space across different views and across different scenes? ", "limitations": "Experiment is performed on only one scene. So it’s unknown whether this method is finetuned for that scene or more general.\n\nFrom L131-133, the sampling grid t_0<...<t_m cannot be guaranteed to cover the non-empty space.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n+The proposed importance sampling for Monte-Carlo (MC) estimate is technically sound by being based on the re-parameterization method. \n\n+The author also shows how to make the importance sampling along the ray differentiable by taking advantage of the re-parameterization. As a result, the new sampling method can be plugged into the neural rendering pipeline and make the implicit function trainable end-to-end.\n\n+The proposed differentiable sampling method has much lower sampling variance as compared to the existing sampling method used in the NeRF-based methods.\n\n+The density field is approximated by splines in closed form. This makes the integration along the ray for getting the light attenuation more efficient to compute, as compared to the previous methods where another integration is performed along the ray.\n\nWeakness\n\n-Although the proposed sampling method and its corresponding backpropagation process is sound and promising, the paper lacks the experiments to validate its effectiveness:\n\n1.The method is only evaluated on a single scene (Lego)\n\n2.The method is only compared to the base NeRF. There are works on speeding up NeRF without sacrificing performance during both training and rendering (eg. Plenoxel NeRF). How does this method compare those methods? And more importantly, can the method be plugged into those methods for further speed-up? Without the corresponding experiment, those questions can hardly be answered.\n\n3.Key experiment missing for trade-off between speed and reconstruction quality: How is reconstruction quality degrades with the decrease # of samples (ie. speed-up) by using the proposed method?\n\n4.Related to the missing speed-quality trade-off: From Tab.2, the estimation time per iteration  increases with the number of points in the splines and exceeds the baseline at around 16 points. So the question is how does the reconstruction quality compare with the baseline at around 16 sampling points. The paper does not include the quality comparison .\n\n\n5.Comparison to other SOTA method missing: How is the proposed method compared to the SOTA NeRF methods in terms of reconstruction quality and computational and memory consumption \n\n\n-Paper needs more refinement: There are multiple question marks in the paper due to error in reference.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gXndcpF5PC", "reviewer": "Reviewer_46SP", "summary": "The paper proposes a reparametrization of volume sampling for faster rendering of neural radiance fields. The key idea is that, when the density value (i.e., \\sigma) is known at a few sparse points, volume rendering can be reparametrized for importance sampling, which reduces the variance and thereby accelerates the rendering of the neural radiance field. The paper provides the demonstration of lower variance (compared to uniform sampling), ablation study, and novel-view synthesis with a different number of samples both qualitatively and quantitatively.", "questions": "Why is the proposed method efficient compared to the hierarchical volume sampling of the original NeRF paper? There is no clear demonstration of this. Both require the coarse sampling stage, which is the same additional cost compared to the naive sampling strategy. If the proposed method modifies the network structure to reduce this additional computational burden, the original NeRF can also benefit from this modification. The main baseline should be the original NeRF paper, and the comparison should be stated much more clearly and fairly in the experiment.", "limitations": "The checklist says the limitations are described in Section 1, but Section 1 does not include any description of the limitation.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While this paper tackles an interesting and important problem in neural rendering, which is an acceleration of NeRF rendering, it needs to be much improved in terms of presentation and evaluation.\n\n- The proposed method is not evaluated properly. The key contribution of this paper is importance sampling that can accelerate NeRF rendering, but the provided evaluation does not demonstrate this contribution. Figure 2 shows the variance compared to uniform sampling, but the baseline should be the importance sampling used in the original NeRF. Any importance sampling should provide better results than uniform sampling, and this experiment cannot show any meaningful demonstration as the proposed method has an additional cost of density sampling in the previous step for the reparametrization. Also, in Section 5.2, the proposed method modified the original NeRF to reduce the computational cost of density sampling (by changing the density network from eight layers to three layers), which will reduce the quality of the density field and make the resultant comparison NOT apple-to-apple. Moreover, the evaluations are conducted only on a single scene (i.e., lego) and the improvement over the baseline is marginal even in the current evaluation.\n- The presentation can be much improved. There are a number of typos and the notation is not consistent throughout the paper, which severely reduces the clarity of the paper considering the fact that the main contribution is in the reparametrization of equations. To list a few:\n    - Line 47: R^3 → \\mathbb{R}^3\n    - Line 48: directionn → direction\n    - Line 48: location x → \\mathbf{x}, direction d → \\mathbf{d}\n    - Line 53: ray r → ray \\mathbf{r}\n    - Line 58: fauithfull → faithful\n    - Line 75: c(o + Td, d) → c and d should be bold\n    - Line 82: \\sigma_r → both \\sigma and r should be bold\n    - Notation for Equations are not consistent; there are Eq n, Eq. n, n, Eq. eq. (n).\n    - Lines 110, 123: Eqs. eqs\n    - Line 130: r = o + td → r, o, d, should be bold\n    - Line 208: Eq. ??\n    - Line 224: Fig. ??\n    - Line 225: ren dering → rendering\n- The paper is based on Equation (2), which is the Riemann sum for the integral of volume rendering. However, it is not directly equivalent to the discrete volume rendering used in the original NeRF paper (Equation (3) in the original NeRF paper that is based on [Max 1995]) because of the small difference in the derivation (e.g., density is locally uniform between the samples). The difference in the equation should be clarified for a better comparison to the original NeRF.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DFRBlP3I_-k", "reviewer": "Reviewer_2V6p", "summary": "A Monte Carlo sampling strategy is proposed for rendering NeRF-based density field representation. The method works by firstly densely querying the density field to compute an inverse CDF, which is later used to reduce the number of subsequent samples of colour network.\n\nThe method is compared to the NeRF baseline and achieved faster speeds (~20%) but at the cost of slightly worse image quality (~0.7dB reduction in PSNR).", "questions": "I would suggest the authors to include missing references, and expand the experiment section.\n\nTo me the main drawback of proposed method is that dense sampling of density network still takes significant time. As such the speed gain is rather limited compared to vanilla NeRF. I wonder whether using a smaller density network can help in this regard.", "limitations": "I would suggest discussing the limitations in detail in section 6.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper does address an important problem, that is to improve the efficiency of NeRF rendering. The proposed method is well motivated and indeed effective in that it does significantly reduce the sampling rate of colour network (8 to 32 samples per ray compared to 128 samples per ray in NeRF). However, on the other hand, it still requires a dense sampling of density network in the first stage, and as such the overall rendering efficiency is only slightly, if at all, improved without sacrificing performance. \n\nMany related papers are missing from related work section and should be cited and compared to, these include faster NeRFs:\n- David B Lindell, Julien NP Martel, and Gordon Wetzstein. 2021. Autoint: Automatic integration for fast neural volume rendering\n- Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. 2021. Derf: Decomposed radiance fields.\n- Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. 2021. FastNeRF: High-Fidelity Neural Rendering at 200FPS\n- Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021. Plenoctrees for real-time rendering of neural radiance fields.\n- Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021. KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. \n\nThere are several papers on implicit surface volumetric sampling, that may be worth citing as well:\n- Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.\n- Michael Oechsle, Songyou Peng, Andreas Geiger. 2021. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction.\n\nExperiments are too limited. There is only one case being the lego scene. Table 1 only includes two training sampling rates, what are the results for 1,2,8,16 samples per ray?\n\nSome typos need to be corrected, e.g.:\nline 71: propose a propose a\nline 208, 224: broken ref\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject