Differentiable Rendering with Reparameterized Volume Sampling
Authors: Kirill Struminsky
Oleg Desheulin
Venue: Submitted to ICLR 2023 (2023)
Keywords: neural radiance fields
differentiable rendering
importance sampling
reparameterization trick

Abstract: We propose an alternative rendering algorithm for neural radiance fields based on importance sampling. In view synthesis, a neural radiance field approximates underlying density and radiance fields based on a sparse set of views of a scene. To generate a pixel of a novel view, it marches a ray through the pixel and computes a weighted sum of radiance emitted from a dense set of ray points. This rendering algorithm is fully differentiable and facilitates gradient-based optimization of the fields. However, in practice, only a tiny opaque portion of the ray contributes most of the radiance to the sum. Therefore, we can avoid computing radiance in the rest part. In this work, we use importance sampling to pick non-transparent points on the ray. Specifically, we generate samples according to the probability distribution induced by the density field. Our main contribution is the reparameterization of the sampling algorithm. It allows end-to-end learning with gradient descent as in the original rendering algorithm. With our approach, we can optimize a neural radiance field with just a few radiance field evaluations per ray. As a result, we alleviate the costs associated with the color component of the neural radiance field.

Under review as a conference paper at ICLR 2023
DIFFERENTIABLE RENDERING WITH REPARAMETERIZED VOLUME SAMPLING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose an alternative rendering algorithm for neural radiance fields based importance sampling. In view synthesis, neural radiance field approximates underlying density and radiance fields based on a sparse set of scene views. To generate pixel of a novel view, it marches ray through the pixel and computes a weighted sum of radiance emitted from a dense set of ray points. This rendering algorithm fully differentiable and facilitates gradient- baased optimization of the fields. However. in practice, only tiny opaque portion of the ray contributes most of the radiance to the sum. Therefore, we can avoid computing radiance the rest part. In this work, we use importance sampling to pick non-transparent points on the ray. Specifically, we generate samples according to the probability distribution induced by the density field. Our main contribution is the reparameterization of the sampling algorithm. It allows end-to-end learning with gradient descent as optimize neural
the original rendering algorithm. With approach, radiance field with just a few radiance field evaluations per ray. As a result, we alleviate the costs associated with the color component of the neural radiance field at the additional cost of the density sampling algorithm.
INTRODUCTION
We propose a volume rendering algorithm for learning 3D scenes and generating novel views. Recently, learning-hasee approaches led to significant progress in this area. As an early instance, (20) represent a scene a density field and radiance (color) field parameterized with an MLP. Using differentiable volume rendering algorithm (18) with the MLP-based fields to produce images, they minimize the discrepancy between the output images and set of reference images to learn scene representation. The algorithm we propose is drop-in replacement for the volume rendering algorithm used in NeRF (20) and follow-ups.
The underlying model in NeRF generates an image point in the following way. It casts ray from a camera through the point and defines the point color as a weighted sum along the ray. The sum aggregates the radiance of each ray point with weights induced by the density field. Each term involves a costly neural network query, and model has a trade-off between rendering quality and computational load. NeRF obtained better trade off with a two-stage sampling algorithm obtaining ray points with higher weights. The algorithm is reminiscent of importance sampling, yet it requires training an auxiliary model.
In this work we propose a rendering algorithm based on importance sampling. Our algorithm also acts in two stages. In the first stage, marches through the ray estimate density. In the second stage, it constructs a Monte-Carbe color approximation using the density to pick points along the ray. Figure [ illustrates the estimates for a varying number of samples. The resulting estimate is fully- differriiiieee and does not require any auxiliary models. Besides, we only need a few samples to construct a precise color approximation. Intuitively, we only need to compute the radiance of the point where ray hits a solid surface. As result, our algorithm is especially suitable for recent architectures (23 36 32) that use distinct models to parameterize radiance and density. Specifically, the first stage only queries the density field, whereas the second stage only queries the radiance field. Compared to the standard rendering algorithm, the second stage of our algorithm avoids redundant radiance queries and reduces the memory required for rendering at the cost of slight estimate variance.

---

Under review as a conference paper at ICLR 2023
/ Figure 1: Novel views of a ship generated with the proposed radiance estimates. For each ray estimate density and then compute radiance at few ray points generated using the ray density. As the above images indicate, render quality gradually improves with the number of ray points and saturates at approximately 16 ray points.
pl PSNR
16 pts ray
64
Original
Below, Section 2 give recap of neural radiance fields. Then we proceed the main contribution of our work in Section 3 namely the rendering algorithm fueled by novel sampling procedure. Finally, in our experiments in Section we evaluate the algorithm in terms of rendering quality, speed and memory requirements.
2 NEURAL RADIANCE FIELDS
Neural radiance fields represent 3D scenes with non-negative scalar density field a 3 - RR+ and a vector radiance field e I x 3 + R Scalar field a represents volume density at each spatial location æ, and c(æ,, d) returns the light emitted from spatial location æ in direction d represented as a normalized three dimensional vector.
For novel view synthesis. NeRF adapts volume rendering algorithm that computes pixel color C(r) (denoted with a capital letter) as expected radiance for a ray r = o + td passing through a pixel from origin E R$ in direction d R33 To ease the notation, will denote density and radiance restricted to ray r as
(t) == (o + td)
(1) c(o td, d).
(2)
With that in mind, the expected radiance along ray r is given as
C(r) C(r)= (t)cr, where (t( (t) exp (-f"ar(s)dd
(3)
Here, in and ty are near and far ray boundaries and (t)(t) is an unnormalized probability density function of random variable t a ray Intuitively, t is the location the ray where portion of light coming into the point o was emitted.
To approximate the nested integrals in Equation with piecewise approximation on a grid fn = to t <... < fm = tt and compute the formula analytically for the approximation. In particular, a piecewise constant approximation, which predominant in NeRF literature, yields formula
3 Max (18) proposed to replace fields o, and Cr
((r) Σ (1 (s)(-or(4488))) exp
-o c(l.), where 8; 4:+1 bg-
(4)
1=1
1=1
Importantly, Equation fully differentiable and can be used as part gradient- -ased learning pipeline.
Given the ground truth expected color Cge(r) along r. the optimization objective in NeRF
T(C(r), (|((r) Cyt (r)
(5)
2

---

Under review as a conference paper at ICLR 2023
captures the difference between Cr) and estimated color C(r). To reconstruct NeRF runs a gradient based optimizer to minimize the objective 3 averaged across multiple rays and multiple viewpoints.
While the above approximation works in practice, it involves multiple evaluations of and a along a dense grid. Besides that, a common situation is when ray intersects solid surface at some point t E f In this case, probability density (t((t) will concentrate its mass near t and will be close zero in other parts of the ray. As result, most of the terms in Equation 4 will make negligible contribution to the sum. In Section 4 we discuss various solutions to picking the grid points that are most likely contribute the sum. As an alternative, in the next section we propose to estimate the expected radiance with stochastic estimates that require only few radiance evaluations.
3 STOCHASTIC ESTIMATES FOR THE EXPECTED COLOR
Monte Carlo method gives a natural way to approximate the expected color. For example, given i..d. samples 11....,4 ts ~ (t) and the normalization constant Vy == f (t)dtt the following sum
Časc(r) = yf k Σ (t.)
(6)
an unbiased estimate of the expected radiance in Equation from high-density regions of pr by design, thus for a degenerate density p,. even a few samples would provide estimate with low variance. Each term in Equation contributes equally to the sum.
3 Moreover, samples 4 ...... , la come
Importantly, unlike the approximation in Equation 4 the Monte-Carlo estimate depends on scene density a implicitly through sampling algorithm and requires custom gradient estimate for the parameters of a. As an illustration, the full NeRF samples points on a ray from the distribution induced by an auxiliary "coarse" density model. These points are then used as grid knots in approximation 4 However, as their sampling algorithm is non con-differentiable and cannot be trained end-to-end, they introduce auxiliary "coarse" radiance field and train "coarse" components separately.
Below, we propose propose principled end-to-end differentiable algorithm to generate samples from pr (t). We then apply the algorithm estimate radiance as Equation and optimize the estimates to reconstruct the density and the radiance field of scene.
3.1 REPARAMETERIZED EXPECTED RADIANCE ESTIMATES
The solution we propose primarily inspired by the reparameterization trick (12 3T). We first
3 For Ft(t) exexp (-fi. (s)s)ss) and y = Ft)tt we write
change the variable in Equation
( o, (t)) = / C(r) = I'd
Uvs
cr c,(1,
The integral boundaries Yn Fy(tn) cumulative distribution function of the variable t with a single exception that, general, y := (ty) # 1. In volume rendering. (t) is called opacity function with ys being equal to overall pixel opaqueness.
and y Ft(t). Function F-(t) acts the
In the right-hand side of Equation 7 the integral boundaries depend on opacity F, and, as consequence, on ray density or. We further simplify the integral by changing the integration boundaries to [o, 1] and substituting yn 0:
J" CFFFF
J. yser (P, (Fr"(
(8)
Given the above derivation, we construct the reparameterized Monte Carlo (R/MC) estimate for the right-hand side integral in Equation with A i..dd U0, 1] samples U1,..., UR:
&
ER ( = y1 k t=1
Σ c
(9)
3

---

Under review as a conference paper at ICLR 2023
In the above estimate, random samples M1..... dg do not depend on volume density or color c,- Essentially, the reparameterized Monte-Catie Cante-Carlo estimate generates samples from pr (t) using inverse cumulative distribution function (yyu).
We further improve the estimate using stratified sampling. We replace uniform samples u1 Wk with uniform independent samples within regular grid bins U2 Uli sirh 1, k and derive the reparameterized stratified Monte Carlo (R/SMC) estimate
k C (Mc = Yf k 1m1
> (FF
(10)
It is easy to show that both 9 and 10 are unbiased estimates of Additionally, the gradient of estimates I is an unbiased estimate of the gradient of the expected color C(r). However, in practice we can only query or at certain ray points and cannot compute F, analytically. Thus, in the following section, we introduce approximations of Fy and its inverse.
3.2 OPACITY APPROXIMATIONS
Expected radiance estimate relies on opacity (t(() = exp (-Jr, (+(s)ds) and its inverse
Fr (y). We propose to approximate the opacity using a piecewise density field approximation. illustrates the approximations and ray samples obtained through opacity inversion.
Figure 2
Density Field
Opacity
0,75
Opacity F(t)
PWC ōg(t)
Approximation folt)
PWL d
0.50
Approsimation Pytt)
0.25
A
0.00
En
tr
Figure Illustration of opacity inversion. On the left, we approximate density field o, with piecewise constant (PWC) and a piecewise linear (PWL) approximation. On the right, we approximate opacity (t) and compute Fr (yru) for u ~ U(0, 1].
To construct the approximation, take grid t, to < t <. tm ty and construct piccewise constant and piecewise línear approximations. In the piecewise linear case, we compute o in the grid points and interpolate the values between the grid point. In the piecewise constant case, pick random point within each bin & S & S. i + and approximate density with (y,(f) inside the corresponding bin. Importantly, for non-negative field these approximations also non-negative.
Then we compute the integral fe, (s(ss used in Ft(t) for E [t4, f++1) analytically as sum of rectangular areas
To(t) Σ (t)tttttt (t) t2)
(11)
for the the piecewise constant approximation and of trapezoidal areas for the piecewise linear approximation
(t,t,) +or(t)-))) 11-1) (or(t,) +or(())))))))))))))))))))))))))) 12),
I(11)
(12)
2
1=1
where (t) = (t,
fontt + (t((1)) - is the interpolated density at t. Given these approxi- i-
mations, we are now able to approximate Fr and yf in Equation 9
We generate samples on a ray based on inverse opacity F- Fy-( by solving the equation
y = Ft(t) eexp (-/," (r(s)
(13)
4

---

Under review as a conference paper at ICLR 2023
fo where [0, 1] is random sample. We rewrite the equation as
- -log(1 - yy = or (s)dss
(14)
and note that integral approximations lo(t) and I1 (t) monotonic piecewise linear and piecewise quadratic functions. We obtain the solution of Equation [4 first finding a bin that contains a solution and then solving linear quadratic equation. Crucially, solution / can be seen differentiable function of the density field or and we can back-propagate the gradients w.r.t. through t.
In the supplementary materials, we provide explicit formulae for & for both approximations and discuss the solutions crucial for the numerical stability. Additionally, we provide an alternative inversion algorithm in the case when Jr, (s)ds can be computed without approximations. In our experiments we report the results only for piecewise linear approximation. our prelimenary experiments, the piecewise constant approximation was faster but delivered worse rendering quality,
RELATED WORK
There are multiple ways to represent the shape for a scene for novel view synthesis. Earlier learning- based approaches rely on such implicit representations as signed distance fields (27 34 35 and
occupancy fields [19 255 to represent non-transparent objects. We concentrate on implicit represen- tations based on density fields pioneered in NeRF (20). Each representation relies on a designated rendering algorithm. In particular, NeRF relies on an mmission-absoopttion optical model developed in (11) with numerical scheme specified in (18).
Monte-Carke estimates for integral approximations. this work, we revisit the algorithm in- troduced to approximate the expected color in (I8). Currently, the algorithm is default solution in multiple of works on neural radiance fields. The authors of (18) approximate density and radi- ance fields with piecewise constant functions along ray and compute 3 as approximation. Instead, we reparameterize Equation 3 and construct Monte-Carlo estimates for the integral. To compute the estimates practice we use piecewise approximations only for the density field. The cumulative density function (CDF) used in our estimates involves integrating density field along a ray. In (15), the authors construct field anti-derivatives to accelerate inference. While they use the anti-derivatives to compute 3 on grid with fewer knots, the anti-derivatives can apply in our framework construct Monte-Carlo approximations based on the inverse CDF without resorting to piecewise approximations.
In the past decade, integral reparameterizations became a common practice generative modeling (13 31 and approximate Bayesian inference 3 7 (22). Similar to Equation 3 objectives in these areas require optimizing expected values with respect to distribution parameters. We refer readers to (2 for systematic overview. Notably, in computer graphics, (17) apply reparameterization to estimate gradients of path traced images with respect to scene parameters.
Algorithms for picking ray points. Opposed to numerical scheme in Equation
algorithm
only requires to evaluate radiance at sparse set of points sampled from the density field. In (20), the authors similar hierarchical scheme to generate ray points using an auxiliary coarse density field. Crucially, unlike our reparameterized importance sampling, the importance sampling algorithm in their work does allow differentiating with respect the coarse model parameters. The ad-hoc solution introduced in (20) is to train the coarse model separately using the same rendering objective Subsequent works propose variations of the scheme: Mip-NeRF (2) merges coarse and fine models using a scale-aware neural field, and Mip-NeRF R 360 (2) distills the coarse density field from a fine field instead of training an auxiliary coarse radiance field. For non-transparent scenes Unisurf (26) treats the density field as occupancy field and gradually incorporates root-finding algorithms into volume sampling. Simultancously, number of works propose training an auxiliary model to return coarse samples for given ray. For instance, DoNeRF (24) uses designated depth oracle network supervised with ground truth depth maps, TermiNeRF (28) foregoes the depth supervision by distilling the sampling network from pre-trained NeRF model. Finally, the authors of (I) train a proposal network to generate points on a ray end-to-end starting with a pre- pre-trained NeRF. The aforementioned works speed up rendering, but the reliance on auxiliary networks hinders using faster
5

---

Under review as a conference paper at ICLR 2023
grid-based architectures and makes the overall scene representation less interpretable. In contrast to the above works, our algorithm learns sampling points on a ray from scratch an end-to-end fashion, works with an arbitrary density field, and does not requires any auxiliary models.
NeRF acceleration through architecture and sparsity. The above algorithms for picking points on ray generally aim to reduce the number of field evaluations during rendering. An alternative optimization approach is to reduce the time required to evaluate the field. In the past few years, a variety of architectures combining Fourier features (33) and grid-based features was proposed (8)
30). Besides grids, some works exploit space partitions based on Voronoi diagrams (29), 32 36
trees 10 37) and even hash tables (23). These architectures generally trade-off inference speed for parameter count. TensorRF (4) stores the grid tensors in a compressed format to achieve both high compression and fast performance.
top of that, skipping the density queries for the empty parts of scene additionally improves rendering time (14). For the novel view synthesis, the idea allows to speed up rendering during training and inference 2 (6). Notably, our rendering algorithm works with arbitrary density fields and, as result, is compatible with the improved field architectures and sparse fields.
EXPERIMENTS
5.1 IMPORTANCE SAMPLING FOR A SINGLE RAY
We begin with comparison of importance sampling color estimates in one-dimensional setting. In this experiment, we assume that we know density in advance and show how the estimate variance depends on number of radiance calls. Compared importance sampling, the standard approximation from Equation 4 has zero variance but does not allow controlling number of radiance calls.
Our experiment models light propagation on a single ray in three typical situations. The upper row of Figure defines scalar radiance field (orange) (t) and opacity functions (blue) F, for
"Foggy" density field. models a semi semi- transpanntt volume. Similar fields occur after model initialization during density field training;
"Glass and wall" density field. Models light passing through nearly transparent volumes such as glass. The light emitted at three points: the inner and outer surface of the transparent volume and opaque volume near the end of the ray:
"Wall" density field. Light emitted from single point on ray. Such fields are most common in applications.
Foggy Density Field
Glass and Wall Density Field
Wall Density Field
Opecity
Opecity
Opecity
Radionce
Estimate Variance
Estimate Variance
Estimate Variance
10
10'
U[t., tcl
with stmtiifed sampling
10
py(C) (ours)
10"
with stunitted sampling
10
10"
10"
Samples
Number Samples
Number Samples
Figure Color estimate variance compared for a varying number of samples. The upper plot illustrates underlying opacity function on ray; the lower graph depicts variance in logarithmic scale. Compared to a naive importance sampling estimate (dashed red), reparameterized sampling exhibits lower variance (dashed green). Stratified sampling improves variance in both setups (solid lines).

---

Under review as a conference paper at ICLR 2023
For the three fields estimated the expected radiance C(r) S. (+()) We considered two bascline methods (both in red in Figure 3: the first obtained with uniform distribution on a ray U [t,, ty]. and its stratified modification with a uniform & denote the number of samples, not the
importance sampling estimate of C
grid in number grid points m in piecewise density approximation):
& i, (note that here
dF, dt
(w((r) = Σ (k--t-1)e(())
with independent T; U(t ty].
(15)
t=n
We compared the baseline against estimate from Equation 9 and its stratified counterpart from Equation TO All estimates are unbiased. Therefore, we only compared the estimates variances for a varying number of samples m.
In all setups, our stratified estimate uniformly outperformed the baselines. For the most challenging "foggy" field, approximately k 32 samples required match the baseline performance for == 128. We matched the baseline with only a k = 4 samples for other fields. Importance sampling requires only few points for degenerate distributions. In further experiments, take 32 to obtain a precise color estimate even when model did not converge to a degenerate distribution.
5.2 SCENE RECONSTRUCTION WITH REPARAMETERIZED VOLUME SAMPLING
Next, apply algorithm 3D scene reconstruction based set of image projections. As a benchmark, we use the synthetic data from NeRF (20). The primary goal of the experiment is to demonstrate computational advantages of algorithm compared the basic volume rendering algorithm.
PSNR (1)
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
NeRF fill
33.00
25.01
30.13
36.18
32.54
29.62
32.91
28.65
31.01
NeRF woo.
31.32
24.55
29.25
35.24
31.42
29.22
31.74
27.73
30.06
NeRF (Curs)
31.35
22.42
28.42
34.36
30.70
28.72
31.18
26.89
29.26
DVGO
33.99
25.33
32.57
36.65
34.58
29.59
33.12
28.93
31.95
DVGO (Oun)
34.24
25.06
30.46
36.76
33.87
29.14
33.08
28.06
31.34
SSIM (1)
Chair
Drums
Fieus
Hotdog
Lego
Materials
Mic
Ship
Avg.
NeRF full
0.967
0.925
0.964
0.974
0.961
0.949
0.980
0.856
0.947
NeRF wio
0.951
0.914
0.956
0.969
0.951
0.944
0.973
0.844
0.938
NeRF (Churs
0.956
0.875
0.949
0.965
0.946
0.940
0.971
0.824
0.928
DVGO
0.976
0.928
0.977
0.980
0.976
0.950
0.983
0.876
0.956
DVGO (Ouns)
0.978
0.925
0.968
0.980
0.973
0.946
0.983
0.866
0,952
LPIPS (L)
Chair
Drums
Ficus
Hotdog
Lego
Materials
Mic
Ship
Avg.
NeRF fill
0.046
0.091
0.044
0.121
0.050
0.063
0.028
0.206
0.081
NeRF h.
0.065
0.177
0.056
0.130
0.072
0.080
0.039
0.249
0.109
NeRF (Clus)
0.065
0.178
0.066
0.078
0.083
0.077
0.040
0.225
0.102
DVGO
0.027
0.083
0.026
0.033
0.027
0.059
0.018
0.163
0.054
DVGO (Ouns)
0.027
0.084
0.040
0.034
0.029
0.062
0.017
0.172
0.058
Table 1: Rendering quality comparison with NeRF and DVGO 32 Metries are calculated over
test views for synthetic scenes (20) with & = 32 points in color estimates and m 256 knots along each ray our NeRF modificatio, for details please see Section 3.2 Our method slightly worse than NeRF without hierarchical sampling (coarse model) in terms of average PSNR and SSIM, although it is slightly better in terms of average I.PPS. As we have only modified the underlying integration scheme, expected the model performance to match the non hon-hierarl NeRF, For LPIPS calculation we used official implementation (38) and VGG features. Similarly, our modification of DVGO is slightly worse than the original DVGO.
7

---

Under review as a conference paper at ICLR 2023
5.2.1 NEURAL RADIANCE FIELDS
As our first model, took the original NeRF's architecture (20) and hyperparameters without any modifications, except for the output activation of field a. In particular, we used Softplus activation with 3 10 instead of ReLU to avoid zero gradients. To form training batch during training we took a random subset of 64 training images and sampled 64 rays per images. To construct piecewise linear density approximation, we slightly perturbed a uniform ray grid with 256 knots. For the proposed importance-sampling reparameterizztion we calculated Equation TQ with k = 32 samples to estimate color. the Table T provide the NeRF metrics with and without hierarchical sampling.
below, we report the obtained results. For reference, we also
With NeRF architecture we expected our algorithm to be comparable to NeRF without hierarchical sampling: these two models use grids of the same density and do not rely hierarchical sampling. However, despite our expectations, the quantitative results of the baseline were slightly better. The only difference between NeRF without hierarchical sampling and expected color approximation. We speculate that the variance of our stochastic estimate prevents model from finding fine optimum. For reference also provide the result for the full NeRF model, however the model is not directly comparable to ours. Even though the full NeRF model also samples points along ray two-stage manner, re-weights the output points using a second "fine" network, whereas samples in our model are weighted with uniform weights (see Equation 10
model is the underlying
Model
PSNR (1)
SSIM (?)
LPIPS (VGG) (D)
Speed (s)(L)
Mem (Gb)(L)
NeRF w/o h.
31.42
0.951
0.072
26.59
6.742
Ours, pt
25.67
0.870
0.1768
20.18
4.189
Ours, pts
28.29
0.909
0.1510
20.31
4.191
Ours, 4 pts
29.94
0.934
0.1187
20.49
4.191
Ours, pts
30.52
0.943
0.0930
20.82
4.191
Ours, 16 pts
30.68
0.946
0.0829
21.50
4.193
Ours, 32 pts
30.72
0.946
0.0801
22.89
4.197
Ours, 64 pts
30.72
0.947
0.0794
25.74
4.209
Table 2: Ablation study and comparison in terms of speed and quality with different number of points improtance-weighted color estimate. We compare inference on views of Lego scene (20). Speed represents the average rendering time of a single 800 800 frame on NVIDIA v v100 GPU. We measured speed and memory usage in pytorch3d's re-implementation of NeRF as our implementation also written in pytorch. Our algorithm slightly improves rendering time and memory footprint.
Then, we evaluated the proposed method with varying number of samples at the inference stage (k in Equation TOp. We took the Lego scene model from the previous experiment and varied the number of points in our reparametrized color estimation. The quantitative results of this experiments can be found in Table land and Figure contain qualitative results. From the rendering quality viewpoint, the three metrics gradually increased with the number of samples and saturated and approximately 16 points. Our algorithm produced sensible renders even for = 1, however noise artifacts only disappeared for k: 8.
pont
poitss
ponts
ports
18 pents
ports
Figure 4: NeRF rendering results with different number of samples in the proposed stratified estimate with re-sampling. From left to right and from top to down: 1, 2, 4, 8, 32 points estimates and ground truth for reference.
8

---

Under review as a conference paper at ICLR 2023
5.2.2 DIRECT VOXEL GRID OPTIMIZATION
We also tested our rendering algorithm on recent voxel-based radiance field model DVGO (32). The model takes only few minutes train thanks lightning fast architecture, progressive scaling, and custom CUDA kernels. We took the official implementation and only replaced the rendering algorithm based on Equation 4 To achieve the rendering performance competitive with their CUDA kernels, we optimized the performance using just-in-ting tntimme compilation module in Pytorch.
We evaluated the two training stages of DVGO with a varying number of radiance samples. On the first "coarse" stage, the model fits low-resolution 3D density grid and a view-independent 3D radiance grid. On the second "fine" stage, the model fits density grid with gradually improving resolution and view-dependent radiance field combining grid and an MLP. Crucially, the second stage relies on a coarse grid from the first stage to skip empty regions space and optimize the performance. Table 3 presents the comparison results synthetic Lego scene. Results for other synthetic scenes for 32 samples are in Table
Radiance calls +
Mem (Gb) +
Training time (8) 4
PSNR
Coarse
Fine
Coarse
Fine
Coarse
Fine
Ours
1
4.7
5.0
39
167
29.84
Ours
8
4.7
5.1
39
205
33.29
Ours
32
32
4.7
5.7
40
362
33.95
Ours
64
64
4.7
7.3
42
581
34.15
DVGO
130.7
13.4
4.7
5.5
54
238
34.60
Table 3: Comparison between the default rendering algorithm and the proposed reparameterized estimates with a varying number of samples for DVGO model. Our algorithm improves training speed the coarse stage by up to 30% and can improve training speed on the fine stage at cost of lower model quality.
On the coarse stage, we observed at least 20% 30% improvement in training time. The improvement can be attributed to fewer radiance samples. Using the auxiliary mask for empty space regions, DVGO significantly reduces the number of radiance calls the second stage. As a result, our rendering algorithm improved training time only when the number of samples is lower than the average number of radiance calls in DVGO (13.4 in this case). At the same time, the PSNR column indicates, the rendering quality deteriorated with fewer radiance samples. Notably, in our experiments, the model trained with 64 samples achieved PSNR 34.33 even with 16 during evaluation stage. We concluded that lower PSNRs are caused by the estimate variance during training.
6 CONCLUSION
In this work, we proposed an alternative rendering algorithm for novel view synthesis models based on radiance fields. The core of our contribution is end-to-end differentiable ray point sampling algorithm. For two pre-existing architectures, we show that the algorithm can achieve competitive rendering quality while reducing training and rendering time, and required GPU memory. Besides that, believe that such an algorithm opens up new possibilities efficient rendering and architecture design that are yet be explored.
REFERENCES
[1] Relja Arandjelović and Andrew Zisserman. Nerf in detail: Learning to sample for view synthesis. arXiv preprint 106.052666 2021.
[2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Brrunn-aaalll,. and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti- anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.

---

Under review as a conference paper at ICLR 2023
[3] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning. pages 1613-1622. PMLR,
2015.
[4] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. arXiv preprini arXiv:2203.09517, 2022.
[5] Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterizatioo gradients. Advances in Neural Information Processing Systems, 31, 2018.
[6] Sara Fridovich-Keie Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501-5510, 2022.
[7] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pages 1050-1059. PMLR, 2016.
[8] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin, Fast- nerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 14346-14355, 2021.
[9] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875-5884, 2021.
[10] Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. Efficientnerf efficient neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12902-12911, 2022.
[11] James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPII computer graphics, 18-3)1155-774 1984.
[12] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.694: 12.6900, 2014.
[13] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 132 14, 2013.
[14] Mare Levoy. Efficient ray tracing of volume data. ACM Transactions on Graphics (TOG), 93)::245-261, 1990.
[15] David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14565,, 2021.
[16] Lingjie Liu, Jiatao Gu, Kyaw 7aw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances Neural Information Processing Systems, 33:15651-1566; 2020.
[17] Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. Reparameterizing discontinuous integrands for differentiable rendering. ACM Transactions Graphics (TOG), 18(6):1-14, 2019.
[18] Nelson Max. Optical models for direct volume rendering, IEEE Transactions on Visualization and Computer Graphics, 1(2):99-108, 1995.
[19] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4460-4470, 2019.
[20] Ben Mildenhall, Pratul P Srinivasan. Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference computer vision, pages 405-421 Springer, 2020.
10

---

Under review as a conference paper at ICLR 2023
[21] Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. J. Mach. Learn. Res., 21(132):1-66 2020.
[22] Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pages 2498-25225 PMLR,
2017.
[23] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv preprint arXiv:2201.05989 2022.
[24] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks. In Computer Graphics Forum, volume 40, pages 45 59, Wiley Online Library, 2021.
[25] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volu- metric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504-3515.
2020.
[26] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589-5599, 2021.
[27] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[28] Martin Piala and Ronald Clark. Terminerf: Ray termination prediction for efficient neural rendering. In 2021 International Conference 3D Vision (3DV), pages 1106-1114. IEEE, 2021.
[29] Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14153-14161 2021.
[30] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 14335-14345 2021.
[31] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1288 PMLR, 2014.
[32] Cheng Sun, Min Sun, and Jwwann-Tzone Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. arXiv preprint arXiv:21 2021.
[33] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Konn Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[34] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv: 106. 10689, 2021.
[35] Lior Yariv, Jiatao Gu. Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4811 2021.
[36] Alex Yu, Sara Fridovich-Keil. Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint 12.0::2/12.05131,
2021. 2021.
11

---

Under review as a conference paper at ICLR 2023
[37] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5752-5761. 2021.
[38] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features perceptual metric. In CVPR, 2018.
A APPENDIX
Below we discuss caveats and implementation details of our sampling algorithm.
A.1 INVERSE FUNCTIONS FOR DENSITY INTEGRALS
In this section, we derive explicit formulae for the density integral inverse used in inverse opacity.
A.1.1 PIRCEWISE CONSTANT APPROXIMATION INVERSE
We start with a formula for the integral
lo(t) Σ
- -t-) 4))
(16)
and solve for equation
y Iott.
(17)
The equation above a linear equation with solution
y - (1)()(1) 11-1)
(18)
(f,
In our implementation we add small to the denominator to improve stability when (f+)) Rz 0.
A.1.2 PIECEWISE LINEAR APPROXIMATION INVERSE
The piecewise linear density approximation yield piecewise quadratic function
[111) (ty) + or((t=1)) (1, -1-1)) (o, + (t) 44)
(19)
2
Again, we solve
y = (t)
(20)
for t. We change the variable to At = t- t, and note that terms and e in quadratic equation
0 att bat +c
(21)
will be
(t++1)) -or(t)))or(tt)
(22)
(1i) + (11-)-1) (t, tg-1)
1 - (tit1 tx))
(23)
and with a few algebraic manipulations we find the linear term
b (44) (44+1 4).
(24)
Since our integral monotonically increases, we can deduce that the root At must be
At b+ /b² 4ac
(25)
2a
However, this root is computationally unstable when a so 0. The standard trick to rewrite the root
At
2c
(26)
For computational stability, we add small e to the square root and denominators use replace the root with A/. / when b 0 and At & when O. See the supplementary notebook for details.
12

---

Under review as a conference paper at ICLR 2023
A.2 NUMERICAL STABILITY IN INVERSE OPACITY
Inverse opacity input is a combination of a uniform sample w and ray opacity Vf = exp (-J (s)s)s)):
1
- log(1 - yyu).
(27)
The expression above is combination of logarithm and exponent. We rewrite it to replace with more reliable logsumexp operator:
y - log [ex((log(1 u)) + exp(log S(' or(()ds))
(28)
In practice, for opaque rays 1 (r(s) = 0 implementation of logsumexp becomes computation- ally unstable. In this case, replace y with they are almost identical.
A.3 IMPLICIT INVERSE OPACITY GRADIENTS
To compute the estimates in Equation 9
we need to compute the inverse opacity (y) along with its gradient. In the main paper, we invert opacity explicitly with a differentiable algorithm. Alternativel., could invert (t)
exp (-fe. (r(s)ds) with binary search.
Opacity (t) is monotonic function and for y E (yn, (F,( Fr(y)) the inverse lies in (fn, 1y). To compute (y). we start with boundaries f = t, and f, = t and gradually decrease the gap between the boundaries based on the comparison of F, ( with Importantly, such procedure is easy parallelize across multiple inputs and multiple rays.
However, we cannot back-propagate through the binary search iterations and need a workaround to compute the gradient (t of t(0) F,- (y, 8). To do this, follow (5) and compute differentials of the right and the left hand side of equation y0) = (t,
dy 10 d0
OF, Öt
+ OF, d0.
(29)
ōt 00 00
By the definition of (t, 0) we have
aF. (1 (t,1, O).
(30)
OF, (1 - (t, a (f(o((s.))s))
(31)
We solve Equation 29 for Øt 76 and substitute the partial derivatives using Equations 30 and 31 to obtain
final expression for the gradient
ōt
By D6
8 -1 (1-F
DB Jr or(s, 0)ds
(32)
00
(1 Ft,t, (t,,,
In our implementation, we use automatic differentiation to compute dy/00 and Jr, o())ds to combine the results in Equation 32
13


Reviews:
[{"review_id": "sa8JZbmj0Mx", "reviewer": "Reviewer_D4iJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a rendering method for a neural radiance field. When marching on each light ray, the authors chose the sampling points using the idea of importance sampling; the samples are drawn in accordance with the contributions to the final rendered color. The method was designed so as to reduce the number of the radiance queries, which (they expected) would bring the computational advantages over the previous methods. They incorporated this sampling strategy into two baseline methods, NeRF and DIGO, and compared the performance with the original methods. ", "summary_of_the_review": "As mentioned in \"Strength and Weaknesses\", I appreciate the logical explanation for the importance sampling for a neural radiance field. However, I think the authors' contribution is insufficient in the current form; they have not established the effectiveness of their proposal: computational advantages over the previous methods. Therefore, I recommend rejection for this submission.", "strength_and_weaknesses": "Strengths\n\n- The idea of importance sampling is explained well in a logical manner. As far as I read, the theory and the related derivations are correct. Moreover, this idea was successfully implemented as a trainable module that can be incorporated into the existing neural rendering methods.\n\nWeaknesses\n\n- The authors failed to establish the effectiveness (computational advantages over the previous methods) of their proposal.  As shown in Tables 2 and 3, the authors' method achieved only a small amount (up to 25 percent) of reduction in computation time  compared to the baselines (NeRF and DVGO) at the cost of quality decline. The authors should notice the fact that the number of sampling points can be changed also for the original methods (NeRF and DVGO) so as to find a desirable trade-off between the computational cost and rendering quality. However, the authors used a fixed number of samples for the baselines,  and thus, I cannot see the advantage of the authors' proposal over the baseline methods. Moreover, the authors did not compare their proposed method against other speed-up methods for neural radiance fields. For example, DONeRF achieved a significant improvement in quality-speed trade-off: 15-78 times fewer computation and comparable rendering quality than/to the baseline NeRF. KiloNeRF achieved more than 2000 times speed up while maintaining the rendering quality. Compared to these works, the contribution of the present submission seems quite limited. \n\n- Insufficient consideration/evaluation for the method's design. The proposed method was designed to reduce the number of \"radiance queries\", but it still needs many \"density queries\" to obtain accurate opacity along the light ray. Moreover, the re-parameterization for the importance sampling introduced additional computational cost. Therefore, it seems that the authors' method cannot save the computation so much as the other speed-up methods. The authors should evaluate computational cost (e.g. in FLOP per pixel) involved in each of these steps (density queries, re-parameterization, and radiance queries) to see the problem clearer. Moreover, it seems that the authors sticked to the idea of using a single radiance field, rejecting the use of auxiliary/multiple networks. However, the use of auxiliary/multiple networks has practical advantages in terms of the speed-quality trade-off.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "f0vl_YPMTk", "reviewer": "Reviewer_yzB7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at avoiding computing radiance in less contributive parts by reparameterizing the sampling algorithm. This can help decrease the number of evaluations to MLPs. This sounds reasonable. But the experiments do not meet the authors’ expectations. First, decreasing the MLP evaluations should indeed improve efficiency a lot. But the results show only minor improvements on NeRF baseline. This reminds us of if the proposed approach itself slows down the efficiency. Second, DVGO is an explicit volume-based approach which is extremely efficient. I have to say the author have chosen a wrong baseline. Because the efficiency of explicit volume-based approach is not severely slow down by the great number of sampling points. Third, a big number of sampling points is vital to rendering quality. So, the rendering quality of this paper degrades. Fourth, the experiments cannot support the claim. Please refer to the weakness for more details.", "summary_of_the_review": "The unimpressive experiment results mostly lead to a negative opinion to this paper. \nThe claimed contribution on efficiency cannot be supported by the experiment. Some constraints (time used on reaching the same psnr, or evaluating the produced psnr using the same training time) have to be set for a fair comparison. And the rendering quality is also not good.\n", "strength_and_weaknesses": "Strength:\n1) The mathematical analysis looks great.\n\nWeakness:\n1)\tThe proposed method can lead to improvements on efficiency but degrades on quality. But the experiments presented in the paper are hard to support the claimed contribution. It would be fair if make experiments of evaluating two methods on the time used of reaching the same PSNR. Or vice versa, evaluating the produced PNSR with the same training time. \n2)\tThe improved efficiency is very slight compared with baseline approach. Normally, a smaller number of evaluations to MLPs should lead to great efficiency improvements. But in this paper, the efficiency improvement is little. The author should analyze the reason deeply, is this caused by the extra time used on solving Eq. 14 and 10?\n3)\tThe reasons of piece wise constant has worse quality than piece wise linear are not analyzed.\n4)\tThe reasons of different performance on three experiment setups (Fig. 3) are not analyzed.\n5)\t“Notably, in our experiments, the model trained with 64 samples achieved PSNR 34.33 even with 16 during evaluation stage.” I cannot find the 34.33 in paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "daUTibIM2k", "reviewer": "Reviewer_bm7A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an importance sampling scheme for the radiance evaluation of NeRF-style volumetric rendering. Importance sampling replaces the coarse network in NeRF with direct sampling of the high resolution density field. This promises a simpler algorithm without the need for a proxy loss on the coarse network, and improved quality/performance, as radiance samples are concentrated in regions of high opacity, while transparent regions are skipped.\n\nThe paper describes the relevant background on volumetric rendering and outlines how the piecewise constant ray integral can be recast, first as a monte-carlo estimate in ray space, and then as as an integral over the normalized opacity along the ray.\n\nThe paper describes in detail how the approximate opacity is computed with a piecewise approximation, and how samples from the distribution are drawn via inverse transform sampling. A complication of the paper is that in order to construct an approximation of the cumulative density function, the technique does require a relatively fine (256) sampling of the density field, so any performance improvement of importance sampling would be gained solely by saving radiance evaluations.\n\nThere is a insightful synthetic experiment that compares the proposed integration scheme to uniform sampling in ray space (with ablations for stratified sampling).\n\nThe paper then presents comparisons to NeRF and a voxel-based technique (DVGO). It appears from these experiments that both baselines (full NeRF with coarse model, and an unguided NeRF without hierarchical sampling) perform better than the proposed technique.\n\n", "summary_of_the_review": "I'm leaning against accepting this paper. The empirical evaluation is unfortunately not very impressive. The technical content is solid, but only addresses a narrow aspect of the overall algorithm, and does not include enough new ideas to put the paper over the bar in light of the mixed experimental results.\n", "strength_and_weaknesses": "The paper is well written and well motivated. The description of the technical contribution is clear, and Figures 1-3 are good visualizations of the technique. \n\nThe outlined idea holds some promise, given that NeRFs coarse-then-fine scheme, despite its effectiveness, is not thoroughly analyzed in the original paper.\n\nA drawback of the proposed technique is the need to finely sample the density field to create the approximation for inverse transform sampling. This seems to preclude major performance benefits that would otherwise be associated with importance sampling.\n\nThe experiments reported in Section 5.2 suggest that the proposed importance sampling scheme just doesn't work very well in its current implementation. The reconstruction quality is consistently lower than NeRF's, even if the coarse network of NeRF is disabled. (Could the authors clarify how many radiance evaluations the NeRF models in Table 1 and 2 perform?)\n\nThe comparison to DVGO also fails to show significant performance or quality improvements.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "CUm5beNa-8F", "reviewer": "Reviewer_XnSd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Volume rendering is a critically important algorithm in NeRF (neural radiance field) methods, which connects 3D real points with 2D image pixels. This paper aims at exploring an alternative rendering algorithm mainly focusing on the importance sampling for integration. A reparameterized Monte Carlo estimation method is proposed to enable end-to-end optimization, where fewer samples are needed for integration to save cost.", "summary_of_the_review": "The main concerns lie in experimental evaluation and comparisons. For its current state, no evident advantages over previous works are shown.", "strength_and_weaknesses": "**Strength**  \nThe topic of designing better sampling methods for volume rendering is interesting and valuable. This paper proposes a reparameterization method to enable end-to-end differentiable optimization, which is reasonable and has been evaluated in NeRF-provided benchmarks.\n\n**Weaknesses**    \n**The biggest concerns come from the experimental evaluation and comparisons.**  \n* The proposed method shows a worse rendering quality while the speedup over the original NeRF and DVGO is not so significant. \n* Related works, e.g. NeRF-ID, DO-NeRF, AutoInt, TermiNeRF, should also be compared in experiments with regard to both rendering quality and speed. \n* Some highly related works are missing, including but not limited to [1, 2]. Please have a careful check.\n* Experimental evaluation is not so comprehensive. Real scenes, e.g. LLFF ones, need to be included.\n\n**Minor**:   \n* The citation style does not follow the ICLR-required author-year ones.\n\n[1] Fang J, Xie L, Wang X, et al. Neusample: Neural sample field for efficient view synthesis[J]. arXiv preprint arXiv:2111.15552, 2021.   \n[2] Kurz A, Neff T, Lv Z, et al. AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields[J]. arXiv preprint arXiv:2207.10312, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject