NeRF Revisited: Fixing Quadrature Instability in Volume Rendering
Authors: Mikaela Angelina Uy
Kiyohiro Nakayama
Guandao Yang
Rahul Krishna Thomas
Leonidas Guibas
Ke Li
Venue: NeurIPS 2023 poster (2023)
Keywords: neural radiance fields
volumetric rendering
nerfs
numerical quadrature
importance sampling
TL;DR: We derive the volume rendering equation under a piecewise linear opacity and piecewise constant color assumption for NeRFs.

Abstract: Neural radiance fields (NeRF) rely on volume rendering to synthesize novel views. Volume rendering requires evaluating an integral along each ray, which is numerically approximated with a finite sum that corresponds to the exact integral along the ray under piecewise constant volume density. As a consequence, the rendered result is unstable w.r.t. the choice of samples along the ray, a phenomenon that we dub quadrature instability. We propose a mathematically principled solution by reformulating the sample-based rendering equation so that it corresponds to the exact integral under piecewise linear volume density. This simultaneously resolves multiple issues: conflicts between samples along different rays, imprecise hierarchical sampling, and non-differentiability of quantiles of ray termination distances w.r.t. model parameters. We demonstrate several benefits over the classical sample-based rendering equation, such as sharper textures, better geometric reconstruction, and stronger depth supervision. Our proposed formulation can be also be used as a drop-in replacement to the volume rendering equation of existing NeRF-based methods. Our project page can be found at pl-nerf.github.io.

NeRF Revisited: Fixing Quadrature Instability in Volume Rendering
Mikaela Angelina Uy!
George Kiyohiro Nakayama'
Guandao Yang!.2 Ke Li
Rahul Krishna Thomas' Leonidas Guibas
Sttanford University *Cornell University {mikacuy, w4756677, guandao, rt03mas, guibas)@stanford edu, keli0sfu.ca
SSmon Fraser University 'Google
Abstract
Neural radiance fields (NeRF) rely on volume rendering to synthesize novel views. Volume rendering requires evaluating an integral along each ray. which is numeri- cally approximated with finite sum that corresponds to the exact integral along the ray under piecewise constant volume density. As a consequence, the rendered result is unstable w.r.t. the choice of samples along the ray, phenomenon that we dub quadrature instability. We propose mathematically principled solution by reformulating the sample-based rendering equation so that it corresponds to the exact integral under piecewise linear volume density. This simultaneously resolves multiple issues: conflicts between samples along difffrrnnt rays, imprecise hierarchical sampling, and non-differentiability of quantiles of ray termination distances w.r.t. model parameters. We demonstrate several benefits over the clas- sical sample-based rendering equation, such as sharper textures, better geometric reconstruction, and stronger depth supervision. Our proposed formulation be also be used as a drop-in replacement to the volume rendering equation for existing methods like NeRFs. Our project page can be found at wenerf..ittub.i..
Introduction
The advent of neural radiance fields (NeRF) [18] has sparked flurry of work neural rendering and has opened the way to many exciting applications [5, 11, 8, 20]. One of the key underpinnings of NeRF is volume rendering [14] it is especially well-suited end-to-end differentiable rendering [9], since the rendered image is a smooth function of the model parameters. This has made it possible to learn the 3D geometry and appearance solely from a 2D photometric loss on rendered images.
In volume rendering, the rendered colour ý for every pixel is an expectation of the colours along the ray cast through the pixel w.r.t. the distribution over ray termination distance [6].
Es-n(s) [c(s)] - / p(s)c(s) ds
(1)
where p(s) denotes the probability density function (PDF) of the distribution over ray termination distance and c(s) denotes the the colour a function of different points along the ray.
In general, p(s) and c(s) can of arbitrary forms, so evaluating this integral analytically is not possible. Therefore, in practice, Eapp(s) [c(s)] is approximated with quadrature. The quadrature formula that is most commonly used the NeRF literature takes the following form:
E Es-p(s)((()]]
Σ T, (1---115600 1=U
where T, exp (- Ei (s - sk)) and T is the opacity evaluated at sample sy along the ray.
This expression is derived from the exact integral under a piecewise constant assumption to the opacity and colour along the given ray [14].
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

However, this seemingly simple. innocuous assumption can result in the rendered image being sensitive to the choice of samples along the ray at which the opacity (s)) and colour c(s) are evaluated. While this does not necessarily cause a practical issue in classical rendering pipelines [14, 15, 10], has surprising consequences when used in neural rendering. Specifically, because the opacity at all points within the interval between two samples is assumed to be the same, there is band near the surface of the geometry where opacity at all points within the band is as high as points on the surface itself. Because different rays cast from different cameras can pass through this band at different angles and offsets, both the number and the positions of samples within this band can be very different across different rays (see Fig 1 and Fig for two example scenarios). Hence, simultaneously supervising these rays to produce the same colour values as in the real image captures can give rise to conflicting supervisory signals, which can result in artifacts like fuzzy surfaces and blurry texture.
Moreover, because the piecewise constant opacity assumption gives rise to a closed form expression that is equivalent to an expectation w.r.t. discrete random variable, it is common practice in the NeRF literature to draw samples along the ray from the discrete distribution [17]. This is commonly used to draw importance samples, and also to supervise the samples along the ray [28]. for example in losses that penalize deviation of the samples from the true depth [6]. Sampling from the discrete distribution requires the definition of continuous surrogate function to the cumulative distribution function (CDF) of the discrete random variable, which unfortunately yields imprecise samples. As result, samples that are drawn may not be close to the surface even if the underlying probability density induced by the NeRF is concentrated at the surface. Additionally, individual supervision cannot be provided to each sample drawn from the surrogate, because the gradient of the loss w.r.t. each sample would be almost zero everywhere.
All these issues, i.e. conflicting supervision, imprecise samples and lack of supervision on the CDF from the samples, stem from the assumption that opacity is piecewise constant causing the sensitivity to the choice of samples both during rendering and sampling. We dub this problem quadrature instability. In this paper, we revisit the quadrature used to approximate volume rendering in NeRF and devise different quadrature formula [14] based on a different approximation to the opacity. We expression can be derived under any piecewise polynomial
first show that interestingly a closed-fore approximation for opacity. When the polynomial degree is O, it reduces to the piecewise constant opacity as in existing literature, and when the degree is or more, we show that it would lead to poor numerical conditioning
Therefore, we further explore a degree of (i.e., piecewise linear) and show that it both resolves quadrature instability and has good numerical conditioning. We derive the rendering equation under piecewise linear opacity explicitly and show that it has a simple and intuitive form. This results in new quadrature method for volume rendering, which can serve as drop-in replacement for existing methods like NeRFs. We demonstrate that this reduces artifacts, improves rendering quality and results in better geometric reconstruction. We also devise new way to sample directly from the distribution of samples along each ray induced by NeRF without going through the surrogate, which opens the way more refined importance sampling approach and more effective method to supervise samples using depth.
2 Related Work
NeRFs. Neural Radiance Field (NeRF) is a powerful representation for novel-view synthesis [18]
that represents a scene using the weights of an MLP that is rendered by volumetric rendering [14].
A key finding to the success of NeRF was the use of positional encoding [26, 29] to effectively
increase the capacity of the MLPs that models the opacity and emitted color as function of 3D)
coordinate and viewing direction. Many works extend NeRF such as handling larger or unbounded scenes [40, 3, 37, 25], unconstrained photo collections [13]. dynamic and deformable scenes [11, 19] and sparser input views [6. 39, 31, 28J. There are a number of papers that aim to improve the rendering quality of NeRF Some do so by utilizing different kinds of supervision such as NeRF in the Dark [16], while others tackle this by improving the model [2. 36, 4]. MipNeRF [2] changes the model input by introducing integrated positional encoding (IPE) to reduce the aliasing effect along the xy coordinates. DiVeR [36] predicts the rendered colour within a line interval directly
from a trilinearly interpolated feature in a voxel-based representation ZipNeRF [4] modifies the proposal network to grid enabling it to be used together with IPE. In contrast, our work focuses on changing the objective function by modifying the rendering equation from piecewise constant opacity to piecewise linear, while keeping the model and supervision fixed. Additionally, ZipNeRF [4] also
2

---

brings up a model specific issue on z-aliasing, where their model struggles under this setting. Similar to z-aliasing observed by ZipNeRF, we consider the setting of having conflicting supervision when presented with training views at different distances from the scene. While they may appear similar on the surface, the phenomena we study different that it is general and independent of the model, on having conflicting ray supervision from camera views, e.g. different camera-to-scens distances and the grazing angle setup.
Importance Sampling on NeRFs. Densely sampling and evaluating NeRF along multiple points in each camera ray is inefficient. Inspired by an early work on volume rendering [10], prior works typically use a coarse-to-fine hierarchical sampling strategy where the final samples are obtained by importance sampling of coarse proposal distribution [18, 2, 3, 8J. These importance samples are drawn using inverse transform sampling where a sample is obtained by taking the inverse of the cumulative density function (CDF) of the proposal ray distribution. However, prior NeRF works that assume piecewise constant opacity result in a non-invertible CDF, and instead introduce a surrogate invertible function derived from the CDF in order to perform importance sampling. In contrast, our work that utilizes a piecewise linear opacity assumption results in invertible CDF and closed-form solution to obtain samples with inverse transform sampling. Other works also attempt to alter sampling using neural networks or occupancy caching [31, 23, 24, 12]. These techniques are orthogonal to our work as they propose changes to the model as opposed to our work where importance sampling is derived from given model.
Volume rendering. Volume rendering is an important technique in various computer graphics and vision applications explored in different classical works [10, 33, 7, 14]. These works include studying ray sampling efficiency [10] and data structures, e.g. octree [22] and volume hierarchy [21] for toarse-to-fine hierarchical sampling. The crux behind volume rendering is the integration over the weighted average of the color along the ray, where the weights is a function of the volume density (opacity). Max and Chen [14, 15] derive the volume rendering equation under the assumption of piecewise constant opacity and color, which NeRF [17] and its succeeding works use to learn their neural scene representation. However, the piecewise constant assumption results in rendering outputs that are sensitive to the choice of samples as well the non-invertible CDF introducing draw backs to NeRF training. Following up on [14]. works also [34] derive the volume rendering equation under the assumption that both opacity and color are piecewise linear that yield unwieldy expressions that lead to numerical issues and/or are expensive compute. Some earlier works on rendering unstructured polygonal meshes that attempt to use this model [35], but it is in general not commonly used in practice due to the mentioned issues and hence has yet to be adopted into learning neural scene representations. In this work, we address both sets of issues that arise from the piecewise constant opacity and color and piecewise linear opacity and color by reformulating the volume rendering equation to assume piecewise linear opacity and piecewise constant color. Our derivation results in simple and closed-fore formulation to volume rendering making it suitable for NeRFs
3 Background
3.1 Volume Rendering Review
Definitions. In classical literature, the process of volume rendering [15] mapping a 3D field of optical properties to 2D image and the visual appearance is computed through the exact integration of these optical properties along the viewing rays. In this optical model, each point in space is an infinitesimal particle with certain opacity that emits varying amounts of light, represented as scalar color c, in all viewing directions. The opacity t is the differential probability of a viewing ray hitting a particle- that is for a viewing ray r(s) is the ray direction, the probability of ray r hitting a particle along an infinitesimal interval ds is ((r(s) yds. Moreover, the transmittance (s(s) is defined as the probability that the viewing ray travels a distance s from the view origin without terminating, i.c. without hitting any particles.
+ sd, where o is the view origin and d
Continuous probability distribution along the ray r. As illustrated by Max and Chen [15], the probability of hitting& a particle s + ds only depends on the probability of hitting a particle at s and not any particles before it the probability of ray terminating at distance s is given by T ((r(s))T,(3) where (s(s) = exp(- - fo (u))du). Hence the continuous probability density function (PDF) of ray rss), which describes the likelihood of a ray terminating and emitting at 8, is given by
p(s) ((r(s)(
(3)
3

---

Rendered Test View
Probability Density
Plecewise Conslant Cpscity
surface
Grazing Angle Views
Oepio of
Perpendicular Views
Plecewise Linear Opadiy
Cumulative Density
surface
Grazing Angle Views
Perpendicular
Views
242
51
22
Figure I: Ray Conflicts: Grazing Angle. (Left) Illustration of conflicting ray supervision the grazing under the piecewise constant opacity. For the constant setting, to render perpendicular rays (yellow) correctly, the model has to store the associated optical properties at a region in front of the surface as a sample takes the values of the left bin boundary. In the presence of a ray near the grazing angle, it will be crossing this region of high opacity (the gradient in front of the surface), associating with conflicting opacity/color signals. (Middle) This results in fuzzier surfaces as shown along the side of the microphone there is conflict in ray supervision between the perpendicular and grazing angle rays. Our piecewise linear opacity assumption alleviates this issue and results clearer rendered view. (Right) As shown, the resulting PDF is peakier and the CDF is sharper for our linear setting, where the plotted distributions correspond to the ray from the marked pixel in red.
where 8 E [0, oo] and r(s) is a point on the ray r. For notational simplicity we omit r and write it as p(s) (s))
Volume Rendering as a Continuous Integral. The observed color of the ray is then the expected value of the colors e(s) of all particles & along the ray weighted by the probability of hitting them. Mathematically, this results in the following continuous integral
/ p(s)c()) L( (s))T(s)c(s)ds
Bemp(e)(((s)
Quadrature under Piecewise Constant Opacity т. Since this integral cannot in general be evaluated analytically, it is approximated with quadrature. Let 81, 82, .... SN be N (ordered) samples on the ray that define the intervals, where I, = [s,. $5+1] is the 1th interval, and In = [0. s1]. IN = [sN, oo]. The volume density for particles along the interval In is then approximated under the assumption that opacity is constant along each interval, making it piecewise constant along the ray [15]. That is, for all j we have:
Ys E [sj. S3411. T()))
for brevity denote T(5;) Tj. i.e. 7, is the opacity for sample sy. Under this piecewise constant opacity assumption, the volume rendering equation Eq. 4 then becomes as follows:
E Exepts) [c(s)] = Σ P&c=- =
N
(1,
Σ
Σ
N
2111
((u)T(w)du C₁ =
N Σ Tí -1(69411111) oj. 1=0
1=
1=0
(6)
PPractically the integral taken with sn and far s bounds.

---

where T exp Eh-oo - -
Here. color çj is also approximated be constant along each interval 1,, and P3 is the probability of each interval. Now, let us define the discrete random variable s = fs), where
-5x(511--5) sk))
80
I S 80
[(x)
8j S x < 88+1 for all j E N -
SN
SN
which gives corresponding probability mass function P(s). Observe that the analytical expression of the integral Eq. 6 turns out to be the same as taking the expectation w.r.t the discrete random variable 3, i.e. [c(ss) = P( [c())] This piecewise constant opacity assumption in the volume rendering equation is used in most, if not all, existing NeRF works. We recommend the reader to read [15] for more detailed derivations and supplementary for more thorough walkthrough.
3.2 Neural Radiance Fields.
Following Max and Chen [14], Mildenhall et.al. [18] introduced neural radiance fields, neural scene representation that uses the volume rendering equation under the piecewise constant opacity assumption for novel view synthesis. A neural radiance field (NeRF) is coordinate-based neural scene representation, where opacity 7 RR3 - R2o and color eV IR$ g - [0. 255] are predicted at each continuous coordinate by parameterizing them as a neural network. To train the neural network, 2D images are used as supervision where each viewing ray associated with a ground truth color. Volume rendering allows for the 3D coordinate outputs to be aggregated into an observed pixel color allowing for end-to-end training with 2D supervision. The supervision signal on the coordinates of the ray samples $1,.... &N. which updates the corresponding output opacity and color at those samples. NeRF uses a importance sampling strategy by drawing samples from the ray distribution from coarse network generate better samples for rendering of their fine network. To sample from a distribution, inverse transform sampling is needed, that is, one draws u ~ (O, 1) then passes it the inverse of cumulative distribution (CDF), i.e. sample x Fu), where F is the CDF of the distribution. Under the piecewise constant assumption, the CDF of the discrete random variable ş is given by:
S 80
F(x) = Exes P(Sk) 1 x < 85+1 for all E (1,..., N 13
(8)
1
SN
This CDF is however non-continuous and non- non-invertibe invertiieeeee.... NeRF's approach to get around this is to define a surrogate invertible function G derived from its CDF, then taking x G (w). Concretely, 5ーー G(y) By 8リ-クパ-1 (s,) + - 5リ- F(s)-1), where y E [s)-1.s)) However, this does not necessarily result in the samples from the actual ray distribution p(s) from the model.
Drawbacks of Piecewise Constant Opacity 7 in NeRFs
Unfortunately, there properties associated with the piecewise constant opacity formulation that may not be desirable in the context of NeRFs. First, it is sensitive to the choice of samples, i.e. sample positions s, along the ray, a phenomenon we dub as quadrature instability. This quadrature instability is due to the assumption that all points in an interval take the opacity of the left bin (Eq 5), making it sensitive sample positions. As illustrated in Figure 1, this would lead to ray conflicts in optimizing a NeRF when you have rays that are directly facing, i.e. perpendicular to, the surface (yellow rays) and rays that are close the grazing angle (red ray), i.c. parallel to, the object. To render the perpendicular rays correctly, vanilla NeRF has to store the optical properties (opacity and color) associated with the perpendicular rays at a point before its intersection with the surface. This creates inaccurate signals to the optimization process when NeRF renders the ray at a grazing angle, as it will cross multiple conflicting opacity/colors (illustrated by the blue gradient). The sample sensitivity issue also arises when having cameras at different distances from the object as illustrated in Fig. 2, as this would lead to shifted sets of samples, causing inconsistencies when rendering at different camera-to-objeet distances. Notice that the noise on the texture of the chair is different across different viewing distances, where the middle view has fewer artifacts compared to the closer and further views.
5

---

Camera to Scone Distance:
1.0x
0.75x
0.5x
Constnt
Surface
Lincer
Ray Conflicts: Different Camera-to-Seene Distances. (Left) Rendered views from cameras at different distances from the object. At all distances, the rendered output for linear have sharper texture than constant because of the latter's sensitivity to the choice of samples. We also highlight the instability of the constant model as shown by the noisier texture of the middle view compared to the closer and further views. (Right) An illustration that moving the camera different distances from the object result in different samples that lead to conflicts.
Figure 2:
The second issue comes from the CDF F being piecewise constant (Eq 8). This leads to two consequences. First, the piecewise constant assumption makes F non-invertible, hence, as mentioned in the previous section, importance sampling needs to be performed via a surrogate function G. This results in uniformity across the samples within the bin samples within bin assigned with equal probability, leading to imprecise importance samples. The second consequence comes from the fact that is not continuous, leading to an issue in training a NeRF that has a loss based on its samples. In other words, there will be vanishing gradient effect when taking the gradient w.r.t. the samples, and one such example of a sample-based loss used for NeRFs is depth [28].
5 Generalized Form for P,
We first show a generalized derivation for the probability Pj of each interval I,. which we use to for- mulate our approach that alleviates the problems described above. From T(x) = exp (- Jo ru)duu). first notice that:
dT exp (- L'' (()) = -T(ss ('(s (s))
ds
This results in the probability of each interval 1j given follows:
P3 (s)) --
- T(5441)
Since sy's are arbitrarily sampled, Pj can be exactly evaluated in a closed-form expression, if and only if T()) is in closed-form.
Our PL-NeRF
We observe from Eq. 15 that we can obtain closed-form expression for Ps for any piecewise polynomial function in r, which can be of any degree d = 0, 1, 2, ..., Commonly used in existing i. piecewise constant, that is unstable w.r.t. the choice of samples as highlighted in the previous section. Interestingly, we also observe and show that the problem making difficult and unstable to optimize. Please
NeRF literature is choosing
becomes numerically ill-conditioned for see supplementary for the full proof. Hence, we propose to make opacity piecewise linear (d = 1). which we call PL-NeRF, leading to a simple and closed-form expression for the volume rendering integral that is numerically stable and is drop-in replacement to existing NeRF-based methods. We show both theoretically and experimentally that the piecewise linear assumption sufficient and alleviates the problems caused by quadrature instability under the piecewise constant assumption.

---

Volume Rendering with Piecewise Linear Opacity. We pro- pose an elegant reformulation the sample-based rendering equation that corresponds to the exact integral under piecewise linear opacity while keeping piecewise constant color leading to simple and closed- expression for the integral. That is, instead of piecewise constant opacity as in Eq 5, we assume a linear opacity for each interval I,. Concretely, for [8j 55+15)+1)
Apeep
where Tj = (52); T++1 Ti+1
T(5++1), we have 8jt1 -
(1)
8-8j 8j+1 8j TJ+1-
Figure 3: Illustration of opacities T along a ray under the piecewise constant (green) and piecewise linear (orange) assumptions.
T(s)
8j+1 8j Tj
(10)
which is linear w.r.t. 8 E [8j, 85+1] 1 as illustrated in Fig.
Now, under the piecewise linear opacity assumption, transmittance is derived as the following closed-form expression:
T(8;) exp [- 1 r(u) du
II exp
7(u) du
H4-1
T(s;) II exp
(Th + Th (se
8k 1)
(11)
2
Together with Eq. 15, this leads to the following simple and closed-form expression for Pj. corre- sponding to the exact integral under the piecewise linear opacity assumption:
P3 = T(s)) (1-osp) [ (T5+1 + T,) {,) 1)
(12)
Precision Importance Sampling. Moreover, it also turns out that with our piecewise linear opacity assumption, we are able to derive an exact closed-form solution for inverse transform sampling. Recall that in Sec 4, we pointed drawback of the CDF Fer being non-invertible and discontinuous under piecewise constant opacity. We show that this is alleviated in our piecewise linear setting. Concretely, given samples $1, ..., &N resulting in interval probabilities PI. Pi...., from our derivation (Eq 12), the CDF for continuous random variable is then given as
F(1) = I ds E
p(s)db=E>, RiCt 25
p(s)
P [" T(s)T(s) ds.
(13)
Note that unlike in piecewise constant opacity, we do not convert a continuous random variable to discrete random variable thus, the resulting CDF F being continuous. Now, assuming that opacity T 2 0 everywhere, from Eq. 13 we see that strictly increasing. Since F is continuous and strictly increasing, then it invertible.
Finally, we have our precision importance sampling, where by inverse transform sampling, we can (w) for UO, 1) from the given ray distribution p(s) since
solve for the exact sample the CDF is invertible under piecewise linear opacity. That is, without loss of generality, let sample (0, 1) fall into the bin [Ck, Cstil, where C2 Ejek PJ. which is equivalent to solving for x E [sk, 5h+1], Reparameterizing I 8k where [0, 88+1 sk]. the exact solution for
sample u is given by
2(T841 - Th)
1- T(sk)
t 8k+1 8k
-Tk +
(14)
Tk+1 Tk
(8k-1
Ske)
Please see supplementary for full derivation. This leads to precisely p(s) resulting in better importance sampling and stronger depth supervision.
from the ray distribution
We note that DS- Ne-F [6] shows that this will sum to assuming opaque far plane. would correspond to the far plane.

---

G
Constine
Linvar
Figure 4: Qualitative Results for Blender and Real Forward Facing.
Blender
Avg.
Chair
Drums
Fieus
Hotdog
Lego
Mat.
Mic
Ship
Const. (Vanilla)
30.61
32.54
24.79
29.63
36.08
32.01
29.31
32.55
27.95
PSNR
Linear (Ours)
31.10
32.92
25.07
30.18
36.46
32.90
29.52
33.08
28.71
Const. (Vanilla)
0.943
0.966
0.918
0.960
0.975
0.959
0.943
0.978
0.846
SSIMI
Linear (Ours)
0.948
0.969
0.923
0.965
0.977
0.966
0.948
0.981
0.857
Const. (Vanilla)
5.17
3.19
7.97
4.14
2,48
2.33
4.32
2.16
14.8
LPIPS4
I Linear (Ours)
4.39
2.85
7.10
3.03
2.28
1.81
3.21
1.73
13.1
LLFF
Avg.
Fern
Flower
Fortress
Horns
Leaves
Orchid
Room
Trex
Const. (Vanilla)
27.53
26.79
28.23
32.53
28.54
22.35
21.20
33.03
27.58
PSNR
Linear (Ours)
28.05
26.85
28.71
32.95
29.38
22.51
21.25
33.99
28.79
Const. (Vanilla)
0.874
0.746
0.886
0.925
0.893
0.816
0.746
0.956
0.916
SSIMT
Linear (Ours)
0.885
0.863
0.902
0.932
0.911
0.826
0.754
0.961
0.933
Const. (Vanilla)
7.37
9.67
6.34
2.92
7.26
11.0
11.8
4.33
5.66
LPIPS
Linear (Ours)
6.06
7.92
4.93
2.46
5.51
9.59
10.2
3.54
4.38
Table Quantitative Results on Blender and LLFF Datasets. LPIPS scores x10
7
Results
In this section, we present our experimental evaluations to demonstrate the advantages our piecewise linear opacity formulation for volume rendering, which we call PL-NeRF.
7.1
Datasets, Evaluation Metrics and Implementation Details.
Datasets and Evaluation Metries. We evaluate our method the standard datasets: Blender and Real Forward Facing (LLFFF datasets as used in [18]. We use the released training and test splits for each. See supplementary for more details. For quantitative comparison,
follow the standard evaluation metrics and report PSNR, SSIM [32] and LPIPS [41] on unseen test views. We also report the waae---aan-suudee---ae
(RSME) on the expected ray termination in our depth experiments.
Implementation Details. PL-NeRF is implemented top of NeRF-Pytorch [38], reproducible Pytorch implementation of the constant (vanilla) NeRF, where we simply change the volume rendering to formulation under piecewise linear opacity and utilize exact importance sampling derivation. Similar to [18] we optimize a separate network for the coarse and fine models that are jointly trained batch size of 1024 rays and learning rate of 5 x 10-4 that decays exponentially to 5 10-5 throughout the course of optimization. We train each scene for 500k iterations which takes ~ 21 hours single Nvidia V100 GPU 3. Our precision importance sampling enables us to use fewer samples for the fine network, hence keeping the total number rendering samples the same, we use 128 coarse samples and 64 fine samples to train and test our method.
with the MSE loss ground truth images. We
7.2 Experiments on Blender and LLFF Datasets
We first evaluate our PL-NeRF
the standard Blender and Real Forward Facing datasets. Table 1 shows that our PL-NeRF (linear) outperforms the vanilla [18] (constant) model that assumes piece- wise constant opacity in all metrics for both the synthetic Blender and Real Forward Facing datasets. Figure and Figure 4 show qualitative results. As shown our PL-NeRF is able to achieve sharper
We rerun and train the vanilla (constant) model using the released reproducible configs from [38].

---

Blender
Avg.
Chair
Drums
Ficus
Hotdog
Lego
Mat.
Mic
Ship
Mip-NeRF
31.76
33.95
24.39
31.20
36.12
33,84
30.55
34.63
29.41
PSNRT
PL-MipNeRF
32.48
35.11
24.92
32.25
36.51
35.15
30.69
35.22
30.00
Mip-NeRF
0.955
0.975
0.921
0.971
0.978
0.971
0.957
0.987
0.876
SSIMI
PL--MipNeRF
0.959
0.981
0.928
0.977
0.980
0.976
0.959
0.989
0.882
Mip-NeRF
3.64
1.80
6.82
2.35
1.97
1.44
2.39
0.973
11.4
LPIPS
PL.-MipNeRF
3.09
1.32
5.78
1.66
1.67
1.07
2.09
0.788
10.3
Table
Quantitative Results of Mip-NeRR v.s. PI.-MipNeRF LPIPS scores x x10
textures shown in the Lego's scooper and the bread's surface in the hotdog. Moreover, our approach is also able to recover less fuzzy surfaces as shown in the microphone scene (Figure 1) where training views are close the grazing angle of its head. As illustrated, the resulting probability density the corresponding to the marked pixel is peakier than constant as our precision importance sampling allows us have better samples closer the surface. We also see clearer ropes in ship, less cloudy interior of the drum, and more solid surfaces such as cleaner leg of the swivel chair in the room scene.
7.3 Geometric Extraction
We also show improvement in geometric reconstruction of PL- NeRF. We extract the geom- etry from the learned density field from the trained models of PL-NeRF and Vanilla NeRF using marching cubes with a
Constant
threshold of 25 following [30].
Figure 5 shows qualitative re- sults the reconstruction of our piecewise linear vs the orig- inal piecewise constant formula- tion. As shown, we are able to better recover the holes on the body and wheels of the Lego scene well as the interior structure inside the Mic. Moreover, interestingly, the surface of the drum is reconstructed to be transparent as visually depicted in the images, opposed to the ground truth being opaque.
Linear
Figure 5: Geometry Extraction Qualitative Examples.
GT
Mip-NeRF
MI-MIENERF
Mip-NeRF
PL.MipNeRR
Figure 6: Qualitative Results for Mip-NeRF vs PI.-MipNeRE. We see that under difficult scenarios such as fine texture details of the chair and grazing angle views on the mic, PL-MipNeRF visually shows significant improvement.
7.4 Effectiveness of our formulation on other Radiance Field Methods
We also demonstrate our formulation's 's effectiveness other radiance field methods and show that our approach can be used a drop-in replacement to existing NeRF-based methods. We integrate our piecewise linear opacity formulation to the volume rendering integral into Mip-NeRF (PL-MipNeRF)) Table shows our quantitative results demonstrating consistent improvement across all scenes in the original hemisphere Blender dataset. Figure 6 shows qualitative examples where we see that under difficult scenarios such as when ray conflicts arise the fine details of the Chair and in presence of grazing angle views in the Mic, our PL-MipNeRR shows significant improvement over the baseline. Our results show that our piecewise linear opacity and piecewise constant color formulation scales well to Mip-NeRF as well. See supplementary for implementation details. We also plug

---

Dist 0.25x
Dist 0.5x
Dist 0.75x
Train Set
PSNR
SSIM
LPIPS]
PSNR1
SSIM1
LPIPS|
PSNR1
SSIMI LPIPS.
Const. (Vanilla)
20.18
0.612
54.1
22.80
0.753
30.4
25.97
0.867
14.1
Hemisphere
Linear (Ours)
20.34
0.637
50.0
23.00
0.767
27.5
26.28
0.876
12.6
Const. (Vanilla)
22.30
0.677
45.7
25.51
0.811
23.1
28.02
0.891
11.3
Multi Dist.
Linear (Ours)
22.66
0.705
41.1
26.04
0.828
20.3
28.55
0.900
9.90
Table 3: Testing on close-up views: Hemisphere are with training cameras located on the original hemisphere. Multi Dist. with training cameras are at random distances across a depth scale range of 0.5 1.0 of the original hemisphere. Reported LPIPS score multiplied by 102.
piecewise linear opacity formulation into DIVeR [36], voxel-based NeRF model, and show that our formulation also an effective drop-in replacement outperforming the original DIVeR [36]. Please see supplementary for experiment results and implementation details.
7.5 Experiments on Close-up Views
We further consider the challenging setting of testing on cameras closer to the objects. Table (top) shows quantitative results when training on the original hemisphere dataset and tested on different close-up views. As shown by the drop in metrics, the difficulty increases as the camera moves closer to the object (0.75x to 0.5x to 0.25x of the original radius) where details get more apparent. Our PL.-NeRF outperforms the vanilla piecewise constant model in all settings and the gap (SSIM and LPIPS) between ours and the constant assumption increases as the setting becomes harder highlighting the importance of recovering shaper texture and less fuzzy surfaces.
We also consider the set-up of training with cameras with different distances to the object that result in different sets of ray samples causing conflicts. We generate training views following the data processing pipeline from [18] with random distance scaling factor sampled from (O0.5 1.0). As shown in Table 3 (bottom), our PL-NeRF outperforms the vanilla constant baseline in all metrics across different camera distances, where the gap (LPIPS and SSIM) is also larger the closer the camera to the object, where details are more apparent. The difficulty for the constant (vanilla) case under multiple camera distances is its sensitivity to the choice of samples along the ray. Figure 2 shows that the conflicting rays cause quadrature instability for under piecewise constant opacity leads to unstable outputs shown by the noisy texture on the chair. For the constant, the level of noise (gold specs) and blurriness vary at different camera distances, whereas our PL-NeRF renders crisper and more consistent outputs even the camera is moved closer or further for the object.
7.6 Experiments with Depth Supervision
Finally, we also show that our PL-NeRF enables stronger depth supervision under our piecewise linear opacity assumption due our precision im- portance sampling that allows for gradients to flow to these more refined samples resulting in more accurate depth. As in previous works [28], we use sample-based loss for to incorporate depth supervision". Table shows quantitative results when training and testing on the less constrained Blender dataset with cameras at random distances from the object, as described in the previous section, with depth supervision. As shown, our PL-NeRF outperforms the vanilla constant baseline on all metrics including depth RSME demonstrating that our approach allows for stronger depth supervision.
PSNR1 SSIM
LPIPS|
RMSEL
Const, (Vanilla)
29.20
0.898
11.2
0.178
Linear (Ours)
29.54
0.905
10.4
0.147
Table 4: Depth Supervision. Reported LPIPS score is multiplied by 10a.
8 Conclusion
We proposed a new way approximate the volume rendering integral that avoids quadrature instability, by considering a piecewise linear approximation to opacity and a piecewise constant approximation to color. We showed that this results in a simple closed-form expression for the integral that is easy to evaluate. We turned this into a new objective for training NeRFs that is a drop-in replacement to existing methods and demonstrated improved rendering quality and geometric reconstruction, more accurate importance sampling and stronger depth supervision.
'We use the original hyperparameters from [28] in this experiment.
10

---

Acknowledgements. This work is supported by Apple Scholars in AI/ML PhD Fellowship, Snap Research Fellowship, a Vannevar Bush Faculty Fellowship, ARL grant W911NF-21-2-0104, a gift from the Adohe corporation, the Natural Sciences and Engineering Research Council of Canada (NSERC), the BC DRI Group and the Digital Research Alliance of Canada.
References
[1] Henrik Aanes, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision, pages 1-16, 2016.
13
[2] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo
and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
2.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360:
Unbounded anti-aliased neural radiance fields. In CVPR, 2022. 2,3
[4] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV, 2023.
[5] Zhiqin Chen. Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In CVPR, 2023.
[6] Kangle Deng, Andrew Liu, Jun- Man Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In CVPR, pages 12882-12891, 2022. 2, 7
[7] Robert Drebin, Loren Carpenter, and Pat Hanrahan. Volume rendering. SIGGRAPH. 22(4):65-74, 1988.
[8] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV, pages 5875 5884, 2021. 1.
[9] Hiroharu Kato, Deniz Beker, Mihai Morariu. Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey. ArXiv, abs/2006. 122p20006.2077.. 2020. 1
[10] Marc Levoy. Efficient ray tracing of volume data, ACM Transactions on Graphics (TOG), 93))245-261,
1990. 2,3 3
[11] Zhengqi Li, Simon Niklaus. Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view
synthesis of dynamic scenes. In CVPR, pages 6498 05086550, 2021. 1.2
[12] Lingjie Liu, Jiatao Gu, Kyaw 7aw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields.
In NeurIPS, 2020. 3
[13] Ricardo Martin-Brualla Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron. Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021. 2
[14] Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics. 1():99--088 1995. 1, 2, 3,5
[15] Nelson Max and Min Chen. Local and global illumination in the volume rendering integral. In Hans Hagen, editor, Scientific Visualization: Advanced Concepts, volume of Dagstuhl Follow-Ups, pages 259-274, Dagstuhl, Germany, 2010. Schloss Pagstuhl-Lcibniz-Zennree
fuer Informatik. 2, 3, 4, 5, 13, 17
[16] Ben Mildenhall, Peter Hedman, Ricardo Martin-Bruall Pratul P. Srinivasan, and Jonathan T. Barron. NeRF in the dark: High dynamic range view synthesis from noisy raw images. In CVPR, 2022. 2 [17] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorti,, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling
guidelines. ACM Transactions on Graphics, 2019. 2, 3
[18] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. In ECCV, 2020. 1, 2, 3, 5, 8, 10, 13
[19] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz,
and Ricardo Martin-Brual Brualla. Nerfies: Deformable neural radiance fields. In ICCV, 2021. 2
[20] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, representation for
Ricardo Martin Brualla. and Steven M. Seitz. Hypernerf: A topologically varying neural radiance fields. ACM Transactions Graphics, 40(6), dec 2021. I
[21] Steven M Rubin and Turner Whitted. A 3-dimensional representation for fast rendering of complex scenes. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques, pages
110-116. 1980.
[22] Hanan Samet. The design and analysis of spatial data structures, volume 85. Addison- Reading,
MA, 1990. 3
[23] Cheng Sun, Min Sun, and Hwann- -aann Chen. Direct voxel grid optimization: Super-fast convergence for
radiance fields reconstruction. In CVPR, 2022. 3
[24] Cheng Sun, Min Sun, and Hwann- -zonn Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprint arXiv:2206. 2000022006.00085, 2022. 3
[25] Matthew Tancik, Vincent Casser. Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022. 2
11

---

[26] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 33:7537-7547, 2020. 2
[27] Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang. Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings. 2023. 28
[28] Mikacla Angelina Uy, Ricardo Hartin-Brualla, Leonidas Guibas, and Ke Li. Scade: Nerfs from space carving with ambiguity-
depth estimates. In CVPR, 2023. 2, 6. 10, 28
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz
Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017. 2
[30] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning
neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurlPS, 2021. 9
[31] Qianqian Wang. Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron,
Ricardo Martin-Brualla. Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based
rendering. In ICCV, 2021. 2, 3
[32] Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004. 8 [33] Lee Westover. Interactive volume rendering. In Proceedings of the 1989 Chapel Hill workshop on Volume
visualization, pages 9 9-16, 1989. 3
[34] Peter L. Williams and Nelson Max. A volume density optical model. pages 61-68, 1992. 3
[35] Peter L Williams, Nelson Max, and Clifford M Stein. A high accuracy volume renderer for unstructured
data. IEEE Transactions on Visualization and Computer Graphics, 4(1):37-55 54, 1998. 3
[36] Liwen Wu, Jae Yong Lee, Anand Bhattad, Yuxiong Wang, and David Forsyth. Diver: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In CVPR, 2022. 2, 10 [37] Yuanbo Xiangli. Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In
ECCV, 2022. 2
[38]
Lin Yen-Chen. Nerf-pytorch. https: // github..... com/yenchenlin wwwethheennnnnnneeeeee http///ffffffooeeee pytopcc/ 2020. 8
[39]
Alex Yu, Vickie Ye, Matthew Tancik. and Angjoo Kanazawa, pixelnerf: Neural radiance fields from few images. In CVPR, pages 4578-4587, 2021. 2,
[40] Kai Zhang. Gernot Riegler. Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural
radiance fields. ArXiv, abs/2010.07492, 2020. 2
[41] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable
effectiveness of deep features as perceptual metric. In CVPR, pages 586-595, 2018. 8
12

---

Appendix
We conduct further experiments, analysis and discussions on our proposed reformulation, where we take piecewise linear approximation to opacity and piecewise constant approximation to color that results in an integral that is a simple and closed-form expression. This allows us to address the drawbacks of current piecewise constant assumption in NeRFs such ray conflicts during optimization and a non-invertible CDF that lead to imprecise importance samples and vanishing gradients. We provide additional results in Sec A.1: ablation study (Sec A.I.I), video demo (Sec A.1. additional results on a real dataset (Sec A.1.3), additional qualitative results (Sec A.1.4), comparison with PL-DIVeR 1.1. additional geometric extraction results 1.6 and comparison with less number of samples A.1.7. We then provide a walkthrough of the piecewise constant derivation from [15] (Sec A.2), which is followed by the thorough step-by-step derivation of our piecewsie linear opacity in volume rendering and precise importance sampling (Sec A.3). We also include analyses on piccewise quadratic and higher order polynomials Sec A.4. Finally, we end with additional implementation and experiment details (Sec A.5), Limitations (Sec A.6) and Societal Impact (Sec A.7).
A.1
Additional Results
A.1.1 Ablation Study
We conduct further ablation study on our precise importance sampling, As described in Sec. 4 the main paper, our piecewise linear opacity approximation allows to solve for closed-form solution for inverse transform sampling leading to the formulation of our precise importance sampling. Unlike the vanilla piecewise constant opacity approximation, our approach results in an invertible CDF, and hence we do not need to define an invertible surrogate function G for inverse transform sampling as in piecewise constant opacity (see Eq. 8 of main paper), which does not necessarily result in samples from the actual ray distribution p(s). We quantitatively ablate the effectiveness of our precise importance sampling (Precise) by replacing our formulation (Eq. 13 main paper) with the surrogate function as the vanilla constant setting (Surrogate). As shown in Table Al (first and second row of each metric), our precise importance sampling consistently outperforms using the surrogate on all metrics across all 8 scenes in the Blender dataset (hemisphere). Morcover, we also show that our precise importance sampling enables to fewer samples for the fine network as it is able to sample correctly from the ray distribution. Hence, keeping the same total number of rendering samples, we are able to achieve further boost in performance by using 128 coarse and 64 fine samples as shown in Table A1 1 (second and third row of each metric). We use N₂ and N₂ to denote the number of coarse and fine samples, respectively.
A.1.2 Video Demo
We also show a video demo of our results. Please see our project page (pl-nerf.github.io) for the video (best viewed in full screen). In the mic scene, we observe that the structure inside the mic is lost in the piecewise constant opacity approximation. Moreover, we see the blurriness along the sides of the body of the mic caused by ray conflicts from the perpendicular and grazing angle views during optimization of the vanilla NeRF model. For the chair scene, we see that our PL-NeRF able to achieve shaper textures on the back of the chair. Moreover, we see that there is an instability to the samples as illustrated by the inconsistencies in the noise the gold specs on the green texture (please view in full screen). This is also evident when vary the camera-to-scent distance. The chair model was trained on disti-distance Blender data to highlight the difference when camera distances vary.
A.1.3 Real Dataset Results
We further evaluate our PL-NeRF on real dataset DTU [1]. We train and evaluate our approach on the 15 test scenes used [39], and report the standard metrics (PSNR, SSIM, LPIPS). We follow the protocol used in [18] for the Real Forward Facing scene where g of the views were held out for testing while the rest are used for training. Table A2 shows the quantitative results averaged over the 15 scenes, where our PL-NeRF outperforms the constant [18] baseline.
13

---

GT
Constant
Linear
Figure A1: DTU Qualitative Result Visualizations of rendered DTU dataset test views. Because of the issue of grazing angle and binning inaccuracy of the piecewise constant opacity assumption, the vanilla NeRF (constant) exhibits blurry geometry, and rendering artifacts in the zoomed-in views (middle column). On the other hand, the piecewise linear opacity assumption in our model (linear) alleviates these issues (right column). Overall, the rendered views exhibit sharper surface boundaries and more faithful reconstruction compared to the constant model.
14

---

GT
Constant
Linear
Figure A2: Blender Qualitative Result Additonal visualizations of rendered Blender dataset test views. We see that our PL-NeRF is able to achieve sharper and crisper texture (chair and hotdog surface), better capture fine geometric detail (hole in lego, rope in ship) and avoid blurriness caused by conflicting rays, e.g. grazing angle views as shown in the mic.
15

---

Metrics Method
Ne
N;
Avg.
Chair
Drums
Ficus
Hotdog
Lego
Mat.
Mic
Ship
Surrogate
64
128
30.25
31.96
24.69
29.05
35.64
31.32
29.10 32.60
27.59
Precise
64
128
30.87
32.85
24.96
29.61
36.54
32.27
29.35
33.21
28.22
PSNR1
Precise
128
64
31.10
32.92
25.07
30.18
36.46
32.90
29.52 33.08
28.71
Surrogate
64
128
0,940
0.966
0.914
0.957
0.973
0.953
0,943
0.978
0.841
Precise
64
128
0.946
0.969
0.921
0.961
0.977
0.962
0.947
0.982
0.848
SSIM
Precise
128
64
0.948
0.969
0.923
0.965
0.977
0.966
0.948
0.981
0.857
Surrogate 64
128
5.50
3.85
8.52
4.15
2.70
2.94
4.04
2.30
15.5
Precise
64
128
4.77
2.94
7.54
3.85
2.27
2.25
3.39
1.59
14.3
LPIPS|
Precise
128
64
4.39
2.85
7.10
3.03
2.28
1.81
3.21
1.73
13.1
Table Al: Ablation Study, Reported LPIPS scores are multiplied by 10a.
We use No and N; to
denote the number coarse and fine samples, respectively.
PSNR
SSIM
LPIPS!
Const. (Vanilla)
27.96
0.909
8.58
Linear (Ours)
28.43
0.918
7.73
Table A2: DTU Quantative Results Metries computed from the average of 15 scenes from DTU dataset. The reported LPIPS score is multiplied by 102.
A.1.4 Additional Qualitative Results
We additionally show qualitative results from DTU dataset in Fig. Al. More qualitative results are also shown in Fig. A2.
Blender
Avg.
Chair
Drums
Ficus
Hotdog
Lego
Mat.
Mic
Ship
DIVeR
30.78
32.01
24.72
30.1
35.94
29.03
29.31
32.10
29.08
PSNR
PL-DIVeR
30.88
32.92
24.7
30.23
35.94
33.42
32.06
33.08
28.99
DIVeR
0.956
0.959
0.917
0.963
0.974
0.965
0.977
0.978
0.870
SSIMT
PL-DIVeR
0.947
0.969
0.916
0.963
0.966
0.966
0.977
0.981
0.871
DIVeR
3.39
2.79
6.13
2.34
1.92
1.46
1.77
2.16
7.77
LPIPS|
PL-DIVeR
3.28
2.85
6.01
2.12
1.83
1.49
1.77
1.73
7.82
Table A3: Quantitative Results of DIVeR v.s. PL-DIVeR Reported LPIPS scores are multiplied by
102
A.1.5 PL-DiVER
We plug our method into DIVeR by using their voxed-based representation and feature integration, and dropping our piecewise linear opacity formulation for volume rendering (PL-DIVeR). Results are shown in Table A3 demonstrating that our approach is on-par if not better across the different scenes in the Blender dataset. We highlight that this shows the improvement of using our piecewise linear opacity formulation, which is a drop-in replacement to existing methods.
A.1.6 Geometric Extraction
We also show quantitative results in geometric extraction improvement of PL-NeRF compared to the original Vanilla NeRF. Table A4 reports the distance between the surface of the ground truth model to the predicted meshes by sampling point clouds via ray casting, We see that our piecewise linear approach achieves lower error compared Vanilla NeRF on almost all the scenes in the Blender dataset.
A.1.7 Comparison with Less Samples
run both our PL-NeRF and Vanilla NeRF with coarse and fine samples results in an average of (30.09, 0.939, 0.056) and 229.86.00 0.937, 0.059) for (PSNR, SSIM, LPIPS), respectively, on the Blender dataset. This shows that under less number samples our piecewise linear opacity formulation is better than the original piecewise constant opacity assumption.
16

---

Blender
Avg.
Chair
Drums Ficus
Hotdog
Lego
Mat.
Mic
Ship
Vanilla NeRF
10.43 5.162
6.842
29.94
7.555
7.474
6.833
5.214
11.44
CD4
PL-NeRF
10.10
4.676
7.754
29.58
7.004
6.825
6.061
5.213
10.44
Table A4: Geometry Extraction Error Distance between the surface of the GT to the predicted
meshes. Scores are x10
A.2 Volume Rendering: Walkthrough of Piecewise Constant Derivation from [15]
From [15], under approximation that both opacity and color are piecewise constant, for 8 (s:) and r(8) = Vs E [8t, 81+1,, the probability of the interval P; is derived
[8s, 85411 I. where
as follows:
%1
P.
T(s)T(s)ds
Ti cxp (- / r(u)du)ds
Ti exxpp (- /// I(u)du) exp (- [ r(u)du)ds
TiT(s:) exp (- / r.dudds
T(T(s)) (2(t) exp (-T:(s 85) )ds
exp (-Ti(8 8:) 86t1
Tj -sT((s) (1 exp -i))
Moreover, under the piecewise constant assumption, transmittance T is derived and given by:
T(s:) =xx)= (- 1" r(u)du)
II exp (- / r(u)du)
-.--
J=1
II exp (- (3)
rj-1du)
j=1
exp (Ty-1(8) s)-1))-
j=1
This is the formulation that is used in most, if not all, NeRF works, which has the drawbacks that we raised such as ray conflicts during NeRF optimization and non-invertible CDF causing imprecise importance sampling and vanishing gradients when defining a loss w.r.t. the samples. We propose to approach this issue by deriving the volume rendering equation under piecewise linear approximation to opacity, which we detail in the next sections.
17

---

A.3 Volume Rendering: Our Piecewise Linear Derivation
We now show full derivation for the volume rendering equation, under the assumption that the opacity 7(8) is piecewise linear, i.e. is linear within each interval [s;, 8142], and piecewise constant color. We then derive the probability of interval under this assumption.
A.3.1 Generalized form for P..
Recall the generalized form of P. derived in the main paper. First from the definition of transmit- lance, we have
S r(u)du)
T(s) exp (-
ds exp (- / r(u)du)r(s) = - (T())(s)) (())) (s))
This results in the exact expression for the probability P. of interval given follows:
8141 T(s)T(s) = --
p1141 f(r (s)s) T(s:) T(s++1)-
P; =
(15)
A.3.2 Evaluating T(s) for s [&s, 8:41]
Let T, 7(si). T+1 7($+41), be sampled points along the ray. For s E [st 8t41), assuming piecewise linear opacity 7, i.e. T(s) is linear within each bin, we have
T(8)
8j-1 )r;
34
T
8541 - 8;
8;-1 - 8,
[(T++] :T)s 8 + (8i+1T: - 8:7441)
8i+1
A.3.3 Transmittance T(s:)
We first derive expression for transmittance T(s:) under the piecewise linear assumption.
18

---

T(s:) exp (- / r(u)du)
II exp (- 1 r(u)du)
1=1
8/--
-1
exp
[(T T3-1)u + (5yTz-1- -s)-17))]du)
==1
8j 8j
18--1
-1
T--1 (8} - 23-1) + (B/T/-1- - -3- -s--i))
exp
8j 8j-1
2
j=1
II exp
j=1
II j=1
exp
Tj8j +
+ 28 28j-177)
2 Ty8 - T38j-1+ + T 13-18)-1
exp
j=1
Thus we get
(T, Tg-1)(8} - 8)-1))
T(s:) = II exp j=1
(16)
2
A.3.4 Probability of interval [st, Sc+1]
From the generalized form for P. derived in the main paper, plug in the expression for transmittance (s;) as derived above to obtain:
8141
P;
(ss)Tss)ds
T(s:) - T(si+1)
II exp (- (Tj + Tj 1)-1) 85-1)
exp (- (Tj + Tj
(_(T)
1)
88 1)
2
==1
Hence, we obtain
P. = T(s:) (1-exp
(Ti+1 + (8)+1 - 82)1)
(17)
2
A.3.5 Our Precision Importance Sampling
To sample from the ray distribution, inverse transform sampling is needed, that is, one draws ut UO, 1) then passes
the inverse of cumulative distribution (CDF), i.e. a sample x (w), where F is the CDF of the distribution. Unlike the piecewise constant case, where F is not invertible, needing for surrogate function G derived from we show that under our piecewise linear opacity assumption, solve for the solution for each corresponding u.
19

---

As illustrated in the main paper, since F is continuous and increasing, under our assumption that т 0 then invertible. Now, without loss generality, let sample fall into the CDF interval [Ck, de+1]. where = Ejzk Pj. We know that the probability of the corresponding interval is Pa as given by Eq. 17. Thus we have:
Ck+1 ck Pk
- ((s)T(s)ds
=T(s)) (1-exp) (- (Te+1 + (x)(1 3k)
We want to solve for sample [sk, 8k+1], such that x FF)(u). Equivalently, since know that x E [sk, Sk+1]. then we reparameterize and let æ 84 + L, where & E [0, 8k+1 sk). We are
solving for x follows:
l (ss)r(s)ds
L T(s)T(s)ds + [ (ss)r(s)ds
ck + [ ((s)
Ch [ ((s)r(s)ds
Now, from the derivation of the general form for P. in Eq. 15, we can similarly obtain
u - Cs = T(ss) - T(x)).
=T(sk) (1 exp (- (-) r(u)du)).
Thus, simplifying we get
u 1 Ck T(sk)
exp (- L T(u)du)
r(u)du) = 1 u Ch T(sR)
exp (-
r(u)du) = in (1 u Ch
T(sk)
which gives us the expression
T(u)du ((u)d==In(T)) (n(((s))) (T((())) - (u - -c))).
(18)
In practice, we can simply add small c, say 10 to the model output resulting in positive make positive everywhere.
20

---

This holds for (sx) + O, which is true under our assumption that 7 > 0, and T(s;) (u c+) 2 0, which we will show in Sec. A.3.8 below.
A.3.6 Evaluating - S., ((u)du.
We evaluate the expression - Sxx r(u)du in order to solve for the exact sample Recall, for E [sk: 8k+1], we have
T(s) = 8k+1
1 +
Tb+1
3k+1
Ki
sk++1 8k
[[Tx+] Tk)s (sk+1Tk skTK+1)]
8k+1 5k
Let constants Tk+1 The b 8K+17k SkTk+1. d 8441-a8"
thus can write r(s) as follows:
7(8) clas b).
(19)
Thus, from Eq 19 we have:
(" d(au bjdu
r(u)du-
d( I" audu + { bdu)
8kk
au²
d
+ b(x - -s)]
2 86
(xx2
+ b(x skh]
a((*
+ +1)² 2)
+ b((sx + [8)]]
2
al2 + 28st)
+ bl]
2
d a
022
+ (as; + b)t]
2
Tk+1
i2 + ((Tx4) - TT)SS + 88417h SkT641)(
8k+1
2
Tk+1
Tk 12 + -
8k+1
8
Tk+1
Tk
2 + Tkt
2(8841 3%)
Hence, plugging this in Eq. 18, we get the quadratic equation
Tk+1 Tk 2 + Tht (In(T(sw)) - In(T(ss) - (u - cx))) 0. 2 2 sk)
We want to solve for t E [0, 5k+1 sk), and the roots of the quadratic equation is given by
21

---

(sk+1 --)
7h + v r + 2 - Te)(nn (ex)
1611-
(20)
(Tx+1 - Tk)
That means to compute for the solution, we need to find the root
(-Th x + 2( J(n T( (nn))]nn([[ (m--x=))))
(5x+1 ---1
(0,
(Tk+1 Th)
which we will show always exists and is unique.
A.3.7 Bounding A =T + 2( 11 T(sc) In) (1-)))))))
(#k+1-9k)
Let us first bound the discriminant of the quadratic formula. We have
Ck S Ck+11 - Ck
k
k-1
Σ Ps Σ P
i=0
j=0
(sk) -1-exp (- (T&+1 + Tx) - 3k)
Thus we have
In([(sk)= (u cx)) 2 (T(sk) - -T(ss) (1 exp (- (Tk+1 + [k)( Sk
2
(Tu+1 + Tx) Tx)($641 8k)
= (n(((())) exp
2
(Th+1 + (ok+2 -8k)
= =In(T(sk))
2
(Te+1 + Txk - -8k)
Ins))s))) (n(((ss)) (u cn))
2
2(Tx+1 - Tk) (n((())) (u cx))) v
2( 2(Tx41 (x)( (Th t Koy KMu1
($k+1 - sk)
(sk-1 - st))
Tk+1
Hence, computing the discriminant we get:
- 2(Tk+1 - TIn T(ss) (In In(T(ss)) (u cx)))
I c + (T -22 = -
(S - Sk)
Similarly, cu 2 0, where equality holds when Ca- This gives us A > +.
Hence, we know that T A
Since we need
Th A
(-Th VA
(0, 1), and
(Tk+1 Th)
(Tk+1 Tk)
22

---

Thus to find the solution we need to take the positive root. We have
(-Th VA)
(-Th + VTz
(Tk+1 - Tk)
(Tk+1 Tk)
( + VD)
(-Tk v TK+1 (Tk+1 Tk)
(Th+1 Th)
This shows that the solution is within the desired interval. Hence, the solution is
(8k+1 - -88) + V T 2(10+1-Te))
(sk41
(21)
(Tk+1 7kk
Hence, for the positive root, we know that t E [0, Sk-1 -s]]
A.3.8 Proof for T(s))= 2 (u ck)
k-
Ck
Pj
)=0
k-1
(T+ + [T)((551 )3,)) $1)))
T(s;) (1 - eexpe-p---e
2
)=0
We know that
(T; + Ti- TT-1)(8s 81-1)
T(sy) = II
exp( 1=1
Let a; = exp(- (1=t)-1(e)- and T(sp) = 1. Hence we have
T(s)) 44
k-1
a = (1sn) - 01) +
| a;) : (1 aaa)))))
j=1
(1 01) + (a1) 02) + (a102)(1 az) +
d a2.....
Since T(s:) a.a2...0k and
Ovi then
T(sk) c = -1002...a + (1 a102...ak)
=1
Gk+1
2 4.0
Note that above proof and solutions hold for т + Th+1 and T(xx) * O. which all holds since we have Tss) > OVs, which is equivalent to F' being increasing function.
23

---

A.3.9 The solution for sample w:
Putting everything together, we have the solution t given as
t= (8k+1 8k)(-Tk + v 2( - -7 (In (s2) (o)
(#6+1 -82)
(22)
(T++1 Tk)
From Sec A.3.8, we have (s*x) = a1 a2.2...a and Ck 01022..00. thus T(s*) - (u ck) Hence we can simplify it to
(sk+1 5h)(-TH T/
2(T411 72ln T(s -(st)- (ak:1 1-) %b)
(Tx+1 - -Th)
2(T2+1 -2)( 1- In f-m! (#e+1 B8k)
(sR+1 sk)(-TH + V Th
(23)
[T++1
Tk)
A.4 Piecewise Quadratic and Higher Order Polynomials
Now, we first consider the full derivation for volume rendering equation under the assumption that opacity piecewise quadratic and color piecewise constant. Consider opacities TI,.... Tr queried at samples . along the ray. Here, we set in and 8n+1 Iy to the near and far plane, with To = 0 and Ta+1 10 denoting empty and opaque space.
To interpolate opacity, because quadratic function can only be uniquely defined with points, choose (s)) to be quadratic within each interval [sj. $5+2] for even values of j. To encapsulate all points, this forces n to be odd, e.g. using 127 coarse samples and 64 fine samples.
A.4.1 Derivation: Computing (s)
In the same way as Sec. A.3.3, we can derive transmittance, which in foosed-form. The only modification is that the formulae for integrals of opacity are different over left and right subintervals 18,88511 and [sj+1, S2+2), with j even (see the next section for a derivation of these integrals):
522
$25
#2f+1
T(s2:) exp (-
(u)ddu) = II exp (-/
((uddu) exp (-
r(u)du).
(24)
j=1
$2j
Substituting the expressions in Eqs. 28 and 29 gives a closed form expression in terms of the 8y ss's and T ss. We can similarly compute
T(82++1) exp(
82641
T(u)du) II exp (-
1 r(u)du)exp)
22
exp
f r(u)du). (25)
g25+1
825-1
Then the probability of the ith interval for each 0 s n is, as before,
P. = T(s:) - ((8+1)).
(26)
This leads to a closed-form expression for P:. This means the behavior P. depends entirely on that of the opacity integral. However, due to the form of f ((s)ds detailed in the next section, this means P, involves a piecewise exponential of a rational function in T 's and 8, 's, which leads to poor numerical conditioning and thus optimization instability.

---

A.4.2 Derivation: Computing and Integrating 7 (s) on Intervals
We now compute r(s) and its integral on each interval. Fix the interval (sj, 8y+2], with odd, and Tj = (s5)), Tj+1 T(5)+1), T+22 =(5)22). By Lagrange interpolation, the quadratic T(s) passing through (5j,7))s (5541)T44) (5542) (5542, Tj+2) is given by:
7(8)= = Tj (s 8s+1) - 8342) Tjt1 (s - 8j)(8 - 85+2) + Tit2 (s - 8j)(s - 89+1); (27)
az73
ajßj
8y7s
aj = sj+1 8/1
Bg = = 8/+11
N/ 8j+2 $j-
Note the integrals of the three monic quadratics over [sy, sJ+1] can be expressed terms of ay 9y.
L s(s-
a
0303
(s -541))( s)+2)ds
+
3
2
a;
01
3
(s 18-38) -s)s(ss -)+2)ds=
s(s 71 7j)ds
j
2
8141 Jaan (8-3)) s;)(s ss+1)ds
a
3
s(s - a;)ds
Thus, using Eq. 27 gives the integral of opacity over 85+1]:
55+1 (s)dds T3
a
a33;
Tj+1
o2
0
T++2
(28)
75
3
2
857t
Similarly, the integrals of those same three quadratics over [55+1, 8542] factor out By:
2:12
(s - s)+1)(s s3+2)ds / 8
s(s s (s)ds
9111
1212
33 8373
(s - s;)(s ss+2)ds=
(s (g)gssss
9111
-8,
/
A3
83
Ba,;
(s - ss) sj+1)ds=
(s + as sds
1311
Then, summing these up along Eq. 27 gives the integral of opacity over S]+2];
f T(s)ds=
Tj ajTs 6 aj
B3 [
Tj+1
83
8373 2
Tj+2
88
3zaj
+
3 +
(29)
3
%
2
Observe Eqs. and 29 that the integral of opacity involves some terms with ojj 33s T5 in the denominator, which do not cancel. Thus, the integral of opacity over an interval is not polynomial in T. 's and s; 's, but is instead a rational function. This contrasts with the linear derivation in Sec. A.3.3, where this integral was degree multivariate polynomial in and s; 's. This caveat causes numerical instability, which will be discussed further in Secs. A.4.3 and A.4.4.
Generally, following the steps of the above derivation shows that if we interpolate 7 piecewise by any degree polynomial, 2 2, then the result is a rational function in s;'s 's and 7, 'ss but not a polynomial, which would also lead to training instability as in the quadratic case.
25

---

A.4.3 Piecewise Quadratic Problem 1: Negative Interpolated Opacity
As seen above, one problem with the piecewise quadratic model is that the integral of opacity is a rational function in ae Bt, ?:. In particular, due the presence of negatives in front of certain rational terms in Eqs. 28 and 29, it may become negative the denominators of these terms approach zero. One example is shown in Figure A.4.3, for samples 81, $2; 83 with Ti = 72 < T3 and 83 - 82 < $1, where the interpolated quadratic dips far below the r-axis. Note this interpolation is physically implausible, as opacity should be nonnegative everywhere.
-1,81,
Furthermore, from Eqs. 24 and 25, when the opacity integral is negative on intervals, transmittance can then be product with exponentials of negative terms, and potentially be greater than 1. This is incompatible with the physical interpretation of transmittance as a probability that a light travels a distance along ray without being absorbed.
Figure A3: Interpolation gives negative 7-values when s-values are close.
A.4.4 Piecewise Quadratic Problem 2: Instability from Sample Proximity
In Eqs. 28 and 29, observe the presence of terms such ajj: (j. " in the denominator. These do not appear in the piecewise linear model. As a result, because these quantities approach zero become arbitrarily
samples 8j. 8 become closer, then the integral of opacity large positive or even negative (as shown qualitatively in the previous section). Referencing the transmittance formulae in Eqs. 24 and 25, this means transmittance can approach zero or infinity and gradients can explode as samples are clustered together. This is fairly common in stratified sampling, and even more so with importance sampling.
To formally describe this instability, we first look at what happens when s, and sy+1 coincide:
lim 8541 -8 Jaj
(ssds= = 0
because T Rgo >> is continuous. There is no instability in this integral when 8,-1 - 85- However, when 13+1 and 8j42 coincide, because Bg - 0+ and y3 - at. the integral approaches
lim 5942-28
Fj+1 T(s)ds= lim
Tj
+
a
Tj+1
a
T2+2
*5+2-+8/+1
B3
2
8,70
11-10
7-11977 +
T3-20]
Tj0g
lim
67
3
8j12 2442-
B
2r 13 - 3T5+1057 + T-320
Tj0j
lim
3
67,8,
8342
T,00
T + +2ay7j - 3333)
T+11)
lim
5342-99111 3
TJ410 + +3y3)(a - 71) + a T1+1)
TjOj.
lim
3
85 8312-28/11
6758,
T2 3 )8, + 03) (TØ42 7/41)
3,0j
lim
3
85 2
6%
26

---

J0
2Tz+10J
a)(Ts+2 - Ts+1)
lim
3
3
6n5
8542 3411
2T
TJ+2 TJ+1
lim
3
6
Bg
8y+2
The behavior of this integral limit depends on the last limit. To analyze this, let h Rz0 - IR be the network function which takes samples on the ray and outputs opacity. The last limit
M+)+22 - h(85+1)
Tj42 Tj+1
lim
lim
(30)
8j+2 - -j+1
Because h is almost always differentiable at 8,41, this becomes h' 1'(s)+1) and so
1512 T(s)ds= = Tjaj 2T a ((
(31)
3
There are no constraints on the value of h h1(5)11) as the network trains, so the above limit can achieve any real value. particular, we can derive a condition for when the limit is negative:
7(8)ds <0 = 1(88++1) 2T, + 4Ty41
lim
(32)
5542
That is, whenever there is a sharp enough increase in opacity (which can happen at, say, a surface crossing) and samples 55+1: 8,42 sufficiently close, the interpolated quadratic can have negative integral on [sj, $j+1], which leads to the issues described in Sec. A.4.3, In other words, the integral of opacity over (51. is unstable as $5+2 + 8 A similar analysis holds to show the instability of the opacity integral on [s5+1, 5542] as 85+1 + 87.
A.4.5 Piecewise Quadratic Problem 3: Importance Sampling
Suppose we wish to importance sample in the same manner as piecewise linear, that is, we use inverse transform sampling. In essence, we draw (O, 1) and then sample F (u), with F the CDF of the distribution. Recall F is computed as
f(x) = | (())r(8)ds=
nSk
' T(s))(s)dss = cx + $ʃ. T(s) (33) 8b
((s)T(s)ds++
eo
As before, the latter equation becomes
f(z) cj T(s*) - T(x).
(34)
Hence, computing amounts solving the equation f(x) u, which becomes from above:
T(x) = + T(xx) u.
(35)
As T is the exponential of a cubic, this amounts to solving a cubic above. This does have a real solution, as F is increasing and continuous, so the Intermediate Value Theorem implies f(x) has a solution; and the solution is unique because F is strictly increasing, as is an exponential of a polynomial. However, the complexity exact importance sampling would be large.
This analysis reveals another downside of higher order polynomials. In general, suppose we wish to interpolate opacity with piecewise degree polynomial. Then following the same method as above, we see transmittance is the piccewise exponential of a degree n + polynomial, which is polynomial. The
the integral of 7. So inverse transform sampling reduces to solving a degree Abel- Mhel-Rufiini Theorem asserts for n + 2 that this polynomial is in general not solvable by radicals. In other words, there is simple closed form for exact importance sampling when 4.
Theoretically, this could be exactly solved for n 2 and n = 3 (i.e. when opacity piecewise quadratic or cubic, respectively). but the formulae to derive cubic and quartic solutions can become sufficiently complicated and the resulting complicated expressions may result numerical instability during optimization, especially when taking the gradient w.r.t. the samples.
27

---

A.5 Experiment details, Reproducibility and Compute
A.5.1
Implementation Details
We include the core code snippets of our implementation of PL-NeRF. Figure A4 shows the volume rendering equation that includes the implementation of Eq. 10 and 11 (main paper). This is direct replacement of the original constant approximation, where we also show the code snippet in the figure reference. Figure A5 shows the implementation of our precise importance sampling from Eq. 13 (main paper), which is also a direct replacement of the constant importance sampling implementation (Figure A6) for reference. We highlight that our formulation is direct replacement of the functions from the original implementation, and hence for the depth experiments, we are also able to directly adapt the codebase from [28]. We use Nvidia v100 and A5000 GPU's for our experiments. Each scene is trained on single GPU and takes 15- 20 hours. We used an internal academic cluster and cloud compute resources train and evaluate our models.
For the MipNeRF- experiments, our experiments are also run on the standard train and test split of the Blender dataset with the official released hyperparameters of Mip-NeRF using the NerfStudio [27] codebase. For PL-MipNeRF, we use the two-MLP training scheme with coarse loss weight of 1.0.
For DIVeR- experiments, we use their official implementation and configuration for DIVeR64 at 128 voxels. For PL-DIVeR, utilize their voxel-basee representation and feature integration and dropping in our piecewise linear opacity rendering formulation. We similarly run the DIVeR models on single Nvidia v100 GPU trained using their default configurations and hyperparameters for DIVeR64 at 128 voxels.
The total training time for 500k iterations on a single Nvidia v100 GPU is 17.78 and 21.43 hours for Vanilla NeRF and PL-NeRF, respectively. Figure 2-c shows the head-to-head comparison of training PSNR (y-axis) with respect to time (x-axis) of Vanilla NeRF vs PL-NeRF on the Lego scene. Rendering single 800x800 image takes 25.59 and 32.35 seconds for Vanilla NeRF and PI.-NeRF, respectively.
A.5.2 Computational complexity under different number of samples
We measured the total rendering time for single 800x800 image under different numbers of samples for our PL-NeRF. The total rendering time for (64+64), (64+128) and (128+64) are 19.20, 25.8 and 32.35 seconds, respectively.
A.5.3 Convergence plots under different number of samples
Figure ?? shows the convergence plots under different numbers of samples of our PL-NeRF Vanilla NeRF. We see that under different number of samples, our linear approach converges to a higher training PSNR.
A.6 Limitations
Our piecewise linear opacity approximation is able to handle arbitrarily small opacity, e.g. 1e-6, however, we cannot handle coordinates with exactly zero opacity. This special case is not an issue in practice. Also theoretically, any atom absorbs light and thus will not have exactly zero opacity, except in vacuum. Another limitation is our method slightly slower than the original piecewise constant approximation, and this is due to requiring more FLOPS for our importance sampling computation (Eq. 13 main paper). Another additional limitation we still assume piecewise color, i.e. within a bin we do not handle color integration. Modeling this can potentially handle difficult scenarios such as double-walled colored glass or atmospheric effects such as fog or smoke. Lastly, we also inherit the limitations of NeRFs in general, such requiring known camera poses.
A.7 Broader Impact
Our models require the usage of GPUs both in training time and rendering time, and GPUs use up energy to run and power them. We acknowledge that this contributes climate change that an important societal issue. Despite this, we observe the improvement in the results that is theoretically
28

---

Lines
tar-falsel:
raw2zegr Lanbdo distei torcheegg seveigl--ee dists)
dists 2.v 2. forch.. p...plorry.ll
ate teil (eter) i. taultar)) varry bin (w1lt hit apenaa suriate)
tau tarch. cet orees (aareess(rrsrpppppeeeeee shape (b), 12, device device)-1e-18, relt..e) rolse, terchworest rer. shape(t), 1), device deitle)- -3)
F.re 1e-6
Lntervel_ave__oe 6.5 [taul...,is] toul...pt 311
Evaluntiing expl-8.3 (tox, (6:3) (x,5))))
expr renzesprt (enterpr(nnervvl.. Lave. tee,teu, dists) PLrays, NLsomates<l)
Tranonittance untsl
curth.curprootterrc. cat (torch.ones: (B)) 1l, device doviee-dvvoo), ciarl, -1), -1) Ilaa,, (Larples12), Thari-l, starts off at
multuply tramunittance
TEs,
sapples--1]
elsed
weights
sae Origina'
Plecosise Constant Ospcity
def compu zo, se ignts(ra,, 2. Jals, rays,,, natse-4, 1:
raude Lembds cay, siate, ecs, renl. 1. tarchccupl oct, dist))
vevttl..r 2:) 2_vatsl...vf
dists torch lelists,,, forch-fell Botch-rilLLtleldatttt (iletattss...... 1e28, device device)), -1) PLreys, fusmptee))
dists dists torch, www.:roys.r.....ooo da..rryy_dl...eeeee Hins,, die
listal
"ssapless]
waights alate terch. www.ourtttrrrccceetttoooo (ttrccheeerccrrrreeooeeoo shaee,)) 11, device dovice), al 1o-101, -1, -3G,,
return weights
Figure A4: Code snippet for volume rendering. The implementation for our piecewise linear opacity approximation is a drop-in replacement from the original piecewise constant.
grounded and believe that it is beneficial for the pursuit of science. We responsibly models by first prototyping on selected scenes before scaling up to different scenes across datasets to minimize its impact to climate change.
29

---

сxe our Piecevise Linear Opacity Reformulation
def sample_ad se_ tanpt__.o_reffomlattttt reformulttionteiee latinnneins,, weighes, tau, T, near, far, M_sapptes, det- Fat-False, Frest-Falss,, 20-4, apsiton -1e-3):
zere, thresthee
and bins z_vals, bin boundaries, input does not include near and far plane yet
#2d veights is the PV of each bin N_samples les +
bins cotch (near, bins, farl, -3]
pdf weights
terch. cunsurlpef,
cati (torch (ske(cf cdf), [batch, ten(eins))
cd
sarples
Linspace(O., steps N_samples, biin.devicee)
u.espand( scdf.shape[: [:-1) (N_samples])
(nnttts(cddf..a ssrt(ccfsshape--eee [N_samples],
overerite
fixed
nev_shape List(cdf.
(N_somples]
Linspace(0., N_samples les)
np.broadcast tolu, shap))
else:
np.randon. (ndd( shape)
terch. Tensor (u)
Inwert CDF
u.contigueus()
Inds borch:earchserteelccff right-True)
below torch.rax(torch.zee (ike -1, inds
above torch.nin (cof, shape(-11-1 torch.ones_ Like__)) inds)
inds_g
strch.stack( (baacckebloo. abovel, -1)
(batch, RLsamples les, 2)
natched_shepe shape Linds, g. g-shape(o), inds _. s.spee cdf. shapee111
esgaadddsststhh (aatched_ segan[ssatchee__hape)), 2, inds_g) bins_g torch, gather gatch. (b (bins. unw.uuezee(e)))) exppnn.eeeaddddtcheee
cdf_g
wurch-gather (cdf. unsqueeze (1) expand
shape), 2, inds_g)
T_a garch.gathertI www.teeeT..uquueeeeeeee.e unsqueeze(()
expand(eatchedd shape), 2, inds.s)
tau_g
garche (tas. unsqueeze (1) expand (satchod___hape)), 2, inds_g)
#2d Get tau diffs
tai_diff tauf....::] taul....t-1]
natchei_shape__t tau Linds_g.shapeln [o], inds_g st, taw_iff.
tau_diff_p torch.gatheri wet.bggsthrrrauuddiiee diff unsqueeze (1) exxand (astchedd stapeedd__se__euu__ 2, below.nnsqueeze (oreeee((1)))))e squeeque
s_left bins bins_gl...... []
s.right bins gl....
T_left T.l....0]
tau, tau.gl....0]
tau_right tau_g(...,]]
dunny torch.onesia shap.
samples1 torchashere (orchh Lagdcal_nd(fauuddff zero_threshold, tau_diff_p -eero_ sro__threeholdd, s_left, dumny)
samples2 (fffffffffdfff_p threshold,
Px_ s_sppppppll le(s, r_right, T_left, tau_left, taghtight, v, apsiloreessioon_). samplesi)
samples
(sampls s_left, samples3)
(tw..unsquuuuee exgand matchedd expand((etccceet shapee,
T_below
tau_below
bin_below bins
samples T.below, tau, below, bin,
def Linear, samplets, Jeft, s_right, T. left, tau, tau,Jaft, tau, right, 4psifan-1e-3)=
in_tem torch. (trorchhaattttrch.ooo Lakeaaness_eeeeeeeeeeeeeee Like(T_ epsilan,
torch.div(3-a diviivviio tarch..axttorch..o ones, LikelT, (efeeT__eft)--eeee_ee -psitann, T_lft) 1)
discrininant 12 torch.divl 2 riaut tau_teft) In_tera
torch. torch.maxttrrhh.nne.. (igh(s_righh)))eaoeeeeeep epsiton, s_right s_left)
torch.divf (s_right left) (-tav_left torch.sart(), torch.nax(torch) torch.nxx(tordhh ono.chtocc.ees Like left) epsilnn, tai_right tau, left))
setssriniinntt-epsiii discriminant)))
#9d clamp to (6, rirggh leftt]]
clach,c torch.ones, Tikett, device selicctt.deviic t.device) eesi s.right s_left)
sample s_left
return sample
Figure A5: Code snippet for our Precise Importance Sampling. The implementation of our precision importance sampling is also direct replacement from the original function from the constant implementation called sample_pdis (See next figure for reference).
30

---

Original Piecewise Constant Opacity def sample_pdf (bins, weights, N_samples, det=False, pytest=False):
weights weights 1e-5 #t prevent nans
weights turcihssu(weighhs,
keepdin:True)
cdf torch. (pf, -1)
cdf totch. (eroc t cdf), -1) (batch, len(bins))
samples
1f det:
torch. Linspace
N_sanples, device sevvicebins..oe bins.device)
u u.expand wcd. 11) [N_samples])
else:
u torch.rand rand (cst 1]) [N_samples), device-bins bins.device)
Pytest, overwrite with numpy's 's fixed randon numbers
1f pytest:
пр. (eorrandm.seed(O))
new_shape scatpc 1]] [N_samples]
det:
np. Linspace 1., N_sanples) ap.bboadcast_tol sewps)
elset
randon. rand(
(awdaaaew__saaaeeeeeeaaeeeeeeeea new_eewwaaee nne__saape)
torch.Tensor(u)
CDF
U. contiguuus ()
inds
sorch.searchsorteelcdf, u, right=True)
torch.nax(torch.zero. Linelisssss 1), above norch.nin( (caf. 1) torch.ones. Like(inds), inds)
inds_g
(below, above],
(batch, N_samples Les,
cdf_g tf.gather(cdf,, inds_g, axis=-1, batch dins=len (inds_p shape)-2)
bins g = sf.ggthee(biin inds 9, axis=-1, batch. dims=len (inds a. shape)-22 -2)
natched_shape (inds_g.shhape inds_g.shape() [1], cdf. [-1]]
cdf_g htt.hgaatterccff.nssueeeeeee (ew.::eeeeeeeeeeeeeeeeeeeeeate (nacched shape), inds_g)
bins_g torch. unsiueezee (esespueeczc(..eeeppn((ttttttt eespan(maacchee_eeaae
inds_g)
denom kcdf_g(..... aff..,,]-cdf_g(...... [dlcdd_gl....0)
torch.where(denee Minbere(deonn-ee-- torch.ones Like (denon), denon)
= (u-cdf_gll samples bins_gl....
foff_gl.... gl...,)) denon
(bins gl..., bins .l__gg...,,))
return samples
Figure A6: This is the original importance sampling for the constant approximation for reference.
31


Reviews:
[{"review_id": "rNpWd6aZBx", "reviewer": "Reviewer_tVLL", "summary": "The paper addresses a fundamental limitation of existing NeRF formulation, namely piecewise constant integration, which results in sensitivity to point sampling, summarized as quadrature instability in this paper. To address this, the paper proposes a new formulation based on piecewise linear quadrature for density and piecewise constant quadrature for color. The paper demonstrates that the proposed method produces sharper and more stable results, and leads to better quantitative metrics. ", "questions": "Is visual difference obvious for a random input, or if the example shown needs to be carefully chosen?\n", "limitations": "The paper does not discuss its limitations. It may happen that the method is robust, but some limitations (including limited improvements) in certain scenarios should still be discussed. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper clearly analyzes the problem of existing methods.\n\nThe proposed solution is theoretically sound.\n\nThe proposed method leads to improvements in NeRF reconstruction, both qualitatively and quantitatively. \n\nThe paper shows detailed theoretical analysis and visual comparisons in supplementary material and video. ", "weaknesses": "The improvements appear to be quite small from the overall quantitative metrics.\n\nIt seems that the improvements may be more visible in specific scenarios, but may be subtle for general inputs. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our paper clear and theoretically sound that addresses a fundamental problem of the existing NeRF formulation leading to quantitatively and qualitatively improvements in NeRF reconstruction. Below we address the questions that were raised.\n\n## Q1 Small improvements from overall quantitative metrics.\nAs emphasized by Reviewer WoBC, our “experiment results are convincing with a non-negligible improvement of 0.5dB in PSNR but also across other metrics (SSIM, LPIPS) on multiple classic datasets (one synthetic and two real).” One advantage of our approach is highlighted in difficult scenarios where sensitivity to samples becomes a problem from the original piecewise constant formulation. The metrics are not designed to be sensitive to those specific scenarios, but as also mentioned by Reviewer am3W the “qualitative improvements are significant in particularly difficult scenarios”.\n\n## Q2 Visual difference of random views\nIn scenes with difficult scenarios as mentioned in the paper, random views will have visual differences. On the other hand, in cases where the piecewise constant opacity assumption suffices, the performance is similar – as with any other rendering methods, if the underlying assumption of a simpler rendering model suffices, then the improvement of the more sophisticated model is less apparent.\n\n## Q3 Limitations\nOne potential limitation is we still assume piecewise color. Enabling a more sophisticated color function can potentially handle difficult scenarios such as double-walled colored glass or atmospheric effects such as fog or smoke. We will add this to our revision, however, we also note a statement from Reviewer WoBC that “given the nature of the contributions, there are no major limitations nor potential negative impacts to discuss.”\n"}, {"review_id": "QXC6qYwajx", "reviewer": "Reviewer_am3W", "summary": "This paper presents a fix for the \"quadrature instability\" arising from sampling of points for numerical integration used by NeRF2020 and its successors.The sampling inconsistence may not be big on classical rendering techniques, but can be significant when using neural architectures. Opacity and color values are assumed to be piecewise constant by NeRF. This paper offers a tractable solution with piecewise linear opacity values and piecewise constant color values. To avoid dense sampling the NeRF and follow up use a importance sampling approach based on corse-to-fine strategy. For drawing these importance samples the NeRF utilizes a surrogate CDF as original CDF is not invertible. The impact is shown on several standard scenes on difficult situations such as close to surfaces, fine feature areas, etc.\n", "questions": "Here are some questions/points:\n- Can this method be integrated into all the other Radiance Field methods that essentially use the same ray-tracing formulation? Has it been done? That will increase its appeal\n- What is the computational overhead of the expanded formulation compared to vanilla NeRF? The paper mentions 18 hours for 500K iterations. I am interested in knowing about head-to-head comparison with vanilla NeRF to know if there is significant performance penalty in using this formulation. I would like to know it for both training and rendering for view generation.\n- Just curious: What is the adverse impact of using piecewise constant color values? What could the advantage be if we use linear color also? Does the other combination of piecewise constant density and piecewise linear color have similar closed-form solutions? What would the visual impact of that combination?\n- Are comparisons with later NeRF such as ZipNeRF/MipNeRF available? It would be interesting to see if the new proposed strategy scales well to those as well.\n- Depth supervised experiments seem valid. Does this pan-well against works like RGBDNeRF (Yuan et al. 2022 TPAMI)?\n \n\n\nMinor points:\n- Citations appear in ordinary parentheses as \"(25)\" instead of the more standard \"[25]\". This was very confusing in the beginning. Is this OK with NeurIPS style?\n- Line 164: Isn't it more correctly R^3 x S^2 --> R^3? Why discretize it to [0-255]? Are you doing something different here?\n\n ", "limitations": "What are the limitations of this approach in the authors' views? None are mentioned in the paper.\n\nShowing the formulations effectiveness on other Radiance Field recovery methods (they all use the same rendering equation) will enhance the paper.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The proposed method is effective and efficient, drawing on existing volume-rendering literature. The modification suggested is rather simple but effective. The modified method should improve the quality of all Radiance Fields methods and hence of interest to researchers working on those areas.", "weaknesses": "The paper proposes a small, clean formulation of ray-integration that improves quality of NeRF reconstruction. The limited nature of the idea is its weakness, if it can be called so. The improvements in quantitative measures like PSNR is marginal, but the qualitative improvements are significant in particularly difficult situations. The usefulness and utility of the formulation could be established more clearly by incorporating it to other NeRF variants whose code is available. I would wholeheartedly support this idea in a vision conference like ICCV/CVPR. It is upto NeurIPS to decide if such a small-but-effective idea is of sufficient interest to its audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our formulation clean that results in a method that is simple but effective. We appreciate that you find our work to be of interest to researchers working in the area of radiance fields and for highlighting that our improvement is significant in particularly difficult situations. We answer the questions raised below. \n\n## Q1 Comparison with Mip-NeRF\nWe integrate our piecewise linear opacity formulation to the volume rendering integral into Mip-NeRF (PL-MipNeRF). Table 1 shows our quantitative results demonstrating consistent improvement across all scenes in the original hemisphere Blender dataset. Our experiments are run on the standard train and test split with the officially released hyperparameters of Mip-NeRF using the NerfStudio codebase. For PL-MipNeRF, we use the two-MLP training scheme with a coarse loss weight of 1.0. Figure 1 shows qualitative examples where we see that under difficult scenarios such as when ray conflicts arise in the fine details of the Chair and in the presence of grazing angle views in the Mic, our PL-MipNeRF shows significant improvement over the baseline. Our results show that our piecewise linear opacity and piecewise constant color formulation scales well to Mip-NeRF as well. We will include this in our revision.\n\n## Q2 Formulation’s effectiveness to other radiance fields methods\nWe also demonstrate our formulation’s effectiveness on other radiance field methods, such as Mip-NeRF as presented above, and additionally DIVeR, a recent voxel–grid method. We plug our method into their approach by utilizing their voxel-based representation and feature integration and dropping in our piecewise linear opacity rendering formulation (PL-DIVeR). We similarly run the DIVeR models on a single Nvidia v100 GPU trained using their default configurations and hyperparameters for DIVeR64 at 128 voxels. Table 2 shows the quantitative results on the standard Blender dataset, showing that our formulation is an effective drop-in replacement in other radiance field methods. We will include these results in our revision.\n\n## Q3 Computational overhead compared to Vanilla NeRF\nThe total training time for 500k iterations on a single Nvidia v100 GPU is 17.78 and 21.43 hours for Vanilla NeRF and PL-NeRF, respectively. Figure 2-c shows the head-to-head comparison of training PSNR (y-axis) with respect to time (x-axis) of Vanilla NeRF vs PL-NeRF on the Lego scene. Rendering a single 800x800 image takes 25.59 and 32.35 seconds for Vanilla NeRF and PL-NeRF, respectively. We will include these findings and specify the training and rendering time in our revision.\n\n## Q4 Possible advantage of piecewise linear color; impact of using piecewise constant color\nAn example scenario where piecewise linear color could have an advantage is a scene with a thin glass (or some non-opaque surface) with one color, e.g. red, on one side and another color, e.g. blue, on the other. Piecewise linear color would smoothly blend the color of the two sides.\n\n## Q5 Would piecewise linear color and piecewise constant opacity have closed-form solutions? Impact of such combinations.\nYes, it is closed-form. Under piecewise constant opacity and piecewise linear color, we can write the expected color of the bin as $\\int_{s_i}^{s_{i+1}} \\tau(s)T(s)c(s)ds = T(s_i)\\tau_i \\int_{s_i}^{s_{i+1}} c(s) \\exp{(-\\tau_i(s-s_i))}ds$ (refer to Sec 2.3 from Max and Chen 2010). This expression is of the form $\\int A(s)\\exp{B(s)}ds$, where $A(s)$ and $B(s)$ are both linear functions, hence its integral is in closed form using integration by parts. This assumption can potentially tackle the situation presented above on the double-colored glass, however, it still has a problem for the scenarios we have presented in the paper such as grazing angle and cameras at different distances.\n\n## Q6 Marginal improvements in quantitative measures\nOur results show significant qualitative improvements in particularly difficult situations as mentioned in your review. Unfortunately, the standard metrics are not designed to be sensitive to those specific scenarios, but nonetheless, we point out that Reviewer WoBC states that our experimental results are convincing with a “non-negligible improvement” across metrics in multiple classic datasets. We also note that as with any other rendering methods, if the underlying simpler assumption suffices, e.g. piecewise constant opacity, suffices for a particular scene, then the improvement of the more sophisticated model, e.g. piecewise linear opacity, is less apparent.\n\n## Q7 RGBDNeRF\nRGBDNeRF is orthogonal to our method as it depth-supervised experiments as it does not use a depth-based loss. As mentioned by HvcE, the results of our depth-supervised experiments demonstrate that our piecewise linear approach improves on the sample-based depth loss of a recent method, SCADE, compared to the original piecewise constant approximation.\n\n## Q8 Citation format\nThank you. We will change the citation format to the more standard “[25]” in our revision.\n\n## Q9 Ln 164\nThanks for catching this technicality. We are not doing anything different here – we will clarify and correct this in our revision. \n\n## Q10 Limitations\nOne potential limitation is our piecewise color assumption, i.e. within a bin we do not integrate color. This assumption will struggle in difficult scenarios such as two-sided colored glass or atmospheric effects such as fog or smoke. We will add this limitation in our revision. However, we also note a statement from Reviewer WoBC that “given the nature of the contributions, there are no major limitations nor potential negative impacts to discuss.”"}, {"review_id": "tJaU37HXAC", "reviewer": "Reviewer_WoBC", "summary": "Neural rendering methods like nerf relies on the integration of contributions (both color and density) along rays to predict views. Since analytical computation of the integral is not possible (the color and the density being both prediction from neural networks, usually MLPs), an approximation is estimated. Traditionally, the hypothesis done is that the color and the density are locally constant thus leading to a rectangle rule approximation. In this work, the authors propose to replace the the constant approximation of the density by a linear approximation. This allows for a more accurate estimation and a closed-form formula for sampling points during training. ", "questions": "RFF are more often referred to as LLFF. I suggest renaming to avoid any confusion (even though LLFF technically refers to the method proposed in the same paper) since a reader would expect that name.", "limitations": "Given the nature of the contributions, there's no major limitations nor potential negative societal impact to discuss. Some limitations are mentioned in the supplementary material but they are more related to the analysis of the method as I mentioned in \"Weaknesses\".", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed modification is simple and elegant, it is mathematically driven leading to a closed-form formula. Moreover it is plug-and-play with most (all?) popular nerf-based methods and have a direct impact on the quality of the density estimation. It offers an important solution to the fuzzyness problem of the density learned by nerf methods, thus allowing for better surface reconstruction, and especially smaller details.\n\nThe paper is clear and well written. It introduces well all the different concepts necessary to understand the contribution. \n\nExperimental results are convincing with a non-negligible  improvement of 0.5dB in PSNR but also across other metrics (SSIM and LPIPS) on multiple classic datasets (one synthetic and two real). ", "weaknesses": "While the paper is very clear and the ideas elegant, I find the experimental section lacking:\n\n* The proposed modification should impact mostly the density and thus the geometry of the reconstruction. The experimental section only focuses on a. While this is can be sufficient, I find disappointing that the authors did not take the opportunity to show the potential improvement of the geometry for the different datasets showed in the paper.\n\n* The choice of baseline nerf is surprising. Given the type of experiments (with large changes of camera-to-scene distances), I think that mip-nerf would have made for a much better baselines. It already fixes a large number of artifacts pointed out by the authors in Figure S1 and therefore would have been a more appropriate comparison. I nonetheless expect that the conclusions can be transcribed to more recent frameworks.\n\n* I find surprising that there is barely any discussion about the computation complexity of the proposed sampling (nor time measurements). I saw later that it is mentioned in a \"Limitations\" section in the supplementary but for me it's not a limitation per say and it should be studied in an ablation inside the paper. Especially since S1.1 shows that a better sampling can provide interesting convergence properties, thus allowing to change the number of samples used during the training. I would have expected a comparison with less samples than the 64 + 128 baseline (64 + 64 seems like an obvious thing to try based on the comments inside the text, I'm surprised that authors didn't even try that), to see the impact and find maybe a better compromise between computation time and performance. looking at the convergence plots would have also been interesting. Indeed, the proposed method might offer a faster convergence compared to the previous sampling, thus having a double impact.\n\n* While slightly outside of the scope of this paper, it still would have been interesting to show the impact on the more practical methods (instant-ngp or voxel based methods for example) that are often preferred because of their fast training speed. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our work simple, very clear and well-written with elegant ideas that are mathematically driven. We appreciated that you highlighted that our approach is a plug-and-play to the popular nerf-based methods and that our experimental results are convincing with non-negligible improvement across metrics on multiple classes datasets. We address your questions below.\n\n## Q1 Geometry Extraction\nThank you for your suggestion. We extract the geometry from the learned density field from the trained models of PL-NeRF and Vanilla NeRF using marching cubes with a threshold of 25. Table 3 reports the distance between the surface of the ground truth model to the predicted meshes by sampling point clouds via ray casting. We see that our piecewise linear approach achieves a lower error compared to Vanilla NeRF on almost all the scenes in the Blender dataset. Figure 2-a shows qualitative results on the reconstruction of our piecewise linear vs the original piecewise constant formulation. As shown, we are able to better recover the holes on the body and wheels of the Lego scene as well as the interior structure inside the Mic. Moreover, interestingly, the numbers on the Drums scene is due to the surface of the drum being visually transparent as shown in the figure. We will include these results in our revision.\n\n## Q2 Mip-NeRF baseline\nWe plug-in our piecewise linear opacity for volume rendering into Mip-NeRF (PL-MipNeRF), and results are shown in Table 1. We demonstrate consistent improvement across all scenes on the Blender dataset using the standard train and test splits. We use the official released hyperparameters for MipNeRF and rerun their models using NeRFStudio. Our PL-MipNeRF uses the two-MLP training scheme with a coarse loss weight of 1.0, keeping all other hyperparameters fixed. Our results show that our modification is a drop-in replacement and can be transcribed to other more recent frameworks. Figure 1 shows qualitative examples where we see that under difficult scenarios such as when ray conflicts arise in the fine details of the Chair and in the presence of grazing angle views in the Mic, our PL-MipNeRF shows significant improvement over the baseline. We will include these results in our revised version.\n\n## Q3 Computational complexity under different number of samples\nWe measured the total rendering time for a single 800x800 image under different numbers of samples for our PL-NeRF. The total rendering time for (64+64), (64+128) and (128+64) are 19.20, 25.85 and 32.35 seconds, respectively. We will report these timings in our revision.\n\n## Q4 Comparison with less samples\nThank you for your suggestion. We run both our PL-NeRF and Vanilla NeRF with 64 coarse and 64 fine samples results in an average of (30.09, 0.939, 0.056) and (29.86, 0.937, 0.059) for (PSNR, SSIM, LPIPS), respectively, on the Blender dataset. This shows that under less number of samples our piecewise linear opacity formulation is better than the original piecewise constant opacity assumption. We will include this in our revision.\n\n## Q5 Convergence plots under different number of samples\nFigure 2-b shows the convergence plots under different numbers of samples of our PL-NeRF vs Vanilla NeRF. We see that under different number of samples, our linear approach converges to a higher training PSNR.\n\n## Q6 Additional results on a practical voxel-based method.\nWe also demonstrate that our approach can also be integrated into a recent voxel-based method, DIVeR. We take its voxel-grid representation and feature integration and plug our piecewise linear opacity rendering formulation (PL-DIVeR). Table 2 shows quantitative results on the Blender dataset showing that even in voxel-based representations, our approach is on-par if not better than the piecewise constant opacity baseline. We will include these additional results in our revised version.\n\n## Q7 Renaming RFF to LLFF\nThank you for clarifying this and we will rename this in our revision to avoid confusion.\n"}, {"review_id": "pVUHPVu80B", "reviewer": "Reviewer_HvcE", "summary": "This paper describes a modification to the way NeRF performs approximate integration for volume rendering.  Instead of assuming piecewise constant opacity, they propose to use piecewise linear opacity.  They argue that this resolves what they dub the \"quadrature instability\" in NeRF.  They re-derive the quadrature integration method using piecewise linear opacity and piecewise constant color, and show that this also enables them to derive a precise inverse of the ray termination CDF for importance sampling.  Through experiments on standard NeRF datasets they show an quantitative improvement in rendering quality and also show qualitatively that their method makes the rendering sharper and more stable across change of viewpoint and camera distance.", "questions": "I think the most critical issue is the comparison with DIVeR.  How does this method compare to DIVeR, which appears to support an even more accurate computation of the integral?\n\nI would also be interested to know whether they expect that this method will improve the surface reconstruction capability of NeRF, and how it compares to NeuS in this regard.\n\nFinally I would like to know if they have a reference regarding the original NeRF's method of inverse transform sampling the CDF -- is this just based on the NeRF code or is there a reference for that?\n", "limitations": "A section on limitations does not appear in the paper.  A frank discussion of the limitations of the approach would strengthen the paper.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper addresses an often overlooked aspect of NeRF which is the specifics of how the volume rendering is implemented and whether it could be improved.  The propose what to the best of my knowledge is a novel modification, which is to approximate the opacity along the ray as piecewise linear (instead of piecewise constant) in order to obtain a more accurate quadrature estimate of integral.  This formulation also leads them to derive a more precise method of importance sampling.\n\nThey analyze conceptually how the piecewise constant assumption in NeRF leads to conflicting ray supervision between perpendicular and grazing angle rays.  They claim that their piecewise linear formulation reduces this problem and leads to a more peaked opacity PDF.\n\nThey provide extensive derivations for the formulas in the supplemental material.  The new formulations are a \"drop-in\" replacement for the equations used in vanilla NeRF, and thus they can easily modify existing NeRF implementations to use them.\n\nIn their evaluation they compare against vanilla NeRF on the standard synthetic and real forward-facing datasets and show a quantitative and qualitative improvement in rendering quality across the board.  \n\nThey also show that their method improves SCADE, a recent method that incorporates depth estimates into NeRF training, and show that swapping in their formulas leads to a modest improvement in quality.  They hypothesize that this is because of their more precise importance sampling.", "weaknesses": "They seemed to have missed a highly relevant previous work:\n\nWu, L., Lee, J. Y., Bhattad, A., Wang, Y. X., & Forsyth, D. (2022). DIVeR: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16200-16209).\n\nThis method is based on a voxel grid approach rather than an MLP.  However, the DIVeR paper also discusses the inaccuracy of integration using piecewise constant opacity.  Their solution allows for integration over more complicated functions than piecewise linear and so might be more accurate than the piecewise linear approach proposed here.  Indeed, DIVeR reports higher average rendering quality metrics on the NeRF synthetic dataset than what is reported here.  \n\nBecause they claim that their method improves sampling and makes the opacity function more peaked at the surface, I would be very curious to see a comparison of surface reconstruction quality compared to NeuS:\n\nWang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., & Wang, W. (2021). NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. Advances in Neural Information Processing Systems, 34, 27171-27183.\n\nThe explanation of inverse transform sampling in the original NeRF method (section 3.2) should be expanded, as this is not explained thoroughly in the original NeRF paper.  Their discussion (L175-178) is quite brief considering how critical it is to the paper.\n\nThe paper has many grammatical errors and incomplete sentences which make it difficult to read.  I included some examples here:\n* L135 \"Hence the continuous probability density function (PDF) the ray r(s)\"\n* L138 \"s is a point on the ray r\" -> it doesn't make sense that s would be a point (since it is a scalar)\n* L152-153 \"P_j is the probability of each interval, which is mathematically equivalent to the probability of the interval.\" This sentence sounds like a meaningless tautology.\n* L154 missing a period\n* L117 \"Then taking x = g^-1(u).\" This is an incomplete sentence.\n* L117-118 \"However, this does not necessarily result in the samples from the actual ray distribution p(s) from the model.\"  This sentence doesn't work grammatically.  Some more analysis of why this doesn't result in sampling from the distribution would be helpful (explanations and visualizations).\n* L203 \"such an example of a sample-based loss used for NeRFs is depth\" \n* L232-3 \"we pointed the drawback\"\n* L238 \"thus the resulting CDF F being continuous\"\n* Table 1 caption: needs a period\n* L273 capitalize Lego", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our formulation to be a novel modification. We address each of the concerns below.\n\n## Q1: DIVeR citation\nThank you for bringing up the paper. We will cite DIVeR in our revision.\n\n## Q2 Comparison with DIVeR\nWe plug our method into DIVeR by using their voxel-based representation and feature integration and dropping in our piecewise linear opacity formulation for volume rendering (PL-DIVeR). Results are shown in Table 2 demonstrating that our approach is on-par if not better across the different scenes in the Blender dataset. We use their official implementation and configuration for DIVeR64 at 128 voxels trained on a single Nvidia v100 GPU for each scene. We include this additional comparison in our revised version. We highlight that this shows the improvement of using our piecewise linear opacity formulation, which is a drop-in replacement to existing methods. Note that DIVeR is not directly comparable to PL-NeRF, but is instead compared against PL-DIVeR. \n\n## Q3: Proof that DIVeR’s solution does not quite integrate over more complicated functions\nWe note that technically it is not quite accurate to claim that “DIVeR’s solution allows for integration over more complicated functions”. We show that there is a tradeoff between the plausibility of the learned radiance field and their $MLP_w$ being an affine transformation and color being piecewise constant. Here we define a plausible radiance field as one where there is a unique opacity defined at each 3D point. We prove mathematically below that the volume rendering integral of DiVER does not integrate over more complicated functions if the underlying opacity field is plausible.\n\n### Claim 1: DIVeR’s volume rendering equation holds if and only if color is piecewise constant. \nFrom Eq. 12 in DIVeR supp A.1., Hölder's inequality becomes equality iff color is constant along the ray in each voxel. The reverse direction is proven by substitution. The forward direction is shown by proving the contrapositive – if the radiance field is not constant in an open interval of the line segment, the integral of radiance over that open interval will be strictly smaller than the $L_\\infty$ norm of radiance field over the same domain.\n\n### Claim 2: A plausible density field, i.e. a unique opacity $\\sigma(x) \\forall x \\in \\mathbb{R}^3$, only exists if DIVeR’s $MLP_w$  is an affine transformation.\nWe show this by showing the Hessian matrix of MLP_w is zero ranked. Let $S_1, … S_6$ be the six sides of a voxel and $S$ be the union of the sides. For $y \\in S_i$, we define a function $$x \\mapsto \\int_0^{\\lVert x-y\\lVert}\\sigma(r_{x, y}(t))dt=MLP_w(\\int_{t^{in}}^{t^{out}}\\hat{f}(r_{x, y}(t))dt) = MLP_w(\\int_0^{\\lVert x-y\\lVert}\\hat{f}(r_{x, y}(t))dt)$$,\nfor $x \\in S - S_i$, where equality follows from Eq. 5 in DIVeR main. By taking the gradient at $x \\in S - S_i$ and rearranging the terms, we derive that $$\\sigma(x)=x\\cdot[\\nabla MLP_w(\\int_0^{\\lVert x-y\\lVert}\\hat{f}(r_{x,y}(t))dt)]C(x).$$ Notice that $C(x)$ only depends on x. Since the RHS depends on $y \\in S_i$ while the LHS does not, we can take the gradient of both sides w.r.t. to y and conclude the Hessian of $f \\mapsto MLP_w(f)$ is rank zero on a convex open set. Then it follows that DIVeR’s $MLP_w$ is an affine transformation. \n\nHence, there is a tradeoff between the plausibility of the learned radiance field and $MLP_w$ being an affine transformation, and under this scenario, it turns out to simply be a trilinear interpolation of opacity.\n## Q4  Our opacity function being more peaked\nWe quantitatively evaluate our precise importance sampling by taking our models trained with depth supervision. Specifically, we compute the average L2 distance between the ground truth depth and 64 random samples drawn from the fine networks using our precise importance sampling for PL-NeRF and the original inverse transform sampling for Vanilla NeRF. PL-NeRF and Vanilla NeRF attain an average error of 0.019 and 0.033, respectively, demonstrating that our approach with precise importance sampling draws samples closer to the ground truth surface.\n\n## Q5 Improvement in geometry for NeRF\nWe show quantitative improvement in our extracted geometry in Table 3. We extract the mesh from the learned opacity fields using marching cubes with a threshold of 25 and compute the distance between the ground truth model and the output mesh. Figure 2 also shows qualitative examples.\n\nWe note that the SDF methods are orthogonal to our contribution as their goal is to learn an SDF field and convert that into density to enable supervision through volume rendering. How to render density is orthogonal to how density is outputted, our focus is the former, while the SDF works explore the latter. In principle, it is possible to use our method as a replacement for the volume rendering integration that these methods use. Exploring this could be an interesting direction for future work.\n\n## Q6 Discussion on original NeRF’s inverse transform sampling\n\nYes, it is based on the official codebase from the original NeRF that its successors are based on. Under the piecewise constant opacity assumption, the PDF of the ray termination distribution is not continuous, making its corresponding CDF also not continuous and thus non-invertible. Hence, to be able to sample with inverse transform sampling, a continuous surrogate CDF function (that we denoted as $G$) is needed, which from the official codebase is a linear interpolation. The code snippet is included in Figure S5 in our supplementary. Because a surrogate function is used, i.e. $G \\neq \\tilde{F}$, then the cumulative distribution described by $G$ is not equal to the cumulative distribution described by $\\tilde{F}$, and hence the resulting samples from inverse transform sampling using $G$ is not equal to the samples drawn from the distribution, i.e. PDF $\\tilde{f}$. We will elaborate on Sec 3.2 of the main paper in our revision."}]

Author rebuttal:
We thank the reviewers for their feedback and for finding our approach novel (HvcE), simple (WoBC, am3W) and effective (am3W) that introduces a clean (am3W), elegant (WoBC) and theoretically sound (tVLL) formulation that is clear and well-written (WoBC). As summarized by the reviewers, we tackle a commonly overlooked problem for NeRFs on how integration is done for volume rendering. Specifically, we propose a piecewise linear opacity and piecewise constant color formulation to volume rendering that alleviates quadrature instability and leads to a simple and closed-form equation. Our new formulation is a plug-and-play (WoBC) or drop-in replacement (HvcE) to existing NeRF-based methods that assumes piecewise constant opacity and color. Our experiments show that this leads to quantitative improvements across different metrics on multiple classic datasets 
(WoBC) that is qualitatively significant in particularly difficult situations (am3W). Below we include a brief summary to each response. Please see individual reviewer responses for more details.

## [tVLL, am3W] Marginal quantitative improvement
As mentioned by WoBC, our experiment results “are convincing with a non-negligible improvement across metrics on multiple datasets,” and the improvements of our approach are highlighted in “particularly difficult situations” (am3W). We note that the metrics are not designed to be sensitive to those specific scenarios.

## [tVLL] Visual difference of random inputs
Random inputs will have visual differences in scenes with difficult scenarios as mentioned in the paper. However, as with any other rendering methods, if the underlying simpler assumption, e.g. piecewise constant, suffices, then the improvement of the more sophisticated model, e.g. piecewise linear, is less apparent.

## [am3W, WoBC] Comparison with Mip-NeRF
We plug-in our piecewise linear opacity formulation into Mip-NeRF (PL-MipNeRF). Table 1 shows consistent improvement across all scenes in all metrics on the Blender dataset, demonstrating that results can scale to more recent frameworks.

## [am3W, WoBC] Integration to other methods
In addition to MipNeRF, we also integrate our approach into DIVeR, a recent voxel-based approach (PL-DIVeR), as shown in Table 2. Results show that our formulation can be utilized as a drop-in replacement to the recent voxel-based approach.

## [am3W] Computation overhead compared to Vanilla NeRF
On a single Nvidia V100 GPU, the total training time for 500k iterations of Vanilla-NeRF and PL-NeRF is 17.78 and 21.43 hours, respectively, while the total rendering time for one 800x800 image is 25.59 and 32.35 seconds, respectively. Figure 2-c shows the training curves with respect to time for the Lego scene.

## [am3W] Possible advantage of linear color
An example scene where this can be advantageous is a thin glass with one color on one side and a different color on the other.

## [am3W] Will there be a similar closed-form expression for piecewise linear color and piecewise constant opacity?
Yes, it turns out that it is in closed-form too. However, that assumption will still have the problems for the difficult scenarios presented in our paper.

## [am3W] RGBDNeRF
RGBDNeRF is orthogonal to our experiments on depth loss with our piecewise linear formulation as this work does not supervise their NeRF training with depth.

## [WoBC] Geometry Extraction
Table 3 shows the average distance between the surface of the ground truth model to the extracted meshes of the trained models for Vanilla NeRF and PL-NeRF. Figure 2-a shows qualitative examples.

## [WoBC] Computational complexity of different number of samples
The total rendering time for a 800x800 image of PL-NeRF is 19.20, 25.85, 32.35 seconds for (64+64), (64+128) and (128+64) samples, respectively.

## [WoBC] Less number of samples (64+64)
The results for PL-NeRF and Vanilla NeRF for (64+64) samples is (30.09, 0.939, 0.056) and (29.86, 0.937, 0.059), respectively, for (PSNR, SSIM, LPIPS).

## [WoBC] Convergence plots
Figure 2-b shows the convergence plots under different number of samples for PL-NeRF and Vanilla NeRF.

## [HvcE] DIVeR citation
We will cite DIVeR in our revision.

## [HvcE] Comparison with DIVeR
We incorporate our approach to DIVeR by using their voxel representation and feature integration and plugging in our piecewise linear opacity volume rendering integration. Table 2 shows quantitative results.

## [HvcE] DIVeR does not quite allow for integration over more complicated functions.
Interestingly, we show that there is a tradeoff between the plausibility of the learned radiance field and DIVeR’s $MLP_w$ being an affine transformation and color being piecewise constant. As a result, it does not quite allow for the integration over more complicated functions.

## [HvcE] Method improves sampling making it more peaky at the surface
We quantitatively evaluate the samples drawn from the learned distributions of PL-NeRF and Vanilla NeRF by computing the average L2 distance between the samples and the ground truth depth. The results are 0.019 and 0.033 for PL-NeRF and Vanilla NeRF, respectively.

## [HvcE] Improvement in geometry for NeRF
Table 3 shows the improvement in the extracted geometry of PL-NeRF compared to Vanilla NeRF across the different scenes in the Blender dataset. Figure 2-a shows qualitative examples. We note that the SDF methods are orthogonal to our contribution as their goal is to learn an SDF field and convert that into density to enable supervision through volume rendering. How to render density is orthogonal to how density is outputted, our focus is the former, while the SDF works explore the latter.

## [HvcE] Reference for inverse transform sampling of the original NeRF
Yes, it is based on their official codebase where linear interpolation is used as the surrogate function $G$. Since a surrogate function is used then the samples are not being drawn from the original underlying distribution.

Meta-review: The paper presents a method for improving the stability of the volume rendering process commonly used in the literature. Most reviewers are positive about this paper, given the simplicity of the technique and its significance to the community. The rebuttal results showcase the wider applicability of the proposed sampling methods, which reviewers appreciate. However, there were concerns about the complicated proof regarding the "plausible density field". Given the promising empirical results, the reviewers agree that this paper has sufficient merits to be accepted but are uncomfortable with having the proof in the camera ready since these theoretical justifications must be properly reviewed before it can be included in the paper. 

Considering the suggestions from the reviewers, the AC decides to accept the paper, but urges the authors *not* to include additional proof in the final submission.
Decision: Accept (poster)