Title: NeRF Revisited: Fixing Quadrature Instability in Volume Rendering
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural radiance fields

volumetric rendering

nerfs

numerical quadrature

importance sampling
Abstract: Neural radiance fields (NeRF) rely on volume rendering to synthesize novel views. Volume rendering requires evaluating an integral along each ray, which is numerically approximated with a finite sum that corresponds to the exact integral along the ray under piecewise constant volume density. As a consequence, the rendered result is unstable w.r.t. the choice of samples along the ray, a phenomenon that we dub quadrature instability. We propose a mathematically principled solution by reformulating the sample-based rendering equation so that it corresponds to the exact integral under piecewise linear volume density. This simultaneously resolves multiple issues: conflicts between samples along different rays, imprecise hierarchical sampling, and non-differentiability of quantiles of ray termination distances w.r.t. model parameters. We demonstrate several benefits over the classical sample-based rendering equation, such as sharper textures, better geometric reconstruction, and stronger depth supervision. Our proposed formulation can be also be used as a drop-in replacement to the volume rendering equation of existing NeRF-based methods. Our project page can be found at pl-nerf.github.io.

NeRF Revisited: Fixing Quadrature Instability in Volume Rendering

Abstract

Neural radiance fields (NeRF) rely on volume rendering to synthesize novel views. Volume rendering requires evaluating an integral along each ray, which is numerically approximated with a finite sum that corresponds to the exact integral along the ray under piecewise constant volume density. As a consequence, the rendered result is unstable w.r.t. the choice of samples along the ray, a phenomenon that we dub quadrature instability . We propose a mathematically principled solution by reformulating the sample-based rendering equation so that it corresponds to the exact integral under piecewise linear volume density. This simultaneously resolves multiple issues: conflicts between samples along different rays, imprecise hierarchical sampling, and non-differentiability of quantiles of ray termination distances w.r.t. model parameters. We demonstrate several benefits over the classical sample-based rendering equation, such as sharper textures, better geometric reconstruction, and stronger depth supervision. Our proposed formulation can be also be used as a drop-in replacement to the volume rendering equation for existing methods like NeRFs. Our project page can be found at pl-nerf.github.io.

1 Introduction

The advent of neural radiance fields (NeRF) [18] has sparked a flurry of work on neural rendering and has opened the way to many exciting applications [5, 11, 8, 20]. One of the key underpinnings of NeRF is volume rendering [14] - it is especially well-suited to end-to-end differentiable rendering [9], since the rendered image is a smooth function of the model parameters. This has made it possible to learn the 3D geometry and appearance solely from a 2D photometric loss on rendered images.

In volume rendering, the rendered colour ˆ y for every pixel is an expectation of the colours along the ray cast through the pixel w.r.t. the distribution over ray termination distance s [6].

where p ( s ) denotes the probability density function (PDF) of the distribution over ray termination distance s and c ( s ) denotes the the colour as a function of different points along the ray.

In general, p ( s ) and c ( s ) can be of arbitrary forms, so evaluating this integral analytically is not possible. Therefore, in practice, E s ∼ p ( s ) [ c ( s )] is approximated with quadrature. The quadrature formula that is most commonly used in the NeRF literature takes the following form:

where T j = exp ( -∑ j k =0 -τ k ( s k +1 -s k ) ) and τ k is the opacity evaluated at a sample s k along the ray.

This expression is derived from the exact integral under a piecewise constant assumption to the opacity and colour along the given ray [14].

However, this seemingly simple, innocuous assumption can result in the rendered image being sensitive to the choice of samples along the ray at which the opacity σ ( s ) and colour c ( s ) are evaluated. While this does not necessarily cause a practical issue in classical rendering pipelines [14, 15, 10], it has surprising consequences when used in neural rendering. Specifically, because the opacity at all points within the interval between two samples is assumed to be the same, there is a band near the surface of the geometry where opacity at all points within the band is as high as points on the surface itself. Because different rays cast from different cameras can pass through this band at different angles and offsets, both the number and the positions of samples within this band can be very different across different rays (see Fig 1 and Fig 2 for two example scenarios). Hence, simultaneously supervising these rays to produce the same colour values as in the real image captures can give rise to conflicting supervisory signals, which can result in artifacts like fuzzy surfaces and blurry texture.

Moreover, because the piecewise constant opacity assumption gives rise to a closed form expression that is equivalent to an expectation w.r.t. a discrete random variable, it is common practice in the NeRF literature to draw samples along the ray from the discrete distribution [17]. This is commonly used to draw importance samples, and also to supervise the samples along the ray [28], for example in losses that penalize deviation of the samples from the true depth [6]. Sampling from the discrete distribution requires the definition of a continuous surrogate function to the cumulative distribution function (CDF) of the discrete random variable, which unfortunately yields imprecise samples. As a result, samples that are drawn may not be close to the surface even if the underlying probability density induced by the NeRF is concentrated at the surface. Additionally, individual supervision cannot be provided to each sample drawn from the surrogate, because the gradient of the loss w.r.t. each sample would be almost zero everywhere.

All these issues, i.e. conflicting supervision, imprecise samples and lack of supervision on the CDF from the samples, stem from the assumption that opacity is piecewise constant causing the sensitivity to the choice of samples both during rendering and sampling. We dub this problem as quadrature instability . In this paper, we revisit the quadrature used to approximate volume rendering in NeRF and devise a different quadrature formula [14] based on a different approximation to the opacity. We first show that interestingly a closed-form expression can be derived under any piecewise polynomial approximation for opacity. When the polynomial degree is 0, it reduces to the piecewise constant opacity as in existing literature, and when the degree is 2 or more, we show that it would lead to poor numerical conditioning

Therefore, we further explore a degree of 1 (i.e., piecewise linear ) and show that it both resolves quadrature instability and has good numerical conditioning. We derive the rendering equation under piecewise linear opacity explicitly and show that it has a simple and intuitive form. This results in a new quadrature method for volume rendering, which can serve as a drop-in replacement for existing methods like NeRFs. We demonstrate that this reduces artifacts, improves rendering quality and results in better geometric reconstruction. We also devise a new way to sample directly from the distribution of samples along each ray induced by NeRF without going through the surrogate, which opens the way to a more refined importance sampling approach and a more effective method to supervise samples using depth.

2 Related Work

NeRFs. Neural Radiance Field (NeRF) is a powerful representation for novel-view synthesis [18] that represents a scene using the weights of an MLP that is rendered by volumetric rendering [14]. A key finding to the success of NeRF was the use of positional encoding [26, 29] to effectively increase the capacity of the MLPs that models the opacity and emitted color as a function of a 3D coordinate and viewing direction. Many works extend NeRF such as handling larger or unbounded scenes [40, 3, 37, 25], unconstrained photo collections [13], dynamic and deformable scenes [11, 19] and sparser input views [6, 39, 31, 28]. There are a number of papers that aim to improve the rendering quality of NeRF. Some do so by utilizing different kinds of supervision such as NeRF in the Dark [16], while others tackle this by improving the model [2, 36, 4]. MipNeRF [2] changes the model input by introducing integrated positional encoding (IPE) to reduce the aliasing effect along the xy coordinates. DiVeR [36] predicts the rendered colour within a line interval directly from a trilinearly interpolated feature in a voxel-based representation ZipNeRF [4] modifies the proposal network to a grid enabling it to be used together with IPE. In contrast, our work focuses on changing the objective function by modifying the rendering equation from piecewise constant opacity to piecewise linear, while keeping the model and supervision fixed. Additionally, ZipNeRF [4] also

brings up a model specific issue on z-aliasing, where their model struggles under this setting. Similar to z-aliasing observed by ZipNeRF, we consider the setting of having conflicting supervision when presented with training views at different distances from the scene. While they may appear similar on the surface, the phenomena we study is different in that it is general and independent of the model, on having conflicting ray supervision from camera views, e.g. different camera-to-scene distances and the grazing angle setup.

Importance Sampling on NeRFs. Densely sampling and evaluating NeRF along multiple points in each camera ray is inefficient. Inspired by an early work on volume rendering [10], prior works typically use a coarse-to-fine hierarchical sampling strategy where the final samples are obtained by importance sampling of a coarse proposal distribution [18, 2, 3, 8]. These importance samples are drawn using inverse transform sampling where a sample is obtained by taking the inverse of the cumulative density function (CDF) of the proposal ray distribution. However, prior NeRF works that assume piecewise constant opacity result in a non-invertible CDF, and instead introduce a surrogate invertible function derived from the CDF in order to perform importance sampling. In contrast, our work that utilizes a piecewise linear opacity assumption results in an invertible CDF and a closed-form solution to obtain samples with inverse transform sampling. Other works also attempt to alter sampling using neural networks or occupancy caching [31, 23, 24, 12]. These techniques are orthogonal to our work as they propose changes to the model as opposed to our work where importance sampling is derived from a given model.

Volume rendering. Volume rendering is an important technique in various computer graphics and vision applications as explored in different classical works [10, 33, 7, 14]. These works include studying ray sampling efficiency [10] and data structures, e.g. octree [22] and volume hierarchy [21] for coarse-to-fine hierarchical sampling. The crux behind volume rendering is the integration over the weighted average of the color along the ray, where the weights is a function of the volume density (opacity). Max and Chen [14, 15] derive the volume rendering equation under the assumption of piecewise constant opacity and color, which NeRF [17] and its succeeding works use to learn their neural scene representation. However, the piecewise constant assumption results in rendering outputs that are sensitive to the choice of samples as well the non-invertible CDF introducing drawbacks to NeRF training. Following up on [14], works also [34] derive the volume rendering equation under the assumption that both opacity and color are piecewise linear that yield unwieldy expressions that lead to numerical issues and/or are expensive to compute. Some earlier works on rendering unstructured polygonal meshes that attempt to use this model [35], but it is in general not commonly used in practice due to the mentioned issues and hence has yet to be adopted into learning neural scene representations. In this work, we address both sets of issues that arise from the piecewise constant opacity and color and piecewise linear opacity and color by reformulating the volume rendering equation to assume piecewise linear opacity and piecewise constant color. Our derivation results in a simple and closed-form formulation to volume rendering making it suitable for NeRFs.

3 Background

3.1 Volume Rendering Review

Definitions. In classical literature, the process of volume rendering [15] mapping a 3D field of optical properties to a 2D image and the visual appearance is computed through the exact integration of these optical properties along the viewing rays. In this optical model, each point in space is an infinitesimal particle with a certain opacity τ that emits varying amounts of light, represented as a scalar color c , in all viewing directions. The opacity τ is the differential probability of a viewing ray hitting a particle - that is for a viewing ray r ( s ) = o + s d , where o is the view origin and d is the ray direction, the probability of ray r hitting a particle along an infinitesimal interval d s is τ ( r ( s ))d s . Moreover, the transmittance T r ( s ) is defined as the probability that the viewing ray r travels a distance s from the view origin without terminating, i.e. without hitting any particles.

Continuous probability distribution along the ray r . As illustrated by Max and Chen [15], the probability of hitting a particle s + ds only depends on the probability of hitting a particle at s and not any particles before it the probability of ray r terminating at distance s is given by τ ( r ( s )) T r ( s ) , where T r ( s ) = exp( -∫ s 0 τ ( u )d u ) . Hence the continuous probability density function (PDF) of ray r ( s ) , which describes the likelihood of a ray terminating and emitting at s , is given by

Rendered Test View

X

X

Grazing

Angle Views

Region of

high opacity

surface

Perpendicular

Views

Piecewise Linear Opacity

Grazing

Angle Views

surface

Perpendicular

Views

Probability Density

Cumulative Density

1.0

Constant

0.8

0.6

0.4

0.2

00

1.8

1.9

2.0

2.1

t values

2.2

2.3

1.0

Constant

Linear

0.8

0.6

0.4

0.2

0.0

1.8

1.9

2.0

t values

2.1

2.2

2.3

2.4

where s ∈ [0 , ∞ ] and r ( s ) is a point on the ray r . For notational simplicity we omit r and write it as p ( s ) = τ ( s ) T ( s ) .

Volume Rendering as a Continuous Integral. The observed color of the ray is then the expected value of the colors c ( s ) of all particles s along the ray weighted by the probability of hitting them. Mathematically, this results in the following continuous integral 1 :

Quadrature under Piecewise Constant Opacity τ . Since this integral cannot in general be evaluated analytically, it is approximated with quadrature. Let s 1 , s 2 , ..., s N be N (ordered) samples on the ray that define the intervals, where I j = [ s j , s j +1 ] is the j th interval, and I 0 = [0 , s 1 ] , I N = [ s N , ∞ ] . The volume density for particles along the interval I j is then approximated under the assumption that opacity is constant along each interval, making it piecewise constant along the ray [15]. That is, for all j we have:

for brevity we denote τ ( s j ) = τ j , i.e. τ j is the opacity for sample s j . Under this piecewise constant opacity assumption, the volume rendering equation Eq. 4 then becomes as follows:

where T j = exp ( -∑ j k =0 -τ k ( s k +1 -s k ) ) . Here, color c j is also approximated to be constant along each interval I j , and P j is the probability of each interval. Now, let us define the discrete random variable ˜ s = ˜ f ( s ) , where

which gives corresponding probability mass function ˜ P ( s ) . Observe that the analytical expression of the integral Eq. 6 turns out to be the same as taking the expectation w.r.t. the discrete random variable ˜ s , i.e. E s ∼ p ( s ) [ c ( s )] = E ˜ s ∼ ˜ P ( s ) [ c (˜ s )] . This piecewise constant opacity assumption in the volume rendering equation is used in most, if not all, existing NeRF works. We recommend the reader to read [15] for more detailed derivations and our supplementary for a more thorough walkthrough.

3.2 Neural Radiance Fields.

Following Max and Chen [14], Mildenhall et.al. [18] introduced neural radiance fields, a neural scene representation that uses the volume rendering equation under the piecewise constant opacity assumption for novel view synthesis. A neural radiance field (NeRF) is a coordinate-based neural scene representation, where opacity τ θ : R 3 → R ≥ 0 and color c ψ : R 3 × S 2 → [0 , 255] 3 are predicted at each continuous coordinate by parameterizing them as a neural network. To train the neural network, 2D images are used as supervision where each viewing ray is associated with a ground truth color. Volume rendering allows for the 3D coordinate outputs to be aggregated into an observed pixel color allowing for end-to-end training with 2D supervision. The supervision signal are on the coordinates of the ray samples s 1 , ..., s N , which updates the corresponding output opacity and color at those samples. NeRF uses a importance sampling strategy by drawing samples from the ray distribution from a coarse network to generate better samples for rendering of their fine network. To sample from a distribution, inverse transform sampling is needed, that is, one draws u ∼ U (0 , 1) then passes it to the inverse of a cumulative distribution (CDF), i.e. a sample x = F -1 ( u ) , where F is the CDF of the distribution. Under the piecewise constant assumption, the CDF of the discrete random variable ˜ s is given by:

This CDF is however non-continuous and non-invertible. NeRF's approach to get around this is to define a surrogate invertible function G derived from its CDF, then taking x = G -1 ( u ) . Concretely, G ( y ) = y -s j -1 s j -s j -1 ˜ F ( s j ) + s j -y s j -s j -1 ˜ F ( s j -1 ) , where y ∈ [ s j -1 , s j ] . However, this does not necessarily result in the samples from the actual ray distribution p ( s ) from the model.

4 Drawbacks of Piecewise Constant Opacity τ in NeRFs

Unfortunately, there are properties associated with the piecewise constant opacity formulation that may not be desirable in the context of NeRFs. First, it is sensitive to the choice of samples, i.e. sample positions s , along the ray, a phenomenon we dub as quadrature instability . This quadrature instability is due to the assumption that all points in an interval take the opacity of the left bin (Eq 5), making it sensitive to sample positions. As illustrated in Figure 1, this would lead to ray conflicts in optimizing a NeRF when you have rays that are directly facing, i.e. perpendicular to, the surface (yellow rays) and rays that are close to the grazing angle (red ray), i.e. parallel to, the object. To render the perpendicular rays correctly, vanilla NeRF has to store the optical properties (opacity and color) associated with the perpendicular rays at a point before its intersection with the surface. This creates inaccurate signals to the optimization process when NeRF renders the ray at a grazing angle, as it will cross multiple conflicting opacity/colors (illustrated by the blue gradient). The sample sensitivity issue also arises when having cameras at different distances from the object as illustrated in Fig. 2, as this would lead to shifted sets of samples, causing inconsistencies when rendering at different camera-to-object distances. Notice that the noise on the texture of the chair is different across different viewing distances, where the middle view has fewer artifacts compared to the closer and further views.

Camerato

1.0x

0.75x

0.5x

Scene Distance:

Constant

Surface

Linear

The second issue comes from the CDF ˜ F being piecewise constant (Eq 8). This leads to two consequences. First, the piecewise constant assumption makes ˜ F non-invertible, hence, as mentioned in the previous section, importance sampling needs to be performed via a surrogate function G . This results in uniformity across the samples within the bin - samples within a bin are assigned with equal probability, leading to imprecise importance samples. The second consequence comes from the fact that ˜ F is not continuous, leading to an issue in training a NeRF that has a loss based on its samples. In other words, there will be a vanishing gradient effect when taking the gradient w.r.t. the samples, and one such example of a sample-based loss used for NeRFs is depth [28].

5 Generalized Form for P j

We first show a generalized derivation for the probability P j of each interval I j , which we use to formulate our approach that alleviates the problems described above. From T ( s ) = exp ( -∫ s 0 τ ( u )d u ) , we first notice that:

This results in the probability of each interval I j given as follows:

Since s j 's are arbitrarily sampled, P j can be exactly evaluated in a closed-form expression, if and only if T ( · ) is in closed-form.

6 Our PL-NeRF

We observe from Eq. 15 that we can obtain a closed-form expression for P j for any piecewise polynomial function in τ , which can be of any degree d = 0 , 1 , 2 , ..., n . Commonly used in existing NeRF literature is choosing d = 0 , i.e. piecewise constant, that is unstable w.r.t. the choice of samples as highlighted in the previous section. Interestingly, we also observe and show that the problem becomes numerically ill-conditioned for d ≥ 2 making it difficult and unstable to optimize. Please see supplementary for the full proof. Hence, we propose to make opacity piecewise linear ( d = 1 ), which we call PL-NeRF , leading to a simple and closed-form expression for the volume rendering integral that is numerically stable and is a drop-in replacement to existing NeRF-based methods. We show both theoretically and experimentally that the piecewise linear assumption is sufficient and alleviates the problems caused by quadrature instability under the piecewise constant assumption.

Volume Rendering with Piecewise Linear Opacity. We propose an elegant reformulation to the sample-based rendering equation that corresponds to the exact integral under piecewise linear opacity while keeping piecewise constant color leading to a simple and closed-form expression for the integral. That is, instead of piecewise constant opacity as in Eq 5, we assume a linear opacity for each interval I j . Concretely, for s ∈ [ s j , s j +1 ] , where τ j = τ ( s j ) , τ j +1 = τ ( s j +1 ) , we have

which is linear w.r.t. s ∈ [ s j , s j +1 ] as illustrated in Fig. 3.

Now, under the piecewise linear opacity assumption, transmittance is derived as the following closed-form expression:

Together with Eq. 15, this leads to the following simple and closed-form expression for P j , corresponding to the exact integral under the piecewise linear opacity assumption:

Precision Importance Sampling. Moreover, it also turns out that with our piecewise linear opacity assumption, we are able to derive an exact closed-form solution for inverse transform sampling. Recall that in Sec 4, we pointed a drawback of the CDF ˜ F being non-invertible and discontinuous under piecewise constant opacity. We show that this is alleviated in our piecewise linear setting. Concretely, given samples s 1 , ..., s N resulting in interval probabilities P 1 , ..., P N 2 from our derivation (Eq 12), the CDF for continuous random variable t is then given as

Note that unlike in piecewise constant opacity, we do not convert a continuous random variable s to a discrete random variable ˜ s , thus, the resulting CDF F being continuous. Now, assuming that opacity τ ≥ 0 everywhere, from Eq. 13 we see that F is strictly increasing. Since F is continuous and strictly increasing, then it is invertible.

Finally, we have our precision importance sampling, where by inverse transform sampling, we can solve for the exact sample x = F -1 ( u ) for u ∼ U (0 , 1) from the given ray distribution p ( s ) since the CDF F is invertible under piecewise linear opacity. That is, without loss of generality, let sample u ∼ U (0 , 1) fall into the bin u ∈ [ C k , C k +1 ] , where C k = ∑ j<k P j , which is equivalent to solving for x ∈ [ s k , s k +1 ] . Reparameterizing x = s k + t , where t ∈ [0 , s k +1 -s k ] , the exact solution for sample u is given by

Please see supplementary for full derivation. This leads to precisely sampling from the ray distribution p ( s ) resulting in better importance sampling and stronger depth supervision.

GT

Constant

Linear

7 Results

In this section, we present our experimental evaluations to demonstrate the advantages our piecewise linear opacity formulation for volume rendering, which we call PL-NeRF .

7.1 Datasets, Evaluation Metrics and Implementation Details.

Datasets and Evaluation Metrics. We evaluate our method on the standard datasets: Blender and Real Forward Facing (LLFF) datasets as used in [18]. We use the released training and test splits for each. See supplementary for more details. For quantitative comparison, we follow the standard evaluation metrics and report PSNR, SSIM [32] and LPIPS [41] on unseen test views. We also report the root-mean-squared-error (RSME) on the expected ray termination in our depth experiments.

Implementation Details. PL-NeRF is implemented on top of NeRF-Pytorch [38], a reproducible Pytorch implementation of the constant (vanilla) NeRF, where we simply change the volume rendering to our formulation under piecewise linear opacity and utilize our exact importance sampling derivation. Similar to [18] we optimize a separate network for the coarse and fine models that are jointly trained with the MSE loss on ground truth images. We use a batch size of 1024 rays and a learning rate of 5 × 10 -4 that decays exponentially to 5 × 10 -5 throughout the course of optimization. We train each scene for 500k iterations which takes ∼ 21 hours on a single Nvidia V100 GPU 3 . Our precision importance sampling enables us to use fewer samples for the fine network, hence keeping the total number of rendering samples the same, we use 128 coarse samples and 64 fine samples to train and test our method.

7.2 Experiments on Blender and LLFF Datasets

We first evaluate our PL-NeRF on the standard Blender and Real Forward Facing datasets. Table 1 shows that our PL-NeRF (linear) outperforms the vanilla [18] (constant) model that assumes piecewise constant opacity in all metrics for both the synthetic Blender and Real Forward Facing datasets. Figure 1 and Figure 4 show qualitative results. As shown our PL-NeRF is able to achieve sharper

textures as shown in the Lego's scooper and the bread's surface in the hotdog. Moreover, our approach is also able to recover less fuzzy surfaces as shown in the microphone scene (Figure 1) where training views are close to the grazing angle of its head. As illustrated, the resulting probability density of the ray corresponding to the marked pixel is peakier than constant as our precision importance sampling allows us to have better samples closer to the surface. We also see clearer ropes in ship, less cloudy interior of the drum, and more solid surfaces such as cleaner leg of the swivel chair in the room scene.

7.3 Geometric Extraction

We also show improvement in geometric reconstruction of PLNeRF . We extract the geometry from the learned density field from the trained models of PL-NeRF and Vanilla NeRF using marching cubes with a threshold of 25 following [30]. Figure 5 shows qualitative results on the reconstruction of our piecewise linear vs the original piecewise constant formulation. As shown, we are able to better recover the holes on the body and wheels of the Lego

G

Constant

Linear

scene as well as the interior structure inside the Mic. Moreover, interestingly, the surface of the drum is reconstructed to be transparent as visually depicted in the images, as opposed to the ground truth being opaque.

GT

Mip-NeRF

PL-MipNeRF

GT

Mip-NeRF

PL-MipNeRF

7.4 Effectiveness of our formulation on other Radiance Field Methods

We also demonstrate our formulation's effectiveness on other radiance field methods and show that our approach can be used as a drop-in replacement to existing NeRF-based methods. We integrate our piecewise linear opacity formulation to the volume rendering integral into Mip-NeRF ( PL-MipNeRF ). Table 2 shows our quantitative results demonstrating consistent improvement across all scenes in the original hemisphere Blender dataset. Figure 6 shows qualitative examples where we see that under difficult scenarios such as when ray conflicts arise in the fine details of the Chair and in the presence of grazing angle views in the Mic, our PL-MipNeRF shows significant improvement over the baseline. Our results show that our piecewise linear opacity and piecewise constant color formulation scales well to Mip-NeRF as well. See supplementary for implementation details. We also plug our

piecewise linear opacity formulation into DIVeR [36], a voxel-based NeRF model, and show that our formulation is also an effective drop-in replacement outperforming the original DIVeR [36]. Please see supplementary for experiment results and implementation details.

7.5 Experiments on Close-up Views

We further consider the challenging setting of testing on cameras closer to the objects. Table 3 (top) shows quantitative results when training on the original hemisphere dataset and tested on different close-up views. As shown by the drop in metrics, the difficulty increases as the camera moves closer to the object (0.75x to 0.5x to 0.25x of the original radius) where details get more apparent. Our PL-NeRF outperforms the vanilla piecewise constant model in all settings and the gap (SSIM and LPIPS) between ours and the constant assumption increases as the setting becomes harder highlighting the importance of recovering shaper texture and less fuzzy surfaces.

We also consider the set-up of training with cameras with different distances to the object that result in different sets of ray samples causing conflicts. We generate training views following the data processing pipeline from [18] with a random distance scaling factor sampled from U (0 . 5 , 1 . 0) . As shown in Table 3 (bottom), our PL-NeRF outperforms the vanilla constant baseline in all metrics across different camera distances, where the gap (LPIPS and SSIM) is also larger the closer the camera is to the object, where details are more apparent. The difficulty for the constant (vanilla) case under multiple camera distances is its sensitivity to the choice of samples along the ray. Figure 2 shows that the conflicting rays cause quadrature instability for under piecewise constant opacity leads to unstable outputs as shown by the noisy texture on the chair. For the constant, the level of noise (gold specs) and blurriness vary at different camera distances, whereas our PL-NeRF renders crisper and more consistent outputs even as the camera is moved closer or further for the object.

7.6 Experiments with Depth Supervision

Finally, we also show that our PL-NeRF enables stronger depth supervision under our piecewise linear opacity assumption due our precision importance sampling that allows for gradients to flow to these more refined samples resulting in more accurate depth. As in previous works [28], we use a sample-based loss for to incorporate depth supervision 4 . Table 4 shows quantitative

results when training and testing on the less constrained Blender dataset with cameras at random distances from the object, as described in the previous section, with depth supervision. As shown, our PL-NeRF outperforms the vanilla constant baseline on all metrics including depth RSME demonstrating that our approach allows for stronger depth supervision.

8 Conclusion

We proposed a new way to approximate the volume rendering integral that avoids quadrature instability, by considering a piecewise linear approximation to opacity and a piecewise constant approximation to color. We showed that this results in a simple closed-form expression for the integral that is easy to evaluate. We turned this into a new objective for training NeRFs that is a drop-in replacement to existing methods and demonstrated improved rendering quality and geometric reconstruction, more accurate importance sampling and stronger depth supervision.

Acknowledgements. This work is supported by a Apple Scholars in AI/ML PhD Fellowship, a Snap Research Fellowship, a Vannevar Bush Faculty Fellowship, ARL grant W911NF-21-2-0104, a gift from the Adobe corporation, the Natural Sciences and Engineering Research Council of Canada (NSERC), the BC DRI Group and the Digital Research Alliance of Canada.

References

Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision , pages 1-16, 2016. 13

Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV , 2021. 2, 3

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR , 2022. 2, 3

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. ICCV , 2023. 2

Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In CVPR , 2023. 1

Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In CVPR , pages 12882-12891, 2022. 1, 2, 7

Robert A Drebin, Loren Carpenter, and Pat Hanrahan. Volume rendering. SIGGRAPH , 22(4):65-74, 1988. 3

Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV , pages 5875-5884, 2021. 1, 3

Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey. ArXiv , abs/2006.12057, 2020. 1

Marc Levoy. Efficient ray tracing of volume data. ACM Transactions on Graphics (TOG) , 9(3):245-261, 1990. 2, 3

Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR , pages 6498-6508, 2021. 1, 2

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. In NeurIPS , 2020. 3

Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR , 2021. 2

Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics , 1(2):99-108, 1995. 1, 2, 3, 5

[15]

Hagen, editor,

Local and global illumination in the volume rendering integral.

Scientific Visualization: Advanced Concepts

, volume 1 of

In Hans

, pages

Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan, and Jonathan T. Barron. NeRF in the dark: High dynamic range view synthesis from noisy raw images. In CVPR , 2022. 2

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics , 2019. 2, 3

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020. 1, 2, 3, 5, 8, 10, 13

Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In ICCV , 2021. 2

Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics , 40(6), dec 2021. 1

Steven M Rubin and Turner Whitted. A 3-dimensional representation for fast rendering of complex scenes. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques , pages 110-116, 1980. 3

Hanan Samet. The design and analysis of spatial data structures , volume 85. Addison-wesley Reading, MA, 1990. 3

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR , 2022. 3

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprint arXiv:2206.05085 , 2022. 3

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR , 2022. 2

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS , 33:7537-7547, 2020. 2

Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings , 2023. 28

Mikaela Angelina Uy, Ricardo Martin-Brualla, Leonidas Guibas, and Ke Li. Scade: Nerfs from space carving with ambiguity-aware depth estimates. In CVPR , 2023. 2, 6, 10, 28

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017. 2

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS , 2021. 9

Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In ICCV , 2021. 2, 3

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004. 8

Lee Westover. Interactive volume rendering. In Proceedings of the 1989 Chapel Hill workshop on Volume visualization , pages 9-16, 1989. 3

Peter L Williams and Nelson Max. A volume density optical model. pages 61-68, 1992. 3

Peter L Williams, Nelson L Max, and Clifford M Stein. A high accuracy volume renderer for unstructured data. IEEE Transactions on Visualization and Computer Graphics , 4(1):37-54, 1998. 3

Liwen Wu, Jae Yong Lee, Anand Bhattad, Yuxiong Wang, and David Forsyth. Diver: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In CVPR , 2022. 2, 10

Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In ECCV , 2022. 2

Lin Yen-Chen. Nerf-pytorch. https://github.com/yenchenlin/nerf-pytorch/ , 2020. 8

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR , pages 4578-4587, 2021. 2, 13

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. ArXiv , abs/2010.07492, 2020. 2

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , pages 586-595, 2018. 8

Appendix

We conduct further experiments, analysis and discussions on our proposed reformulation, where we take a piecewise linear approximation to opacity and piecewise constant approximation to color that results in an integral that is a simple and closed-form expression. This allows us to address the drawbacks of current piecewise constant assumption in NeRFs such as ray conflicts during optimization and a non-invertible CDF that lead to imprecise importance samples and vanishing gradients. We provide additional results in Sec A.1: ablation study (Sec A.1.1), a video demo (Sec A.1.2), additional results on a real dataset (Sec A.1.3), additional qualitative results (Sec A.1.4), comparison with PL-DIVeR A.1.5, additional geometric extraction results A.1.6 and comparison with less number of samples A.1.7. We then provide a walkthrough of the piecewise constant derivation from [15] (Sec A.2), which is followed by the thorough step-by-step derivation of our piecewsie linear opacity in volume rendering and precise importance sampling (Sec A.3). We also include analyses on piecewise quadratic and higher order polynomials in Sec A.4. Finally, we end with additional implementation and experiment details (Sec A.5), Limitations (Sec A.6) and Societal Impact (Sec A.7).

A.1 Additional Results

A.1.1 Ablation Study

We conduct a further ablation study on our precise importance sampling. As described in Sec. 4 in the main paper, our piecewise linear opacity approximation allows to solve for a closed-form solution for inverse transform sampling leading to the formulation of our precise importance sampling. Unlike the vanilla piecewise constant opacity approximation, our approach results in an invertible CDF, and hence we do not need to define an invertible surrogate function G for inverse transform sampling as in piecewise constant opacity (see Eq. 8 of main paper), which does not necessarily result in samples from the actual ray distribution p ( s ) . We quantitatively ablate the effectiveness of our precise importance sampling ( Precise ) by replacing our formulation (Eq. 13 main paper) with the surrogate function G as in the vanilla constant setting ( Surrogate ). As shown in Table A1 (first and second row of each metric), our precise importance sampling consistently outperforms using the surrogate on all metrics across all 8 scenes in the Blender dataset (hemisphere). Moreover, we also show that our precise importance sampling enables us to use fewer samples for the fine network as it is able to sample correctly from the ray distribution. Hence, keeping the same total number of rendering samples, we are able to achieve a further boost in performance by using 128 coarse and 64 fine samples as shown in Table A1 (second and third row of each metric). We use N c and N i to denote the number of coarse and fine samples, respectively.

A.1.2 Video Demo

We also show a video demo of our results. Please see our project page (pl-nerf.github.io) for the video (best viewed in full screen). In the mic scene, we observe that the structure inside the mic is lost in the piecewise constant opacity approximation. Moreover, we see the blurriness along the sides of the body of the mic caused by ray conflicts from the perpendicular and grazing angle views during optimization of the vanilla NeRF model. For the chair scene, we see that our PL-NeRF is able to achieve shaper textures on the back of the chair. Moreover, we see that there is an instability to the samples as illustrated by the inconsistencies in the noise - the gold specs on the green texture (please view in full screen). This is also evident when we vary the camera-to-scene distance. The chair model was trained on multi-distance Blender data to highlight the difference when camera distances vary.

A.1.3 Real Dataset Results

We further evaluate our PL-NeRF on a real dataset - DTU [1]. We train and evaluate our approach on the 15 test scenes used in [39], and report the standard metrics (PSNR, SSIM, LPIPS). We follow the protocol used in [18] for the Real Forward Facing scene where 1 8 of the views were held out for testing while the rest are used for training. Table A2 shows the quantitative results averaged over the 15 scenes, where our PL-NeRF outperforms the constant [18] baseline.

GT

Constant

Linear

GT

Constant

Linear

A.1.4 Additional Qualitative Results

We additionally show qualitative results from DTU dataset in Fig. A1. More qualitative results are also shown in Fig. A2.

A.1.5 PL-DiVER

We plug our method into DIVeR by using their voxed-based representation and feature integration, and dropping in our piecewise linear opacity formulation for volume rendering (PL-DIVeR). Results are shown in Table A3 demonstrating that our approach is on-par if not better across the different scenes in the Blender dataset. We highlight that this shows the improvement of using our piecewise linear opacity formulation, which is a drop-in replacement to existing methods.

A.1.6 Geometric Extraction

We also show quantitative results in geometric extraction improvement of PL-NeRF compared to the original Vanilla NeRF. Table A4 reports the distance between the surface of the ground truth model to the predicted meshes by sampling point clouds via ray casting. We see that our piecewise linear approach achieves a lower error compared to Vanilla NeRF on almost all the scenes in the Blender dataset.

A.1.7 Comparison with Less Samples

We run both our PL-NeRF and Vanilla NeRF with 64 coarse and 64 fine samples results in an average of (30.09, 0.939, 0.056) and (29.86, 0.937, 0.059) for (PSNR, SSIM, LPIPS), respectively, on the Blender dataset. This shows that under less number of samples our piecewise linear opacity formulation is better than the original piecewise constant opacity assumption.

A.2 Volume Rendering: Walkthrough of Piecewise Constant Derivation from [15]

From [15], under the approximation that both opacity and color are piecewise constant, for s ∈ [ s i , s i +1 ] , where τ i = τ ( s i ) and τ ( s ) = τ i ∀ s ∈ [ s i , s i +1 ] , the probability of the interval P i is derived as follows:

Moreover, under the piecewise constant assumption, transmittance T is derived and given by:

This is the formulation that is used in most, if not all, NeRF works, which has the drawbacks that we raised such as ray conflicts during NeRF optimization and non-invertible CDF causing imprecise importance sampling and vanishing gradients when defining a loss w.r.t. the samples. We propose to approach this issue by deriving the volume rendering equation under a piecewise linear approximation to opacity, which we detail in the next sections.

A.3 Volume Rendering: Our Piecewise Linear τ Derivation

We now show our full derivation for the volume rendering equation, under the assumption that the opacity τ ( s ) is piecewise linear, i.e. it is linear within each interval [ s i , s i +1 ] , and piecewise constant color. We then derive the probability of an interval under this assumption.

A.3.1 Generalized form for P i .

Recall the generalized form of P i as derived in the main paper. First from the definition of transmittance, we have

This results in the exact expression for the probability P i of an interval given as follows:

A.3.2 Evaluating τ ( s ) for s ∈ [ s i , s i +1 ] .

Let τ j = τ ( s i ) , τ i +1 = τ ( s i +1 ) , be sampled points along the ray. For s ∈ [ s i , s i +1 ] , assuming piecewise linear opacity τ , i.e. τ ( s ) is linear within each bin, we have

A.3.3 Transmittance T ( s i )

We first derive expression for transmittance T ( s i ) under the piecewise linear τ assumption.

Thus we get

A.3.4 Probability of interval [ s i , s i +1 ]

From the generalized form for P i as derived in the main paper, we plug in the expression for transmittance T ( s i ) as derived above to obtain:

Hence, we obtain

A.3.5 Our Precision Importance Sampling

To sample from the ray distribution, inverse transform sampling is needed, that is, one draws u ∼ U (0 , 1) then passes it to the inverse of a cumulative distribution (CDF), i.e. a sample x = F -1 ( u ) , where F is the CDF of the distribution. Unlike the piecewise constant case, where F is not invertible, needing for a surrogate function G derived from F, we show that under our piecewise linear opacity assumption, we can solve for the solution x for each corresponding u .

As illustrated in the main paper, since F is continuous and increasing, under our assumption that τ > 0 5 , then F is invertible. Now, without loss of generality, let sample u fall into the CDF interval [ c k , c k +1 ] , where c k = ∑ j<k P j . We know that the probability of the corresponding interval is P k as given by Eq. 17. Thus we have:

We want to solve for sample x ∈ [ s k , s k +1 ] , such that x = F -1 ( u ) . Equivalently, since we know that x ∈ [ s k , s k +1 ] , then we reparameterize and let x = s k + t , where t ∈ [0 , s k +1 -s k ] . We are solving for x as follows:

Now, from the derivation of the general form for P i in Eq. 15, we can similarly obtain

which gives us the expression

Thus, simplifying we get

̸

This holds for T ( s k ) = 0 , which is true under our assumption that τ > 0 , and T ( s k ) -( u -c k ) ≥ 0 , which we will show in Sec. A.3.8 below.

A.3.6 Evaluating -∫ x s k τ ( u )d u .

We evaluate the expression -∫ x s k τ ( u )d u in order to solve for the exact sample x . Recall, for s ∈ [ s k , s k +1 ] , we have

Let constants a = τ k +1 -τ k , b = s k +1 τ k -s k τ k +1 , d = 1 s k +1 -s k , thus we can write τ ( s ) as follows:

Thus, from Eq 19 we have:

Hence, plugging this in Eq. 18, we get the quadratic equation

We want to solve for t ∈ [0 , s k +1 -s k ] , and the roots of the quadratic equation is given by

That means to compute for the solution, we need to find the root

which we will show always exists and is unique.

Let us first bound the discriminant of the quadratic formula. We have

Thus we have

Hence, computing the discriminant we get:

Similarly, u -c k ≥ 0 , where equality holds when u = c k . This gives us ∆ ≥ τ 2 k .

Hence, we know that τ 2 k +1 ≥ ∆ ≥ τ 2 k . Since we need

Thus to find the solution t , we need to take the positive root. We have

This shows that the solution is within the desired interval. Hence, the solution is

Hence, for the positive root, we know that t ∈ [0 , s k -1 -s k ] .

A.3.8 Proof for T ( s k ) ≥ ( u -c k )

We know that

Let a i = exp( -( τ i + τ i -1 )( s i -s i -1 ) 2 ) , and T ( s 0 ) = 1 . Hence we have

Since T ( s k ) = a 1 a 2 ...a k and a i > 0 ∀ i then

̸

̸

Note that above proof and solutions hold for τ k = τ k +1 and T ( s k ) = 0 , which is all holds since we have τ ( s ) > 0 ∀ s , which is equivalent to F being an increasing function.

A.3.9 The solution for sample u :

Putting everything together, we have the solution t given as

From Sec A.3.8, we have T ( s k ) = a 1 a 2 ...a k and c k = 1 -a 1 a 2 ...a k , thus T ( s k ) -( u -c k ) = 1 -u . Hence we can simplify it to

A.4 Piecewise Quadratic and Higher Order Polynomials

Now, we first consider the full derivation for volume rendering equation under the assumption that opacity is piecewise quadratic and color is piecewise constant. Consider opacities τ 1 , . . . , τ n queried at n samples s 1 , . . . , s n along the ray. Here, we set s 0 = t n and s n +1 = t f to the near and far plane, with τ 0 = 0 and τ n +1 = 10 10 denoting empty and opaque space.

To interpolate opacity, because a quadratic function can only be uniquely defined with 3 points, we choose τ ( s ) to be quadratic within each interval [ s j , s j +2 ] for even values of j . To encapsulate all points, this forces n to be odd , e.g. using 127 coarse samples and 64 fine samples.

A.4.1 Derivation: Computing T ( s )

In the same way as Sec. A.3.3, we can derive transmittance, which is in closed-form. The only modification is that the formulae for integrals of opacity are different over left and right subintervals [ s j , s j +1 ] and [ s j +1 , s j +2 ] , with j even (see the next section for a derivation of these integrals):

Substituting the expressions in Eqs. 28 and 29 gives a closed form expression in terms of the s j 's and τ j 's. We can similarly compute

Then the probability of the i th interval for each 0 ≤ i ≤ n is, as before,

This leads to a closed-form expression for P i . This means the behavior of P i depends entirely on that of the opacity integral. However, due to the form of ∫ τ ( s ) ds detailed in the next section, this means P i involves a piecewise exponential of a rational function in τ j 's and s j 's , which leads to poor numerical conditioning and thus optimization instability.

A.4.2 Derivation: Computing and Integrating τ ( s ) on Intervals

We now compute τ ( s ) and its integral on each interval. Fix the interval [ s j , s j +2 ] , with j odd, and τ j = τ ( s j ) , τ j +1 = τ ( s j +1 ) , τ j +2 = τ ( s j +2 ) . By Lagrange interpolation , the quadratic τ ( s ) passing through ( s j , τ j ) , ( s j +1 , τ j +1 ) , ( s j +2 , τ j +2 ) is given by:

Note the integrals of the three monic quadratics over [ s j , s j +1 ] can be expressed in terms of α j , β j , γ j :

Thus, using Eq. 27 gives the integral of opacity over [ s j , s j +1 ] :

Similarly, the integrals of those same three quadratics over [ s j +1 , s j +2 ] factor out β j :

Then, summing these up along Eq. 27 gives the integral of opacity over [ s j +1 , s j +2 ] :

Observe in Eqs. 28 and 29 that the integral of opacity involves some terms with α j , β j , γ j in the denominator, which do not cancel . Thus, the integral of opacity over an interval is not a polynomial in τ i 's and s i 's , but is instead a rational function . This contrasts with the linear derivation in Sec. A.3.3, where this integral was a degree 2 multivariate polynomial in τ i 's and s i 's. This caveat causes numerical instability, which will be discussed further in Secs. A.4.3 and A.4.4.

Generally, following the steps of the above derivation shows that if we interpolate τ piecewise by any degree d polynomial, d ≥ 2 , then the result is a rational function in s i 's and τ i 's, but not a polynomial, which would also lead to training instability as in the quadratic case.

A.4.3 Piecewise Quadratic Problem 1: Negative Interpolated Opacity

As seen above, one problem with the piecewise quadratic model is that the integral of opacity is a rational function in α i , β i , γ i . In particular, due to the presence of negatives in front of certain rational terms in Eqs. 28 and 29, it may become negative as the denominators of these terms approach zero. One example is shown in Figure A.4.3, for samples s 1 , s 2 , s 3 with τ 1 = τ 2 < τ 3 and s 3 -s 2 ≪ s 2 -s 1 , where the interpolated quadratic dips far below the x -axis. Note this interpolation is physically implausible, as opacity should be nonnegative everywhere .

Furthermore, from Eqs. 24 and 25, when the opacity integral is negative on intervals, transmittance can then be a product with exponentials of negative terms, and potentially be greater than 1 . This is incompatible with the physical interpretation of transmittance as a probability that a light travels a distance along a ray without being absorbed.

A.4.4 Piecewise Quadratic Problem 2: Instability from Sample Proximity

In Eqs. 28 and 29, we observe the presence of terms such as α j , β j , γ j in the denominator. These do not appear in the piecewise linear model. As a result, because these quantities approach zero as samples s j , s j +1 , s j +2 become closer, then the integral of opacity can become an arbitrarily large positive or even negative (as shown qualitatively in the previous section). Referencing the transmittance formulae in Eqs. 24 and 25, this means transmittance can approach zero or infinity and gradients can explode as samples are clustered together. This is fairly common in stratified sampling, and even more so with importance sampling.

To formally describe this instability, we first look at what happens when s j and s j +1 coincide:

because τ : R ≥ 0 → R is continuous. There is no instability in this integral when s j +1 → s + j . However, when s j +1 and s j +2 coincide, because β j → 0 + and γ j → α + j , the integral approaches

The behavior of this integral limit depends on the last limit. To analyze this, let h : R ≥ 0 → R be the network function which takes in samples on the ray and outputs opacity. The last limit is

Because h is almost always differentiable at s j +1 , this becomes h ′ ( s j +1 ) , and so

There are no constraints on the value of h ′ ( s j +1 ) as the network trains, so the above limit can achieve any real value. In particular, we can derive a condition for when the limit is negative:

That is, whenever there is a sharp enough increase in opacity (which can happen at, say, a surface crossing) and samples s j +1 , s j +2 are sufficiently close, the interpolated quadratic can have a negative integral on [ s j , s j +1 ] , which leads to the issues described in Sec. A.4.3. In other words, the integral of opacity over [ s j , s j +1 ] is unstable as s j +2 → s + j +1 . Asimilar analysis holds to show the instability of the opacity integral on [ s j +1 , s j +2 ] as s j +1 → s + j .

A.4.5 Piecewise Quadratic Problem 3: Importance Sampling

Suppose we wish to importance sample in the same manner as piecewise linear, that is, we use inverse transform sampling. In essence, we draw u ∼ U (0 , 1) and then sample x = F -1 ( u ) , with F the CDF of the distribution. Recall F is computed as

As before, the latter equation becomes

Hence, computing x amounts to solving the equation f ( x ) = u , which becomes from above:

As T is the exponential of a cubic, this amounts to solving a cubic above. This does have a real solution, as F is increasing and continuous, so the Intermediate Value Theorem implies f ( x ) = u has a solution; and the solution is unique because F is strictly increasing, as it is an exponential of a polynomial. However, the complexity of exact importance sampling would be large.

This analysis reveals another downside of higher order polynomials. In general, suppose we wish to interpolate opacity τ with a piecewise degree n polynomial. Then following the same method as above, we see transmittance T is the piecewise exponential of a degree n +1 polynomial, which is the integral of τ . So inverse transform sampling reduces to solving a degree n +1 polynomial. The Abel-Ruffini Theorem asserts for n +1 ≥ 5 that this polynomial is in general not solvable by radicals. In other words, there is no simple closed form for exact importance sampling when n ≥ 4 .

Theoretically, this could be exactly solved for n = 2 and n = 3 (i.e. when opacity is piecewise quadratic or cubic, respectively), but the formulae to derive cubic and quartic solutions can become sufficiently complicated and the resulting complicated expressions may result in numerical instability during optimization, especially when taking the gradient w.r.t. the samples.

A.5 Experiment details, Reproducibility and Compute

A.5.1 Implementation Details

We include the core code snippets of our implementation of PL-NeRF . Figure A4 shows the volume rendering equation that includes the implementation of Eq. 10 and 11 (main paper). This is a direct replacement of the original constant approximation, where we also show the code snippet in the figure as reference. Figure A5 shows the implementation of our precise importance sampling from Eq. 13 (main paper), which is also a direct replacement of the constant importance sampling implementation (Figure A6) for reference. We highlight that our formulation is a direct replacement of the functions from the original implementation, and hence for the depth experiments, we are also able to directly adapt the codebase from [28]. We use Nvidia v100 and A5000 GPU's for our experiments. Each scene is trained on a single GPU and takes 15 -20 hours. We used an internal academic cluster and cloud compute resources to train and evaluate our models.

For the MipNeRF-based experiments, our experiments are also run on the standard train and test split of the Blender dataset with the official released hyperparameters of Mip-NeRF using the NerfStudio [27] codebase. For PL-MipNeRF, we use the two-MLP training scheme with a coarse loss weight of 1.0.

For DIVeR-based experiments, we use their official implementation and configuration for DIVeR64 at 128 voxels. For PL-DIVeR, we utilize their voxel-based representation and feature integration and dropping in our piecewise linear opacity rendering formulation. We similarly run the DIVeR models on a single Nvidia v100 GPU trained using their default configurations and hyperparameters for DIVeR64 at 128 voxels.

The total training time for 500k iterations on a single Nvidia v100 GPU is 17.78 and 21.43 hours for Vanilla NeRF and PL-NeRF, respectively. Figure 2-c shows the head-to-head comparison of training PSNR (y-axis) with respect to time (x-axis) of Vanilla NeRF vs PL-NeRF on the Lego scene. Rendering a single 800x800 image takes 25.59 and 32.35 seconds for Vanilla NeRF and PL-NeRF, respectively.

A.5.2 Computational complexity under different number of samples

We measured the total rendering time for a single 800x800 image under different numbers of samples for our PL-NeRF. The total rendering time for (64+64), (64+128) and (128+64) are 19.20, 25.85 and 32.35 seconds, respectively.

A.5.3 Convergence plots under different number of samples

Figure ?? shows the convergence plots under different numbers of samples of our PL-NeRF vs Vanilla NeRF. We see that under different number of samples, our linear approach converges to a higher training PSNR.

A.6 Limitations

Our piecewise linear opacity approximation is able to handle arbitrarily small opacity, e.g. 1 e -6 , however, we cannot handle coordinates with exactly zero opacity. This special case is not an issue in practice. Also theoretically, any atom absorbs light and thus will not have exactly zero opacity, except in a vacuum. Another limitation is our method is slightly slower than the original piecewise constant approximation, and this is due to requiring more FLOPS for our importance sampling computation (Eq. 13 main paper). Another additional limitation is we still assume piecewise color, i.e. within a bin we do not handle color integration. Modeling this can potentially handle difficult scenarios such as double-walled colored glass or atmospheric effects such as fog or smoke. Lastly, we also inherit the limitations of NeRFs in general, such as requiring known camera poses.

A.7 Broader Impact

Our models require the usage of GPUs both in training time and rendering time, and GPUs use up energy to run and power them. We acknowledge that this contributes to climate change that is an important societal issue. Despite this, we observe the improvement in the results that is theoretically

grounded and believe that it is beneficial for the pursuit of science. We responsibly ran our models by first prototyping on selected scenes before scaling up to different scenes across datasets to minimize its impact to climate change.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes a modification to the way NeRF performs approximate integration for volume rendering.  Instead of assuming piecewise constant opacity, they propose to use piecewise linear opacity.  They argue that this resolves what they dub the \"quadrature instability\" in NeRF.  They re-derive the quadrature integration method using piecewise linear opacity and piecewise constant color, and show that this also enables them to derive a precise inverse of the ray termination CDF for importance sampling.  Through experiments on standard NeRF datasets they show an quantitative improvement in rendering quality and also show qualitatively that their method makes the rendering sharper and more stable across change of viewpoint and camera distance.\n\nSTRENGTHS:\nThis paper addresses an often overlooked aspect of NeRF which is the specifics of how the volume rendering is implemented and whether it could be improved.  The propose what to the best of my knowledge is a novel modification, which is to approximate the opacity along the ray as piecewise linear (instead of piecewise constant) in order to obtain a more accurate quadrature estimate of integral.  This formulation also leads them to derive a more precise method of importance sampling.\n\nThey analyze conceptually how the piecewise constant assumption in NeRF leads to conflicting ray supervision between perpendicular and grazing angle rays.  They claim that their piecewise linear formulation reduces this problem and leads to a more peaked opacity PDF.\n\nThey provide extensive derivations for the formulas in the supplemental material.  The new formulations are a \"drop-in\" replacement for the equations used in vanilla NeRF, and thus they can easily modify existing NeRF implementations to use them.\n\nIn their evaluation they compare against vanilla NeRF on the standard synthetic and real forward-facing datasets and show a quantitative and qualitative improvement in rendering quality across the board.  \n\nThey also show that their method improves SCADE, a recent method that incorporates depth estimates into NeRF training, and show that swapping in their formulas leads to a modest improvement in quality.  They hypothesize that this is because of their more precise importance sampling.\n\nWEAKNESSES:\nThey seemed to have missed a highly relevant previous work:\n\nWu, L., Lee, J. Y., Bhattad, A., Wang, Y. X., & Forsyth, D. (2022). DIVeR: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16200-16209).\n\nThis method is based on a voxel grid approach rather than an MLP.  However, the DIVeR paper also discusses the inaccuracy of integration using piecewise constant opacity.  Their solution allows for integration over more complicated functions than piecewise linear and so might be more accurate than the piecewise linear approach proposed here.  Indeed, DIVeR reports higher average rendering quality metrics on the NeRF synthetic dataset than what is reported here.  \n\nBecause they claim that their method improves sampling and makes the opacity function more peaked at the surface, I would be very curious to see a comparison of surface reconstruction quality compared to NeuS:\n\nWang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., & Wang, W. (2021). NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. Advances in Neural Information Processing Systems, 34, 27171-27183.\n\nThe explanation of inverse transform sampling in the original NeRF method (section 3.2) should be expanded, as this is not explained thoroughly in the original NeRF paper.  Their discussion (L175-178) is quite brief considering how critical it is to the paper.\n\nThe paper has many grammatical errors and incomplete sentences which make it difficult to read.  I included some examples here:\n* L135 \"Hence the continuous probability density function (PDF) the ray r(s)\"\n* L138 \"s is a point on the ray r\" -> it doesn't make sense that s would be a point (since it is a scalar)\n* L152-153 \"P_j is the probability of each interval, which is mathematically equivalent to the probability of the interval.\" This sentence sounds like a meaningless tautology.\n* L154 missing a period\n* L117 \"Then taking x = g^-1(u).\" This is an incomplete sentence.\n* L117-118 \"However, this does not necessarily result in the samples from the actual ray distribution p(s) from the model.\"  This sentence doesn't work grammatically.  Some more analysis of why this doesn't result in sampling from the distribution would be helpful (explanations and visualizations).\n* L203 \"such an example of a sample-based loss used for NeRFs is depth\" \n* L232-3 \"we pointed the drawback\"\n* L238 \"thus the resulting CDF F being continuous\"\n* Table 1 caption: needs a period\n* L273 capitalize Lego\n\nQUESTIONS:\nI think the most critical issue is the comparison with DIVeR.  How does this method compare to DIVeR, which appears to support an even more accurate computation of the integral?\n\nI would also be interested to know whether they expect that this method will improve the surface reconstruction capability of NeRF, and how it compares to NeuS in this regard.\n\nFinally I would like to know if they have a reference regarding the original NeRF's method of inverse transform sampling the CDF -- is this just based on the NeRF code or is there a reference for that?\n\nLIMITATIONS:\nA section on limitations does not appear in the paper.  A frank discussion of the limitations of the approach would strengthen the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nNeural rendering methods like nerf relies on the integration of contributions (both color and density) along rays to predict views. Since analytical computation of the integral is not possible (the color and the density being both prediction from neural networks, usually MLPs), an approximation is estimated. Traditionally, the hypothesis done is that the color and the density are locally constant thus leading to a rectangle rule approximation. In this work, the authors propose to replace the the constant approximation of the density by a linear approximation. This allows for a more accurate estimation and a closed-form formula for sampling points during training.\n\nSTRENGTHS:\nThe proposed modification is simple and elegant, it is mathematically driven leading to a closed-form formula. Moreover it is plug-and-play with most (all?) popular nerf-based methods and have a direct impact on the quality of the density estimation. It offers an important solution to the fuzzyness problem of the density learned by nerf methods, thus allowing for better surface reconstruction, and especially smaller details.\n\nThe paper is clear and well written. It introduces well all the different concepts necessary to understand the contribution. \n\nExperimental results are convincing with a non-negligible  improvement of 0.5dB in PSNR but also across other metrics (SSIM and LPIPS) on multiple classic datasets (one synthetic and two real).\n\nWEAKNESSES:\nWhile the paper is very clear and the ideas elegant, I find the experimental section lacking:\n\n* The proposed modification should impact mostly the density and thus the geometry of the reconstruction. The experimental section only focuses on a. While this is can be sufficient, I find disappointing that the authors did not take the opportunity to show the potential improvement of the geometry for the different datasets showed in the paper.\n\n* The choice of baseline nerf is surprising. Given the type of experiments (with large changes of camera-to-scene distances), I think that mip-nerf would have made for a much better baselines. It already fixes a large number of artifacts pointed out by the authors in Figure S1 and therefore would have been a more appropriate comparison. I nonetheless expect that the conclusions can be transcribed to more recent frameworks.\n\n* I find surprising that there is barely any discussion about the computation complexity of the proposed sampling (nor time measurements). I saw later that it is mentioned in a \"Limitations\" section in the supplementary but for me it's not a limitation per say and it should be studied in an ablation inside the paper. Especially since S1.1 shows that a better sampling can provide interesting convergence properties, thus allowing to change the number of samples used during the training. I would have expected a comparison with less samples than the 64 + 128 baseline (64 + 64 seems like an obvious thing to try based on the comments inside the text, I'm surprised that authors didn't even try that), to see the impact and find maybe a better compromise between computation time and performance. looking at the convergence plots would have also been interesting. Indeed, the proposed method might offer a faster convergence compared to the previous sampling, thus having a double impact.\n\n* While slightly outside of the scope of this paper, it still would have been interesting to show the impact on the more practical methods (instant-ngp or voxel based methods for example) that are often preferred because of their fast training speed.\n\nQUESTIONS:\nRFF are more often referred to as LLFF. I suggest renaming to avoid any confusion (even though LLFF technically refers to the method proposed in the same paper) since a reader would expect that name.\n\nLIMITATIONS:\nGiven the nature of the contributions, there's no major limitations nor potential negative societal impact to discuss. Some limitations are mentioned in the supplementary material but they are more related to the analysis of the method as I mentioned in \"Weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a fix for the \"quadrature instability\" arising from sampling of points for numerical integration used by NeRF2020 and its successors.The sampling inconsistence may not be big on classical rendering techniques, but can be significant when using neural architectures. Opacity and color values are assumed to be piecewise constant by NeRF. This paper offers a tractable solution with piecewise linear opacity values and piecewise constant color values. To avoid dense sampling the NeRF and follow up use a importance sampling approach based on corse-to-fine strategy. For drawing these importance samples the NeRF utilizes a surrogate CDF as original CDF is not invertible. The impact is shown on several standard scenes on difficult situations such as close to surfaces, fine feature areas, etc.\n\nSTRENGTHS:\nThe proposed method is effective and efficient, drawing on existing volume-rendering literature. The modification suggested is rather simple but effective. The modified method should improve the quality of all Radiance Fields methods and hence of interest to researchers working on those areas.\n\nWEAKNESSES:\nThe paper proposes a small, clean formulation of ray-integration that improves quality of NeRF reconstruction. The limited nature of the idea is its weakness, if it can be called so. The improvements in quantitative measures like PSNR is marginal, but the qualitative improvements are significant in particularly difficult situations. The usefulness and utility of the formulation could be established more clearly by incorporating it to other NeRF variants whose code is available. I would wholeheartedly support this idea in a vision conference like ICCV/CVPR. It is upto NeurIPS to decide if such a small-but-effective idea is of sufficient interest to its audience.\n\nQUESTIONS:\nHere are some questions/points:\n- Can this method be integrated into all the other Radiance Field methods that essentially use the same ray-tracing formulation? Has it been done? That will increase its appeal\n- What is the computational overhead of the expanded formulation compared to vanilla NeRF? The paper mentions 18 hours for 500K iterations. I am interested in knowing about head-to-head comparison with vanilla NeRF to know if there is significant performance penalty in using this formulation. I would like to know it for both training and rendering for view generation.\n- Just curious: What is the adverse impact of using piecewise constant color values? What could the advantage be if we use linear color also? Does the other combination of piecewise constant density and piecewise linear color have similar closed-form solutions? What would the visual impact of that combination?\n- Are comparisons with later NeRF such as ZipNeRF/MipNeRF available? It would be interesting to see if the new proposed strategy scales well to those as well.\n- Depth supervised experiments seem valid. Does this pan-well against works like RGBDNeRF (Yuan et al. 2022 TPAMI)?\n \n\n\nMinor points:\n- Citations appear in ordinary parentheses as \"(25)\" instead of the more standard \"[25]\". This was very confusing in the beginning. Is this OK with NeurIPS style?\n- Line 164: Isn't it more correctly R^3 x S^2 --> R^3? Why discretize it to [0-255]? Are you doing something different here?\n\nLIMITATIONS:\nWhat are the limitations of this approach in the authors' views? None are mentioned in the paper.\n\nShowing the formulations effectiveness on other Radiance Field recovery methods (they all use the same rendering equation) will enhance the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper addresses a fundamental limitation of existing NeRF formulation, namely piecewise constant integration, which results in sensitivity to point sampling, summarized as quadrature instability in this paper. To address this, the paper proposes a new formulation based on piecewise linear quadrature for density and piecewise constant quadrature for color. The paper demonstrates that the proposed method produces sharper and more stable results, and leads to better quantitative metrics.

STRENGTHS:
The paper clearly analyzes the problem of existing methods.

The proposed solution is theoretically sound.

The proposed method leads to improvements in NeRF reconstruction, both qualitatively and quantitatively. 

The paper shows detailed theoretical analysis and visual comparisons in supplementary material and video.

WEAKNESSES:
The improvements appear to be quite small from the overall quantitative metrics.

It seems that the improvements may be more visible in specific scenarios, but may be subtle for general inputs.

QUESTIONS:
Is visual difference obvious for a random input, or if the example shown needs to be carefully chosen?

LIMITATIONS:
The paper does not discuss its limitations. It may happen that the method is robust, but some limitations (including limited improvements) in certain scenarios should still be discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07