SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mikaela Angelina Uy<sup>1</sup> George Kiyohiro Nakayama<sup>1</sup> Guandao Yang<sup>1,2</sup>
Rahul Krishna Thomas $^{1}$  Leonidas Guibas $^{1}$  Ke Li $^{3,4}$
$^{1}$ Stanford University  $^{2}$ Cornell University  $^{3}$ Simon Fraser University  $^{4}$ Google {mikacuy, w4756677, guandao, rt03mas, guibas}@stanford.edu, keli@sfu.ca
Neural radiance fields (NeRF) rely on volume rendering to synthesize novel views. Volume rendering requires evaluating an integral along each ray, which is numerically approximated with a finite sum that corresponds to the exact integral along the ray under piecewise constant volume density. As a consequence, the rendered result is unstable w.r.t. the choice of samples along the ray, a phenomenon that we dub quadrature instability. We propose a mathematically principled solution by reformulating the sample-based rendering equation so that it corresponds to the exact integral under piecewise linear volume density. This simultaneously resolves multiple issues: conflicts between samples along different rays, imprecise hierarchical sampling, and non-differentiability of quantiles of ray termination distances w.r.t. model parameters. We demonstrate several benefits over the classical sample-based rendering equation, such as sharper textures, better geometric reconstruction, and stronger depth supervision. Our proposed formulation can be also be used as a drop-in replacement to the volume rendering equation for existing methods like NeRFs. Our project page can be found at pl-nerf.github.io.
The advent of neural radiance fields (NeRF) [18] has sparked a flurry of work on neural rendering and has opened the way to many exciting applications [5, 11, 8, 20]. One of the key underpinnings of NeRF is volume rendering [14] – it is especially well-suited to end-to-end differentiable rendering [9], since the rendered image is a smooth function of the model parameters. This has made it possible to learn the 3D geometry and appearance solely from a 2D photometric loss on rendered images.
In volume rendering, the rendered colour  $\hat{y}$  for every pixel is an expectation of the colours along the ray cast through the pixel w.r.t. the distribution over ray termination distance  $s$  [6].
$$
\hat {y} = \mathbb {E} _ {s \sim p (s)} [ c (s) ] = \int_ {0} ^ {\infty} p (s) c (s) \mathrm {d} s \tag {1}
$$
where  $p(s)$  denotes the probability density function (PDF) of the distribution over ray termination distance  $s$  and  $c(s)$  denotes the colour as a function of different points along the ray.
In general,  $p(s)$  and  $c(s)$  can be of arbitrary forms, so evaluating this integral analytically is not possible. Therefore, in practice,  $\mathbb{E}_{s\sim p(s)}[c(s)]$  is approximated with quadrature. The quadrature formula that is most commonly used in the NeRF literature takes the following form:
$$
\mathbb {E} _ {s \sim p (s)} [ c (s) ] = \sum_ {j = 0} ^ {N} T _ {j} \left(1 - e ^ {- \tau_ {j} \left(s _ {j + 1} - s _ {j}\right)}\right) c _ {j}, \tag {2}
$$
where  $T_{j} = \exp \left(-\sum_{k=0}^{j} -\tau_{k}(s_{k+1} - s_{k})\right)$  and  $\tau_{k}$  is the opacity evaluated at a sample  $s_{k}$  along the ray.
This expression is derived from the exact integral under a piecewise constant assumption to the opacity and colour along the given ray [14].
However, this seemingly simple, innocuous assumption can result in the rendered image being sensitive to the choice of samples along the ray at which the opacity  $\sigma(s)$  and colour  $c(s)$  are evaluated. While this does not necessarily cause a practical issue in classical rendering pipelines [14, 15, 10], it has surprising consequences when used in neural rendering. Specifically, because the opacity at all points within the interval between two samples is assumed to be the same, there is a band near the surface of the geometry where opacity at all points within the band is as high as points on the surface itself. Because different rays cast from different cameras can pass through this band at different angles and offsets, both the number and the positions of samples within this band can be very different across different rays (see Fig 1 and Fig 2 for two example scenarios). Hence, simultaneously supervising these rays to produce the same colour values as in the real image captures can give rise to conflicting supervisory signals, which can result in artifacts like fuzzy surfaces and blurry texture.
Moreover, because the piecewise constant opacity assumption gives rise to a closed form expression that is equivalent to an expectation w.r.t. a discrete random variable, it is common practice in the NeRF literature to draw samples along the ray from the discrete distribution [17]. This is commonly used to draw importance samples, and also to supervise the samples along the ray [28], for example in losses that penalize deviation of the samples from the true depth [6]. Sampling from the discrete distribution requires the definition of a continuous surrogate function to the cumulative distribution function (CDF) of the discrete random variable, which unfortunately yields imprecise samples. As a result, samples that are drawn may not be close to the surface even if the underlying probability density induced by the NeRF is concentrated at the surface. Additionally, individual supervision cannot be provided to each sample drawn from the surrogate, because the gradient of the loss w.r.t. each sample would be almost zero everywhere.
All these issues, i.e. conflicting supervision, imprecise samples and lack of supervision on the CDF from the samples, stem from the assumption that opacity is piecewise constant causing the sensitivity to the choice of samples both during rendering and sampling. We dub this problem as quadrature instability. In this paper, we revisit the quadrature used to approximate volume rendering in NeRF and devise a different quadrature formula [14] based on a different approximation to the opacity. We first show that interestingly a closed-form expression can be derived under any piecewise polynomial approximation for opacity. When the polynomial degree is 0, it reduces to the piecewise constant opacity as in existing literature, and when the degree is 2 or more, we show that it would lead to poor numerical conditioning
Therefore, we further explore a degree of 1 (i.e., piecewise linear) and show that it both resolves quadrature instability and has good numerical conditioning. We derive the rendering equation under piecewise linear opacity explicitly and show that it has a simple and intuitive form. This results in a new quadrature method for volume rendering, which can serve as a drop-in replacement for existing methods like NeRFs. We demonstrate that this reduces artifacts, improves rendering quality and results in better geometric reconstruction. We also devise a new way to sample directly from the distribution of samples along each ray induced by NeRF without going through the surrogate, which opens the way to a more refined importance sampling approach and a more effective method to supervise samples using depth.
NeRFs. Neural Radiance Field (NeRF) is a powerful representation for novel-view synthesis [18] that represents a scene using the weights of an MLP that is rendered by volumetric rendering [14]. A key finding to the success of NeRF was the use of positional encoding [26, 29] to effectively increase the capacity of the MLPs that models the opacity and emitted color as a function of a 3D coordinate and viewing direction. Many works extend NeRF such as handling larger or unbounded scenes [40, 3, 37, 25], unconstrained photo collections [13], dynamic and deformable scenes [11, 19] and sparser input views [6, 39, 31, 28]. There are a number of papers that aim to improve the rendering quality of NeRF. Some do so by utilizing different kinds of supervision such as NeRF in the Dark [16], while others tackle this by improving the model [2, 36, 4]. MipNeRF [2] changes the model input by introducing integrated positional encoding (IPE) to reduce the aliasing effect along the xy coordinates. DiVeR [36] predicts the rendered colour within a line interval directly from a trilinearly interpolated feature in a voxel-based representation ZipNeRF [4] modifies the proposal network to a grid enabling it to be used together with IPE. In contrast, our work focuses on changing the objective function by modifying the rendering equation from piecewise constant opacity to piecewise linear, while keeping the model and supervision fixed. Additionally, ZipNeRF [4] also
brings up a model specific issue on z-aliasing, where their model struggles under this setting. Similar to z-aliasing observed by ZipNeRF, we consider the setting of having conflicting supervision when presented with training views at different distances from the scene. While they may appear similar on the surface, the phenomena we study is different in that it is general and independent of the model, on having conflicting ray supervision from camera views, e.g. different camera-to-scene distances and the grazing angle setup.
Importance Sampling on NeRFs. Densely sampling and evaluating NeRF along multiple points in each camera ray is inefficient. Inspired by an early work on volume rendering [10], prior works typically use a coarse-to-fine hierarchical sampling strategy where the final samples are obtained by importance sampling of a coarse proposal distribution [18, 2, 3, 8]. These importance samples are drawn using inverse transform sampling where a sample is obtained by taking the inverse of the cumulative density function (CDF) of the proposal ray distribution. However, prior NeRF works that assume piecewise constant opacity result in a non-invertible CDF, and instead introduce a surrogate invertible function derived from the CDF in order to perform importance sampling. In contrast, our work that utilizes a piecewise linear opacity assumption results in an invertible CDF and a closed-form solution to obtain samples with inverse transform sampling. Other works also attempt to alter sampling using neural networks or occupancy caching [31, 23, 24, 12]. These techniques are orthogonal to our work as they propose changes to the model as opposed to our work where importance sampling is derived from a given model.
Volume rendering. Volume rendering is an important technique in various computer graphics and vision applications as explored in different classical works [10, 33, 7, 14]. These works include studying ray sampling efficiency [10] and data structures, e.g. octree [22] and volume hierarchy [21] for coarse-to-fine hierarchical sampling. The crux behind volume rendering is the integration over the weighted average of the color along the ray, where the weights is a function of the volume density (opacity). Max and Chen [14, 15] derive the volume rendering equation under the assumption of piecewise constant opacity and color, which NeRF [17] and its succeeding works use to learn their neural scene representation. However, the piecewise constant assumption results in rendering outputs that are sensitive to the choice of samples as well the non-invertible CDF introducing drawbacks to NeRF training. Following up on [14], works also [34] derive the volume rendering equation under the assumption that both opacity and color are piecewise linear that yield unwieldy expressions that lead to numerical issues and/or are expensive to compute. Some earlier works on rendering unstructured polygonal meshes that attempt to use this model [35], but it is in general not commonly used in practice due to the mentioned issues and hence has yet to be adopted into learning neural scene representations. In this work, we address both sets of issues that arise from the piecewise constant opacity and color and piecewise linear opacity and color by reformulating the volume rendering equation to assume piecewise linear opacity and piecewise constant color. Our derivation results in a simple and closed-form formulation to volume rendering making it suitable for NeRFs.
3.1 Volume Rendering Review
**Definitions.** In classical literature, the process of volume rendering [15] mapping a 3D field of optical properties to a 2D image and the visual appearance is computed through the exact integration of these optical properties along the viewing rays. In this optical model, each point in space is an infinitesimal particle with a certain opacity  $\tau$  that emits varying amounts of light, represented as a scalar color  $c$ , in all viewing directions. The opacity  $\tau$  is the differential probability of a viewing ray hitting a particle – that is for a viewing ray  $\mathbf{r}(s) = \mathbf{o} + s\mathbf{d}$ , where  $\mathbf{o}$  is the view origin and  $\mathbf{d}$  is the ray direction, the probability of ray  $\mathbf{r}$  hitting a particle along an infinitesimal interval  $\mathrm{ds}$  is  $\tau(\mathbf{r}(s))\mathrm{ds}$ . Moreover, the transmittance  $T_{\mathbf{r}}(s)$  is defined as the probability that the viewing ray  $\mathbf{r}$  travels a distance  $s$  from the view origin without terminating, i.e. without hitting any particles.
Continuous probability distribution along the ray  $\mathbf{r}$ . As illustrated by Max and Chen [15], the probability of hitting a particle  $s + ds$  only depends on the probability of hitting a particle at  $s$  and not any particles before it the probability of ray  $\mathbf{r}$  terminating at distance  $s$  is given by  $\tau(\mathbf{r}(s)) T_{\mathbf{r}}(s)$ , where  $T_{\mathbf{r}}(s) = \exp(-\int_0^s \tau(\mathbf{u}) \, \mathrm{d}u)$ . Hence the continuous probability density function (PDF) of ray  $\mathbf{r}(s)$ , which describes the likelihood of a ray terminating and emitting at  $s$ , is given by
$$
p (s) = \tau (\mathbf {r} (s)) T _ {\mathbf {r}} (s), \tag {3}
$$
Figure 1: Ray Conflicts: Grazing Angle. (Left) Illustration of conflicting ray supervision at the grazing under the piecewise constant opacity. For the constant setting, to render perpendicular rays (yellow) correctly, the model has to store the associated optical properties at a region in front of the surface as a sample takes the values of the left bin boundary. In the presence of a ray near the grazing angle, it will be crossing this region of high opacity (the gradient in front of the surface), associating it with conflicting opacity/color signals. (Middle) This results in fuzzier surfaces as shown along the side of the microphone as there is a conflict in ray supervision between the perpendicular and grazing angle rays. Our piecewise linear opacity assumption alleviates this issue and results in a clearer rendered view. (Right) As shown, the resulting PDF is peakier and the CDF is sharper for our linear setting, where the plotted distributions correspond to the ray from the marked pixel in red.
where  $s \in [0, \infty]$  and  $\mathbf{r}(s)$  is a point on the ray  $\mathbf{r}$ . For notational simplicity we omit  $\mathbf{r}$  and write it as  $p(s) = \tau(s) T(s)$ .
Volume Rendering as a Continuous Integral. The observed color of the ray is then the expected value of the colors  $c(s)$  of all particles  $s$  along the ray weighted by the probability of hitting them. Mathematically, this results in the following continuous integral<sup>1</sup>:
$$
\mathbb {E} _ {s \sim p (s)} [ c (s) ] = \int_ {0} ^ {\infty} p (s) c (s) \mathrm {d} s = \int_ {0} ^ {\infty} \tau (s) T (s) c (s) \mathrm {d} s. \tag {4}
$$
Quadrature under Piecewise Constant Opacity  $\tau$ . Since this integral cannot in general be evaluated analytically, it is approximated with quadrature. Let  $s_1, s_2, \ldots, s_N$  be  $N$  (ordered) samples on the ray that define the intervals, where  $I_j = [s_j, s_{j+1}]$  is the  $j^{\text{th}}$  interval, and  $I_0 = [0, s_1]$ ,  $I_N = [s_N, \infty]$ . The volume density for particles along the interval  $I_j$  is then approximated under the assumption that opacity is constant along each interval, making it piecewise constant along the ray [15]. That is, for all  $j$  we have:
$$
\forall s \in \left[ s _ {j}, s _ {j + 1} \right], \tau (s) = \tau \left(s _ {j}\right), \tag {5}
$$
for brevity we denote  $\tau(s_j) = \tau_j$ , i.e.  $\tau_j$  is the opacity for sample  $s_j$ . Under this piecewise constant opacity assumption, the volume rendering equation Eq. 4 then becomes as follows:
$$
\mathbb {E} _ {s \sim p (s)} [ c (s) ] = \sum_ {j = 0} ^ {N} P _ {j} c _ {j} = \sum_ {j = 0} ^ {N} \left(\int_ {s _ {j}} ^ {s _ {j + 1}} \tau (u) T (u) \mathrm {d} u\right) c _ {j} = \sum_ {j = 0} ^ {N} T _ {j} \left(1 - e ^ {- \tau_ {j} \left(s _ {j + 1} - s _ {j}\right)}\right) c _ {j}, \tag {6}
$$
Practically the integral is taken with near  $s_n$  and far  $s_f$  bounds.
where  $T_{j} = \exp \left(-\sum_{k=0}^{j} -\tau_{k}(s_{k+1} - s_{k})\right)$ . Here, color  $c_{j}$  is also approximated to be constant along each interval  $I_{j}$ , and  $P_{j}$  is the probability of each interval. Now, let us define the discrete random variable  $\tilde{s} = \tilde{f}(s)$ , where
$$
\tilde {f} (x) = \left\{ \begin{array}{l l} s _ {0} & x \leq s _ {0} \\ s _ {j} & s _ {j} \leq x <   s _ {j + 1} \text {f o r a l l} j \in \{1, \dots , N - 1 \}, \\ s _ {N} & x > s _ {N} \end{array} \right. \tag {7}
$$
which gives corresponding probability mass function  $\tilde{P}(s)$ . Observe that the analytical expression of the integral Eq. 6 turns out to be the same as taking the expectation w.r.t. the discrete random variable  $\tilde{s}$ , i.e.  $\mathbb{E}_{s \sim p(s)}[c(s)] = \mathbb{E}_{\tilde{s} \sim \tilde{P}(s)}[c(\tilde{s})]$ . This piecewise constant opacity assumption in the volume rendering equation is used in most, if not all, existing NeRF works. We recommend the reader to read [15] for more detailed derivations and our supplementary for a more thorough walkthrough.
3.2 Neural Radiance Fields.
Following Max and Chen [14], Mildenhall et.al. [18] introduced neural radiance fields, a neural scene representation that uses the volume rendering equation under the piecewise constant opacity assumption for novel view synthesis. A neural radiance field (NeRF) is a coordinate-based neural scene representation, where opacity  $\tau^{\theta}:\mathbb{R}^{3}\to \mathbb{R}_{\geq 0}$  and color  $c^{\psi}:\mathbb{R}^{3}\times \mathbb{S}^{2}\rightarrow [0,255]^{3}$  are predicted at each continuous coordinate by parameterizing them as a neural network. To train the neural network, 2D images are used as supervision where each viewing ray is associated with a ground truth color. Volume rendering allows for the 3D coordinate outputs to be aggregated into an observed pixel color allowing for end-to-end training with 2D supervision. The supervision signal are on the coordinates of the ray samples  $s_1,\ldots ,s_N$  , which updates the corresponding output opacity and color at those samples. NeRF uses a importance sampling strategy by drawing samples from the ray distribution from a coarse network to generate better samples for rendering of their fine network. To sample from a distribution, inverse transform sampling is needed, that is, one draws  $u\sim U(0,1)$  then passes it to the inverse of a cumulative distribution (CDF), i.e. a sample  $x = F^{-1}(u)$  , where  $F$  is the CDF of the distribution. Under the piecewise constant assumption, the CDF of the discrete random variable  $\tilde{s}$  is given by:
$$
\tilde {F} (x) = \left\{ \begin{array}{l l} 0 & x \leq s _ {0} \\ \sum_ {k <   j} \tilde {P} \left(s _ {k}\right) & 1 \leq x <   s _ {j + 1} \text {f o r a l l} j \in \{1, \dots , N - 1 \}. \\ 1 & x > s _ {N} \end{array} \right. \tag {8}
$$
This CDF is however non-continuous and non-invertible. NeRF's approach to get around this is to define a surrogate invertible function  $G$  derived from its CDF, then taking  $x = G^{-1}(u)$ . Concretely,  $G(y) = \frac{y - s_{j-1}}{s_j - s_{j-1}} \tilde{F}(s_j) + \frac{s_j - y}{s_j - s_{j-1}} \tilde{F}(s_{j-1})$ , where  $y \in [s_{j-1}, s_j]$ . However, this does not necessarily result in the samples from the actual ray distribution  $p(s)$  from the model.
4 Drawbacks of Piecewise Constant Opacity  $\tau$  in NeRFs
Unfortunately, there are properties associated with the piecewise constant opacity formulation that may not be desirable in the context of NeRFs. First, it is sensitive to the choice of samples, i.e. sample positions  $s$ , along the ray, a phenomenon we dub as quadrature instability. This quadrature instability is due to the assumption that all points in an interval take the opacity of the left bin (Eq 5), making it sensitive to sample positions. As illustrated in Figure 1, this would lead to ray conflicts in optimizing a NeRF when you have rays that are directly facing, i.e. perpendicular to, the surface (yellow rays) and rays that are close to the grazing angle (red ray), i.e. parallel to, the object. To render the perpendicular rays correctly, vanilla NeRF has to store the optical properties (opacity and color) associated with the perpendicular rays at a point before its intersection with the surface. This creates inaccurate signals to the optimization process when NeRF renders the ray at a grazing angle, as it will cross multiple conflicting opacity/colors (illustrated by the blue gradient). The sample sensitivity issue also arises when having cameras at different distances from the object as illustrated in Fig. 2, as this would lead to shifted sets of samples, causing inconsistencies when rendering at different camera-to-object distances. Notice that the noise on the texture of the chair is different across different viewing distances, where the middle view has fewer artifacts compared to the closer and further views.
Figure 2: Ray Conflicts: Different Camera-to-Scene Distances. (Left) Rendered views from cameras at different distances from the object. At all distances, the rendered output for linear have sharper texture than constant because of the latter's sensitivity to the choice of samples. We also highlight the instability of the constant model as shown by the noisier texture of the middle view compared to the closer and further views. (Right) An illustration that moving the camera to different distances from the object result in different samples that lead to conflicts.
The second issue comes from the CDF  $\tilde{F}$  being piecewise constant (Eq 8). This leads to two consequences. First, the piecewise constant assumption makes  $\tilde{F}$  non-invertible, hence, as mentioned in the previous section, importance sampling needs to be performed via a surrogate function  $G$ . This results in uniformity across the samples within the bin - samples within a bin are assigned with equal probability, leading to imprecise importance samples. The second consequence comes from the fact that  $\tilde{F}$  is not continuous, leading to an issue in training a NeRF that has a loss based on its samples. In other words, there will be a vanishing gradient effect when taking the gradient w.r.t. the samples, and one such example of a sample-based loss used for NeRFs is depth [28].
5 Generalized Form for  $P_{j}$
We first show a generalized derivation for the probability  $P_{j}$  of each interval  $I_{j}$ , which we use to formulate our approach that alleviates the problems described above. From  $T(s) = \exp \left(-\int_0^s\tau (u)\mathrm{d}u\right)$ , we first notice that:
$$
\frac {\mathrm {d} T}{\mathrm {d} s} = - \exp (- \int_ {0} ^ {s} \tau (u) \mathrm {d} u) \tau (s) = - T (s) \tau (s)
$$
$$
T ^ {\prime} (s) = - T (s) \tau (s).
$$
This results in the probability of each interval  $I_{j}$  given as follows:
$$
P _ {j} = \int_ {s _ {j}} ^ {s _ {j + 1}} \tau (s) T (s) \mathrm {d} s = - \int_ {s _ {j}} ^ {s _ {j + 1}} T ^ {\prime} (s) \mathrm {d} s = T (s _ {j}) - T (s _ {j + 1}). \tag {9}
$$
Since  $s_j$ 's are arbitrarily sampled,  $P_j$  can be exactly evaluated in a closed-form expression, if and only if  $T(\cdot)$  is in closed-form.
6 Our PL-NeRF