SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\alpha_ {j} = s _ {j + 1} - s _ {j}, \quad \beta_ {j} = s _ {j + 2} - s _ {j + 1}, \quad \gamma_ {j} = s _ {j + 2} - s _ {j}.
$$
Note the integrals of the three monic quadratics over  $[s_j, s_{j+1}]$  can be expressed in terms of  $\alpha_j, \beta_j, \gamma_j$ :
$$
\int_ {s _ {j}} ^ {s _ {j + 1}} (s - s _ {j + 1}) (s - s _ {j + 2}) \mathrm {d} s = \int_ {- \alpha_ {j}} ^ {0} s (s - \beta_ {j}) \mathrm {d} s = \frac {\alpha_ {j} ^ {3}}{3} + \frac {\alpha_ {j} ^ {2} \beta_ {j}}{2},
$$
$$
\int_ {s _ {j}} ^ {s _ {j + 1}} (s - s _ {j}) (s - s _ {j + 2}) \mathrm {d} s = \int_ {0} ^ {\alpha_ {j}} s (s - \gamma_ {j}) \mathrm {d} s = \frac {\alpha_ {j} ^ {3}}{3} - \frac {\alpha_ {j} ^ {2} \gamma_ {j}}{2},
$$
$$
\int_ {s _ {j}} ^ {s _ {j + 1}} (s - s _ {j}) (s - s _ {j + 1}) \mathrm {d} s = \int_ {0} ^ {\alpha_ {j}} s (s - \alpha_ {j}) \mathrm {d} s = - \frac {\alpha_ {j} ^ {3}}{6}.
$$
Thus, using Eq. 27 gives the integral of opacity over  $[s_j, s_{j+1}]$ :
$$
\int_ {s _ {j}} ^ {s _ {j + 1}} \tau (s) \mathrm {d} s = \frac {\tau_ {j}}{\gamma_ {j}} \cdot \left[ \frac {\alpha_ {j} ^ {2}}{3} + \frac {\alpha_ {j} \beta_ {j}}{2} \right] - \frac {\tau_ {j + 1}}{\beta_ {j}} \left[ \frac {\alpha_ {j} ^ {2}}{3} - \frac {\alpha_ {j} \gamma_ {j}}{2} \right] + \frac {\tau_ {j + 2}}{\beta_ {j} \gamma_ {j}} \cdot \left[ - \frac {\alpha_ {j} ^ {3}}{6} \right]. \tag {28}
$$
Similarly, the integrals of those same three quadratics over  $[s_{j + 1}, s_{j + 2}]$  factor out  $\beta_j$ :
$$
\int_ {s _ {j + 1}} ^ {s _ {j + 2}} (s - s _ {j + 1}) (s - s _ {j + 2}) \mathrm {d} s = \int_ {0} ^ {\beta_ {j}} s (s - \beta_ {j}) \mathrm {d} s = - \frac {\beta_ {j} ^ {3}}{6},
$$
$$
\int_ {s _ {j + 1}} ^ {s _ {j + 2}} (s - s _ {j}) (s - s _ {j + 2}) \mathrm {d} s = \int_ {- \beta_ {j}} ^ {0} (s + \gamma_ {j}) s \mathrm {d} s = \frac {\beta_ {j} ^ {3}}{3} - \frac {\beta_ {j} ^ {2} \gamma_ {j}}{2},
$$
$$
\int_ {s _ {j + 1}} ^ {s _ {j + 2}} (s - s _ {j}) (s - s _ {j + 1}) \mathrm {d} s = \int_ {0} ^ {\beta_ {j}} (s + \alpha_ {j}) s \mathrm {d} s = \frac {\beta_ {j} ^ {3}}{3} + \frac {\beta_ {j} ^ {2} \alpha_ {j}}{2}.
$$
Then, summing these up along Eq. 27 gives the integral of opacity over  $[s_{j + 1}, s_{j + 2}]$ :
$$
\int_ {s _ {j + 1}} ^ {s _ {j + 2}} \tau (s) \mathrm {d} s = \frac {\tau_ {j}}{\alpha_ {j} \gamma_ {j}} \cdot \left[ - \frac {\beta_ {j} ^ {3}}{6} \right] - \frac {\tau_ {j + 1}}{\alpha_ {j}} \left[ \frac {\beta_ {j} ^ {2}}{3} - \frac {\beta_ {j} \gamma_ {j}}{2} \right] + \frac {\tau_ {j + 2}}{\gamma_ {j}} \cdot \left[ \frac {\beta_ {j} ^ {2}}{3} + \frac {\beta_ {j} \alpha_ {j}}{2} \right]. \tag {29}
$$
Observe in Eqs. 28 and 29 that the integral of opacity involves some terms with  $\alpha_{j},\beta_{j},\gamma_{j}$  in the denominator, which do not cancel. Thus, the integral of opacity over an interval is not a polynomial in  $\tau_{i}$ 's and  $s_i$  's, but is instead a rational function. This contrasts with the linear derivation in Sec. A.3.3, where this integral was a degree 2 multivariate polynomial in  $\tau_{i}$ 's and  $s_i$  's. This caveat causes numerical instability, which will be discussed further in Secs. A.4.3 and A.4.4.
Generally, following the steps of the above derivation shows that if we interpolate  $\tau$  piecewise by any degree  $d$  polynomial,  $d \geq 2$ , then the result is a rational function in  $s_i$ 's and  $\tau_i$ 's, but not a polynomial, which would also lead to training instability as in the quadratic case.
A.4.3 Piecewise Quadratic Problem 1: Negative Interpolated Opacity
As seen above, one problem with the piecewise quadratic model is that the integral of opacity is a rational function in  $\alpha_{i},\beta_{i},\gamma_{i}$ . In particular, due to the presence of negatives in front of certain rational terms in Eqs. 28 and 29, it may become negative as the denominators of these terms approach zero. One example is shown in Figure A.4.3, for samples  $s_1,s_2,s_3$  with  $\tau_{1} = \tau_{2} < \tau_{3}$  and  $s_3 - s_2\ll s_2 - s_1$ , where the interpolated quadratic dips far below the  $x$ -axis. Note this interpolation is physically implausible, as opacity should be nonnegative everywhere.
Furthermore, from Eqs. 24 and 25, when the opacity integral is negative on intervals, transmittance can then be a product with exponentials of negative terms, and potentially be greater than 1. This is incompatible with the physical interpretation of transmittance as a probability that a light travels a distance along a ray without being absorbed.
Figure A3: Interpolation gives negative  $\tau$ -values when  $s$ -values are close.
A.4.4 Piecewise Quadratic Problem 2: Instability from Sample Proximity
In Eqs. 28 and 29, we observe the presence of terms such as  $\alpha_{j},\beta_{j},\gamma_{j}$  in the denominator. These do not appear in the piecewise linear model. As a result, because these quantities approach zero as samples  $s_j,s_{j + 1},s_{j + 2}$  become closer, then the integral of opacity can become an arbitrarily large positive or even negative (as shown qualitatively in the previous section). Referencing the transmittance formulae in Eqs. 24 and 25, this means transmittance can approach zero or infinity and gradients can explode as samples are clustered together. This is fairly common in stratified sampling, and even more so with importance sampling.
To formally describe this instability, we first look at what happens when  $s_j$  and  $s_{j+1}$  coincide:
$$
\lim  _ {s _ {j + 1} \rightarrow s _ {j} ^ {+}} \int_ {s _ {j}} ^ {s _ {j + 1}} \tau (s) \mathrm {d} s = 0
$$
because  $\tau : \mathbb{R}_{\geq 0} \to \mathbb{R}$  is continuous. There is no instability in this integral when  $s_{j+1} \to s_j^+$ . However, when  $s_{j+1}$  and  $s_{j+2}$  coincide, because  $\beta_j \to 0^+$  and  $\gamma_j \to \alpha_j^+$ , the integral approaches
$$
\begin{array}{l} \lim _ {s _ {j + 2} \to s _ {j + 1} ^ {+}} \int_ {s _ {j}} ^ {s _ {j + 1}} \tau (s) \mathrm {d} s = \lim _ {s _ {j + 2} \to s _ {j + 1} ^ {+}} \left[ \frac {\tau_ {j}}{\gamma_ {j}} \cdot \left[ \frac {\alpha_ {j} ^ {2}}{3} + \frac {\alpha_ {j} \beta_ {j}}{2} \right] - \frac {\tau_ {j + 1}}{\beta_ {j}} \left[ \frac {\alpha_ {j} ^ {2}}{3} - \frac {\alpha_ {j} \gamma_ {j}}{2} \right] + \frac {\tau_ {j + 2}}{\beta_ {j} \gamma_ {j}} \cdot \left[ - \frac {\alpha_ {j} ^ {3}}{6} \right] \right] \\ = \frac {\tau_ {j} \alpha_ {j}}{3} - \lim _ {s _ {j + 2} \to s _ {j + 1} ^ {+}} \frac {\frac {\tau_ {j + 1} \alpha_ {j} ^ {2}}{3} - \frac {\tau_ {j + 1} \alpha_ {j} \gamma_ {j}}{2} + \frac {\tau_ {j + 2} \alpha_ {j} ^ {3}}{6 \gamma_ {j}}}{\beta_ {j}} \\ = \frac {\tau_ {j} \alpha_ {j}}{3} - \lim _ {s _ {j + 2} \to s _ {j + 1} ^ {+}} \frac {2 \tau_ {j + 1} \alpha_ {j} ^ {2} \gamma_ {j} - 3 \tau_ {j + 1} \alpha_ {j} \gamma_ {j} ^ {2} + \tau_ {j + 2} \alpha_ {j} ^ {3}}{6 \gamma_ {j} \beta_ {j}} \\ = \frac {\tau_ {j} \alpha_ {j}}{3} - \lim  _ {s _ {j + 2} \to s _ {j + 1} ^ {+}} \frac {\tau_ {j + 1} \alpha_ {j} \left(\alpha_ {j} ^ {2} + 2 \alpha_ {j} \gamma_ {j} - 3 \gamma_ {j} ^ {2}\right) + \alpha_ {j} ^ {3} \left(\tau_ {j + 2} - \tau_ {j + 1}\right)}{6 \gamma_ {j} \beta_ {j}} \\ = \frac {\tau_ {j} \alpha_ {j}}{3} - \lim  _ {s _ {j + 2} \rightarrow s _ {j + 1} ^ {+}} \frac {\tau_ {j + 1} \alpha_ {j} (\alpha_ {j} + 3 \gamma_ {j}) (\alpha_ {j} - \gamma_ {j}) + \alpha_ {j} ^ {3} (\tau_ {j + 2} - \tau_ {j + 1})}{6 \gamma_ {j} \beta_ {j}} \\ = \frac {\tau_ {j} \alpha_ {j}}{3} - \lim  _ {s _ {j + 2} \rightarrow s _ {j + 1} ^ {+}} \frac {- \tau_ {j + 1} \alpha_ {j} (\alpha_ {j} + 3 \gamma_ {j}) \beta_ {j} + \alpha_ {j} ^ {3} (\tau_ {j + 2} - \tau_ {j + 1})}{6 \gamma_ {j} \beta_ {j}} \\ \end{array}
$$
$$
\begin{array}{l} = \frac {\tau_ {j} \alpha_ {j}}{3} + \frac {2 \tau_ {j + 1} \alpha_ {j}}{3} - \lim  _ {s _ {j + 2} \rightarrow s _ {j + 1} ^ {+}} \frac {\alpha_ {j} ^ {3} (\tau_ {j + 2} - \tau_ {j + 1})}{6 \gamma_ {j} \beta_ {j}} \\ = \frac {\tau_ {j} \alpha_ {j}}{3} + \frac {2 \tau_ {j + 1} \alpha_ {j}}{3} - \frac {\alpha_ {j} ^ {2}}{6} \lim _ {s _ {j + 2} \to s _ {j + 1} ^ {+}} \frac {\tau_ {j + 2} - \tau_ {j + 1}}{\beta_ {j}}. \\ \end{array}
$$
The behavior of this integral limit depends on the last limit. To analyze this, let  $h: \mathbb{R}_{\geq 0} \to \mathbb{R}$  be the network function which takes in samples on the ray and outputs opacity. The last limit is
$$
\lim  _ {s _ {j + 2} \rightarrow s _ {j + 1} ^ {+}} \frac {\tau_ {j + 2} - \tau_ {j + 1}}{\beta_ {j}} = \lim  _ {s _ {j + 2} \rightarrow s _ {j + 1} ^ {+}} \frac {h \left(s _ {j + 2}\right) - h \left(s _ {j + 1}\right)}{s _ {j + 2} - s _ {j + 1}}. \tag {30}
$$
Because  $h$  is almost always differentiable at  $s_{j+1}$ , this becomes  $h'(s_{j+1})$ , and so
$$
\lim  _ {s _ {j + 2} \rightarrow s _ {j + 1} ^ {+}} \int_ {s _ {j}} ^ {s _ {j + 1}} \tau (s) \mathrm {d} s = \frac {\tau_ {j} \alpha_ {j}}{3} + \frac {2 \tau_ {j + 1} \alpha_ {j}}{3} - \frac {\alpha_ {j} ^ {2}}{6} h ^ {\prime} \left(s _ {j + 1}\right). \tag {31}
$$
There are no constraints on the value of  $h^{\prime}(s_{j + 1})$  as the network trains, so the above limit can achieve any real value. In particular, we can derive a condition for when the limit is negative:
$$
\lim  _ {s _ {j + 2} \rightarrow s _ {j + 1} ^ {+}} \int_ {s _ {j}} ^ {s _ {j + 1}} \tau (s) \mathrm {d} s <   0 \Longleftrightarrow h ^ {\prime} (s _ {j + 1}) > \frac {2 \tau_ {j} + 4 \tau_ {j + 1}}{\alpha_ {j}}. \tag {32}
$$
That is, whenever there is a sharp enough increase in opacity (which can happen at, say, a surface crossing) and samples  $s_{j+1}, s_{j+2}$  are sufficiently close, the interpolated quadratic can have a negative integral on  $[s_j, s_{j+1}]$ , which leads to the issues described in Sec. A.4.3. In other words, the integral of opacity over  $[s_j, s_{j+1}]$  is unstable as  $s_{j+2} \to s_{j+1}^+$ . A similar analysis holds to show the instability of the opacity integral on  $[s_{j+1}, s_{j+2}]$  as  $s_{j+1} \to s_j^+$ .
A.4.5 Piecewise Quadratic Problem 3: Importance Sampling
Suppose we wish to importance sample in the same manner as piecewise linear, that is, we use inverse transform sampling. In essence, we draw  $u \sim U(0,1)$  and then sample  $x = F^{-1}(u)$ , with  $F$  the CDF of the distribution. Recall  $F$  is computed as
$$
f (x) = \int_ {s _ {0}} ^ {x} T (s) \tau (s) d s = \int_ {s _ {0}} ^ {s _ {k}} T (s) \tau (s) d s + \int_ {s _ {k}} ^ {x} T (s) \tau (s) d s = c _ {k} + \int_ {s _ {k}} ^ {x} T (s) \tau (s) d s. \tag {33}
$$
As before, the latter equation becomes
$$
f (x) = c _ {k} + T \left(s _ {k}\right) - T (x). \tag {34}
$$
Hence, computing  $x$  amounts to solving the equation  $f(x) = u$ , which becomes from above:
$$
T (x) = c _ {k} + T \left(s _ {k}\right) - u. \tag {35}
$$
As  $T$  is the exponential of a cubic, this amounts to solving a cubic above. This does have a real solution, as  $F$  is increasing and continuous, so the Intermediate Value Theorem implies  $f(x) = u$  has a solution; and the solution is unique because  $F$  is strictly increasing, as it is an exponential of a polynomial. However, the complexity of exact importance sampling would be large.
This analysis reveals another downside of higher order polynomials. In general, suppose we wish to interpolate opacity  $\tau$  with a piecewise degree  $n$  polynomial. Then following the same method as above, we see transmittance  $T$  is the piecewise exponential of a degree  $n + 1$  polynomial, which is the integral of  $\tau$ . So inverse transform sampling reduces to solving a degree  $n + 1$  polynomial. The Abel-Ruffini Theorem asserts for  $n + 1 \geq 5$  that this polynomial is in general not solvable by radicals. In other words, there is no simple closed form for exact importance sampling when  $n \geq 4$ .
Theoretically, this could be exactly solved for  $n = 2$  and  $n = 3$  (i.e. when opacity is piecewise quadratic or cubic, respectively), but the formulae to derive cubic and quartic solutions can become sufficiently complicated and the resulting complicated expressions may result in numerical instability during optimization, especially when taking the gradient w.r.t. the samples.
A.5 Experiment details, Reproducibility and Compute
A.5.1 Implementation Details
We include the core code snippets of our implementation of PL-NeRF. Figure A4 shows the volume rendering equation that includes the implementation of Eq. 10 and 11 (main paper). This is a direct replacement of the original constant approximation, where we also show the code snippet in the figure as reference. Figure A5 shows the implementation of our precise importance sampling from Eq. 13 (main paper), which is also a direct replacement of the constant importance sampling implementation (Figure A6) for reference. We highlight that our formulation is a direct replacement of the functions from the original implementation, and hence for the depth experiments, we are also able to directly adapt the codebase from [28]. We use Nvidia v100 and A5000 GPU's for our experiments. Each scene is trained on a single GPU and takes  $15 - 20$  hours. We used an internal academic cluster and cloud compute resources to train and evaluate our models.
For the MipNeRF-based experiments, our experiments are also run on the standard train and test split of the Blender dataset with the official released hyperparameters of Mip-NeRF using the NerfStudio [27] codebase. For PL-MipNeRF, we use the two-MLP training scheme with a coarse loss weight of 1.0.
For DIVeR-based experiments, we use their official implementation and configuration for DIVeR64 at 128 voxels. For PL-DIVeR, we utilize their voxel-based representation and feature integration and dropping in our piecewise linear opacity rendering formulation. We similarly run the DIVeR models on a single Nvidia v100 GPU trained using their default configurations and hyperparameters for DIVeR64 at 128 voxels.
The total training time for 500k iterations on a single Nvidia v100 GPU is 17.78 and 21.43 hours for Vanilla NeRF and PL-NeRF, respectively. Figure 2-c shows the head-to-head comparison of training PSNR (y-axis) with respect to time (x-axis) of Vanilla NeRF vs PL-NeRF on the Lego scene. Rendering a single  $800 \times 800$  image takes 25.59 and 32.35 seconds for Vanilla NeRF and PL-NeRF, respectively.
A.5.2 Computational complexity under different number of samples
We measured the total rendering time for a single  $800 \times 800$  image under different numbers of samples for our PL-NeRF. The total rendering time for (64+64), (64+128) and (128+64) are 19.20, 25.85 and 32.35 seconds, respectively.
A.5.3 Convergence plots under different number of samples
Figure ?? shows the convergence plots under different numbers of samples of our PL-NeRF vs Vanilla NeRF. We see that under different number of samples, our linear approach converges to a higher training PSNR.
Our piecewise linear opacity approximation is able to handle arbitrarily small opacity, e.g.  $1e^{-6}$ , however, we cannot handle coordinates with exactly zero opacity. This special case is not an issue in practice. Also theoretically, any atom absorbs light and thus will not have exactly zero opacity, except in a vacuum. Another limitation is our method is slightly slower than the original piecewise constant approximation, and this is due to requiring more FLOPS for our importance sampling computation (Eq. 13 main paper). Another additional limitation is we still assume piecewise color, i.e. within a bin we do not handle color integration. Modeling this can potentially handle difficult scenarios such as double-walled colored glass or atmospheric effects such as fog or smoke. Lastly, we also inherit the limitations of NeRFs in general, such as requiring known camera poses.
Our models require the usage of GPUs both in training time and rendering time, and GPUs use up energy to run and power them. We acknowledge that this contributes to climate change that is an important societal issue. Despite this, we observe the improvement in the results that is theoretically
Figure A4: Code snippet for volume rendering. The implementation for our piecewise linear opacity approximation is a drop-in replacement from the original piecewise constant.
grounded and believe that it is beneficial for the pursuit of science. We responsibly ran our models by first prototyping on selected scenes before scaling up to different scenes across datasets to minimize its impact to climate change.
Figure A5: Code snippet for our Precise Importance Sampling. The implementation of our precision importance sampling is also a direct replacement from the original function from the constant implementation called sample(pdf (See next figure for reference).
def Original Piecewise Constant Opacity
def sample(pdf(bins, weights, N_samples, det=False, pytest=False):
    # Get pdf
    weights = weights + 1e-5 # prevent nans
    pdf = weights / torch.sum(weights, -1, keepdim=True)
    cdf = torch.cumsum(pdf, -1)
    cdf = torch.cat([[torch.zeros_likecdf[.,:1]], cdf], -1) # (batch, len(bins))
    # Take uniform samples
    if det:
        u = torch.linspace(0., 1., steps=N_samples, device=bins_device)
        u = uexpand(list(cdf.shape[:1]) + [N_samples])
    else:
        u = torch.randint(list(cdf.shape[:1]) + [N_samples], device=bins_device)
# Pytest, overwrite u with numpy's fixed random numbers
if pytest:
    np.random.seed(0)
    new_shape = list(cdf.shape[:1] + [N_samples])
    if det:
        u = np.linspace(0., 1., N_samples)
        u = nproadcast_to(u, new_shape)
    else:
        u = np.random.randint(*new_shape)
    u = torch.Tensor(u)
# Invert CDF
u = u.contiguous()
inds = torch.searchsortedcdf, u, right=True)
below = torch.max(torch.zeros_like(inds-1),inds-1)
above = torch.min((cdf.shape[-1]-1) * torch.ones_like(inds),inds)
inds_g = torch.stack([-1], -1) # (batch, N_samples, 2)
# cdf_g = tf.gathercdf,inds_g, axis=-1, batch_dims=len(inds_g.shape)-2)
# bins_g = tf.gather(bins,inds_g, axis=-1, batch_dims=len(inds_g.shape)-2)
matched_shape = [inds_g.shape[0],inds_g.shape[1],cdf.shape[-1]]
cdf_g = torch.gathercdf_unsqueeze(1).expand(matched_shape), 2,inds_g)
bins_g = torch.gather(bins_unsqueeze(1).expand(matched_shape), 2,inds_g)
denom = (cdf_g[,...,1] - cdf_g[,...,0])
denom = torch.where(len(1e-5, torch.ones_like denom), denom)
t = (u-cdf_g[,...,0])/denom
samples = bins_g[,...,0] + t * (bins_g[,...,1] - bins_g[,...,0])
return samples
Figure A6: This is the original importance sampling for the constant approximation for reference.