SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We observe from Eq. 15 that we can obtain a closed-form expression for  $P_{j}$  for any piecewise polynomial function in  $\tau$ , which can be of any degree  $d = 0, 1, 2, \dots, n$ . Commonly used in existing NeRF literature is choosing  $d = 0$ , i.e. piecewise constant, that is unstable w.r.t. the choice of samples as highlighted in the previous section. Interestingly, we also observe and show that the problem becomes numerically ill-conditioned for  $d \geq 2$  making it difficult and unstable to optimize. Please see supplementary for the full proof. Hence, we propose to make opacity piecewise linear ( $d = 1$ ), which we call PL-NeRF, leading to a simple and closed-form expression for the volume rendering integral that is numerically stable and is a drop-in replacement to existing NeRF-based methods. We show both theoretically and experimentally that the piecewise linear assumption is sufficient and alleviates the problems caused by quadrature instability under the piecewise constant assumption.
Volume Rendering with Piecewise Linear Opacity. We propose an elegant reformulation to the sample-based rendering equation that corresponds to the exact integral under piecewise linear opacity while keeping piecewise constant color leading to a simple and closed-form expression for the integral. That is, instead of piecewise constant opacity as in Eq 5, we assume a linear opacity for each interval  $I_{j}$ . Concretely, for  $s \in [s_j, s_{j+1}]$ , where  $\tau_{j} = \tau(s_{j})$ ,  $\tau_{j+1} = \tau(s_{j+1})$ , we have
$$
\tau (s) = \left(\frac {s _ {j + 1} - s}{s _ {j + 1} - s _ {j}}\right) \tau_ {j} + \left(\frac {s - s _ {j}}{s _ {j + 1} - s _ {j}}\right) \tau_ {j + 1}. \tag {10}
$$
which is linear w.r.t.  $s \in [s_j, s_{j+1}]$  as illustrated in Fig. 3.
Figure 3: Illustration of opacities  $\tau$  along a ray under the piecewise constant (green) and piecewise linear (orange) assumptions.
Now, under the piecewise linear opacity assumption, transmittance is derived as the following closed-form expression:
$$
T (s _ {j}) = \exp \left[ - \int_ {0} ^ {s _ {j}} \tau (u) \mathrm {d} u \right] = \prod_ {k = 1} ^ {i} \exp \left[ - \int_ {s _ {k - 1}} ^ {s _ {k}} \tau (u) \mathrm {d} u \right],
$$
$$
\boxed {T (s _ {j}) = \prod_ {k = 1} ^ {i} \exp \left[ - \frac {\left(\tau_ {k} + \tau_ {k - 1}\right) \left(s _ {k} - s _ {k - 1}\right)}{2} \right].} \tag {11}
$$
Together with Eq. 15, this leads to the following simple and closed-form expression for  $P_{j}$ , corresponding to the exact integral under the piecewise linear opacity assumption:
$$
\boxed {P _ {j} = T (s _ {j}) \cdot \left(1 - \exp \left[ - \frac {\left(\tau_ {j + 1} + \tau_ {j}\right) \left(s _ {j + 1} - s _ {j}\right)}{2} \right]\right).} \tag {12}
$$
**Precision Importance Sampling.** Moreover, it also turns out that with our piecewise linear opacity assumption, we are able to derive an exact closed-form solution for inverse transform sampling. Recall that in Sec 4, we pointed a drawback of the CDF  $\tilde{F}$  being non-invertible and discontinuous under piecewise constant opacity. We show that this is alleviated in our piecewise linear setting. Concretely, given samples  $s_1, \ldots, s_N$  resulting in interval probabilities  $P_1, \ldots, P_N^2$  from our derivation (Eq 12), the CDF for continuous random variable  $t$  is then given as
$$
F (t) = \int_ {0} ^ {t} p (s) \mathrm {d} s = \sum_ {s _ {j} <   t} P _ {j} + \int_ {s _ {j}} ^ {t} p (s) \mathrm {d} s = \sum_ {s _ {j} <   t} P _ {j} + \int_ {s _ {j}} ^ {t} \tau (s) T (s) \mathrm {d} s. \tag {13}
$$
Note that unlike in piecewise constant opacity, we do not convert a continuous random variable  $s$  to a discrete random variable  $\tilde{s}$ , thus, the resulting CDF  $F$  being continuous. Now, assuming that opacity  $\tau \geq 0$  everywhere, from Eq. 13 we see that  $F$  is strictly increasing. Since  $F$  is continuous and strictly increasing, then it is invertible.
Finally, we have our precision importance sampling, where by inverse transform sampling, we can solve for the exact sample  $x = F^{-1}(u)$  for  $u \sim U(0,1)$  from the given ray distribution  $p(s)$  since the CDF  $F$  is invertible under piecewise linear opacity. That is, without loss of generality, let sample  $u \sim U(0,1)$  fall into the bin  $u \in [C_k, C_{k+1}]$ , where  $C_k = \sum_{j < k} P_j$ , which is equivalent to solving for  $x \in [s_k, s_{k+1}]$ . Reparameterizing  $x = s_k + t$ , where  $t \in [0, s_{k+1} - s_k]$ , the exact solution for sample  $u$  is given by
$$
\boxed {t = \frac {s _ {k + 1} - s _ {k}}{\tau_ {k + 1} - \tau_ {k}} \left[ - \tau_ {k} + \sqrt {\tau_ {k} ^ {2} + \frac {2 \left(\tau_ {k + 1} - \tau_ {k}\right) \left(- \ln \frac {1 - u}{T \left(s _ {k}\right)}\right)}{\left(s _ {k + 1} - s _ {k}\right)} \right].} \tag {14}
$$
Please see supplementary for full derivation. This leads to precisely sampling from the ray distribution  $p(s)$  resulting in better importance sampling and stronger depth supervision.
2We note that DS-NeRF [6] shows that this will sum to 1 assuming an opaque far plane.  $s_{N + 1}$  would correspond to the far plane.
Figure 4: Qualitative Results for Blender and Real Forward Facing.
Blender	Avg.	Chair	Drums	Ficus	Hotdog	Lego	Mat.	Mic	Ship
PSNR↑	Const. (Vanilla)	30.61	32.54	24.79	29.63	36.08	32.01	29.31	32.55	27.95
Linear (Ours)	31.10	32.92	25.07	30.18	36.46	32.90	29.52	33.08	28.71
SSIM↑	Const. (Vanilla)	0.943	0.966	0.918	0.960	0.975	0.959	0.943	0.978	0.846
Linear (Ours)	0.948	0.969	0.923	0.965	0.977	0.966	0.948	0.981	0.857
LPIPS↓	Const. (Vanilla)	5.17	3.19	7.97	4.14	2.48	2.33	4.32	2.16	14.8
Linear (Ours)	4.39	2.85	7.10	3.03	2.28	1.81	3.21	1.73	13.1
LLFF	Avg.	Fern	Flower	Fortress	Horns	Leaves	Orchid	Room	Trex
PSNR↑	Const. (Vanilla)	27.53	26.79	28.23	32.53	28.54	22.35	21.20	33.03	27.58
Linear (Ours)	28.05	26.85	28.71	32.95	29.38	22.51	21.25	33.99	28.79
SSIM↑	Const. (Vanilla)	0.874	0.746	0.886	0.925	0.893	0.816	0.746	0.956	0.916
Linear (Ours)	0.885	0.863	0.902	0.932	0.911	0.826	0.754	0.961	0.933
LPIPS↓	Const. (Vanilla)	7.37	9.67	6.34	2.92	7.26	11.0	11.8	4.33	5.66
Linear (Ours)	6.06	7.92	4.93	2.46	5.51	9.59	10.2	3.54	4.38
Table 1: Quantitative Results on Blender and LLFF Datasets. LPIPS scores  $\times  {10}^{2}$  .
In this section, we present our experimental evaluations to demonstrate the advantages our piecewise linear opacity formulation for volume rendering, which we call PL-NeRF.
7.1 Datasets, Evaluation Metrics and Implementation Details.
Datasets and Evaluation Metrics. We evaluate our method on the standard datasets: Blender and Real Forward Facing (LLFF) datasets as used in [18]. We use the released training and test splits for each. See supplementary for more details. For quantitative comparison, we follow the standard evaluation metrics and report PSNR, SSIM [32] and LPIPS [41] on unseen test views. We also report the root-mean-squared-error (RSME) on the expected ray termination in our depth experiments.
Implementation Details. PL-NeRF is implemented on top of NeRF-Pytorch [38], a reproducible Pytorch implementation of the constant (vanilla) NeRF, where we simply change the volume rendering to our formulation under piecewise linear opacity and utilize our exact importance sampling derivation. Similar to [18] we optimize a separate network for the coarse and fine models that are jointly trained with the MSE loss on ground truth images. We use a batch size of 1024 rays and a learning rate of  $5 \times 10^{-4}$  that decays exponentially to  $5 \times 10^{-5}$  throughout the course of optimization. We train each scene for 500k iterations which takes  $\sim 21$  hours on a single Nvidia V100 GPU<sup>3</sup>. Our precision importance sampling enables us to use fewer samples for the fine network, hence keeping the total number of rendering samples the same, we use 128 coarse samples and 64 fine samples to train and test our method.
7.2 Experiments on Blender and LLFF Datasets
We first evaluate our PL-NeRF on the standard Blender and Real Forward Facing datasets. Table 1 shows that our PL-NeRF (linear) outperforms the vanilla [18] (constant) model that assumes piecewise constant opacity in all metrics for both the synthetic Blender and Real Forward Facing datasets. Figure 1 and Figure 4 show qualitative results. As shown our PL-NeRF is able to achieve sharper
3We rerun and train the vanilla (constant) model using the released reproducible config from [38].
Blender	Avg.	Chair	Drums	Ficus	Hotdog	Lego	Mat.	Mic	Ship
PSNR↑	Mip-NeRF	31.76	33.95	24.39	31.20	36.12	33.84	30.55	34.63	29.41
PL-MipNeRF	32.48	35.11	24.92	32.25	36.51	35.15	30.69	35.22	30.00
SSIM↑	Mip-NeRF	0.955	0.975	0.921	0.971	0.978	0.971	0.957	0.987	0.876
PL-MipNeRF	0.959	0.981	0.928	0.977	0.980	0.976	0.959	0.989	0.882
LPIPS↓	Mip-NeRF	3.64	1.80	6.82	2.35	1.97	1.44	2.39	0.973	11.4
PL-MipNeRF	3.09	1.32	5.78	1.66	1.67	1.07	2.09	0.788	10.3
Table 2: Quantitative Results of Mip-NeRF v.s. PL-MipNeRF LPIPS scores  $\times  {10}^{2}$  .
textures as shown in the Lego's scooper and the bread's surface in the hotdog. Moreover, our approach is also able to recover less fuzzy surfaces as shown in the microphone scene (Figure 1) where training views are close to the grazing angle of its head. As illustrated, the resulting probability density of the ray corresponding to the marked pixel is peakier than constant as our precision importance sampling allows us to have better samples closer to the surface. We also see clearer ropes in ship, less cloudy interior of the drum, and more solid surfaces such as cleaner leg of the swivel chair in the room scene.
7.3 Geometric Extraction
We also show improvement in geometric reconstruction of PL-NeRF. We extract the geometry from the learned density field from the trained models of PL-NeRF and Vanilla NeRF using marching cubes with a threshold of 25 following [30]. Figure 5 shows qualitative results on the reconstruction of our piecewise linear vs the original piecewise constant formulation. As shown, we are able to better recover the holes on the body and wheels of the Lego
Figure 5: Geometry Extraction Qualitative Examples.
scene as well as the interior structure inside the Mic. Moreover, interestingly, the surface of the drum is reconstructed to be transparent as visually depicted in the images, as opposed to the ground truth being opaque.
Figure 6: Qualitative Results for Mip-NeRF vs PL-MipNeRF. We see that under difficult scenarios such as in fine texture details of the chair and grazing angle views on the mic, PL-MipNeRF visually shows significant improvement.
7.4 Effectiveness of our formulation on other Radiance Field Methods
We also demonstrate our formulation's effectiveness on other radiance field methods and show that our approach can be used as a drop-in replacement to existing NeRF-based methods. We integrate our piecewise linear opacity formulation to the volume rendering integral into Mip-NeRF (PL-MipNeRF). Table 2 shows our quantitative results demonstrating consistent improvement across all scenes in the original hemisphere Blender dataset. Figure 6 shows qualitative examples where we see that under difficult scenarios such as when ray conflicts arise in the fine details of the Chair and in the presence of grazing angle views in the Mic, our PL-MipNeRF shows significant improvement over the baseline. Our results show that our piecewise linear opacity and piecewise constant color formulation scales well to Mip-NeRF as well. See supplementary for implementation details. We also plug our
Train Set	Dist 0.25x	Dist 0.5x	Dist 0.75x
PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓	PSNR↑	SSIM↑	LPIPS↓
Hemisphere	Const. (Vanilla)	20.18	0.612	54.1	22.80	0.753	30.4	25.97	0.867	14.1
Linear (Ours)	20.34	0.637	50.0	23.00	0.767	27.5	26.28	0.876	12.6
Multi Dist.	Const. (Vanilla)	22.30	0.677	45.7	25.51	0.811	23.1	28.02	0.891	11.3
Linear (Ours)	22.66	0.705	41.1	26.04	0.828	20.3	28.55	0.900	9.90
Table 3: Testing on close-up views: Hemisphere are with training cameras located on the original hemisphere. Multi Dist. with training cameras are at random distances across a depth scale range of  ${0.5} - {1.0}$  of the original hemisphere. Reported LPIPS score is multiplied by  ${10}^{2}$  .
piecewise linear opacity formulation into DIVeR [36], a voxel-based NeRF model, and show that our formulation is also an effective drop-in replacement outperforming the original DIVeR [36]. Please see supplementary for experiment results and implementation details.
7.5 Experiments on Close-up Views
We further consider the challenging setting of testing on cameras closer to the objects. Table 3 (top) shows quantitative results when training on the original hemisphere dataset and tested on different close-up views. As shown by the drop in metrics, the difficulty increases as the camera moves closer to the object (0.75x to 0.5x to 0.25x of the original radius) where details get more apparent. Our PL-NeRF outperforms the vanilla piecewise constant model in all settings and the gap (SSIM and LPIPS) between ours and the constant assumption increases as the setting becomes harder highlighting the importance of recovering shaper texture and less fuzzy surfaces.
We also consider the set-up of training with cameras with different distances to the object that result in different sets of ray samples causing conflicts. We generate training views following the data processing pipeline from [18] with a random distance scaling factor sampled from  $U(0.5, 1.0)$ . As shown in Table 3 (bottom), our PL-NeRF outperforms the vanilla constant baseline in all metrics across different camera distances, where the gap (LPIPS and SSIM) is also larger the closer the camera is to the object, where details are more apparent. The difficulty for the constant (vanilla) case under multiple camera distances is its sensitivity to the choice of samples along the ray. Figure 2 shows that the conflicting rays cause quadrature instability for under piecewise constant opacity leads to unstable outputs as shown by the noisy texture on the chair. For the constant, the level of noise (gold specs) and blurriness vary at different camera distances, whereas our PL-NeRF renders crisp and more consistent outputs even as the camera is moved closer or further for the object.
7.6 Experiments with Depth Supervision
Finally, we also show that our PL-NeRF enables stronger depth supervision under our piecewise linear opacity assumption due our precision importance sampling that allows for gradients to flow to these more refined samples resulting in more accurate depth. As in previous works [28], we use a sample-based loss for to incorporate depth supervision<sup>4</sup>. Table 4 shows quantitative
results when training and testing on the less constrained Blender dataset with cameras at random distances from the object, as described in the previous section, with depth supervision. As shown, our PL-NeRF outperforms the vanilla constant baseline on all metrics including depth RSME demonstrating that our approach allows for stronger depth supervision.
PSNR↑	SSIM↑	LPIPS↓	RMSE↓
Const. (Vanilla)	29.20	0.898	11.2	0.178
Linear (Ours)	29.54	0.905	10.4	0.147
Table 4: Depth Supervision. Reported LPIPS score is multiplied by  $10^{2}$ .
We proposed a new way to approximate the volume rendering integral that avoids quadrature instability, by considering a piecewise linear approximation to opacity and a piecewise constant approximation to color. We showed that this results in a simple closed-form expression for the integral that is easy to evaluate. We turned this into a new objective for training NeRFs that is a drop-in replacement to existing methods and demonstrated improved rendering quality and geometric reconstruction, more accurate importance sampling and stronger depth supervision.
4We use the original hyperparameters from [28] in this experiment.
Acknowledgements. This work is supported by a Apple Scholars in AI/ML PhD Fellowship, a Snap Research Fellowship, a Vannevar Bush Faculty Fellowship, ARL grant W911NF-21-2-0104, a gift from the Adobe corporation, the Natural Sciences and Engineering Research Council of Canada (NSERC), the BC DRI Group and the Digital Research Alliance of Canada.
[26] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 33:7537–7547, 2020. 2
[27] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, 2023. 28
[28] Mikaela Angelina Uy, Ricardo Martin-Brualla, Leonidas Guibas, and Ke Li. Scade: Nerfs from space carving with ambiguity-aware depth estimates. In CVPR, 2023. 2, 6, 10, 28
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 2
[30] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021. 9
[31] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In ICCV, 2021. 2, 3
[32] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004. 8
[33] Lee Westover. Interactive volume rendering. In Proceedings of the 1989 Chapel Hill workshop on Volume visualization, pages 9-16, 1989. 3
[34] Peter L Williams and Nelson Max. A volume density optical model. pages 61-68, 1992. 3
[35] Peter L Williams, Nelson L Max, and Clifford M Stein. A high accuracy volume renderer for unstructured data. IEEE Transactions on Visualization and Computer Graphics, 4(1):37-54, 1998. 3
[36] Liwen Wu, Jae Yong Lee, Anand Bhattachad, Yuxiong Wang, and David Forsyth. Diver: Real-time and accurate neural radiance fields with deterministic integration for volume rendering. In CVPR, 2022. 2, 10
[37] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In ECCV, 2022. 2
[38] Lin Yen-Chen. Nerf-pytorch. https://github.com/yenchenlin/nerf-pytorch/, 2020.8
[39] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR, pages 4578-4587, 2021. 2, 13
[40] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. ArXiv, abs/2010.07492, 2020. 2
[41] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595, 2018. 8
We conduct further experiments, analysis and discussions on our proposed reformulation, where we take a piecewise linear approximation to opacity and piecewise constant approximation to color that results in an integral that is a simple and closed-form expression. This allows us to address the drawbacks of current piecewise constant assumption in NeRFs such as ray conflicts during optimization and a non-invertible CDF that lead to imprecise importance samples and vanishing gradients. We provide additional results in Sec A.1: ablation study (Sec A.1.1), a video demo (Sec A.1.2), additional results on a real dataset (Sec A.1.3), additional qualitative results (Sec A.1.4), comparison with PL-DiVeR A.1.5, additional geometric extraction results A.1.6 and comparison with less number of samples A.1.7. We then provide a walkthrough of the piecewise constant derivation from [15] (Sec A.2), which is followed by the thorough step-by-step derivation of our piecewise linear opacity in volume rendering and precise importance sampling (Sec A.3). We also include analyses on piecewise quadratic and higher order polynomials in Sec A.4. Finally, we end with additional implementation and experiment details (Sec A.5), Limitations (Sec A.6) and Societal Impact (Sec A.7).
A.1.1 Ablation Study
We conduct a further ablation study on our precise importance sampling. As described in Sec. 4 in the main paper, our piecewise linear opacity approximation allows to solve for a closed-form solution for inverse transform sampling leading to the formulation of our precise importance sampling. Unlike the vanilla piecewise constant opacity approximation, our approach results in an invertible CDF, and hence we do not need to define an invertible surrogate function  $G$  for inverse transform sampling as in piecewise constant opacity (see Eq. 8 of main paper), which does not necessarily result in samples from the actual ray distribution  $p(s)$ . We quantitatively ablate the effectiveness of our precise importance sampling (Precise) by replacing our formulation (Eq. 13 main paper) with the surrogate function  $G$  as in the vanilla constant setting (Surrogate). As shown in Table A1 (first and second row of each metric), our precise importance sampling consistently outperforms using the surrogate on all metrics across all 8 scenes in the Blender dataset (hemisphere). Moreover, we also show that our precise importance sampling enables us to use fewer samples for the fine network as it is able to sample correctly from the ray distribution. Hence, keeping the same total number of rendering samples, we are able to achieve a further boost in performance by using 128 coarse and 64 fine samples as shown in Table A1 (second and third row of each metric). We use  $N_{c}$  and  $N_{i}$  to denote the number of coarse and fine samples, respectively.
A.1.2 Video Demo
We also show a video demo of our results. Please see our project page (pl-nerf.github.io) for the video (best viewed in full screen). In the mic scene, we observe that the structure inside the mic is lost in the piecewise constant opacity approximation. Moreover, we see the blurriness along the sides of the body of the mic caused by ray conflicts from the perpendicular and grazing angle views during optimization of the vanilla NeRF model. For the chair scene, we see that our PL-NeRF is able to achieve shaper textures on the back of the chair. Moreover, we see that there is an instability to the samples as illustrated by the inconsistencies in the noise – the gold specs on the green texture (please view in full screen). This is also evident when we vary the camera-to-scene distance. The chair model was trained on multi-distance Blender data to highlight the difference when camera distances vary.
A.1.3 Real Dataset Results
We further evaluate our PL-NeRF on a real dataset - DTU [1]. We train and evaluate our approach on the 15 test scenes used in [39], and report the standard metrics (PSNR, SSIM, LPIPS). We follow the protocol used in [18] for the Real Forward Facing scene where  $\frac{1}{8}$  of the views were held out for testing while the rest are used for training. Table A2 shows the quantitative results averaged over the 15 scenes, where our PL-NeRF outperforms the constant [18] baseline.
GT
Constant
Linear
Figure A1: DTU Qualitative Result Visualizations of rendered DTU dataset test views. Because of the issue of grazing angle and binning inaccuracy of the piecewise constant opacity assumption, the vanilla NeRF (constant) exhibits blurry geometry, and rendering artifacts in the zoomed-in views (middle column). On the other hand, the piecewise linear opacity assumption in our model (linear) alleviates these issues (right column). Overall, the rendered views exhibit sharper surface boundaries and more faithful reconstruction compared to the constant model.