SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 5: Real-world results. PyNeRF reconstructs higher-fidelity details (such as the spokes on the bicycle and the lettering within the boat) than other methods.
Table 2: Real-world results. PyNeRF outperforms all baselines in PSNR and average error, and trains  $40 \times$  faster than Mip-NeRF 360 and  $100 \times$  faster than Exact-NeRF (the next best methods).
Boat [24]	Mip-NeRF 360 [3]
↑PSNR	↑SSIM	↓LPIPS	↓Avg Error	↓Train Time (h)	↑PSNR	↑SSIM	↓LPIPS	↓Avg Error	↓Train Time (h)
Plenoxels [25]	17.05	0.505	0.617	0.185	2:14	21.88	0.606	0.524	0.117	1:00
K-Planes [9]	18.00	0.501	0.590	0.168	2:41	21.53	0.577	0.500	0.120	1:08
TensoRF [6]	14.75	0.398	0.630	0.234	2:30	18.07	0.439	0.677	0.181	1:07
iNGP [20]	15.34	0.433	0.646	0.222	1:42	21.14	0.568	0.521	0.126	0:40
Nerfacto [28]	19.27	0.570	0.425	0.135	2:12	22.47	0.616	0.431	0.105	1:02
Mip-NeRF 360 w/ GLO [3]	20.03	0.595	0.416	0.124	37:28	22.76	0.664	0.342	0.095	37:35
Mip-NeRF 360 w/o GLO [3]	15.92	0.480	0.501	0.194	37:10	22.70	0.664	0.342	0.095	37:22
Exact-NeRF w/ GLO [14]	20.21	0.601	0.425	0.123	109:11	21.40	0.619	0.416	0.121	110:06
Exact-NeRF w/o GLO [14]	16.33	0.489	0.510	0.187	107:52	22.56	0.619	0.410	0.121	108:11
PyNeRF	20.43	0.601	0.422	0.121	2:12	23.09	0.654	0.358	0.094	1:00
interest from different camera distances. For further comparison, we construct a multiscale version of the outdoor scenes in the Mip-NeRF 360 [3] dataset using the same protocol as Multiscale Blender [2].
Baselines. We compare PyNeRF to the same fast-rendering approaches as in Section 4.2, along with two unbounded Mip-NeRF variants: Mip-NeRF 360 [3] and Exact-NeRF [14]. We report numbers on each variant with and without generative latent optimization [18] to account for lighting changes.
Results. We summarize our results in Table 2 along with qualitative results in Figure 5. Once again, PyNeRF outperforms all baselines, trains  $40 \times$  faster than Mip-NeRF 360, and  $100 \times$  faster than Exact-NeRF (the next best alternatives).
4.4 Additional Backbones
Methods. We demonstrate how PyNeRF can be applied to any grid-based NeRF method by evaluating it with K-Planes [9] and TensoRF [6] in addition to our default iNGP-based implementatino. We take advantage of the inherent multi-resolution structure of iNGP and K-Planes by reusing the same feature grid across PyNeRF levels and train a separate feature grid per level in our TensoRF variant.
Results. We train the PyNeRF variants along with their backbones across the datasets described in Section 4.2 and Section 4.3, and summarize the results in Table 3. All PyNeRF variants show clear improvements over their base methods.
4.5 City-Scale Convergence
Dataset. We evaluate PyNeRF's convergence properties on the the Argoverse 2 [35] Sensor dataset (to our knowledge, the largest city-scale dataset publicly available). We select the largest overlapping subset of logs and filter out moving objects through a pretrained segmentation model [7]. The resulting training set contains 400 billion rays across 150K video frames.
Methods. We use SUDS [31] as the backbone model in our experiments. We begin training our method on  $8\times$  downsampled images (containing  $64\times$  fewer rays) for 5,000 iterations and then on
Table 3: Additional backbones. We train the PyNeRF variants along with their backbones across the datasets described in Section 4.2 and Section 4.3 All PyNeRF variants outperform their baselines by a wide margin.
Synthetic	Real-World
↑PSNR	↑SSIM	↓LPIPS	↓Avg Error	↑PSNR	↑SSIM	↓LPIPS	↓Avg Error
iNGP [20]	28.86	0.916	0.087	0.032	19.94	0.541	0.537	0.146
K-Planes [9]	27.90	0.865	0.131	0.047	20.54	0.553	0.520	0.136
TensoRF [6]	29.12	0.902	0.100	0.042	17.21	0.421	0.696	0.200
PyNeRF	36.22	0.979	0.013	0.004	22.65	0.645	0.369	0.098
PyNeRF-K-Planes	35.42	0.975	0.014	0.005	22.00	0.622	0.405	0.108
PyNeRF-TensoRF	35.67	0.976	0.015	0.005	21.35	0.568	0.482	0.122
Table 4: City-scale convergence. We track rendering quality over the first four hours of training. PyNeRF achieves the same rendering quality as SUDS  $2 \times$  faster.
↑ PSNR	↑ SSIM
Time (h)	1:00	2:00	3:00	4:00	Time (h)	1:00	2:00	3:00	4:00
SUDS [31]	16.01	17.41	18.08	18.53	SUDS [31]	0.570	0.600	0.602	0.606
PyNeRF	17.17	18.44	18.59	18.73	PyNeRF	0.614	0.618	0.619	0.621
↓LPIPS	↓Avg Error
Time (h)	1:00	2:00	3:00	4:00	Time (h)	1:00	2:00	3:00	4:00
SUDS [31]	0.531	0.496	0.470	0.466	SUDS [31]	0.182	0.160	0.150	0.145
PyNeRF	0.521	0.485	0.469	0.465	PyNeRF	0.165	0.146	0.144	0.142
progressively higher resolutions (downsampled to  $4 \times$ ,  $2 \times$ , and  $1 \times$ ) every 5,000 iterations hereafter. We compare to the original SUDS method as a baseline.
Metrics. We report the evolution of the quality metrics used in Section 4.2 and Section 4.3 over the first four hours of the training process.
Results. We summarize our results in Table 4. PyNeRF converges more rapidly than the SUDS baseline, achieving the same rendering quality at 2 hours as SUDS after 4.
4.6 Diagnostics
Methods. We validate our design decisions by testing several variants. We ablate our MLP-level interpolation described in Equation 5 and compare it to the GausssPyNeRF and LaplacianPyNeRF variants described in Section 3.2 along with another that instead interpolates the learned grid feature vectors (which avoids the need for an additional MLP evaluation per sample). As increased storage footprint is a potential drawback method, we compare our default strategy of sharing the same multi-resolution feature grid across PyNeRF levels to the naive implementation that trains a separate grid per level. We also explore using 3D sample volumes instead of projected 2D pixel areas to determine voxel levels  $l$ .
Results. We train our method and variants as described in Section 4.2 and Section 4.3, and summarize the results (averaged across datasets) in Table 5. Our proposed interpolation method strikes a good balance — its performance is near-identical to the full LaplacianPyNeRF approach while training  $3 \times$  faster (and is significantly better than the other interpolation methods). Our strategy of reusing the same feature grid across levels performs comparably to the naive implementation while training faster due to fewer feature grid lookups. Using 2D pixel areas instead of 3D volumes to determine voxel level  $l$  provides an improvement.
Table 5: Diagnostics. The rendering quality of our interpolation method is near-identical to the full residual approach while training  $3 \times$  faster, and is significantly better than other alternatives. Reusing the same feature grid across levels performs comparably to storing separate hash tables per level while training faster.
Method	Our Interp.	Shared Features	2D Area	↑PSNR	↑SSIM	↓LPIPS	↓Avg Error	↓ Train Time (h)
GaussPyNeRF (Eq. 3)	X	✓	✓	28.72	0.803	0.201	0.056	0:43
LaplacianPyNeRF (Eq. 4)	X	✓	✓	29.48	0.813	0.190	0.052	2:44
Feature grid interpolation	X	X	✓	28.45	0.767	0.244	0.070	0:46
Separate hash tables	✓	X	✓	29.41	0.813	0.196	0.054	0:52
Levels w/ 3D Volumes	✓	✓	X	29.19	0.811	0.184	0.054	0:48
PyNeRF	✓	✓	✓	29.44	0.812	0.191	0.053	0:48
Although our method generalizes to any grid-based method (Section 4.4), it requires a larger on-disk serialization footprint due to training a hierarchy of spatial grid NeRFs. This can be mitigated by reusing the same feature grid when the underlying backbone uses a multi-resolution feature grid [9, 20], but this is not true of all methods [6, 25].
6 Societal Impact
Our method facilitates the rapid construction of high-quality neural representations in a resource efficient manner. As such, the risks inherent to our work is similar to those of other neural rendering papers, namely privacy and security concerns related to the intentional or inadvertent capture or privacy-sensitive information such as human faces and vehicle license plate numbers. While we did not apply our approach to data with privacy or security concerns, there is a risk, similar to other neural rendering approaches, that such data could end up in the trained model if the employed datasets are not properly filtered before use. Many recent approaches [15, 16, 29, 31, 42] distill semantics into NeRF's representation, which may be used to filter out sensitive information at render time. However this information would still reside in the model itself. This could in turn be mitigated by preprocessing the input data used to train the model [32].
We propose a method that significantly improves the anti-aliasing properties of fast volumetric renderers. Our approach can be easily applied to any existing grid-based NeRF, and although simple, provides state-of-the-art reconstruction results against a wide variety of datasets (while training  $60 - 100\times$  faster than existing anti-aliasing methods). We propose several synthetic scenes that model common aliasing patterns as few existing NeRF datasets cover these scenarios in practice. Creating and sharing additional real-world captures would likely facilitate further research.
Acknowledgements. HT and DR were supported in part by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number 140D0423C0074. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government.