Title: PyNeRF: Pyramidal Neural Radiance Fields
Venue: NeurIPS 2023
Accepted: yes
Keywords: view synthesis

3d reconstruction

scene representation

3d deep learning
Abstract: Neural Radiance Fields (NeRFs) can be dramatically accelerated by spatial grid representations. However, they do not explicitly reason about scale and so introduce aliasing artifacts when reconstructing scenes captured at different camera distances. Mip-NeRF and its extensions propose scale-aware renderers that project volumetric frustums rather than point samples. But such approaches rely on positional encodings that are not readily compatible with grid methods. We propose a simple modification to grid-based models by training model heads at different spatial grid resolutions. At render time, we simply use coarser grids to render samples that cover larger volumes. Our method can be easily applied to existing accelerated NeRF methods and significantly improves rendering quality (reducing error rates by 20–90% across synthetic and unbounded real-world scenes) while incurring minimal performance overhead (as each model head is quick to evaluate). Compared to Mip-NeRF, we reduce error rates by 20% while training over 60x faster.

PyNeRF: Pyramidal Neural Radiance Fields

Abstract

Neural Radiance Fields (NeRFs) can be dramatically accelerated by spatial grid representations [6, 9, 20, 25]. However, they do not explicitly reason about scale and so introduce aliasing artifacts when reconstructing scenes captured at different camera distances. Mip-NeRF and its extensions propose scale-aware renderers that project volumetric frustums rather than point samples but such approaches rely on positional encodings that are not readily compatible with grid methods. We propose a simple modification to grid-based models by training model heads at different spatial grid resolutions. At render time, we simply use coarser grids to render samples that cover larger volumes. Our method can be easily applied to existing accelerated NeRF methods and significantly improves rendering quality (reducing error rates by 20-90% across synthetic and unbounded real-world scenes) while incurring minimal performance overhead (as each model head is quick to evaluate). Compared to Mip-NeRF, we reduce error rates by 20% while training over 60 × faster.

1 Introduction

Recent advances in neural volumetric rendering techniques, most notably around Neural Radiance Fields [19] (NeRFs), have lead to significant progress towards photo-realistic novel view synthesis. However, although NeRF provides state-of-the-art rendering quality, it is notoriously slow to train and render due in part to its internal MLP representation. It further assumes that scene content is equidistant from the camera and rendering quality degrades due to aliasing and excessive blurring when that assumption is violated.

Recent methods [6, 9, 20, 25] accelerate NeRF training and rendering significantly through the use of grid representations. Others, such as Mip-NeRF [2], address aliasing by projecting camera frustum volumes instead of point-sampling rays. However, these anti-aliasing methods rely on the base NeRF MLP representation (and are thus slow) and are incompatible with grid representations due to their reliance on non-grid-based inputs.

Inspired by divide-and-conquer NeRF extensions [22, 23, 27, 30] and classical approaches such as Gaussian pyramids [1] and mipmaps [34], we propose a simple approach that can easily be applied to any existing accelerated NeRF implementation. We train a pyramid of models at different scales, sample along camera rays (as in the original NeRF), and simply query coarser levels of the pyramid for samples that cover larger volumes (similar to voxel cone tracing [8]). Our method is simple to implement and significantly improves the rendering quality of fast rendering approaches with minimal performance overhead.

X

(x）x

C,

E[y(x)]→

→（C,)

(a) NeRF

(b) Mip-NeRF

→Fi(x)→

→(CI, 01)

→F(x)

→(Cr，o=)

(C,o)

(c)Grid Methods (ea:iNGP)

(d）PvNeRE

Contribution: Our primary contribution is a partitioning method that can be easily adapted to any existing grid-rendering approach. We present state-of-the-art reconstruction results against a wide range of datasets, including on novel scenes we designed that explicitly target common aliasing patterns. We evaluate different posssible architectures and demonstrate that our design choices provide a high level of visual fidelity while maintaining the rendering speed of fast NeRF approaches.

2 Related Work

The now-seminal Neural Radiance Fields (NeRF) paper [19] inspired a vast corpus of follow-up work. We discuss a non-exhaustive list of such approaches along axes relevant to our work.

Grid-based methods. The original NeRF took 1-2 days to train, with extensions for unbounded scenes [3, 40] taking longer. Once trained, rendering takes seconds per frame and is far below interactive thresholds. NSVF [17] combines NeRF's implicit representation with a voxel octree that allows for empty-space skipping and improves inference speeds by 10 × . Follow-up works [10, 11, 39] further improve rendering to interactive speeds by storing precomputed model outputs into auxiliary grid structures that bypass the need to query the original model altogether at render time. Plenoxels [25] and DVGO [26] accelerate both training and rendering by directly optimizing a voxel grid instead of an MLP to train in minutes or even seconds. TensoRF [6] and K-Planes [9] instead use the product of low-rank tensors to approximate the voxel grid and reduce memory usage, while Instant-NGP [20] (iNGP) encodes features into a multi-resolution hash table. The main goal of our work is to combine the speed benefits of grid-based methods with an approach that maintains quality across different rendering scales.

Divide-and-conquer. Several works note the diminishing returns in using large networks to represent scene content, and instead render the area of interest with multiple smaller models. DeRF [22] and KiloNeRF [23] focus on inference speed while Mega-NeRF [30], Block-NeRF [27], and SUDS [31] use scene decomposition to efficiently train city-scale neural representations. Our method is similar in philosophy, although we partition across different resolutions instead of geographical area.

Aliasing. The original NeRF assumes that scene content is captured at roughly equidistant camera distances and emits blurry renderings when the assumption is violated. Mip-NeRF [2] reasons about the volume covered by each camera ray and proposes an integrated positional encoding that alleviates aliasing. Mip-NeRF 360 [3] extends the base method to unbounded scenes. Exact-NeRF [14] derives a more precise integration formula that better reconstructs far-away scene content. Bungee-NeRF [36] leverages Mip-NeRF and further adopts a coarse-to-fine training approach with residual blocks to train on large-scale scenes with viewpoint variation. LIRF [37] proposes a multiscale image-based representation that can generalize across scenes. The methods all build upon the original NeRF MLP model and do not readily translate to accelerated grid-based methods.

Concurrent work. Several contemporary efforts explore the intersection of anti-aliasing and fast rendering. Zip-NeRF [4] combines a hash table representation with a multi-sampling method that approximates the true integral of features contained within each camera ray's view frustum. Although it trains faster than Mip-NeRF, it is explicitly not designed for fast rendering as the multi-sampling adds significant overhead. Mip-VoG [12] downsamples and blurs a voxel grid according to the volume of each sample in world coordinates. We compare their reported numbers to ours in Section 4.2. Tri-MipRF [13] uses a similar prefiltering approach, but with triplanes instead of a 3D voxel grid.

Classical methods. Similar to PyNeRF, classic image processing methods, such as Gaussian [1] and Laplacian [5] hierarchy, maintain a coarse-to-fine pyramid of different images at different resolutions. Compared to Mip-NeRF, which attempts to learn a single MLP model across all scales, one could argue that our work demonstrates that the classic pyramid approach can be efficiently adapted to neural volumetric models. In addition, our ray sampling method is similar to Crassin et al.'s approach [8], which approximates cone tracing by sampling along camera rays and querying different mipmap levels according the spatial footprint of each sample (stored as a voxel octree in their approach and as a NeRF model in ours).

(a)

(

c

8

,

σ

8

) =

f

8

(

x

,

d

)

(

c

9

,

σ

9

) =

f

9

(

x

,

d

)

(b)

Model Evaluation

c

= 0

.

4

c

8

+0

.

6

c

9

σ

= 0

.

4

σ

8

+0

.

6

σ

9

(c)

Interpolation

11.2

8.6

5.7

3 Approach

3.1 Preliminaries

NeRF. NeRF [19] represents a scene within a continuous volumetric radiance field that captures geometry and view-dependent appearance. It encodes the scene within the weights of a multilayer perceptron (MLP). At render time, NeRF casts a camera ray r for each image pixel. NeRF samples multiple positions x i along each ray and queries the MLP at each position (along with the ray viewing direction d ) to obtain density and color values σ i and c i . To better capture high-frequency details, NeRF maps x i and d through an L -dimensional positional encoding (PE) γ ( x ) = [sin(2 0 π x ) , cos(2 0 π x ) , . . . , sin(2 L π x ) , cos(2 L π x )] instead of directly using them as MLP inputs. It then composites a single color prediction ˆ C ( r ) per ray using numerical quadrature ∑ N -1 i =0 T i (1 -exp( -σ i δ i )) c i , where T i = exp( -∑ i -1 j =0 σ j δ j ) and δ i is the distance between samples. The training process optimizes the model by sampling batches R of image pixels and minimizing the loss ∑ r ∈ R ∥ ∥ C ( r ) -ˆ C ( r ) ∥ ∥ 2 . We refer the reader to Mildenhall et al. [19] for details.

∥ ∥ Anti-aliasing. The original NeRF suffers from aliasing artifacts when reconstructing scene content observed at different distances or resolutions due to its reliance on point-sampled features. As these features ignore the volume viewed by each ray, different cameras viewing the same position from different distances may produce the same ambiguous feature. Mip-NeRF [2] and variants instead reason about volumes by defining a 3D conical frustum per camera pixel. It featurizes intervals within the frustum with a integrated positional encoding (IPE) that approximates each frustum as a multivariate Gaussian to estimate the integral E [ γ ( x )] over the PEs of the coordinates within it.

Grid-based acceleration. Various methods [6, 9, 20, 25, 26] eschew NeRF's positional encoding and instead store learned features into a grid-based structure, e.g. implemented as an explicit voxel grid, hash table, or a collection of low-rank tensors. The features are interpolated based on the position of each sample and then passed into a hard-coded function or much smaller MLP to produce density and color, thereby accelerating training and rendering by orders of magnitude. However, these approaches all use the same volume-insensitive point sampling of the original NeRF and do not have a straightforward analogy to Mip-NeRF's IPE as they no longer use positional encoding.

3.2 Multiscale sampling

Assume that each sample x (where we drop the i index to reduce notational clutter) is associated with an integration volume. Intuitively, samples close to a camera correspond to small volumes, while samples far away from a camera correspond to large volumes (Figure 3). Our crucial insight for enabling multiscale sampling with grid-based approaches is remarkably simple: we train separate NeRFs at different voxel resolutions and simply use coarser NeRFs for samples covering larger volumes . Specifically, we define a hierarchy of L resolutions that divide the world into voxels of length 1 /N 0 , ..., 1 /N L -1 , where N l +1 = sN l and s is a constant scaling factor. We also define a function f l ( x , d ) at each level that maps from sample location x and viewing direction d to color c and density σ . f l can be implemented by any grid-based NeRF; in our experiments, we use a

Algorithm 1 PyNeRF rendering function

Input: m rays r , L pyramid levels, hierarchy mapping function M , base resolution N 0 , scaling factor s

hash table followed by small density and color MLPs, similar to iNGP. We further define a mapping function M that assigns the integration volume of sample x to the hierarchy level l . We explore different alternatives, but find that selecting the level whose voxels project to the 2D pixel area P ( x ) used to define the integration volume works well:

P

(

x

))

))

(2)

where /ceilingleft · /ceilingright is the ceiling function. Such a model can be seen as a (Gaussian) pyramid of spatial grid-based NeRFs (Fig. 2). If the final density and color were obtained by summing across different pyramid levels, the resulting levels would learn to specialize to residual or 'band-pass' frequencies (as in a 3D Laplacian pyramid [5]):

Our experiments show that such a representation is performant, but expensive since it requires l model evaluations per sample. Instead, we find a good tradeoff is to linearly interpolate between two model evaluations at the levels just larger than and smaller than the target integration volume:

This adds the cost of only a single additional evaluation (increasing the overall rendering time from 0.0045 to 0.005 ms per pixel) while maintaining rendering quality (see Section 4.6). Our algorithm is summarized in Algorithm 1.

Matching areas vs volumes. One might suspect it may be better to select the voxel level l whose volume best matches the sample's 3D integration volume. We experimented with this, but found it more effective to match the projected 2D pixel area rather than volumes. Note that both approaches would produce identical results if the 3D volume was always a cube, but volumes may be elongated along the ray depending on the sampling pattern. Matching areas is preferable because most visible 3D scenes consist of empty space and surfaces, implying that when computing the composite color for a ray r , most of the contribution will come from a few samples x lying near the surface of intersection. When considering the target 3D integration volume associated with x , most of the contribution to the final composite color will come from integrating along the 2D surface (since the rest of the 3D volume is either empty or hidden). This loosely suggests we should select levels of the voxel hierarchy based on (projected) area rather than volume.

Hierarchical grid structures. Our method can be applied to any accelerated grid method irrespective of the underyling storage. However, a drawback of this approach is an increased on-disk serialization footprint due to training a hierarchy of spatial grid NeRFs. A possible solution is to exploit hierarchical grid structures that already exist within the base NeRF. Note that multi-resolution grids such as those

used by iNGP [20] or K-Planes [9] already define a scale hierarchy that is a natural fit for PyNeRF. Rather than learning a separate feature grid for each model in our pyramid, we can reuse the same multi-resolution features across levels (while still training different MLP heads).

Multi-resolution pixel input. One added benefit of the above is that one can train with multiscale training data, which is particularly helpful for learning large, city-scale NeRFs [27, 30, 31, 36, 38]. For such scenarios, even storing high-resolution pixel imagery may be cumbersome. In our formulation, one can store low-resolution images and quickly train a coarse scene representation. The benefits are multiple. Firstly, divide-and-conquer approaches such as Mega-NeRF [31] partition large scenes into smaller cells and train using different training pixel/ray subsets for each (to avoid training on irrelevant data). However, in the absence of depth sensors or a priori 3D scene knowledge, Mega-NeRF is limited in its ability to prune irrelevant pixels/rays (due to intervening occluders) which empirically bloat the size of each training partition by 2 × [30]. With our approach, we can learn a coarse 3D knowledge of the scene on downsampled images and then filter higher-resolution data partitions more efficiently. Once trained, lower-resolution levels can also serve as an efficient initialization for finer layers. In addition, many contemporary NeRF methods use occupancy grids [20] or proposal networks [3] to generate refined samples near surfaces. We can quickly train these along with our initial low-resolution model and then use them to train higher-resolution levels in a sample-efficient manner. We show in our experiments that such course-to-fine multiscale training can speed up convergence (Section 4.5).

Unsupervised levels. A naive implementation of our method will degrade when zooming in and out of areas that have not been seen at training time. Our implementation mitigates this by maintaining an auxiliary data structure (similar to an occupancy grid [20]) that tracks the coarsest and finest levels queried in each region during training. We then use the structure at inference time to only query levels that were supervised during training.

4 Experiments

We first evaluate PyNeRF's performance by measuring its reconstruction quality on bounded synthetic (Section 4.2) and unbounded real-world (Section 4.3) scenes. We demonstrate PyNeRF's generalizability by evaluating it on additional NeRF backbones (Section 4.4) and then explore the convergence benefits of using multiscale training data in city-scale reconstruction scenarios (Section 4.5). We ablate our design decisions in Section 4.6.

4.1 Experimental Setup

Training. We implement PyNeRF on top of the Nerfstudio library [28] and train on each scene with 8,192 rays per batch by default for 20,000 iterations on the Multiscale Blender and Mip-NeRF 360 datasets, and 50,000 iterations on the Boat dataset and Blender-A. We train a hierarchy of 8 PyNeRF levels backed by a single multi-resolution hash table similar to that used by iNGP [20] in Section 4.2 and Section 4.3 before evaluating additional backbones in Section 4.4. We use 4 features per level with a hash table size of 2 20 by default, which we found to give the best quality-performance trade-off on the A100 GPUs we use in our experiments. Each PyNeRF uses a 64-channel density MLP with one hidden layer followed by a 128-channel color MLP with two hidden layers. We use similar model capacities in our baselines for fairness. We sample rays using an occupancy

Plenoxels,30FPS

K-Planes,25FPS

iNGP, 40 FPS

Mip-NeRF,0.1 FPS PyNeRF (Ours),33 FPS

Ground Truth

Mic

Ship

Brick

Checkerboard

grid [20] on the Multiscale Blender dataset, and with a proposal network [3] on all others. We use gradient scaling [21] to improve training stability in scenes with that capture content at close distance (Blender-A and Boat). We parameterize unbounded scenes with Mip-NeRF 360's contraction method.

Metrics. We report quantitative results based on PSNR, SSIM [33], and the AlexNet implementation of LPIPS [41], along with the training time in hours as measured on a single A100 GPU. For ease of comparison, we also report the 'average' error metric proposed by Mip-NeRF [2] composed of the geometric mean of MSE = 10 -PSNR / 10 , √ 1 -SSIM , and LPIPS.

4.2 Synthetic Reconstruction

Datasets. We evaluate PyNeRF on the Multiscale Blender dataset proposed by Mip-NeRF along with our own Blender scenes (which we name 'Blender-A') intended to further probe the anti-aliasing ability of our approach (by reconstructing a slanted checkerboard and zooming into a brick wall).

Baselines. We compare PyNeRF to several fast-rendering approaches, namely Instant-NGP [20] and Nerfacto [28], which store features within a multi-resolution hash table, Plenoxels [25] which optimizes an explicit voxel grid, and TensoRF [6] and K-Planes [9], which rely on low-rank tensor decomposition. We also compare our Multiscale Blender results to those reported by Mip-VoG [12], a contemporary fast anti-aliasing approach, and to Mip-NeRF [2] on both datasets.

Results. We summarize our results in Table 1 and show qualitative examples in Figure 4. PyNeRF outperforms all fast rendering approaches as well as Mip-VoG by a wide margin and is slightly better than Mip-NeRF on Multiscale Blender while training over 60 × faster. Both PyNeRF and Mip-NeRF properly reconstruct the brick wall in the Blender-A dataset, but Mip-NeRF fails to accurately reconstruct checkerboard patterns.

4.3 Real-World Reconstruction

Datasets. We evaluate PyNeRF on the Boat scene of the ADOP [24] dataset, which to our knowledge is one of the only publicly available unbounded real-world captures that captures its primary object of

Plenoxels,3FPS

K-Planes,3FPS

iNGP,4 FPS

Mip-NeRF 360,0.04 FPS PyNeRF(OurS),4 FPS

Ground Truth

Bicycle

Boat

interest from different camera distances. For further comparison, we construct a multiscale version of the outdoor scenes in the Mip-NeRF 360 [3] dataset using the same protocol as Multiscale Blender [2].

Baselines. We compare PyNeRF to the same fast-rendering approaches as in Section 4.2, along with two unbounded Mip-NeRF variants: Mip-NeRF 360 [3] and Exact-NeRF [14]. We report numbers on each variant with and without generative latent optimization [18] to account for lighting changes.

Results. We summarize our results in Table 2 along with qualitative results in Figure 5. Once again, PyNeRF outperforms all baselines, trains 40 × faster than Mip-NeRF 360, and 100 × faster than Exact-NeRF (the next best alternatives).

4.4 Additional Backbones

Methods. Wedemonstrate how PyNeRF can be applied to any grid-based NeRF method by evaluating it with K-Planes [9] and TensoRF [6] in addition to our default iNGP-based implementatino. We take advantage of the inherent multi-resolution structure of iNGP and K-Planes by reusing the same feature grid across PyNeRF levels and train a separate feature grid per level in our TensoRF variant.

Results. We train the PyNeRF variants along with their backbones across the datasets described in Section 4.2 and Section 4.3, and summarize the results in Table 3. All PyNeRF variants show clear improvements over their base methods.

4.5 City-Scale Convergence

Dataset. We evaluate PyNeRF's convergence properties on the the Argoverse 2 [35] Sensor dataset (to our knowledge, the largest city-scale dataset publicly available). We select the largest overlapping subset of logs and filter out moving objects through a pretrained segmentation model [7]. The resulting training set contains 400 billion rays across 150K video frames.

Methods. We use SUDS [31] as the backbone model in our experiments. We begin training our method on 8 × downsampled images (containing 64 × fewer rays) for 5,000 iterations and then on

progressively higher resolutions (downsampled to 4 × , 2 × , and 1 × ) every 5,000 iterations hereafter. We compare to the original SUDS method as a baseline.

Metrics. We report the evolution of the quality metrics used in Section 4.2 and Section 4.3 over the first four hours of the training process.

Results. We summarize our results in Table 4. PyNeRF converges more rapidly than the SUDS baseline, achieving the same rendering quality at 2 hours as SUDS after 4.

4.6 Diagnostics

Methods. We validate our design decisions by testing several variants. We ablate our MLP-level interpolation described in Equation 5 and compare it to the GausssPyNeRF and LaplacianPyNeRF variants described in Section 3.2 along with another that instead interpolates the learned grid feature vectors (which avoids the need for an additional MLP evaluation per sample). As increased storage footprint is a potential drawback method, we compare our default strategy of sharing the same multi-resolution feature grid across PyNeRF levels to the naive implementation that trains a separate grid per level. We also explore using 3D sample volumes instead of projected 2D pixel areas to determine voxel levels l .

Results. We train our method and variants as described in Section 4.2 and Section 4.3, and summarize the results (averaged across datasets) in Table 5. Our proposed interpolation method strikes a good balance - its performance is near-identical to the full LaplacianPyNeRF approach while training 3 × faster (and is significantly better than the other interpolation methods). Our strategy of reusing the same feature grid across levels performs comparably to the naive implementation while training faster due to fewer feature grid lookups. Using 2D pixel areas instead of 3D volumes to determine voxel level l provides an improvement.

5 Limitations

Although our method generalizes to any grid-based method (Section 4.4), it requires a larger on-disk serialization footprint due to training a hierarchy of spatial grid NeRFs. This can be mitigated by reusing the same feature grid when the underlying backbone uses a multi-resolution feature grid [9, 20], but this is not true of all methods [6, 25].

6 Societal Impact

Our method facilitates the rapid construction of high-quality neural representations in a resource efficient manner. As such, the risks inherent to our work is similar to those of other neural rendering papers, namely privacy and security concerns related to the intentional or inadvertent capture or privacy-sensitive information such as human faces and vehicle license plate numbers. While we did not apply our approach to data with privacy or security concerns, there is a risk, similar to other neural rendering approaches, that such data could end up in the trained model if the employed datasets are not properly filtered before use. Many recent approaches [15, 16, 29, 31, 42] distill semantics into NeRF's representation, which may be used to filter out sensitive information at render time. However this information would still reside in the model itself. This could in turn be mitigated by preprocessing the input data used to train the model [32].

7 Conclusion

We propose a method that significantly improves the anti-aliasing properties of fast volumetric renderers. Our approach can be easily applied to any existing grid-based NeRF, and although simple, provides state-of-the-art reconstruction results against a wide variety of datasets (while training 60-100 × faster than existing anti-aliasing methods). We propose several synthetic scenes that model common aliasing patterns as few existing NeRF datasets cover these scenarios in practice. Creating and sharing additional real-world captures would likely facilitate further research.

Acknowledgements. HT and DR were supported in part by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number 140D0423C0074. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government.

References

E. Adelson, C. Anderson, J. Bergen, P. Burt, and J. Ogden. Pyramid methods in image processing. RCA Eng. , 29, 11 1983.

J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In ICCV , 2021.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR , 2022.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-NeRF: Anti-aliased grid-based neural radiance fields. In ICCV , 2023.

P. Burt and E. Adelson. The laplacian pyramid as a compact image code. IEEE Transactions on Communications , 31(4):532-540, 1983. doi: 10.1109/TCOM.1983.1095851.

A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. TensoRF: Tensorial radiance fields. In ECCV , 2022.

B. Cheng, M. D. Collins, Y. Zhu, T. Liu, T. S. Huang, H. Adam, and L.-C. Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR , 2020.

C. Crassin, F. Neyret, M. Sainz, S. Green, and E. Eisemann. Interactive indirect illumination using voxel cone tracing: A preview. In Symposium on Interactive 3D Graphics and Games , 2011.

S. Fridovich-Keil, G. Meanti, F. R. Warburg, B. Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR , 2023.

S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin. FastNeRF: High-fidelity neural rendering at 200fps. In ICCV , 2021.

P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV , 2021.

D. Hu, Z. Zhang, T. Hou, T. Liu, H. Fu, and M. Gong. Multiscale representation for real-time anti-aliasing neural rendering. arxiv:2304.10075, 2023.

W. Hu, Y. Wang, L. Ma, B. Yang, L. Gao, X. Liu, and Y. Ma. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In ICCV , 2023.

B. K. S. Isaac-Medina, C. G. Willcocks, and T. P. Breckon. Exact-NeRF: An exploration of a precise volumetric parameterization for neural radiance fields. CVPR , 2023.

J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. LERF: Language embedded radiance fields. In ICCV , 2023.

S. Kobayashi, E. Matsumoto, and V. Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems , volume 35, 2022.

L. Liu, J. Gu, K. Z. Lin, T.-S. Chua, and C. Theobalt. Neural sparse voxel fields. NIPS , 2020.

R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In CVPR , 2021.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. , 41(4):102:1-15, July 2022. doi: 10.1145/3528223.3530127.

J. Philip and V. Deschaintre. Floaters No More: Radiance Field Gradient Scaling for Improved NearCamera Training. In T. Ritschel and A. Weidlich, editors, Eurographics Symposium on Rendering . The Eurographics Association, 2023. ISBN 978-3-03868-229-5. doi: 10.2312/sr.20231122.

D. Rebain, W. Jiang, S. Yazdani, K. Li, K. Yi, and A. Tagliasacchi. DeRF: Decomposed radiance fields. In CVPR , 2021.

C. Reiser, S. Peng, Y. Liao, and A. Geiger. KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. In ICCV , 2021.

D. Rückert, L. Franke, and M. Stamminger. Adop: Approximate differentiable one-pixel point rendering. ACM Trans. Graph. , 41(4):99:1-14, July 2022. ISSN 0730-0301.

Sara Fridovich-Keil and Alex Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR , 2022.

C. Sun, M. Sun, and H. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR , 2022.

M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar. Block-NeRF: Scalable large scene neural view synthesis. In CVPR , pages 8248-8258, June 2022.

M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings , SIGGRAPH '23, 2023.

V. Tschernezki, I. Laina, D. Larlus, and A. Vedaldi. Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representation. In International Conference on 3D Vision (3DV) , 2022.

H. Turki, D. Ramanan, and M. Satyanarayanan. Mega-NERF: Scalable construction of large-scale NeRFs for virtual fly-throughs. In CVPR , pages 12922-12931, June 2022.

H. Turki, J. Y. Zhang, F. Ferroni, and D. Ramanan. SUDS: Scalable urban dynamic scenes. In CVPR , 2023.

J. Wang, B. Amos, A. Das, P. Pillai, N. Sadeh, and M. Satyanarayanan. A scalable and privacy-aware IoT service for live video analytics. In Multimedia Systems Conference , pages 38-49, 2017.

Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing , 13(4):600-612, 2004.

L. Williams. Pyramidal parametrics. Computer Graphics , 17:1-11, July 1983.

B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In NeurIPS Datasets and Benchmarks , 2021.

Y. Xiangli, L. Xu, X. Pan, N. Zhao, A. Rao, C. Theobalt, B. Dai, and D. Lin. BungeeNeRF: Progressive neural radiance field for extreme multi-scale scene rendering. In ECCV , 2022.

H. Xin, Z. Qi, F. Ying, L. Xiaoyu, W. Xuan, and W. Qing. Local implicit ray function for generalizable radiance field representation. In CVPR , 2023.

L. Xu, Y. Xiangli, S. Peng, X. Pan, N. Zhao, C. Theobalt, B. Dai, and D. Lin. Grid-guided neural radiance fields for large urban scenes. In CVPR , 2023.

A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV , 2021.

K. Zhang, G. Riegler, N. Snavely, and V. Koltun. NeRF++: Analyzing and improving neural radiance fields. arXiv:2010.07492, 2020.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

S. Zhi, T. Laidlow, S. Leutenegger, and A. Davison. In-place scene labelling and understanding with implicit scene representation. In ICCV , 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works tackles the problem of antialiasing in grid based Neural Radiance Field representations (e.g. INGP, DirectVoxGo). To this end, a very simple solution is proposed: *instead of training a single NeRF with multiscale features, train separate NeRFs at different resolutions and decide which to use based on the distance of the sample from the camera (size of the pixel footprint)*. Specifically, two levels (closest smaller and larger) are used to predict the color and density, which are then linearly combined into a single prediction per sample along the ray. This requires one additional MLP evaluation per each sample, but is still much more efficient than the multisamples proposed e.g. in ZipNeRF. The proposed method is evaluated on the Multiscale Blender and Multiscale Mip-Nerf360 datasets, where it consistently outperforms the state of the art.\n\nSTRENGTHS:\nIn my opinion, the main strength of this paper is its simplicity. The proposed idea is very general and could easily be implemented in the grid based representation at only a minor cost (one additional MLP evaluation per each sample). The increased storage is also not a large problem, as shown in the supplementary, as features can be shared across level resulting only in the overhead of MLP weights (very small compared to the feature volumes). I really like such simple ideas that lead to good performance improvement.\n\nThe paper is also well presented and easy to understand, the ablation studies support the main ideas (Pyramid training, simplification of the Laplacian pyramid). I especially liked the analysis in the supplementary material (it is maybe unfortunate to have that only in the supplement and could maybe be moved to the main paper for the camera ready version).\n\nWEAKNESSES:\nI only have one minor weakness - sometimes some details are missing, or the description is slightly ambiguous. For example, the number of levels, size of the features per level, MLP dimensions are missing. In the supplementary, it is not fully clear to me what the shared INGP table denotes? In the original INGP each level has its own hash-table, does this imply that a single one is used for all level (what is the dimension and number of the features in this hash-table?)\n\nQUESTIONS:\n**Comments - potential improvement**:\n- The proposed method is very general, but it is only evaluated on the INGP representation. While TensorRF is used in the ablation in the supplementary, it is not clear if the proposed multilevel training also improves its performance. It would be great to support the claims of the generality by applying the ideas to at least one more representation. \n\n**Questions**:\n- The gradients for each camera sample are only propagated back to 2 levels. If the camera poses are very different, this could mean that in some areas the finest levels are never supervised - what happens if the novel view is closer than camera of the training view? It would mean that the levels that are sampled were never supervised or? \n\n- In the representation in the main paper, I don't fully understand why the storage requirements are larger? Is it because there is no concatenation of the features, and hence the feature vectors of each level are of higher dimension? It would be good to clarify this\n\n- If I understand correctly, the method proposed in the supplementary with the shared features across the levels is even simpler (only adds L-1 MLPs to the formulation) and achieves comparable performance. Is there are good reason to not make that the main method? I guess it is not trivial to share the features in the methods that store them explicitly (tensorRF or DirectVoxGo).\n\nLIMITATIONS:\nThe authors describe the limitations of their method and potential negative societal impacts adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPyNeRF replaces the implicit representation in Mip-NeRF with a voxel-based representation method, which combines the cone sampling method and explicit structural representation by interpolating on different voxels based on coordinates of different scales. This method can be easily applied to existing accelerated NeRF methods. PyNeRF improves training speed by 60 times compared to Mip-NeRF while ensuring anti-aliasing effects and reducing errors by 20%.\n\nSTRENGTHS:\n* Experiments are performed on multiple datasets, including synthetic, real, and large-scale datasets, which demonstrate the superior performance and training speed of the proposed method. PyNeRF combines the anti-aliasing capability of MiP-NeRF with the fast fitting of explicit structure, showcasing the advantages of the two methods.\n* Three strategies are adopted for the interpolation method (GaussPyNeRF, LaplacianPyNeRF, pyNeRF), demonstrating that the PyNeRF approach can save storage, improve rendering speed, and ensure the same rendering quality as interpolating on multiple voxels. This verifies the rationality of PyNeRF's interpolation strategy.\n\nWEAKNESSES:\n* The paper states that the proposed method can be applied to existing accelerated NeRF approaches (L9). However, the results show only slight improvements when applied to other acceleration methods (PyNeRF - TensoRF-CP, PyNeRF - TensoRF-VM in Table 5 of the supplementary). It appears that the use of hash tables is the only approach that shows significant improvement.\n* Although the multi-scale sampling and interpolation methods in this paper play a role in anti-aliasing and accelerating training speed, they also increase the consumption of storage space. According to Table 5, using a shared hash table can achieve the same performance, so why not use a shared hash table? This would save storage space.\n* Moreover, adopted methods, e.g. multiscale sampling and multi-resolution voxels, have been proposed and widely used in many existing methods, which makes the novelty somewhat limited.\n\nQUESTIONS:\n* Further analysis is needed to demonstrate why the gain on TensoRF is marginal.\n* The feasibility of the method can be tested on more accelerated NeRF approaches. The storage size of this method may be the main factor that impedes its wider application.\n\nLIMITATIONS:\nThe authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a pyramidal radiance field reconstruction method, which reuses multi-scale feature grid representation and area matching algorithm for level indexing. Specifically, the method trains a pyramid of models at different scales and interpolates point features between neighboring levels determined by the cone size to which the point belongs to. The effectiveness of the proposed method is evaluated on the multi-resolution version of the blender, ADOP and Argoverse sensor dataset. The experiments show that the proposed method is able to provide better rendering quality while preserving high reconstruction speed.\n\nSTRENGTHS:\n1. The motivation is clear. The manuscript addresses an interesting and important anti-aliasing problem for grid-based representation. It's good to a paper on that. \n\n2. The manuscript is well-written. Figure 1 and algorithm 1 are very helpful for understanding the manuscript, good job.\n\n3. The quantitive results reported in the tables are quite good.\n\nWEAKNESSES:\n1. The quantitive result on the standard resolution is missing, such as single resolution for nerf synthetic, mipnerf 360, leading it hard to judge the general performance.\n\n2. The optimization progress is unclear to me. Is the different level feature optimized with the specific scale of the dataset (L113-115) or supervised only on the final rendering (L129)?\n\n3. Experiment details are missing, such as level numbers used for each dataset, the grid resolutions, and the feature channel, representation (dense grid?).\n\n4. Since the manuscript is focused on anti-aliasing, but I could not find the zoom-in results in the appendix video, are there flickers when rendering sequences?\n\nQUESTIONS:\n1. Can this representation be trained and evaluated on a standard multi-view dataset?\n\n2. Table 1 and 2, any insights on why the quantitive (Plenoxels, K-Planes, TensoRF and iNGP)) and the rendering video of iNGP is much worse than the result reported in the original papers? Is that because they are evaluated on multi-resolution? In this case, I would expect the rendering quality to be the same.\n\nLIMITATIONS:\nYes, the authors provide a limitation section and make sense to me.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This work presents a method for anti-aliased renderings for grid-based NeRF representations by jointly optimizing a hierarchy of coarse-to-fine grids. The idea is neat and well justified by empirical evaluations that show quantitative and qualitative improvements over baselines.

STRENGTHS:
1. The proposed method seems simple and easy to reproduce.

2. Experiments are extensive, and demonstrate the effectiveness of the proposed method.

WEAKNESSES:
1. The method is built upon the nerfstudio library. But the empirical results of that base model seems missing. Would be great to include those results to help better see the improvements made by the proposed anti-aliasing techniques.

QUESTIONS:
1. It says that the method is implemented on top of the nerfstudio library. Which specific model was built upon? Is it the nerfacto one, or others?

2. What do the numbers 11.2,8.6,5.7 mean in Fig. 3(a)?

3. Is the MLP shared across the different mipmap levels? What is the MLP size?

4. Does the method apply to other grid-based representations like TensoRF, or K-planes?

LIMITATIONS:
Limitations are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to address the aliasing artifacts in grid-based NeRF. It introduces a pyramid of grids of different resolutions to represent a scene. To query the color and density of a 3D point with a certain integration volume, the method finds the two pyramid levels that best describe the point and performs a weighted combination between the outputs of the two levels. Experiments on both synthetic and real datasets demonstrate the advantages of PyNeRF over previous methods in rendering quality and speed.\n\nSTRENGTHS:\n- How to address alias artifacts is an important problem in NeRF. While mip-NeRF proposes an elegant solution for positional-encoding-based NeRF, extending this to grid-based NeRF is not straightforward. This paper presents a reasonable and novel approach to address the aliasing issue in grid-based NeRF. This is achieved by maintaining a grid pyramid with different levels of resolutions and only using suitable levels during rendering.\n\n- The proposed PyNeRF shows clear advantages over previous approaches, as demonstrated in experiments. The rendering accuracy is significantly better than other grid-based NeRF and in most cases better than mip-NeRF. Qualitative results also show that PyNeRF better captures fine details. The running speed is on par with other grid-based NeRF and is much faster than MLP-based NeRF.\n\n- The effectiveness of the proposed design choice is verified in ablation studies.\n\nWEAKNESSES:\n- The technical contribution is weak, as the proposed method is rather simple and does not involve much technical challenge. The iNGP already has a multi-resolution grid and this work simply chooses to use the two suitable for the scale of the point. Although this strategy is shown to be effective, I doubt if the technical contribution is enough to be presented at NeurIPS.\n\n- Some descriptions are not clear or reasonable. For example:\n\n  - The sentence on Line 121 and 122 is unclear. It's not well explained how P(x) is calculated. It seems to be the projected area from the voxel to the 2D plane. If this is the case, then it is a somewhat odd design. A voxel is a cube, thus its projected area to the 2D plane is view-dependent. But the selection of scale level should not depend on view but depend on the distance between the 3D point and the camera.\n\n  - The first version of the method is called GaussPyNeRF. However, it is not explained how the method is related to Gaussian. I do not see a clear connection.\n\n  - Line 156: \"We can quickly train these along with our initial low-resolution model and then use them to train higher-resolution levels in a sample-efficient manner.\" It's not clearly described how this is done.\n\nQUESTIONS:\n- Authors can respond to my concerns mentioned in weaknesses.\n\n- Line 155: \"many contemporary NeRF methods use occupancy grids or proposal networks to generate refined samples near surfaces\". I suggest adding related citations here.\n\n- Line 35: fast-rendering approach -> grid-based approach. Fast rendering includes not only grid-based approaches but also others like point-based approaches. This paper targets grid-based approaches.\n\n- The title does not mention anti-aliasing, which is the main problem to be solved in this paper. A minor suggestion is to mention it in the title.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}