PyNeRF: Pyramidal Neural Radiance Fields
Authors: Haithem Turki
Michael Zollhöfer
Christian Richardt
Deva Ramanan
Venue: NeurIPS 2023 poster (2023)
Keywords: view synthesis
3d reconstruction
scene representation
3d deep learning
TL;DR: We propose an accelerated, scale-aware NeRF method that is competitive with Mip-NeRF (while training much faster)

Abstract: Neural Radiance Fields (NeRFs) can be dramatically accelerated by spatial grid representations. However, they do not explicitly reason about scale and so introduce aliasing artifacts when reconstructing scenes captured at different camera distances. Mip-NeRF and its extensions propose scale-aware renderers that project volumetric frustums rather than point samples. But such approaches rely on positional encodings that are not readily compatible with grid methods. We propose a simple modification to grid-based models by training model heads at different spatial grid resolutions. At render time, we simply use coarser grids to render samples that cover larger volumes. Our method can be easily applied to existing accelerated NeRF methods and significantly improves rendering quality (reducing error rates by 20–90% across synthetic and unbounded real-world scenes) while incurring minimal performance overhead (as each model head is quick to evaluate). Compared to Mip-NeRF, we reduce error rates by 20% while training over 60x faster.

PyNeRF: Pyramidal Neural Radiance Fields
Haithem Turki
Michael Zollhöfer Meta Reality Labs Research zollhoef wreetaa.rrrrea....mommmmma......
Carnegie Mellon University hturki@cs. cmu. edu
wweea. com
Christian Richardt Meta Reality Labs Research cri rrichardt@met... com
Deva Ramanan Carnegie Mellon University devaccs cmu edu
Abstract
Neural Radiance Fields (NeRFs) can be dramatically accelerated by spatial grid representations [6, 9, 20, 25J. However, they do not explicitly reason about scale and so introduce aliasing artifacts when reconstructing scenes captured at different camera distances. Mip-NeRI and its extensions propose scale-aware renderers that project volumetric frustums rather than point samples but such approaches rely on positional encodings that are not readily compatible with grid methods. We propose a simple modification to grid-based models by training model heads at different spatial grid resolutions. render samples that cover larger volumes. Our method can be easily applied to existing accelerated NeRF methods and significantly improves rendering quality (reducing error rates by 20-90% across synthetic and unbounded real- weal-world scenes) while incurring minimal performance overhead (as each model head is quick to evaluate). Compared to Mip-NeRF, we reduce enror rates by 20% while training over 60x faster.
render time, we simply use coarser grids
Introduction
Recent advances in neural volumetric rendering techniques, most notably around Neural Radiance Fields [19] (NeRFs), have lead significant progress towards photo-realistic novel view synthesis. However, although NeRF provides state-of-the-art rendering quality, it is notoriously slow train and render due in part to its internal MLP representation. It further assumes that scene content is equidistant from the camera and rendering quality degrades due to aliasing and excessive blurring when that assumption is violated.
Recent methods [6, 9, 20, 25] accelerate NeRF training and rendering significantly through the use of grid representations. Others, such as Mip-NeRF [2), address aliasing by projecting camera frustum volumes instead of point-sampling rays. However, these anti-aliasing methods rely the base NeRF ML.P representation (and are thus slow) and are incompatible with grid representations due their reliance on non-grid-based inputs.
Inspired by divide-and-come aand-couuuer NeRF extensions [22, 23, 27, 30] and classical approaches such as Gaussian pyramids [1] and mipmaps [34], we propose a simple approach that can easily be applied to any existing accelerated NeRF implementation. We train pyramid of models different scales, sample along camera rays (as in the original NeRF), and simply query coarser levels of the pyramid for samples that cover larger volumes (similar voxel cone tracing [8]). Our method simple to implement and significantly improves the rendering quality of fast rendering approaches with minimal performance overhead.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

v(x)
(c, ø)
Ely(o)
(a) NeRF
(b) Mip-NeRF
F((x)+
F(x)
(c, a)
c.
Fi(X)*
an)
(c) Grid Methods (eg: iNGP)
(d) PyNeRF
Figure 1: Comparison of methods. (a) NeRF traces ray from the camera's ' center of projection through each pixel and samples points x along each ray. Sample locations are then encoded with a positional encoding produce a feature 7xx) that is fed into an MLP. (b) Mip-NeRF instead reasons about volumes by defining a 3D conical frustum per camera pixel. It splits the frustum into sampled volumes, approximates them multivariate Gaussians, and computes the integral of the positional encodings of the coordinates contained within the Gaussians. Similar to NeRF, these features are then fed into an MLP. (c) Accelerated grid methods, such as iNGP, sample points as in NeRF, but do not use positional encoding and instead featurize each point by interpolating between vertices in a feature grid. These features are then passed into much smaller MLP, which greatly accelerates training and rendering. (d) PyNeRF also uses feature grids, but reasons about volumes by training separate models at different scales (similar to a mipmap). calculates the area covered by each sample in world coordinates, queries the models at the closest corresponding resolutions, and interpolates their outputs.
Our primary contribution a partitioning method that can be easily adapted to any existing grid-rendering approach. We present state-of-the-art reconstruction results against wide range of datasets, including on novel scenes we designed that explicitly target common aliasing patterns. We evaluate different posssible architectures and demonstrate that design choices provide high level of visual fidelity while maintaining the rendering speed of fast NeRF approaches.
Contribution:
Related Work
The now-seminal Neural Radiance Fields (NeRF) paper [19] inspired vast corpus of follow-up work. We discuss non-exhaustive list of such approaches along axes relevant to our work.
Grid-based methods. The original NeRF took 1-2 days train, with extensions for unbounded scenes [3, 40] taking longer. Once trained, rendering takes seconds per frame and is far below interactive thresholds. NSVF [17] combines NeRF's implicit representation with voxel octree that allows for empty-space skipping and improves inference speeds by 10x. Follow-up works [10, 11, 39] further improve rendering to interactive speeds by storing precomputed model outputs into auxiliary grid structures that bypass the need to query the original model altogether at render time. Plenoxels [25] and DVGO [26] accelerate both training and rendering by directly optimizing a voxel grid instead of an MLP to train in minutes or even seconds. TensoRF [6] and K-Planes [9] instead use the product of low-rank tensors to approximate the voxel grid and reduce memory usage, while Instant-NGP [20] (iNGP) encodes features into a multi-resolution hash table. The main goal of our work to combine the speed benefits of grid-based methods with an approach that maintains quality across different rendering scales.

---

Level 1
Level 2
Level 3
Level 4
Figure 2: We visualize renderings from a pyramid spatial grid-based NeRFs trained for different voxel resolutions. Models at finer pyramid levels tend to capture finer content.
Divide-and-conque Several works note the diminishing returns in using large networks to represent scene content, and instead render the area interest with multiple smaller models. DeRF [22] and KiloNeRF [23] focus on inference speed while Mega-NeRF [30], Block-NeRF [27], and SUDS [31] use scene decomposition efficiently train city-scale neural representations. Our method similar in philosophy, although partition across different resolutions instead of geographical area.
Aliasing. The original NeRF assumes that scene content is captured at roughly equidistant camera distances and emits blurry renderings when the assumption is violated. Mip-NeRF [2] reasons about volume covered by each camera ray and proposes an integrated positional encoding that alleviates aliasing. Mip-NeRF 360 [3] extends the base method to unbounded scenes. Exact-NeRF [ [14] derives a more precise integration formula that better reconstructs far-away scene content. Bungee-NeRFF [36] leverages Mip-NeRF and further adopts coarse-to- tonfeett-nneee training approach with residual blocks to train on large-scale scenes with viewpoint variation. LIRF [37] proposes a multiscale image-based based representation that can generalize across scenes. The methods all build upon the original NeRF MLP model and do not readily translate to accelerated grid-bases based methods.
Concurrent work. Several contemporary efforts explore the intersection of anti-aliasing and fast rendering. Zip-NeRF [4] combines a hash table representation with multi-sane samplinm method that approximates the true integral of features contained within each camera ray's view frustum. Although trains faster than Mip-NeRF, it is explicitly not designed for fast rendering as the multi-sampling adds significant overhead. VoG [12] downsamples and blurs voxel grid according the volume of each sample in world coordinates. We compare their reported numbers to ours in Section 4.2. Tri- Tri-MipRF [13] uses similar prefiltering approach, but with triplanes instead of 3D voxel grid.
Classical methods. Similar PyNeRF, classic image processing methods, such as Gaussian [1] and Laplacian [5] hierarchy, maintain a coarse-to-fine pyramid of different images at different resolutions. Compared to Mip-NeRF, which attempts to learn single MLP model across all scales, one could argue that our work demonstrates that the classic pyramid approach can be efficiently adapted to neural volumetric models. In addition, our ray sampling method is similar to Crassin al.'s approach [8], which approximates cone tracing by sampling along camera rays and querying different mipmap levels according the spatial footprint of each sample (stored as voxel octree their approach and as NeRF model in ours).

---

(cs, as) fa(x, d)
0.4cş + 0.6cg
O 0.40g + 0.60g
11.2
8.6
5.7
(co, .9)) = fo(x, (x.d) d)
(a) Point Sampling
(b) Model Evaluation
(c) Interpolation
Figure 3: Overview. (a) We sample frustums along the camera ray corresponding each pixel and
derive the scale of each sample according to its width in world coordinates. (b) We query the model heads closest to the scale of each sample. (c) We derive single color and density value for each sample by interpolating between model outputs according to scale.
Approach
3.1 Preliminaries
NeRE NeRF [19] represents a scene within a continuous volumetric radiance field that captures geometry and view-dependent appearance. It encodes the scene within the weights of a multi- layer perceptron (MLP). At render time, NeRF casts camera ray for each image pixel. NeRF samples multiple positions x; along each ray and queries the MLP at each position (along with the ray viewing direction d) to obtain density and color values σ; and C,. To better capture high-frequency details, NeRF maps X; and d through an L-dimensional positional encoding (PE) Tx), min))"rrz)
instead of directly using them as MLP inputs. It then composites a single color prediction C(r) per ray using numerical quadrature Σ T2(1 exp(-o,):)) c, where T; exp(- Ei= oÿτ;)) and 8, is the distance between samples. The training process optimizes the model by sampling batches R of image pixels and minimizing the loss ErEr C(r) - C(+)|' (()))-(r)](' We refer the reader Mildenhall al. [19] for details.
y(z) = [sin(2" nT), mz)"............................ sin
Anti-aliasing, The original NeRF suffers from aliasing artifacts when reconstructing scene content observed at different distances resolutions due its reliance on point-sampled features. As these features ignore the volume viewed by each ray, different cameras viewing the same position from different distances may produce the same ambiguous feature. Mip-NeRF [2] and variants instead reason about volumes by defining a 3D conical frustum per camera pixel. It featurizes intervals within frustum with integrated positional encoding (IPE) that approximates each frustum as multivariate Gaussian estimate the integral [[7(x)) over the PEs of the coordinates within it.
Grid-hased acceleration. Various methods [6, 9, 20, 25, 26] eschew NeRF's positional encoding and instead store learned features into a based structure. implemented as explicit voxel grid, hash table, collection of low-rank tensors. The features are interpolated based on the position of each sample and then passed into hard-coded function or much smaller MLP to produce density and color, thereby accelerating training and rendering by orders of magnitude. However, these approaches all use the same volume-insensie point sampling of the original NeRF and do not have a straightforward analogy to Mip-NeRF's IPE they no longer use positional encoding.
3.2 Multiscale sampling
Assume that each sample x (where we drop the index to reduce notational clutter) is associated with an integration volume. Intuitively, samples close camera correspond small volumes, while samples far away from camera correspond to large volumes (Figure 3). Our crucial insight for approaches remarkably simple: we train separate
enabling multiscale sampling with grid- NeRFs at different voxel resolutions and simply use coarser NeRFs for samples covering larger volumes. Specifically, we define a hierarchy of resolutions that divide the world into voxels of length 1/No. 1/N1.11, where N = sN; and is constant scaling factor. We also define function Ji(x, d) at each level that maps from sample location x and viewing direction color and density ft can be implemented by any grid-based NeRF; in experiments, use

---

Algorithm PyNeRF rendering function
Input: m rays r, L pyramid levels, hierarchy mapping function M, base resolution No. scaling
factor
Output: m estimated colors
x, d, P(x) < sample(r)
> Saamle points x along each ray with direction d and area P(x)
(P))x) t log, (P(x)
> Equation
min 1, max(0, (P()))))
Equation
w - - 111-MMM MP()x)
b Eqqation
model_out < zeros(len(x)
Zero-initialize model outputs for each sample
for i in unique(!) do
P Iterate over sample levels
model_oute
+= w|l = [x = i] d +))
model_oull{ += (1 w) (1-i)(=ifi--(()) i). dll i
(x
end for composite(moode__oo
Composite model outputs into per-ray color
return c
hash table followed by small density and color MLPs, similar to iNGP. We further define a mapping function M that assigns the integration volume of sample x to the hierarchy level I. We explore different alternatives, but find that selecting the level whose voxels project to the 2D pixel area P(x) used define the integration volume works well:
M(P(x) log, (P) 1= mimmin - 1, max(0, [M((()))))
(1)
(2)
0, (x)
[GaussPyNeRF]
where [.] is the ceiling function. Such model can be seen (Gaussian) pyramid of spatial grid-based NeRFs (Fig. 2). If the final density and color were obtained by summing across different pyramid levels, the resulting levels would learn to specialize residual or "band pass" frequencies (as in a 3D Laplacian pyramid [5]):
G,C Σ dx. 1=0
[LaplacianPyNeRF] (4)
Our experiments show that such a representation is performant, but expensive since it requires / model evaluations per sample. Instead, we find a good tradeoff is to linearly interpolate between two model evaluations at the levels just larger than and smaller than the target integration volume:
0, c = (x,, d) + (1 - 1 d), where w = M(P(x)). (Default) [PyNeRF] (5)
This adds the cost of only a single additional evaluation (increasing the overall rendering time from 0.0045 to 0.005 ms per pixel) while maintaining rendering quality (see Section 4.6). Our algorithm is summarized in Algorithm 1.
Matching areas volumes. One might suspect it may be better to select the voxel level I whose volume best matches the sample's 's 3D integration volume. We experimented with this, but found more effective to match the projected 2D pixel area rather than volumes. Note that both approaches would produce identical results the 3D volume was always a cube, but volumes may elongated along the ray depending on the sampling pattern. Matching areas is preferable because most visible 3D scenes consist of empty space and surfaces, implying that when computing the composite color for ray most of the contribution will come from a few samples lying near the surface of intersection. When considering the target 3D integration volume associated with x, most of the contribution to the final composite color will come from integrating along the 2D surface (since the rest of the 3D volume is either empty or hidden). This loosely suggests we should select levels of the voxel hierarchy based on (projected) area rather than volume.
Hierarchical grid structures. Our method can be applied to any accelerated grid method irrespective of the underyling storage. However, drawback of this approach is an increased on-disk serialization footprint due to training hierarchy of spatial grid NeRFs. A possible solution to exploit hierarchical grid structures that already exist within the base NeRF. Note that multi-resolution grids such those
5

---

Table 1: Synthetic results. PyNeRF outperforms all baselines and trains 60x faster than Mip- NeRF. Both PyNeRF and Mip-NeRF properly reconstruct the brick wall in the Blender-A dataset, but Mip-NeRF fails to accurately reconstruct checkerboard patterns.
Multiscale Blender
Blender-A
JPSNR
1SSIM
JIPIIPS
JAvg Error
Train Time (h)
1PSNR
1SSIM
L.PPPS
LAvg Frorr
1 Train Time (h)
Pienoxels [25]
24.98
0.843
0.161
0:28
0.511
0.190
0:40
K-Planes [9]
29.88
0.946
0.058
0.022
0:32
21.17
0.593
0.641
0.405
1:22
TensoRF [6]
30.04
0,948
0,056
0.021
0:27
27.01
0.785
0.197
0.054
1:20
iNGP [20]
30.21
0.040
0.022
20.85
0.767
0.089
0:56
Nerfacto [28]]
29.56
0,947
0.051
0.022
0:25
27.46
0.796
0.195
0.053
1:07
Mip- VoG [12]
30.42
0,954
0,053
Mip-NeRFI 12]
34.50
0.974
0.017
0,009
29:49
31.33
0.894
0.098
0.063
30:12
PyNeRF
34.78
0.976
0.015
0,008
0:25
41.99
0,986
0,007
0,004
1:10
used by iNGP [20] or K-Planes [9] already define scale hierarchy that is natural fit for PyNeRF. Rather than learning a separate feature grid for each model in our pyramid, we can reuse the same multi-resolution features across levels (while still training different MLP heads).
Multi-resolution pixel input. One added benefit of the above is that one can train with multiscale training data, which is particularly helpful for learning large, city-scale NeRFs 30,,30, 31, 31,36, 38]. For such scenarios, even storing high-resolution pixel imagery may be cumbersome. our formulation, one can store low-resolution images and quickly train a coarse scene representation. The benefits are multiple. Firstly, divide-and-conque approaches such as Mega-NeRF [31] partition large scenes into smaller cells and train using different training pixel/ray subsets for each (to avoid training on irrelevant data). However, in the absence of depth sensors or a priori 3D scene knowledge, Mega- NeRg is limited in its ability to prune irrelevant pixels/rays (due intervening occluders) which empirically bloat the size of each training partition by 2x [30]. With our approach, we can learn a coarse 31) knowledge of the scene downsampled images and then filter higher-resolution data partitions more efficiently. Once trained, lower-resolution levels can also serve as an efficient initialization for finer layers. In addition, many contemporary NeRF methods use occupancy grids [20] or proposal networks [3] to generate refined samples near surfaces. We can quickly train these along with our initial low-resolution model and then use them train higher-resolution levels in sample-efficient manner. We show in our experiments that such course-to-fine multiscale training can speed up convergence (Section 4.5).
Unsupervised levels. A naive implementation of our method will degrade when zooming in and out of areas that have not been seen training time. Our implementation mitigates this by maintaining an auxiliary data structure (similar to an occupancy grid [20]) that tracks the coarsest and finest levels queried in each region during training. We then use the structure at inference time to only query levels that were supervised during training.
Experiments
We first evaluate PyNeRF's performance by measuring its reconstruction quality on bounded synthetic (Section 4.2) and unbounded real-world (Section 4.3) scenes. We demonstrate PyNeRF's generaliz- ability by evaluating it on additional NeRF backbones (Section 4.4) and then explore the convergence benefits of using multiscale training data in city-scale reconstruction scenarios (Section 4.5). We ablate our design decisions in Section 4.6.
4.1 Experimental Setup
Training. We implement PyNeRF on top of the Nerfstudio library [28] and train on each scene with 8. rays per batch by default for 20,000 iterations on the Multiscale Blender and Mip-NeRF 360 datasets, and 50,000 iterations on the Boat dataset and Blender-A. We train a hierarchy of 8 PyNeRF levels backed by a single multi-resolutie
hash table similar to that used by iNGP [20] in Section 4.2 and Section 4.3 before evaluating additional backbones in Section 4.4. We use 4 features per level with a hash table size of 220 by default, which we found to give the best trade-off on the A 1100 GPUs we use in our experiments. Each PyNeRF uses a 64-channel density MI.P with one hidden layer followed by a 128-channel color MI.P with two hidden layers. We use similar model capacities in our baselines for faimess. We sample rays using an occupancy

---

Plenoxels, Planes,
Mip NcRF, PyNeRF (Ours),
Mic
Ship
Brick
Checkerboand
Figure 4: Synthetic results. PyNeRF and Mip-NeRF provide comparable results on the first three scenes that are crisper than those of the other fast renderers. Mip-NeRF does not accurately render tiles the last row while PyNeRF recreates them near-perfectly.
grid [20] the Multiscale Blender dataset, and with a proposal network [3] all others. We use gradient scaling [21] to improve training stability in scenes with that capture content at close distance (Blender-A and Boat). We parameterize unbounded scenes with Mip-NeRF 360's contraction method.
Metrics. We report quantitative results based on PSNR, SSIM [33], and the AlexNet implementation of LPIPS [41], along with the training time in hours measured comparison, we also report the "average" error metric proposed by Mip-NeRF [2] composed of the geometric mean of MSE 1S-PSNR/10 /I SSIM, and LPIPS.
single A100 GPU. For ease of
4.2 Synthetic Reconstruction
Datasets. We evaluate PyNeRF on the Multiscale Blender dataset proposed by Mip-NeRF along with our own Blender scenes (which we name "Blender-A") intended to further probe the anti-aliasing ability of our approach (by reconstructing slanted checkerboard and zooming into brick wall).
Baselines. We compare PyNeRF to several fast-rendering approaches, namely Instant-NGPP [20] and Nerfacto [28], which store features within multi-resolution hash table, Plenoxels [25] which optimizes an explicit voxel grid, and TensoRF [6] and K-Planes [9], which rely on low-rank tensor decomposition. We also compare our Multiscale Blender results to those reported by Mip-VoG [12], a contemporary fast anti-alian aati-aaiasing approach, and to Mip-NeRF [2] on both datasets.
Results. We summarize our results in Table I and show qualitative examples in Figure 4. PyNeRF outperforms all fast rendering approaches well as Mip-VoG VoG by a wide margin and slightly better than Mip-NeRF on Multiscale Blender while training over 60x faster. Both PyNeRF and Mip-NeRF properly reconstruct the brick wall in the Blender-A dataset, Mip-NeRF fails to accurately reconstruct checkerboard patterns.
4.3 Real-World Reconstruction
Datasets. We evaluate PyNeRF on the Boat scene of the ADOP [24] dataset, which to our knowledge one of the only publicly available unbounded real-world captures that captures its primary object of
7

---

FPS
INGP.
Mio-NeRF
Ground
Bicycle
Boat
Figure 5: Real-world results. PyNeRF reconstructs higher-fidelity details (such as the spokes on the bicycle and the lettering within the boat) than other methods.
Table Real-world results. PyNeRF outperforms all baselines PSNR and average error, and trains 40x faster than Mip-NeRF F 360 and 100x faster than Exact-NeRF (the next best methods).
Bont [4]
Mip-NeRF 360 [3]
TSNR
15SIM
LLPIPS
[Ang Error
|Triin Time (h)
PPSR
15SIM
_LPIPS
[Avg Error
Troin Time (h)
Plenoxels
17.05
0.505
0.617
0,185
2:14
21.88
0.606
0.524
0.117
1:00
K-Planes
18.00
0.501
0.590
0.168
2:41
21.53
0.577
0.500
0.120
1:08
TensoRF
iNGP[20 [20]
15.34
0:433
0.222
1:42
21.14
0.568
0.521
0.126
0:40
Nerfacio [2S]
19.27
0.570
0.425
0.135
2:12
22.47
0.616
0.431
0,105
1:02
Mip-NeRFF 360 GL.O [3]
20.03
0.505
0.416
0.124
37:28
22.76
0.664
0.342
D1005
37:35
Mip-NeRF 360 wro GLO [3]
15.92
0,460
0.194
37:10
22.70
0.664
0.342
0.085
37:22
Exact-NeRF [14]
11006
Exact-NeRF [14]
16.33
0.187
107:52
22.56
0.619
0.410
PyNoRF
20.43
06601
0.422
0.121
2:12
23.09
0.654
0.358
0.094
1:00
interest from different camera distances. For further comparison, construct multiscale version of the outdoor scenes in the Mip-NeRF 360 [3] dataset using the same protocol as Multiscale Blender [2].
Baselines. We compare PyNeRF to the same fast-rendering approaches in Section 4.2, along with two unbounded Mip-NeRF variants: Mip-NeRF 360 [3] and Exact-NeRF [14]. We report numbers on each variant with and without generative latent optimization [18] to account for lighting changes.
Results. We summarize our results in Table 2 along with qualitative results in Figure 5. Once again, PyNeRF outperforms all baselines, trains 40x faster than Mip-NeRF 360, and 100x faster than Exact-NeRF (the next best alternatives).
4.4 Additional Backbones
Methods. We demonstrate how PyNeRF can be applied to any grid-based NeRF method by evaluating it with K-Planes [9] and TensoRF [6] in addition to our default iNGP-based implementatino. We take advantage of the inherent multi-resolution structure of iNGP and K-Planes by reusing the same feature grid across PyNeRF levels and train separate feature grid per level in our TensoRF variant.
Results. We train the PyNeRF variants along with their backbones across the datasets described in Section 4.2 and Section 4.3, and summarize the results in Table 3. All PyNeRF variants show clear improvements over their base methods.
4.5 City-Scale Convergence
Dataset. We evaluate PyNeRF's convergence properties on the the Argoverse 2 [35] Sensor dataset (to our knowledge, the largest city-scale dataset publicly available). We select the largest overlapping subset of logs and filter out moving objects through pretrained segmentation model [7]. The resulting training contains 400 billion rays across 1.5KK video frames.
Methods. We use SUDS [31] as the backbone model in our experiments. We begin training our method on 8x downsampled images (containing 64x fewer rays) for 5,000 iterations and then

---

Table Additional backbones. We train the PyNeRF variants along with their backbones across the datasets described in Section 4.2 and Section 4.3 All PyNeRF variants outperform their baselines by a wide margin.
Synthetic
Real-World
1PSNR
1SSIM
JLPIPS
Avv Error
1PSNR
1SSIM
JLPIPS
LAvg Error
iNGP [20]
28.86
0.916
0.087
0.032
19.94
0.541
0.537
0.146
K-Planes [9]
27.90
0.865
0.131
0.047
20.54
0.553
0.520
0.136
TensoRF [6]
29.12
0.902
0.100
0.042
17.21
0.421
0.696
0.200
PyNeRF
36.22
0.979
0.013
0.004
22.65
0.645
0.369
0.098
PyNeRF-K-Planes
35.42
0.975
0.014
0.005
22.00
0.622
0.405
0.108
PyNeRF-TensoRF
35.67
0.976
0.015
0.005
21.35
0.568
0.482
0.122
Table 4: City-scale convergence. We track rendering quality over the first four hours of training. PyNeRF achieves the same rendering quality as SUDS 2x faster.
PSNR
SSIM
Time (h)
1:00
2:00
3:00
Time (h)
1:00
2:00
3:00
4:00
SUDS [31] PyNeRF
16.01
17.41
18.08
18.53
SUDS [3]
0.570
0.600
0.602
0.606
17.17
18.44
18.59
18.73
PyNeRF
0.614
0.618
0.619
0.621
4 L.PPSS
4 Avg Error
Time (h)
1:00
2:00
3:00
4:00
Time (h)
1:00
2:00
3:00
4:00
SUDS [31]
0.531
0.496
0.470
0.466
SUDS [31]
0.182
0.160
0.150
0.145
PyNeRF
0.521
0.485
0.469
0,465
PyNeRF
0.165
0.146
0.144
0.142
progressively higher resolutions (downsampled 4x, 2x, and 1x) every 5,000 iterations hereafter. We compare to the original SUDS method as a baseline.
Metrics. We report the evolution of the quality metrics used in Section 4.2 and Section 4.3 over the first four hours of the training process.
Results. We summarize our results in Table 4. PyNeRF converges more rapidly than the SUDS baseline, achieving the same rendering quality at 2 hours as SUDS after 4.
4.6 Diagnostics
Methods. We validate our design decisions by testing several variants. We ablate our MLP-level interpolation described in Equation 5 and compare it to the GausssPyNeRF and NePR variants described in Section 3.2 along with another that instead interpolates the learned grid feature vectors (which avoids the need for an additional MIP evaluation per sample). increased storage footprint is a potential drawback method, we compare our default strategy of sharing the same multi-resolution feature grid across PyNeRF levels to the naive implementation that trains separate grid per level. We also explore using 3D sample volumes instead of projected 2D pixel areas to determine voxel levels l.
Results. We train our method and variants as described in Section 4.2 and Section 4.3, and summarize the results (averaged across datasets) in Table Our proposed interpolation method strikes good balance - its performance is near-identical to the full LaplacianPyNeel 3x faster (and significantly better than the other interpolation methods). Our strategy reusing the same feature grid across levels performs comparably to the naive implementation while training faster due to fewer feature grid lookups. Using 2D pixel areas instead of 3D volumes to determine voxel level provides improvement.
approach while training

---

Table Diagnostics. The rendering quality our interpolation method near-identical to the full residual approach while training 3x faster, and is significantly better than other alternatives. Reusing the same feature grid across levels performs comparably to storing separate hash tables per level while training faster.
Our Interp.
Shared Features
2D Area
$ Avg Error
Train Time (h)
Method
tPSNR
1SSIM
JLPIPS
GaussPyNeRF (Eq. 3)
28.72
0.803
0.201
0.056
0:43
LaplacianPyNett (Eq. Feature grid interpolation Separate hash tables Levels w/ 3D Volumes
29.48
0.813
0.190
0.052
2:44
28.45
0.767
0.244
0.070
0:46
29.41
0.813
0.196
0.054
0:52
29.19
0.811
0.184
0.054
0:48
PyNeRF
29.44
0.812
0.191
0.053
0:48
5 Limitations
Although our method generalizes to any grid-based method (Section 4.4), it requires a larger on-disk serialization footprint due to training hierarchy of spatial grid NeRFs. This can be mitigated by reusing the same feature grid when the underlying backbone uses a multi-resolution feature grid [9, 20], but this is not true of all methods [6, 25].
6 Societal Impact
Our method facilitates the rapid construction of high-quality neural representations in a resource efficient manner. As such, the risks inherent our work is similar to those of other neural rendering papers, namely privacy and security concerns related to the intentional or inadvertent capture or privacy-sensitive information such as human faces and vehicle license plate numbers. While we did not apply our approach to data with privacy or security concerns, there is a risk, similar to other neural rendering approaches, that such data could end up the trained model if the employed datasets are not properly filtered before use. Many recent approaches [15, 16, 29, 31, 42] distill semantics into NeRF's s representation, which may be used to filter out sensitive information at render time. However this information would still reside in the model itself. This could in turn be mitigated by preprocessing the input data used train the model [32].
Conclusion
We propose method that significantly improves the anti-aliasing properties of fast volumetric renderers. Our approach can be easily applied to any existing grid-based NeRF, and although simple, provides ffate-of-the-ar reconstruction results against wide variety of datasets (while training 60-100x faster than existing anti-aliasing methods). We propose several synthetic scenes that model common aliasing patterns few existing NeRF datasets cover these scenarios in practice. Creating and sharing additional real-world captures would likely facilitate further research.
Acknowledgements. HT and DR were supported in part by the Intelligence Advanced Research Projects Activity (IARPA) via Department Interior/ Interior Business Center (DOI/IBC) contract number 1 140D0423C0074. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOL/IBĆ, or the U.S. Govermment.
10

---

References
E. Adelson, C. Anderson, J. Bergen. P. Burt, and J. Ogden. Pyramid methods in image processing. RCA Eng., 29, 1983.
[2] Barron, B. Mildenhall, M. Tancik, Hedman, Martin Brualla, and P. Srinivasan. Mip-NeRF: multiscale representation anti-aliasing neural radiance fields. ICCV, 2021.
[3] Barron, B. Mildenhall, Verbin, P. P. Srinivasan, and P. Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022.
[4] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-NeRF: Anti-aliased grid-based neural radiance fields. In ICCV, 2023.
[5] P. Burt and E. Adelson. The laplacian pyramid as compact image code. IEEE Trunsactions on Communications, 31(4):532-540, 1983. doi: 10.1109/COM.199 1983. 1983.109851..
[6] A. Chen, Z. Xu, A. Geiger. J. Yu, and H. Su. TensoRF: Tensorial radiance fields. In ECCV, 2022.
[7] B. Cheng, M. D. Collins, Y. Zhu, T. Liu, T. S. Huang, H. Adam, and L.-C. Chen. Panoptic-deeplab: simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, 2020.
[8] C. Crassin, F. Neyret, M. Sainz, S. Green, and E. Eisemann, Interactive indirect illumination using voxel tracing: A preview. In Symposium Interactive 3D Graphics and Games, 2011.
[9] S. Fridovich-Keil. G. Meanti, F. R. Warburg, B. Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
[10] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and Valentin. FastNeRF: High-fidelity neural rendering at 200fps. ICCV, 2021.
[11] P. Hedman, Srinivasan, Mildenhall, Barron, and Debevee, Baking neural radiance fields for real-time view synthesis. In ICCV, 2021.
[12] D. Zhang, Hou. Liu, Fu, and M. Gong. Multiscale representation for real-time anti aliasing neural rendering. arxiv:2304. 10075,
[13] Hu, Y. Wang. L. Ma, B. Yang, L. Gao, X. Liu, and Ma. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In ICCV, 2023.
[14] B. K. S. Isaac-Medina C. G. Willcocks, and T. P. Breckon. Exact-NeRF: An exploration of precise volumetric parameterization for neural radiance fields. CVPR, 2023.
[15] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. LERF: Language embedded radiance fields. In ICCV, 2023.
[16] S. Kobayashi, E. Matsumoto, and V. Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems, volume 35, 2022.
[17] L. Liu, J. Gu, K. Z. Lin, T.SS. Chua, and C. Theobalt. Neural sparse voxel fields. NIPS, 2020.
[18] R. Martin-Brualla N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth, NeRF in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
[19] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields view synthesis. ECCV, 2020.
[20] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1-13 July 2022. doi: 10.114553522233353000
[21] Philip and V. Deschaintre. Floaters No More: Radiance Field Gradient Scaling Improved Near- Camera Training. In T. Ritschel and A. Weidlich, editors, Eurographics Symposiun on Rendering. The Eurographics Association, 2023. ISBN 978-3-03868-229- doi: 10.2312/sr.20231 122.
[22] D. Rebain, W. Jiang, S. Yazdani, K. Li, K. Yi, and A. Tagliasacchi. DeRF: Decomposed radiance fields. In CVPR, 2021.
[23] C. Reiser. S. Peng. Y. Liao, and A. Geiger. KiloNeRF: Speeding neural radiance fields with thousands of tiny MLPs. In ICCV, 2021.

---

[24] D. Rückert, L. Franke, and M. Stamminger. Adop: Approximate differentiable one-pixel point rendering. ACM Trany. Graph., 91:41:99:111, July 2022. ISSN 0730-0301.
[25] Sara Fridovich-Keil and Alex Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[26] C. Sun, M. Sun, and H. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.
[27] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar. Block-NeRIF: Scalable large neural view synthesis. In CVPR, pages $248-8258, June 2022.
[28] M. Tancik, E. Weher, E. Ng. Li, Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, and A. Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH '23, 2023.
[29] V. Tschemezii, I. Laina, D. Larlus, and A. Vedaldi. Neural Feature Fusion Fields: 3D distillation of self-super 2D image representation. International Conference on 3D Vision (3DV), 2022.
[30] H. Turki, D. Ramanan, and M. Satyanarayanan. Mega-NERF: Scalable construction of large-scale NeRPs virtual thytthoougs.. CVPR, pages 12922-12931, June 2022.
[31] H. Turki, Y. Zhang, F. Ferroni, and Ramanan. SUDS: Scalable urban dynamic scenes. CVPR, 2023.
[32] J. Wang, B. Amos, Das, Pillai, N. Sadch, and M. Satyanarayanan, scalable and privacy-aware IoT service live video analytics. Multimedia Systems Conference, pages 38 38-49, 2017.
[33] Z. Wang, A. Bovik, H. Sheikh, and Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions on Image Processing, 13(4)6000-12, 2004.
[34] L. Williams. Pyramidal parametrics. Computer Graphics, 17:1-11, July 1983.
[35] B. Wilson, W. Qi. T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, D. Ramanan, P. Carr, and J. Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. NeurlPS Datasets and Benchmarks, 2021.
[36] Y. Xiangli, Xu, X. Pan, N. Zhao, Rao, C. Theobalt, Dai, and Lin. BungeeNeRF: Progressive neural radiance field for extreme multi-scale scene rendering. In ECCV, 2022.
[37] H. Xin, Qi, F. Ying, Xiaoyu, W. Xuan, and W. Qing. Local implicit ray function for generalizable radiance field representation. CVPR, 2023.
[38] L. Xu, Xiangli, S. Peng. Pan, N. Zhao, Theobalt, Dai, and Lin. Grid-guided neural radiance fields for large urban scenes. CVPR, 2023.
[39] A. Yu, Li, Tancik, H. Ng, and A. Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In /CCV, 2021.
[40] Zhang, G. Riegler, N. Snavely, and Koltun. NeRF++: Analyzing and improving neural radiance fields. arXiv:2010.07492, 2020.
[41] R. Zhang, Isola, A. Efros, Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
[42] S. Zhi, T. Laidlow. Leutenegger. and Davison. In-place scene labelling understanding with implicit seene representation. In /CCV, 2021.
12


Reviews:
[{"review_id": "E72lWg9SAw", "reviewer": "Reviewer_LPiw", "summary": "This paper presents a method to address the aliasing artifacts in grid-based NeRF. It introduces a pyramid of grids of different resolutions to represent a scene. To query the color and density of a 3D point with a certain integration volume, the method finds the two pyramid levels that best describe the point and performs a weighted combination between the outputs of the two levels. Experiments on both synthetic and real datasets demonstrate the advantages of PyNeRF over previous methods in rendering quality and speed.", "questions": "- Authors can respond to my concerns mentioned in weaknesses.\n\n- Line 155: \"many contemporary NeRF methods use occupancy grids or proposal networks to generate refined samples near surfaces\". I suggest adding related citations here.\n\n- Line 35: fast-rendering approach -> grid-based approach. Fast rendering includes not only grid-based approaches but also others like point-based approaches. This paper targets grid-based approaches.\n\n- The title does not mention anti-aliasing, which is the main problem to be solved in this paper. A minor suggestion is to mention it in the title.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- How to address alias artifacts is an important problem in NeRF. While mip-NeRF proposes an elegant solution for positional-encoding-based NeRF, extending this to grid-based NeRF is not straightforward. This paper presents a reasonable and novel approach to address the aliasing issue in grid-based NeRF. This is achieved by maintaining a grid pyramid with different levels of resolutions and only using suitable levels during rendering.\n\n- The proposed PyNeRF shows clear advantages over previous approaches, as demonstrated in experiments. The rendering accuracy is significantly better than other grid-based NeRF and in most cases better than mip-NeRF. Qualitative results also show that PyNeRF better captures fine details. The running speed is on par with other grid-based NeRF and is much faster than MLP-based NeRF.\n\n- The effectiveness of the proposed design choice is verified in ablation studies.\n\n", "weaknesses": "- The technical contribution is weak, as the proposed method is rather simple and does not involve much technical challenge. The iNGP already has a multi-resolution grid and this work simply chooses to use the two suitable for the scale of the point. Although this strategy is shown to be effective, I doubt if the technical contribution is enough to be presented at NeurIPS.\n\n- Some descriptions are not clear or reasonable. For example:\n\n  - The sentence on Line 121 and 122 is unclear. It's not well explained how P(x) is calculated. It seems to be the projected area from the voxel to the 2D plane. If this is the case, then it is a somewhat odd design. A voxel is a cube, thus its projected area to the 2D plane is view-dependent. But the selection of scale level should not depend on view but depend on the distance between the 3D point and the camera.\n\n  - The first version of the method is called GaussPyNeRF. However, it is not explained how the method is related to Gaussian. I do not see a clear connection.\n\n  - Line 156: \"We can quickly train these along with our initial low-resolution model and then use them to train higher-resolution levels in a sample-efficient manner.\" It's not clearly described how this is done.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the valuable feedback! \n\n**Contribution.** Rather than seeing the simplicity of our method as a weakness, we humbly agree with Reviewer LTbu's statement that \"the main strength of this paper *is* its simplicity\". As you state, our simple method significantly outperforms the state of the art.\n\n**Lines 121-122.** We calculate P(x) solely based on the distance between the 3D point and camera similar to Nerfstudio's utility method [1], and then compare it to the resolution hierarchy described in lines 115-116. For example, for a point along the camera ray with a projected pixel area of 0.125, and a PyNeRF hierarchy corresponding to resolutions [2, 4, 8, 16], we query the third level of the hierarchy (with resolution 1 / 8 = 0.125). We will clarify in the camera-ready.\n\n**GaussPyNeRF.** Our intent was to draw an analogy between the resolution and details captured by the different levels in our NeRF hierarchy (Fig. 2) and those characteristic of Gaussian pyramids used in classic image processing [2]. We will clarify.\n\n**Line 155.** SOTA methods either use a proposal network (Mip-NeRF 360, Nerfacto, SUDS, K-Planes) or an occupancy grid (iNGP) to learn a coarse 3D representation and guide sampling near surfaces (vs wastefully querying empty space). Occupancy grids / proposal networks are usually trained jointly with the main NeRF network, which takes significant time at city scale. Selectively training in a coarse-to-fine manner with the low-resolution PyNeRF levels as described in Sec. 4.4 speeds up convergence.\n\n**Line 35.** Agreed - we will amend!\n\nReferences:\n\n[1] https://github.com/nerfstudio-project/nerfstudio/blob/9b6010ea0d20e7ca2a68496bb33d6dcd03bf9b91/nerfstudio/cameras/cameras.py#L849\n\n[2] E. Adelson, C. Anderson, J. Bergen, P. Burt, and J. Ogden. Pyramid methods in image processing. RCA250 Eng., 29, 11 1983.\n"}, {"review_id": "3R5VvWpjbU", "reviewer": "Reviewer_PaSq", "summary": "This work presents a method for anti-aliased renderings for grid-based NeRF representations by jointly optimizing a hierarchy of coarse-to-fine grids. The idea is neat and well justified by empirical evaluations that show quantitative and qualitative improvements over baselines.", "questions": "1. It says that the method is implemented on top of the nerfstudio library. Which specific model was built upon? Is it the nerfacto one, or others?\n\n2. What do the numbers 11.2,8.6,5.7 mean in Fig. 3(a)?\n\n3. Is the MLP shared across the different mipmap levels? What is the MLP size?\n\n4. Does the method apply to other grid-based representations like TensoRF, or K-planes?\n", "limitations": "Limitations are adequately addressed. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method seems simple and easy to reproduce.\n\n2. Experiments are extensive, and demonstrate the effectiveness of the proposed method.", "weaknesses": "1. The method is built upon the nerfstudio library. But the empirical results of that base model seems missing. Would be great to include those results to help better see the improvements made by the proposed anti-aliasing techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper! We’re glad that you appreciate the simplicity and effectiveness of our approach. \n\n**Nerfstudio.** Our implementation is indeed built on the library (we will release our code as a plugin) and is closest to Nerfacto. We list comparison numbers in the tables attached as part of our global response which we will add to our camera ready. We show a 1-7db improvement in PSNR.\n\n**MLPs.** We use separate MLPs per level. We use a 64 channel density MLP with 1 hidden layer followed by a 128 channel color MLP with 2 hidden layers. We use the same size MLPs across the iNGP/K-Planes/Nerfacto baselines for fairness.\n\n**Other backbones.** A strength of our approach is its applicability to other grid-based representations - we list results with a K-Planes backbone as part of our global rebuttal which shows a significant improvement over its baseline (2-6 db PSNR) and report a 4-6 db improvement in PSNR when using TensoRF.\n\n**Fig 3(a).** The numbers are to illustrate the different hierarchy levels each ray sample could fall under (Equation 1 in the paper) - we will clarify."}, {"review_id": "qOkaw31XwW", "reviewer": "Reviewer_JoFm", "summary": "The authors introduce a pyramidal radiance field reconstruction method, which reuses multi-scale feature grid representation and area matching algorithm for level indexing. Specifically, the method trains a pyramid of models at different scales and interpolates point features between neighboring levels determined by the cone size to which the point belongs to. The effectiveness of the proposed method is evaluated on the multi-resolution version of the blender, ADOP and Argoverse sensor dataset. The experiments show that the proposed method is able to provide better rendering quality while preserving high reconstruction speed.", "questions": "1. Can this representation be trained and evaluated on a standard multi-view dataset?\n\n2. Table 1 and 2, any insights on why the quantitive (Plenoxels, K-Planes, TensoRF and iNGP)) and the rendering video of iNGP is much worse than the result reported in the original papers? Is that because they are evaluated on multi-resolution? In this case, I would expect the rendering quality to be the same.", "limitations": "Yes, the authors provide a limitation section and make sense to me.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is clear. The manuscript addresses an interesting and important anti-aliasing problem for grid-based representation. It's good to a paper on that. \n\n2. The manuscript is well-written. Figure 1 and algorithm 1 are very helpful for understanding the manuscript, good job.\n\n3. The quantitive results reported in the tables are quite good.", "weaknesses": "1. The quantitive result on the standard resolution is missing, such as single resolution for nerf synthetic, mipnerf 360, leading it hard to judge the general performance.\n\n2. The optimization progress is unclear to me. Is the different level feature optimized with the specific scale of the dataset (L113-115) or supervised only on the final rendering (L129)?\n\n3. Experiment details are missing, such as level numbers used for each dataset, the grid resolutions, and the feature channel, representation (dense grid?).\n\n4. Since the manuscript is focused on anti-aliasing, but I could not find the zoom-in results in the appendix video, are there flickers when rendering sequences?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading our work and for the constructive feedback!  \n\n**Single vs multi-resolution.** You ask why existing methods such as iNGP perform worse on multi-resolution datasets. Mip-NeRF [1] originally points out that prior work struggles on scenes where the same scene content is viewed from different distances across training and test cameras. For scenes when the camera distance remains roughly constant, PyNeRF performs similarly to existing SOTA, as shown in Table 3 in our rebuttal PDF. mip-NeRF [1] simulates views from different distances by adding 2×–8× downsampled images to training and test sets. Similar to mip-NeRF [1], our rebuttal PDF includes performance numbers on each individual downsampling level in Tables 1 and 2.\n\n**Optimization.** You ask whether we train different pyramid levels separately or jointly with the final rendering loss (L129). We train jointly; different samples along a camera ray typically map to different levels of the hierarchy (Figure 3(a)), but all samples are composited into a single color that is supervised with a per-pixel L2 loss.\n\n**Experiment details.** We will add more details to our camera-ready version. Our main experiments use a PyNeRF hierarchy of 8 levels corresponding to resolutions 128 to 16,384. Each level of the hierarchy is backed by a feature hashtable (same as iNGP), followed by small density and color MLPs (1 layer/64 channels and 2 layers/128 channels, respectively). We will commit to releasing code/data upon acceptance to aid reproducibility. We will also add videos with zoom in effects (which do not exhibit flickering).\n\nReferences:\n\n[1] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021"}, {"review_id": "AlNAWOaeHr", "reviewer": "Reviewer_pHej", "summary": "PyNeRF replaces the implicit representation in Mip-NeRF with a voxel-based representation method, which combines the cone sampling method and explicit structural representation by interpolating on different voxels based on coordinates of different scales. This method can be easily applied to existing accelerated NeRF methods. PyNeRF improves training speed by 60 times compared to Mip-NeRF while ensuring anti-aliasing effects and reducing errors by 20%.", "questions": "* Further analysis is needed to demonstrate why the gain on TensoRF is marginal.\n* The feasibility of the method can be tested on more accelerated NeRF approaches. The storage size of this method may be the main factor that impedes its wider application.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Experiments are performed on multiple datasets, including synthetic, real, and large-scale datasets, which demonstrate the superior performance and training speed of the proposed method. PyNeRF combines the anti-aliasing capability of MiP-NeRF with the fast fitting of explicit structure, showcasing the advantages of the two methods.\n* Three strategies are adopted for the interpolation method (GaussPyNeRF, LaplacianPyNeRF, pyNeRF), demonstrating that the PyNeRF approach can save storage, improve rendering speed, and ensure the same rendering quality as interpolating on multiple voxels. This verifies the rationality of PyNeRF's interpolation strategy.", "weaknesses": "* The paper states that the proposed method can be applied to existing accelerated NeRF approaches (L9). However, the results show only slight improvements when applied to other acceleration methods (PyNeRF - TensoRF-CP, PyNeRF - TensoRF-VM in Table 5 of the supplementary). It appears that the use of hash tables is the only approach that shows significant improvement.\n* Although the multi-scale sampling and interpolation methods in this paper play a role in anti-aliasing and accelerating training speed, they also increase the consumption of storage space. According to Table 5, using a shared hash table can achieve the same performance, so why not use a shared hash table? This would save storage space.\n* Moreover, adopted methods, e.g. multiscale sampling and multi-resolution voxels, have been proposed and widely used in many existing methods, which makes the novelty somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading our paper and for the constructive feedback.\n\n**Contribution.** To the best of our knowledge, our work is among the first to combine fast NeRF rendering with anti-aliasing. We agree that testing on more accelerated NeRF approaches would improve our paper. To that effect, we present additional results with K-Planes in the global rebuttal PDF which show a 2–6 dB improvement over the standard K-Planes baseline. Per your suggestion, we also reexamined our TensoRF results. As its authors state, it is designed for bounded scenes [1] and our initial results on unbounded scenes were especially poor. By adding optimizations not present in the original TensoRF method (namely Mip-NeRF 360's scene contraction [2]), we're able to improve baseline TensoRF performance (from 14.75 to 17.21 dB PSNR on unbounded scenes) and obtain a large improvement when combined with PyNeRF (21.35 dB). We will add these updated numbers and experiment details to the camera-ready version.\n\n**Storage.** As noted in the Limitations section, we agree that storage space is a potential tradeoff of our approach. Our intent with the shared hashtable approach in the supplement was to provide a mitigation for methods that use multi-resolution structures (iNGP, K-Planes), at the cost of generalizability to those that don't (TensoRF, Plenoxels). We will make this more clear in the revised paper by presenting both the general PyNeRF method and the storage-optimized version (for multi-resolution backbones) in the main paper.\n\nReferences:\n\n[1] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022. TensoRF: Tensorial Radiance Fields. In ECCV 2022.\n\n[2] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022."}, {"review_id": "km7KTXIeUJ", "reviewer": "Reviewer_LTbu", "summary": "This works tackles the problem of antialiasing in grid based Neural Radiance Field representations (e.g. INGP, DirectVoxGo). To this end, a very simple solution is proposed: *instead of training a single NeRF with multiscale features, train separate NeRFs at different resolutions and decide which to use based on the distance of the sample from the camera (size of the pixel footprint)*. Specifically, two levels (closest smaller and larger) are used to predict the color and density, which are then linearly combined into a single prediction per sample along the ray. This requires one additional MLP evaluation per each sample, but is still much more efficient than the multisamples proposed e.g. in ZipNeRF. The proposed method is evaluated on the Multiscale Blender and Multiscale Mip-Nerf360 datasets, where it consistently outperforms the state of the art.", "questions": "**Comments - potential improvement**:\n- The proposed method is very general, but it is only evaluated on the INGP representation. While TensorRF is used in the ablation in the supplementary, it is not clear if the proposed multilevel training also improves its performance. It would be great to support the claims of the generality by applying the ideas to at least one more representation. \n\n**Questions**:\n- The gradients for each camera sample are only propagated back to 2 levels. If the camera poses are very different, this could mean that in some areas the finest levels are never supervised - what happens if the novel view is closer than camera of the training view? It would mean that the levels that are sampled were never supervised or? \n\n- In the representation in the main paper, I don't fully understand why the storage requirements are larger? Is it because there is no concatenation of the features, and hence the feature vectors of each level are of higher dimension? It would be good to clarify this\n\n- If I understand correctly, the method proposed in the supplementary with the shared features across the levels is even simpler (only adds L-1 MLPs to the formulation) and achieves comparable performance. Is there are good reason to not make that the main method? I guess it is not trivial to share the features in the methods that store them explicitly (tensorRF or DirectVoxGo). ", "limitations": "The authors describe the limitations of their method and potential negative societal impacts adequately. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "In my opinion, the main strength of this paper is its simplicity. The proposed idea is very general and could easily be implemented in the grid based representation at only a minor cost (one additional MLP evaluation per each sample). The increased storage is also not a large problem, as shown in the supplementary, as features can be shared across level resulting only in the overhead of MLP weights (very small compared to the feature volumes). I really like such simple ideas that lead to good performance improvement.\n\nThe paper is also well presented and easy to understand, the ablation studies support the main ideas (Pyramid training, simplification of the Laplacian pyramid). I especially liked the analysis in the supplementary material (it is maybe unfortunate to have that only in the supplement and could maybe be moved to the main paper for the camera ready version). ", "weaknesses": "I only have one minor weakness - sometimes some details are missing, or the description is slightly ambiguous. For example, the number of levels, size of the features per level, MLP dimensions are missing. In the supplementary, it is not fully clear to me what the shared INGP table denotes? In the original INGP each level has its own hash-table, does this imply that a single one is used for all level (what is the dimension and number of the features in this hash-table?) ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the helpful comments! \n\n**Generalization.** With regards to improvements being marginal for other backbones such as TensoRF, we admittedly focused on hashtable approaches for our original submission. As the authors of TensoRF state, it is mainly designed for bounded scenes [1] and our initial evaluation of TensoRF on unbounded 360 views yielded poor results. In response to reviewer concerns, we have added optimizations not present in the original TensoRF method (such as Mip-NeRF 360's scene contraction [2]), improving the baseline TensoRF from 14.75 to 17.21 dB PSNR, and now see a large improvement when combined with PyNeRF (21.35 dB). We also ran experiments with K-Planes which show similar improvements with PyNeRF (2–6 dB gain in PSNR). We have included updated tables in our rebuttal PDF.  \n\n**Unsupervised levels.** You are correct that one of the limitations of PyNeRF is that performance will degrade when zooming in and out of areas that have not been seen at training time. \"Zoom-out\" (rendering far-away views) can be handled by simulating far-away views during training, by simply adding downsampled training images. \"Zoom-in\" (rendering nearby views) is fundamentally challenging since one cannot easily simulate such views without hallucination-based methods such as super-resolution. Instead, we can force renderings to query only the finest level that was supervised at train-time (by maintaining an occupancy grid-like data structure), which should result in the same blurry artifacts that typical NeRF approaches exhibit during excessive \"zoom-in\". We will note these limitations in the camera-ready paper.\n\n**Storage.** To clarify why storage requirements are larger with our base method, each level is backed by an entirely separate feature grid (which in the case of iNGP happens to be a multi-resolution grid). In the general case (which doesn't assume anything about the underlying storage of the NeRF backbone), we create a separate table for each level (up to the target resolution of the level). In the case of iNGP, this is redundant and we can just use a single multi-resolution table for all levels. As you correctly assumed, we don't do this in the general case since this does not apply to backbones such as TensoRF or Plenoxels. That said, the use of a multi-resolution backbone is particularly natural for our method, and so we plan to present both the general and storage-optimized variant (for multi-resolution backbones) in the main paper."}]

Author rebuttal:
We are glad that reviewers agree that we address "an interesting and important anti-aliasing problem," (JoFm), appreciate "simple ideas that lead to good performance improvement," (LTbu), and acknowledge that "experiments are extensive, and demonstrate the effectiveness of the proposed method." (PaSq)

Two common concerns reviewers raised were that our contribution would be stronger with additional results on non-iNGP backbones (PaSq, pHej, LTbu) and that we should include additional details on our model architecture and experimental setup (LTbu, JoFm).  We present additional experiments with K-Planes and TensoRF backbones in the attached PDF (Tables 1 and 2). Both variants show clear improvements over their base methods. We will gladly add more model and hyperparameter details to the camera ready and commit to releasing code and data to address any remaining reproducibility concerns.

Meta-review: The paper presents a method for multiscale neural radiance fields that provides improved rendering via approximate anti-aliasing. Specifically, multiple feature grids with different resolutions are used to render from the radiance field, and the coarser resolution grids are used to render regions with lower spatial resolutions. All reviewers were positive towards accepting the paper, though some reviewers requested additional experimental details and evaluating the method on different NeRF backbones (e.g., TensoRF and K-Planes in addition to Instant NGP). The authors addressed these concerns in the rebuttal, and the AC finds that the paper is suitable for acceptance. The authors should include the results from the rebuttal in the camera ready version.
Decision: Accept (poster)