Mip-Grid: Anti-aliased Grid Representations for Neural Radiance Fields
Authors: Seungtae Nam
Daniel Rho
Jong Hwan Ko
Eunbyung Park
Venue: NeurIPS 2023 poster (2023)
Keywords: Novel view synthesis
Neural radiance fields

Abstract: Despite the remarkable achievements of neural radiance fields (NeRF) in representing 3D scenes and generating novel view images, the aliasing issue, rendering 'jaggies' or 'blurry' images at varying camera distances, remains unresolved in most existing approaches. The recently proposed mip-NeRF has effectively addressed this challenge by introducing integrated positional encodings (IPE). However, it relies on MLP architecture to represent the radiance fields, missing out on the fast training speed offered by the latest grid-based methods. In this work, we present mip-Grid, a novel approach that integrates anti-aliasing techniques into grid-based representations for radiance fields, mitigating the aliasing artifacts while enjoying fast training time. Notably, the proposed method uses a single-scale shared grid representation and a single-sampling approach, which only introduces minimal additions to the model parameters and computational costs. To handle scale ambiguity, mip-Grid generates multiple grids by applying simple convolution operations over the shared grid and uses the scale-aware coordinate to retrieve the appropriate features from the generated multiple grids. To test the effectiveness, we incorporated the proposed approach into the two recent representative grid-based methods, TensoRF and K-Planes. The experimental results demonstrated that mip-Grid greatly improved the rendering performance of both methods and showed comparable performance to mip-NeRF on multi-scale datasets while achieving significantly faster training time.

Mip-Grid: Anti-aliased Grid Representations for Neural Radiance Fields
Seungtae Nam
Daniel Rho?
Jong Hwan Ko
Eunbyung Park
Department of Artificial Intelligence, Sungkyunkwan University
2 A12XL, KT
Departmentt of Electrical and Computer Engineering, Sungkyunkwan University
Abstract
Despite the remarkable achievements of neural radiance fields (NeRF) in repre- senting 3D scenes and generating novel view images, the aliasing issue, rendering "jaggies" or "blurry" images at varying camera distances, remains unresolved in most existing approaches. The recently proposed mip-NeRF has addressed this challenge by rendering conical frustums instead of rays. However, it relies on MLP architecture to represent the radiance fields, missing out on the fast training speed offered by the latest grid-based methods. In this work, we present mip-Grid, a novel approach that integrates anti-ut aliasing techniques into grid-based repre- sentations for radiance fields, mitigating the aliasing artifacts while enjoying fast training time. The proposed method generates multi-scale grids by applying simple convolution operations over shared grid representation and uses the scale-aware coordinate to retrieve features at different scales from the generated multi-scale grids. To test the effectiveness, recent representative grid-based methods, TensoRF and K- Planes. Experimental results demonstrate that mip- Grid greatly improves the rendering performance both methods and even outperforms mip-NeRF on multi-scale datasets while achieving significantly faster training time. For code and demo videos, please bitts://stnamjer gigttb............ io/iio//a/ ht/mipggid....ee github. io/
integrated the proposed method into the two
Introduction
Neural radiance fields (NeRF [19] have been remarkably successful in modeling and reconstructing 3D scenes. Since its seminal contributions to novel view synthesis, researchers have investigated
potential applications of NeRF across various domains in 3D computer vision and graphics, such
as dynamic scene representation 43 24 25 15), multi-view image generation (30 6
29
32
3D
surface reconstruction [14), 3D content creation 26 16 18. 17), and SLAM
33 47
28
In typical neural rendering setup, NeRF represents a particular scene, mapping input coordinates (x, d), where the spatial coordinate, and is the view direction, its corresponding RGB colors and density values (r,, g, b, .7). Due to its one-to-one mapping, NeRF suffers from aliasing effects when rendering scenes varying image resolutions; is incapable of producing different values for different resolutions given the same input coordinates. Mip-NeRF [1 has addressed this issue by introducing integrated positional encoding (IPE), which takes the volume of 3D conical frustums into account. This allows MLPs to reason about the size and shape of each conical frustum, eliminating the inherent ambiguity present the original NeRF, While this approach provides a successful multi-scale representation, using variant of positional encoding necessitates MLP architectures for the scene representations, which often leads to a long training time, even with contemporary GPUs.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) A quantitative comparison
Rendering samples
36
Reselation
35
34-
PSNR
33-
32
Mip NeRF
1.8 Resolation
TensoRE (original)
TenseRF (single-scale)
30
TensoRF (mulii-scale)
Mip- Miip-TensoRF
29
1.0
0.5
0.0
0.5
1.0
1.5
2.0
TonsoRF (original)
TonsoRF (singlo-scale)
Mip-TensoRtF
Ground Truth
Training time (hours; log scale)
Figure 1: (a) A quantitative comparison of mip-TensoRE against its three baseline models and mip-NeRF (see Sec. for details). (b) Rendering samples of the lego scene in two different scales.
To accelerate the training time of NeRF, the grid-based representations with or without MLPs have been extensively studied 21 8 These approaches allow faster retrieval of color and density values at given input coordinates, reducing training time to less than an hour while maintaining rendering performance. Furthermore, the recent hashing technique with sophisticated code optimization has demonstrated that the training can be done in a few minutes [21]. However, akin to the original NeRF, these methods still implement the one-to-one mapping between input coordinates and the radiance values, resulting in multi-scale ambiguity. this paper, we seek to address this limitation, combining grid-based representations for faster training and multi-scale representations for anti-aliasing.
naive approach to implementing multi-scale representations using grid-based data structures, drawing inspiration from mipmap approaches [11, could involve training separate grid representations for different scales. During the rendering process, we could look up or interpolate the appropriate grid representations. While such an approach could resolve aliasing issues, it would substantially increase the storage requirements. As NeRF has emerged as a new media paradigm, such as 3D photos and videos, large data sizes would unduly waste valuable communication and storage resources.
With only a single-scale grid representation, an effective technique for mitigating aliasing effects would be the multisampling [IO] (or supersampling [40) method. Given the spatial coordinates and scales, we can sample multiple points within the conical frustums and subsequently average features extracted from the single-scale grid representation. This method can reduce the aliasing effects by reasoning scales during the rendering process. However, this approach comes at high computational cost, directly proportional to the number of samples. Morcover, similar study with MLP architecture has shown requires considerable amounts of samples to achieve comparable performance ([].
This paper proposes mip-Grid, a novel method for anti-aliased grid representations that employs a shared single-scale grid representation and single-sami sampling approach. We generate multi-scale grid representations by applying convolutions to the shared single-scale grid representation. Then, the features are extracted by interpolating the generated multi-scale grid representations using an additional input coordinate that can reason about the scales. Since the multi-scale grids are generated during the rendering process, proposed method minimizes additional storage costs. Furthermore, the convolutions to generate multi-scale grids operate on reduced dimensions (2D or 1D instead of 3D convolutions). thereby not incurring substantial computational and memory costs. Lastly, the proposed method can easily be integrated into the recently proposed grid representations, such as factorized tensor representations 7 or plane-base based representations [8), with minimal modifications.
We integrated mip-Grid into two state of-te-of-theaan art grid-based NeRFs: TensoRF and K-Planes. The resulting models, Fe and mip-K- Klaanees, achieved significantly better performance com- pared to the original TensoRF and K-Planes, and min-TensoRF even outperformed mip-NeRF while achieving significantly faster training time (Fig. (a)). We observed that the aliasing artifacts (e.g., "jaggies") are effectively removed from images rendered with the proposed method (Fig. (b)). which are visually more pleasing as well. Additionally, we conducted extensive ablation studies and visualized the learned kernels to help readers understand how the proposed method works internally.

---

2 Related Work
2.1 Grid-based neural radiance fields
Representing continuous radiance fields with coordinate-based neural networks, NeRF can synthesize view-consistent and photorealistic novel views. However, it requires a long training time due to the spectral bias inherent in neural network training [27] and the need to query deep neural networks thousands of times to render single image. A line of work has addressed this issue by encoding input coordinates with Fourier features [37] or learnable positional parameters that are jointly optimized during the training. The latter approach uses various data structures, such as voxel grids 34]. octrees [44 35 38), and hash tables [2 39], to store the learnable parameters, each of which is responsible for learning the location-aware features that are decoded into color and density values. Encoding the fow-dimensional input coordinates with the learnable parameters significantly accelerates the training and reduces the rendering time, mitigating the inherent spectral bias and eliminating the need to query the deep neural networks.
The dense voxel grid is a straightforward data structure to organize the learnable parameters in 3D space. Plenoxels [9] bake the radiance fields into dense voxel grids, resulting 100x faster convergence compared to the NeRF. However, cubically increasing memory usage makes the model unscalable. PlenOctrees [44] reduce the memory footprint of Plenoxels at inference time by replacing the dense grids with sparse octrees after the model convergence. This allows the model to discard the empty space of a scene, but the memory requirement still increase cubically during training. TensoRF (7) and other works 88 5 4 decompose the dense voxel grids into set of matrices or vectors, improving parameter efficiency and rendering quality.
The primary focus of our work lies in developing an anti-aliasing framework for decomposed grid representations. While mip-NeRF has addressed the aliasing issue by introducing integrated positional encoding, the discussion was limited to MLP-based radiance fields, and the direct application to grid representations is not straightforward. We show that our method successfully removes the aliasing artifacts in grid representations, significantly reducing the training time compared to mip-NeRF.
2.2 Anti-aliasing and mipmap
Representing a discrete signal through sampling requires a higher sampling rate than Nyquist rate [31 22 (twice the highest frequency of a given signal), and violating this rate occur aliasing artifacts, such as jaggedness and flickering, resulting in a low-quality visualization. Anti-aliasing has been widely studied in computer graphics, and it can be divided into two categories: post- filtering and pre-filtering. Supersampling (40] is one of the most representative post-filtering methods, which renders an image at higher resolution than the target output resolution. Though effective, supersampling requires high computational and memory costs. The pre-filtering methods (23 42)
reduce the Nyquist rate by filtering a target signal into the low- wwwfirqueeccy region before sampling in order to meet the sampling thcorem. Mipmap [41] is one of the pre- pre-filtering methods designed for real-time multi-scale rendering. It exploits signals, in which the appropriate level is selected based on the desired scale to render. Mipmap can also represent scene arbitrary scales by interpolating the different levels of filtered signals.
mre-fileeeed hierarchy of progressively lower resolution
Two concurrent works, Zip-NeRF 3 and Tri-MipRF IT, incorporate anti anti-aliasing techniques into grid-based NeRFs. Zip-NeRF approximates the integration of the grid representations by using multisampling techniques [00. which requires around 50 samplings to query point during rendering. On the other hand, Tri-MipRR introduces tri-plane mipmaps to represent multi-scale 3D scenes, where the base level mipmap is trained during the reconstruction and the other levels of mipmaps are constructed by downscaling the base level mipmap. Both methods are implemented based on the tiny-cuda-nn 20 a highly optimized deep learning framework, achieving fast training time.
Our work builds on the concept of mipmap in graphics to address the aliasing in grid representations. We share some similarities with Tri-MipRF by incorporating hierarchy of features that can be decoded into color and density values at queried scales. However, while Tri-MipRR construct the low-level mipmap by downscaling the previous level mipmap by a factor 2, our method learns the optimal kernels that can generate multi-scale grid representations. Moreover, unlike both Zip-NeRF and Tri-MipRE, we aim to develop a simple framework that can be easily integrated into existing grid-based NcRFs without extensive modifications and hyperparameter tuning.
3

---

Feature
(ptt)
Aggregation
Feature (pt2)
Feature (pt, pt2)
(a) Multi-resolution Grid
(b) Multi-scale Grid
Figure 2: Multi-resolution and multi-scale grid representations: the ptl and pt2 are the same spatial coordinates in 3D space. When rendering from varying camera distances, (a) multi-resolution grid representations still retrieve the same features for both ptl and pt2, while (b) multi-scale grid representations have distinct grids for each ptl and pt2, hence, effectively resolving scale ambiguity.
3
Method
3.1 Background: grid representations for neural radiance fields
Grid representations have proven to be highly successful in expediting the training time of NeRF [21
7 [9]
Combined with lightweight MLPs, typically followed by feature extraction from the grids,
reduced the training times from days to hours or minutes. Furthermore, unlike the original NeRF that
relies on MLPs, grid-based approaches do not exhibit spectral bias inherent in neural networks [27]. Through the regularization and training techniques. it often surpasses the performance of the MLPs- based methods while achieving significantly faster training time [2T] 77. In a typical setup, two distinct grids represent density and RGB colors separately. Formally,
(1) where a E [0, oc) is density, e E [0, 1]3 RGB colors, 9, E BHxWxLXC. is 3D grid for the density features, and G. # is 4D grid for appearance features. H, W, T, are the resolutions of the grids along each spatial axis X, , Z. while C. and C. are the number of density and appearance feature channels, respectively. Given an input coordinate, fo interpolates the density features in Go and applies non-linear activation function that outputs density values. Similarly, fo extracts the appearance features from Ge. and normally takes additional directional input d, and then these inputs are passed through an MLP to generate RGB colors. The final RGB colors for a given ray are generated using differentiable volume rendering [12). and simple reconstruction loss is used to train the Go. 9.. and the parameters the MLP 0.
a = fo [x;
c= Jo(x (.tr,d;09a. 0,9,),
3.2 Multi-resolution and multi-scale grid representations
Many recent studies have successfully utilized multi-resolution (or multi-level) grid representations to encode various signals 21 g. To avoid confusion, we would like to clarify that multi-resolution grids refer to the usage of grids with different resolutions, while multi-scale grids refer to grids tailored to each image scale. As illustrated Fig. 2 (a), the multi-resolution grid representations are designed to extract features from multiple grids with different resolutions, followed by aggregating operations to combine the extracted features, such as concatenation, addition, or multiplication. This approach draws inspiration from signal processing, encouraging the low-resolution grids to capture global and smooth parts the signal now-frequency components), while high resolution grids represent fine details of the signal thigh-frequeet components). This decomposed representation is often parameter efficient and better reconstructs the signals. In addition, the recent hashing-band based technique 21 also leverages multi-resolution grids to disambiguate hash collisions.
However, due to inherent one-to-one mapping nature, the current multi-resslution grid representa- tions have limitations in addressing the aliasing issue. When provided with identical input coordinates, same features are extracted regardless representations tend to learn to render the average scale images when training on multi-scale images (or images from varying camera distances). This results in "blurred" images when rendering higher resolutions than the average training resolutions and "jaggies" when rendering at lower resolutions.
camera distances (or scales). Consequently, the grid

---

Fig. (b) depicts multi-scale grid representations, where individual grids (or groups of neighboring grids) are responsible for different scale images. In this setup. features are exclusively extracted from single grid or interpolated from neighboring grid representations. As result, this method can effectively address scale ambiguity by introducing distinct grids for different scales. While a promising approach, it dramatically increases the space complexity, making it an impractical solution for many potential NeRF applications, excessively exhausting communication and storage resources.
3.3 Shared grid and generating multi-scale grids
Since having distinct grid representations for different scales is not a viable solution, we propose to generate multi-scale grids with minimal storage requirements. Inspired by mipmapping techniques commonly used in computer graphics 4T, and also motivated by the fact that a significant amount of information shared across different scales, we use a shared grid representation and generate multi-scale grids by applying a simple convolution over the shared grid. The modified radiance fields with the generated multi-scale grids can be written as follows:
== fa s; Go), c fc s, s,d;0,
where G. E R is the generated multi-scale grids for the density features, and G. E R#x xLXSxCe is the generated multi-scale grids for the appearance features. S is the number of the generated multi-scale grids, and s is a scale-aware input coordinate.
There are several options for generating S. We can simply use Gaussian filtering, which has been widely used in conventional computer graphics. However, it may not be the optimal filter for many real-world scenarios. Alternatively, we can utilize convolution with learnable filters or leverage multi- layer neural networks such as convolutional neural networks (CNNs) or Transformers. Although complex neural networks can potentially yield better multi-scale representations, there is a trade-off' in terms of computational complexity and performance. This work primarily investigates a single convolutional layer within our proposed multi-scal representations.
3.4 Scale-aware input coordinates
In order to reason about the scale, mip-NeRF introduced the integrated positional encoding to augment input coordinates to disambiguate the radiance fields from different camera distances: the original NeRF: (x. d) -> (a, c) and mip-NeRF: (æc, d, of a²) - (a, c), where of E I and of E TR variances of the conical frustum with respect its points interval along the ray and radius, respectively. As a result, mip-NeRF can provide different radiance values for the same coordinates (z,d), reflecting the radii of sampled points (až of) to effectively mitigate the aliasing artifacts.
In a similar vein, we introduce additional scale-aware coordinate, which is used to extract the features from the generated multi-scale grid representations. Specifically, introduce three types of scale coordinates: the discrete scale coordinate, the continuous scale coordinate, and the 2-dimensional scale coordinate. The discrete scale coordinate is defined the mean of the pixel width and height in the world coordinates scaled by 2/ /12 (T]. Since this value is unique for each image scale, we can simply use the same value for all input coordinates (i.e., sampled points) of each image scale to extract the features, not introducing extra computation to get per-point scale coordinates.
The continuous scale coordinate is the discrete scale coordinate scaled by the distance from the ray origin to the sampled point. Training the model with the discrete scale coordinate alone can lead to overfitting to given scales, making it challenging for the model to render images at unseen scales. For instance, if dataset includes four scales of images, the model might generate multi-scale grids that overfit to those specific scales. In contrast, the continuous scale coordinate is not unique for each image scale. Each sampled point has distinct scale value, which allows the model to generate multi-scale grids that can be smoothly interpolated into features across a broader range of scales.
The 2-dimensional scale coordinate consists of the continuous scale coordinate and the distance from the ray origin to the sampled point. While reflecting the discrete or continuous scale coordinate can effectively resolves scale ambiguity, we introduce the third one to provide readers with an understanding of how high-dimensional scale coordinates can integrated into our method. During the rendering process, we apply convolution to the shared grid using two sets of kernels. We then independently extract features from each multi-scale grids based on the respective scale coordinates and average the feature sets to get the final features.
5

---

Feature (pt))
Pei
P12
pti
Feature (pt2)
Shared Grid
Factorized Tensors and Convolutions
Genernted Multiscale Grids
Figure 3: The overall feature extraction pipeline the proposed mip-TensoRF.
3.5 Mip-Grid
Current grid-based NeRFs typically involves certain versions of tensor factorization schemes. For ex- ample, TensoRF used CP decomposition for compact models and proposed a new VM (vector-matrix) factorization method for better rendering performance. K-Planes used plane-based factorization, which has been very successful not only in 3D, but also in 2D and 4D scene generation. Thus, we have integrated the proposed method into two prominent grid-based NeRFs: TensoRF and K-Planes.
3.5.1 Mip-TensoRF
In TensoRF, VM factorization for density grid G is formulated as follows (we omitted the subscript for brevity):
R
x + 0 M G = Σ= V + +ox o M2
(3)
where nn TRd denote the vector for X axis, MY2 z E W is the matrix for Y and axes, is the rank, and o is tensor product, hence ex MY E Hx We generate the multi-scale vectors and matrices by applying ID or 2D depth-wise convolutions over vectors and matrices. This is equivalent to applying 3D convolutions over 3D tensors thanks to the linearity of convolution operators. The i-th scale of a multi-scale grid ② E xxxxxxx can be defined as follows:
R
G. Σ (of sk (M
1e + 4 )o A ( *k (M MXY
X
where * is the convolution operator, 9, HXWXL is i-th scale of the grid, kp" E RRR is i-th depth-wise convolution filter for the vector vxx hr,"" RK is the filter for the rank r, and K is the kernel size. Similarly, A.M E K is i-th depth-wise convolution filter for the matrix M. and krr X2 E RR is the filter for the rank Note that have separate learnable kernels for each rank, axis, and scale, which are more expressible than fixed Gaussian filters.
3.5.2 Mip-K-Planes
Similar to TensoRR.,R we apply depth-wise convolutions over the plane-based grid representations. In K-Planes, a grid g E D2Dx x x can be formulated as follows:
xDx
(5) where M E RIXD,XD,XR M E RDaxIxDxxR and X E RDDxD,xIxRR denote three grids for 3D scene representations, denotes broadcast element-wise multiplication. For example, Y 3 X2 will have the shape of DD,,, xD, x Dz, D,, and D. denote the grid resolutions in X, Y, and Z axes. Similar to Eq. 4 multi-scale grid G% E DDDxDD,DD.. xR for i-th scale can be
g Y2 @ X2 XY
written as follows:
CH -= 3 X2 *xx2 O (M XXY) where K.2 2 E RIIKXKXR RXZ RRXIIXR and IX RRxxxII denote i-th depth-wise convolution filters.
(6)

---

Full Res.
1/2 Res.
1/4 Res.
1/8 Res.
Original
Single-scale
Ours
GT
Figure 4: A qualitative comparison of miipTennoRRF against the original TensoRF and the single-scale TensoRF. The zoom-in parts rendering samples in four different scales are shown. As resolution decreases, two baseline models exhibit aliasing artifacts. Best viewed in color and zoom-in.
4 Experiment
4.1 Experimental setup
Implementation details We implemented two variants of mip-Grid: mip-TensoRE and mip-K- K- Planes. Both models apply convolution shared grid representation using four learnable kernels of size 3, where each kernel is responsible for generating different scales of grids. We trained mip- TensoRF for 40k iterations on the multi-scale Blender dataset and 25k iterations on the multi-scale LLFF dataset. All of its three baseline models were also trained for the same number of iterations. We scaled the loss of each pixel following mip-NeRF [ that we began kernel training after the grid upsampling we trained mip-K-Plaan training hyperparameters remained the same the original TensoRF- VIRFE--M-199 and K- K-Planes.
to account for the scale differences. Note ? was completed. Unlike mip-TensoRE, K-Planes for 30k iterations and optimized the kernels from the beginning. All other
As outlined in the Sec. B] mip-Grid introduces three kinds of scale coordinates to extract features from the generated multi-scale grids. We used the discrete scale coordinate to train mip-TensoRF and mip-K Hap-K-Planes and for Tin-Tens we conducted further evaluation using the continuous scale coordinate and the 2D scale coordinate, examining the effectiveness of using more sophisticated scale coordinates. In all tables showing the results of mip-Grid integrated models, abbreviations of the scale coordinate types provided in parentheses along with the name of each model.
Baseline models We compare our method against three baseline models: the original model, the single-scale model, and the multi-scale model. The original models are the same as the TensoRF. VM-192 and K-Planes, but we test them on the multi-scale dataset. The single-scale model is trained on the multi-scale dataset with loss multipliers A and tested on the multi-scale dataset. The multi-scale model is set of four independent original models, each which is trained and tested on different scales of dataset. We include the latter two baseline models to show that naively training the single-scale representations with multi-scale dataset cannot remove aliasing artifacts effectively, and using four independent models requires four times more parameters than the original model.
7

---

Table 1: A model evaluation on the test set of multi-scale Blender dataset. The elapsed training time, evaluated using single RTX 4090 GPU, are shown at the rightmost column. For mip-NeRF, the time elapsed for 100 iterations were measured and multiplied by 10000 to estimate the total runtime for 1 million iterations. Note that the multi-scale model has four times more parameters than the original model.
PSNRT
SSIM 1/
LPIPS
Time (hours)
Avg.
Full Res.
1/ Res.
Res. Res.
Full
V/ Res. 16 Res.
Res.
Res. Res.
Mip-NeRR 1
34.51
32.63
34.34
35:47
35.06
0.9579
0,9703 0.9786 0.9833
0,0469
0.0260 0.0168 0.0120|
>30
TemoRE (original)
30.70
33.13
33.27
30.02
26.40
0.9630
0.9704
0.9624 0,9358
0.0474
0.0337
0,0488 0,0802
0.16
TensoRF ciinle-scalle)
30.61
28.96
32.42
32.53
0.9295
0.9618
0.9703 0,9514
0.1054
0.0608
0.0474 0.0697
TeasoRF (multi-scale)
34.77
33,33
38.74
35.44
34.57
0.9637
0,9757
0.9766 0,9775
0.0456
0.0301 0.0279 0.0238
0.66
Mip-TeasoRF
35.23
32.53
35:40
36.58
36,41
0.9587
0.9757
0,9828 0.9857
0.0533
0.0278 0.0200 0,0163
0.27
K- K-Planes (original)
29.76
32.40
31.83
28.95
25.85
0.9615
0.9663
0,9570 0,9321
0.0491
0.0369
0,0539 0.0861
0.51
K. Planes (single scale)
30.07
32.13
32.35
29.54
0.9590
0.9667 0.9597 0,9156
0.0525
0.0497 0.0813
K- K-Planes (multi-scale)
32.24
32.59
33.60
32.34
30.58
0.9605
0,9696 0 9646 0,9580
0.0508
0.0314
0,0396 0.0415
2.02
Mip-K-Plande
32.27
33.20
33.03
32.73
31.12
0.9601
0.9691
0.9724 0.9689
0.0512
0.0329 0.0291 0.0302
0.60
Table 2: A model evaluation on the test set of multi-scale LLFF dataset. Note that the multi-scale model has four times more parameters than the original model.
PSNR1
SSIM
LPIPS
Avg.
Full Ros.
X6 Ros.
1/ Ros. 15 Res.
Pall Res.
16 Ros. 1/4 Res. 1/ Res.
Full Res.
16 Ros. X Ros. 15 Ros.
TensoRF (original)
26.03
26.73
27.89
26.70
22.81
0,8386
0,8932
0.8964 0.8063
0,2044
0.1069
0.1099 0,1685
TensoRF (single scale)
27.33
24.48
28.25
29.92
26.64
0,7457
0,8863
0.9375 0.8964
0.3114
0.1450 0.0851 0.1259
TensoRF imulli-scaale)
30.18
26.75
29.58
31.33
33.06
0.8378
0,9067
0.9487 0.9731
0.1992
0,1065 0.0632 0.0339
Mip-TensoRF 30.48
26.51
29.69
32.18
33,52
0,8310
0,9165
0.9590 0.9762
0,2152
0.0946 0.0477 0.0316
Datasets All models were evaluated on the multi-scale Blender dataset [T] with three different metrics: PSNR, SSIM, and LPIPS (VGG) [46]. Mip-TensoRF and its baseline models were further evaluated on the multi-scale LLFF dataset. To convert single-scale dataset to multi-scale dataset, we bilinearly downsampled the images by factor of 1, 2, 4, and 8, and rescaled the focal length accordingly as typically done in other multi-scale NeRF literature [13]
4.2 Results
Tab. shows the evaluation results on the multi-scale Blender dataset. Mip- Tin-oensoRF and mip-K- Planes outperform their baseline models, achieving the highest average PSNR. Notably, mip-TensoRE even surpasses mip-NeRF while significantly reducing the training time. The performance of the original models experiences substantial drop as the rendering resolution decreases, leading to noticeable "jaggies" in the images rendered at the two lowest resolutions (Fig. ( The single-scale models achieve better performance at the two lowest resolutions compared to the original models but show worse results at other resolutions. This suggests that, to some extent, training with a multi-scale dataset and employing the loss multipliers helps the model in reasoning about different scales, but this approach does not entirely eliminate aliasing artifacts, On the other hand, the multi-scale models can effectively represent different scales of scene, achieving the best PSNR among the three baseline models. Nevertheless, these models require four times more parameters than the original models, which is undesirable in practical applications. Unlike others, our models successfully mitigate aliasing artifacts while minimizing the additional number of parameters and computational cost.
We provide additional results of training mip-TensoRR with continuous and 2-dimensional scale coordinates in Tab. 3 The first two models show only a marginal difference when evaluated at the training resolutions. A similar trend is observed in evaluations at unseen resolutions, but there is a clear difference when the image resolution is reduced from 4/8 resolution to 3/8 resolution, suggesting that the continuous scale coordinate improves the model's generalization performance. Our method generates multiple sets of feature grids from shared representation, and these feature sets share many similarities. This allows different features to be smoothly interpolated and rendered into visually pleasing images. However, more information needs to be removed as the output images decrease size, leading to relatively large differences between low-scale features. This explains the marginal drop in PSNR at lower resolutions. Moreover, the model trained with 2D scale coordinate outperforms the others, indicating that the additional scale coordinate (the distance from the ray origin to the sampled point) can be beneficial for resolving the remaining scale ambiguity.
8

---

Table 3: An evaluation of mip-TensoRF trained with three different types of scale coordinates. All models were trained on the multi-scale Blender dataset (with four different resolutions; Full Res., 4/8 Res., 2/8 Res., and 1/8 Res.) and tested on the test set of Blender dataset rescaled to eight resolutions.
PSNR1 % Ros.
Avg.
Full Res.
14 Res.
% Ros.
Res.
16 Ros.
16 Ros.
1/ Res.
Mip-TensoRF (disc.)
34.54
32.53
33.73
34.70
35.40
34.33
35.93
35.50
Mip-TensoRE (cont.)
34.73
32.46
33.82
34.29
34.74
35.34
35.64
35.89
35.66
Mip TensoRF (2D)
32.56
35.77
Shared grid
Full Res.
1/2 Res.
1/4 Res.
1/8 Res.
Learned kernels
Figure 5: Visualization of generated multi-scale grids and learned kernels of mip- TensoRF. A single channel the X plane of the appearance grids and the first four channels the learned kernels are shown. The first column of the first row displays shared grid representation, and the generated multi-scale grids are displayed from the second to the last column of the first row. Each column of the second row shows the learned kernels that is used to generate the corresponding feature grids.
4.3 Visualization of learned kernels and multi-scale grids
We visualize the learned kernels and the generated multi-scale grids of mip-TensoRF illustrate how the learned patterns vary across scales. As depicted in Fig. 5 the high-scale kernels look similar Gaussian distribution, and their shape (i.e., the standard deviation) becomes larger as the corresponding scale decreases. This observation aligns with our intuition that the kernels of low-scale features should be smoother than those of high-scale features and demonstrates that our method effectively learns optimal kernels for each scale. A similar phenomenon can be observed in the visualization of feature grids, where high-frequency details are progressively removed as the feature scale decreases. Although we do not provide the visualization of density kernels and grids due to the limited space, the learned patterns closely resemble those the appearance kernels and grids.
4.4 Ablation studies
Fixed Gaussian kernels and learnable kernels We compare mip-TensoRF with fixed Gaussian kernels against the same model with learnable kernels, Both models were initialized with 2 of standard deviations (1.0, 1.0, 1.0, 1.0) and [1.0, 1.5, 2.5, 4.0), where the leftmost value was used to set the kernel of the highest scale, and the rightmost value was used for the lowest scale. The second set is give an inductive bias that the low-scale features should be smoothed more than high-scale features. The fixed Gaussian kernel initialized with the second set standard deviations significantly outperforms the other (Tab, 4.. This suggests that increasing standard deviations can be a beneficial inductive bias. However, these fixed Gaussian kernels may not be optimal for each scale. Instead of using heuristically selected kernels, method fine-tunes the Gaussian kernels and finds the optimal weights during the training. Experimental results demonstrate that the learnable approach outperforms the fixed Gaussian approach, further confirming the effectiveness of our method (Tab, (
The number of multi-scale grids and kernel size The proposed method involves two key hy- perparameters: the number of multi-scale grids and the kernel size. We provide evaluation of mip-TennoRRF with nine different combinations of these two values to give readers guidance on hyperparameter configuration. As shown in Tab. the number of multi-scale grids has little effect

---

Table 4: An evaluation of mip- TensoRF using fixed Gaussian kernels and learnable kernels, initialized with different standard deviations. Trained and tested on the multi-scale Blender dataset.
PSNR
SSIM1
LPIPS va Res. 14 Res. V/ Ros.
Stdew,
Avg.
Full Res.
1/ Ros.
14 Ros. 1/4 Res.
Full Res. VA Ros. 14 Res. V6 Ros.
Full Res.
Fixed
1,0; 1.0; 1.0; 1.0
30,99
28.59
32.46
33.75
29.15
0,9267
0.9627 0,9763 0,9574
0.0991
0.0494
0,0332 0.0593
1.0; 1.5; 2.5; 4.0
34.05
30.84
34,10
35.37
35.87
0.9478
0.9698 0,9787 0.9829
0.0697
0.0355 0.0255 0.0211
Leumablee
1,0; 1.0; 1.0; 1.0
35.31
32.57
35.43
36.64
36.59
0.9588
0.9759
0,9829 0,9863
0.0535
0,0275
0.0196 0.0155
1.0; 1.5; 2.5;
35.40
32.62
35.47
36.70
36.81
0.9593
0.9762
0.9832 0.9868
0.0529
0.0272
0.0192
Table 5: An evaluation of mip- TensoRF varying two hyperparameters: N the number of multi-scale
grids, K - the kernel size. Tested on the hotdog scene of the multi-scale Blender dataset.
PSNR
SSIM
LPIPS
K
Avg.
Full Res.
1/ Res.
94 Res.
1/ Ros.
Full Res.
15 Res. 54 Res. 1/ Ros.
Full Res.
V4 Res. 14 Res. v Res.
3
39.43
36
39.72
40.66
40.49
0,9792
0.9887
0.9917 0.9931
0.0379
0.0164
0.0104
0.0078
5
39.58
36.84
39.78
40.86
40.85
0.9790
0,9888
0.9918 0.9934
0.0382
0.0163
0.0099
0.0069
1I
39.63
36.83
39.81
40.89
41.01
0.9789
0.9890
0.9920 0.9937
0.0381
0,0160
0.0096
0.0066
37.01
39.85
40.57
0.9795
0,9888
0.9918
0.9932
0.0375
0.0164
0.0102
0.0080
39.66
36,98
39.90
40.98
40.78
0.9796
0.9890
0.9920
0.9935
0.0367
0.0155
0.0095
0.0069
11
39.65
36.92
39.83
40.95
40.90
0.9793
0.9890
0.9920
0.9937
0.0371
0.0157
0.0093
0.0067
3
39.60
37.03
39.90
40.92
40.57
0.9797
0.9888
0.9919
0.9952
0.0367
0,0161
0.0099
0.0079
37.06
39.97
0,9890
0.9920
0,9936
0.0095
39.78
37.02
39.97
41.07
41.06
0.9796
0.9890
0.9921
0.9938
0.0363
0.0158
0.0093 0.0068
on the rendering quality, but variations in kernel sizes result in relatively bigger differences in the PSNR of low-scale images. Although different scales of grid representations share many similarities, the low-scale grid should be smoothed enough to produce image without aliasing artifacts, and the kernel size is a critical factor controlling the smoothness of the grids. Using too small kernels may introduce unnecessary feghffreueecc components in the low-scale grid. Nevertheless, there is always a trade-off between rendering quality and speed, and users should consider their specific usage purpose when determining the kernel size.
Limitations and discussion
The proposed method has not been tested on scenes involving more complex camera poses, such as 360-degree scenes [2] 45 36 or free trajectories (39]. However, addressing these scenarios would require further technical development and a thorough parameter tuning. We plan to investigate how the proposed method, or even the grid representation itself, can be extended to these scenarios.
Although the proposed method is significantly faster than mip-NeRF, it introduces additional com- plexity the system. Especially in terms of training time, our method needs more computational resources to generate multi-scale grids and retrieve features. We implemented our model based on TensoRF and K-Planes with minimal modifications, which makes the training time heavily rely on the base models. Also, the implementation of the multi-scale grid generation and the feature retrieval process has not been optimized yet. We strongly believe the training time and memory requirements can be substantially improved by fusing the multi-scale grid generation and the feature retrieval process as a single CUDA kernel or using the NeRF accelerating tool [13
Conclusion
We proposed mip-Grid, anti-aliased grid representations for NeRF. The proposed approach can easily be integrated into the existing grid-based NeRFs, and the two methods using our approach, mip-TensoRF and mip-K. K-Planes, have demonstrated that aliasing artifacts can be effectively removed. Since we generate multi-scale grids from a shared grid representation and do not rely on supersam- pling, the proposed method minimizes additional number of parameters and trains significantly faster than the existing Mased anti-aliased NeRFs. We believe our work paves the way for new research direction toward alias-free NeRF leveraging the training efficiency of grid representations.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the Institute of Information and Communication Technology Planning Evaluation (IITP) grants LITP-2019-0-00422 IITP-2022-0-02066) and the National Research Foun- dation (NRF) grants (2022R 1064 184, RS-2023-00245322) funded by the Ministry of Science and ICT (MSIT) of Korea.
References
[1] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Bruallae and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[2] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[3] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Anti- aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[4] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. arXiv preprint arXiv:2301.09633 2023.
[5] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2022.
[6] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[7] Aupei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
[8] Sara -ridovich-Keil, Giacomo Meanti, Frederik Rahbæe Warburg, Benjamin Recht, and Angjoo Kanazawa. K- X-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[9] Sara Fridovich-Keit Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[10] Ned Greene and Paul S Heckbert. Creating raster omnimax images from multiple perspective views using the elliptical weighted average filter. IEEE Computer Gruphics and Applications, 1986.
[11] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In Proceedings the IERR/CVF International Conference on Computer Vision (ICCV), 2023.
[12] James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics. 1984.
[13] Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerfacc: A general nerf acceleration toolbox. arXiv preprint arXiv:2210.0483 2022.
[14] Zhaoshuo Li, Thomas Müller, Alex Evans, Russell Taylor, Mathias Unberath, Ming- Yu Liu, and Chen- Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putfern Recognition (CVPR), 2023.
[15] Zhengqi Li. Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[16] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang. Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference Computer Vision and Patern Recognition (CVPR), 2023.
11

---

[17] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: sero image 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[18] Luke Melas Sariaziyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusiin: 360deg reconstruc- tion of any ohject from a single image. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[19] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[20] Thomas Müller. tiny-cuda-nn, 2021.
[21] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (TOG), 2022.
[22] Harry Nyguist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 1928.
[23] Mare Olano and Dan Baker. Lean mapping. Proceedings of the ACM SIGGRAPH symposium on Interactive Gruphics and Games, 2010.
[24] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven Seitz, and Ricardo Martin- Brrtln-Buualla.... Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[25] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ri- representation for topologically
cardo Martin- Brualla, and Steven M. Seitz. Hypernerf: A varying neural radiance fields. ACM Transactions on Graphics (TOG), 2021.
[26] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. International Conference on Learning Representations (ICLR), 2022.
[27] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Learning Representations (ICLR), 2019.
[28] Antoni Rosinol, John Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. In JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2023.
[29] Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, and Deqing Sun. VQ3D: Learming 3D-aware generative model on ImageNet. In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[30] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. 2020.
[31] Claude E Shannon. Communication in the presence of noise. 1949.
[32] Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin- -ing Lee, Peter Wonka, and Sergey Tulyakov. 3d generation on imagenet. In International Conference on Learning Representations (ICLR), 2022.
[33] Edgar Sucar. Shikun Liu, Joseph Ortiz. and Andrew Davison. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[34] Cheng Sun, Min Sun, and Hwann-Tee Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[35] Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. In ACM SIGGRAPH, 2022.
[36] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar, Block- Blrck-nerf:: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
12

---

[37] Matthew Tancik. Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions dimensional domains. Advances in Neural Information Prucessing Systems (NeurIPS), 2020.
[38] Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao. Yanshun Zhang. Yingliang Zhang, Minye Wu. Jingyi Yu, and Lan Xu, Fourier plenoctrees for dynamic radiance field rendering in real-time. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[39] Peng Wang. Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Patern Recognition (CVPR), 2023.
[40] Turner Whitted. An improved illumination model for shaded display. In ACM SIGGRAPH. 2005.
[41] Lance Williams. Pyramidal parametrics. In Proceedings of the 10th annual conference on Computer graphics and interactive techniques, 1983.
[42] Lifan Wu, Shuang Zhao, Ling-Qi Yan, and Ravi Ramamoorthi. Accurate appearance preserving prefiltering for rendering displacement-mappee mapped surfaces. ACM Transactions on Graphics (TOG), 2019.
[43] Gengshan Yang. Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[44] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. Plenoctrees for real-tite val-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
[45] Kai Zhang, Gemoot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.0749) 0..499. 2020.
[46] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metrie. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[47] Zihan Zhu Songyou Peng Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin Oswald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2022.
13


Reviews:
[{"review_id": "EtDkVxyk5E", "reviewer": "Reviewer_rA7o", "summary": "The paper proposes a method to deal with the aliasing issue in factorised grid-based NeRF methods, e.g. TensoRF and K-plane, by applying convolution operations to decomposed vectors and matrices.\n\n---\n**After rebuttal**: I have read authors' rebuttal and it addresses my concerns.", "questions": "See the weakness section above.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The goal of handling aliasing in grid-based NeRF is valuable and the proposed method is novel.\n* Source code is provided in supp mat.", "weaknesses": "About evaluation: \n* I am interested to see what is the performance difference between using convolution operations and using a simple Gaussian?\n\nAbout methodology: \n* I got the impression that the proposed method can only handle a number of *discrete* scales after training, since the convolution kernels are optimised for each scale. If I understand correctly, that means a set of convolution kernels is optimised for a specific scale. If a NeRF is optimised for $4$ scales, for example, scale=$1, \\frac{1}{2}, \\frac{1}{4}$, and $\\frac{1}{8}$, then during inference, it’s impossible to render an image for scale=$\\frac{1}{3}$ and $\\frac{1}{6}$ from this NeRF?\n* Following the point above, if my understanding above is correct, it would be much better to be able to render images from continuous scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison between learnable and fixed Guassian kernel**\nWe trained mip-TensoRF (disc.) with fixed Gaussian kernels on multi-scale Blender dataset and compared them against the same model with learnable kernels.\nWe initialized both models with 2 sets of standard deviations [1.0, 1.0, 1.0, 1.0] and [1.0, 1.5, 2.5, 4.0], where the second one aligns with our intuition that low-scale features should be smoothed more than high-scale features.\nGiven the fact that the Gaussian kernels initialized with the second set significantly outperform the other (Table 4), the increasing standard deviations appear to be a good inductive bias, but they may not be the optimal kernels for each scale.\nInstead of using heuristically selected kernels, our method fine-tunes the Gaussian kernels and finds the optimal weights during the training.\nThe experimental results also show the learnable approach is better than the fixed Gaussian (Table 4).\n\n**Model evaluations at continuous resolutions**\nAs briefly mentioned in the global response section, our model can render images at arbitrary resolutions even when trained with the discrete scale coordinates.\nAlthough the discrete scale coordinates force the model to make features that corresponds to a small number of given scales, the different scales of features can be smoothly interpolated as they share many similarities.\nThe problem occurs when the image resolution is reduced below 400 and the difference between features increases accordingly (Table 2).\nHowever, this minor drawback can be mitigated by using the continuous scale coordinates and additional distance coordinates.\nTable 2 shows that our two revised models can render images at a wide range of resolutions without degrading the visual quality."}, {"review_id": "uYgtcnNwqw", "reviewer": "Reviewer_9wTm", "summary": "This work proposes a simple anti-aliasing method for NeRF representations based on factorized grids, e.g., TensoRF and K-planes. The key idea is to learn a lowpass filtering kernel ", "questions": "See the first and second bullet points in the weakness section.", "limitations": "The proposed method doesn't seem to be easily applicable to the multi-resolution hash grid proposed by Instant-NGP. It might be better to mention this.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method seems simple and easy to reproduce. Example implementation is also included in the supplementary.\n2. The method is evaluated against baseline methods without the proposed anti-aliasing technique, and clear improvements in anti-aliasing can be seen both quantitatively and qualitatively. ", "weaknesses": "1. It doesn't seem clear how the multi-scale grid is used exactly in rendering; for example, are all the scales used to get a point's feature, or only a heuristically selected scale is used?\n\n2. One thing I'm a bit unsure is that: why does the same kernel size seem to be used across all scales? Would it makes more sense to use bigger kernel size for lower-resolution grids?\n\n3. A visualization of the kernel weights will be helpful in understanding what filtering patterns have been learned, especially the relationship of the grid scale and learned kernel weights. \n\n4. In table 1 and table 2, best models should be highlighted to facilitate reading.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Explanation of how scale coordinates are used**\nPlease refer to the global response.\n\n**Same size of kernels across all resolutions**\nWe deliberately use the same size of kernels to boost training and inference speeds by leveraging group convolution.\nHowever, it makes sense to use small kernels for high-scale features and large kernels for low-scale features.\nWe trained mip-TensoRF (disc.) on a multi-scale Blender dataset with 3 sets of different kernel sizes: [5, 7, 9, 11], [11, 13, 15, 17], and [3, 5, 9, 17].\nWe found the difference in performance to be insignificant, and the kernel sizes barely affected the final output (Table 5).\nThis suggests that, even if kernel sizes are the same, kernels for low-scale features can be optimized to have large standard deviations, and vice versa.\n\n**Kernel visualization**\nWe visualize the appearance grid kernels of mip-TensoRF (disc.) trained on the Lego scene of the multi-scale Blender dataset to show how the learned patterns of different kernels vary with scale.\nWe select the first 16 channels from each resolution and bilinear upsample the kernels by a factor of 8.\nAlthough we do not provide visualization of density kernels due to the limited space, the learned patterns are very similar to those of appearance kernels.\nAs can be seen in Figure 1, the kernel of the highest-scale features looks similar to a Gaussian distribution, and its shape (i.e., standard deviation) becomes larger as the corresponding feature scale decreases.\nThis is consistent with our intuition that the kernel of the low-scale features should be smoother than that of the high-scale features and shows that our method learns optimal Gaussian kernels for each scale.\n\n**How to apply the proposed method to Instant-NGP**\nThe main focus of our work is to make multiple sets of grid features that can be rendered into different resolutions of images without aliasing artifacts while minimizing the number of additional parameters.\nFiltering the shared representations with learnable kernels is one of many options to achieve this goal.\nIn this sense, we can achieve the same goal with Instant-NGP by adding extra hash tables only at the top level and sharing the remaining levels.\nEach table at the top level will learn features suitable for a given scale, and the remaining levels will learn information shared across different scales.\nThis approach will also require significantly fewer extra parameters than using multiple hash tables at all levels.\nAlthough we believe this is an exciting research direction, we leave it as future work as it is hard to build such a model in a short time with limited resources.\n\n**Minor issues**\nWe will highlight the best models in all tables."}, {"review_id": "nQZH30Iiht", "reviewer": "Reviewer_8s5r", "summary": "This paper proposes mip-Grid to integrate anti-aliasing techniques into grid-based representations for radiance fields. Mip-Grid generates multiple grids by applying convolution operations over the shared single-scale grid representation. Then it uses the scale-aware coordinate to retrieve the appropriate features from the generated multiple grids. This anti-aliased grid representation combines grid-based representations for faster training and Mip-NeRF-like multi-scale representations for anti-aliasing. The key idea is to use a shared single-scale grid representation and a single sampling but generate multiple grid representations by applying convolutions to the shared single-scale grid representation. Then, the features are extracted by interpolating the generated multiple grid representations using an additional input coordinate.", "questions": "I would suggest including the comparison with Zip-NeRF since an unofficial PyTorch implementation of Zip-NeRF is available.", "limitations": "Lack of experiments of unbounded scenes (such as Mip-NeRF 360).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work mitigates the aliasing artifacts when rendering images from varying camera distances, such as blurry and jaggies in the images, while enjoying fast training time.\n\nThis work uses a single-scale shared grid representation and a single-sampling approach, which only introduces minimal additions to the model parameters and computational costs.\n\nThe convolutions to generate multiple grids are lightweight and do not incur substantial computational and memory costs as it operates on reduced dimensions.\n\nThis work shows comparable performance to mip-NeRF on multi-scale datasets while achieving significantly faster training time.\n\nThis work improves the rendering performance of TensoRF and K-Planes on multi-scale datasets while introducing tolerable training time.", "weaknesses": "Given the fact that Zip-NeRF is implemented on the hash-based Instant-NGP, thus it achieves fast training time despite using multisampling techniques. I will concern about the contribution of this work, as Zip-NeRF shows good performance for both anti-aliasing and training time. I would like to see the attitude of other reviewers.\n\nI believe \"unbounded\" scenes are the current research topics for Anti-Aliased NeRFs. Though I understand that addressing these scenarios would require other techniques, the lack of experiments of unbounded scenes (such as Mip-NeRF 360) will degree the rating of this work.\n\nLack of visualization videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison against Zip-NeRF**\nZip-NeRF [2] achieves great visual quality at different image resolutions while decreasing the training time to under an hour for the multi-scale 360 dataset [1].\nHowever, we want to remind reviewers that the authors of Zip-NeRF used 8 V100 GPUs to train their model.\nIn our experiments, it took more than 5 hours to train Zip-NeRF on each scene of the multi-scale Blender dataset with a single A100 GPU.\nOurs take less than an hour on the same GPU, achieving better performance in PSNR (Table 1).\nPlease note that we used an unofficial code for the experiments, and, thus, these should not be regarded as conclusive results.\nMoreover, our implementation has not been fully optimized yet, and we strongly believe there is substantial room for improvement in training speed, e.g., by using fused-cuda kernels.\n\n**Lack of experiments on unbounded scenes**\nWe agree that unbounded scenes are the main focus of current research on anti-aliased NeRF.\nAlthough it does not cover the scene in 360 degrees, the LLFF dataset consists of real-world images, which are also unbounded scenes.\nTable 3 shows that mip-TensoRF (disc.) significantly outperforms baseline models in all three metrics, including mip-NeRF, on the LLFF dataset.\nMoreover, we trained K-planes and mip-K-planes on the garden scene of the multi-scale 360 dataset [1].\nWe used the same contraction function as adopted in K-planes and did not change any hyper-parameters of K-planes regarding both training and network configurations on the LLFF dataset.\nEarly results show that our method improves the PSNR of the baseline model at two lower resolutions (Table 6).\nAlthough our primary model is still behind other models targeting unbounded 360 scenes, there are many fancy techniques that are widely used to extend NeRF-like models to unbounded scenes.\nWe strongly believe our method can be greatly improved by adopting those techniques.\nHowever, building such a model for unbounded scenes requires significant modifications, which pose a challenge that extends beyond the scope of our current paper.\nConsidering the current scope and focus of our research, we leave the exploration of this particular aspect for future investigations.\n\n\n**Reference**\n\n[1] Johnatan T. Barron, et al. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022.\n\n[2] Johnatan T. Barron, et al. Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv, 2023. \n\n[3] Sara Fridovich-Keil, et al. K-planes: Explicit radiance fields in space, time, and appearance, CVPR, 2023."}, {"review_id": "x724cUpXHO", "reviewer": "Reviewer_QWBQ", "summary": "This paper studies the anti-alias problem of grid-based static scene representation.\nThe existing anti-alias methods like Mip-NeRF still require intensive computations, while current grid/factorization-based methods don't show good anti-alias results. \nThis paper proposes a general idea to solve the anti-alias problem: it uses a single-scale grid as the representation while applying learnable convolution to generate multi-scale grids. \nCombining with scale-aware position encoding, this paper shows superior results compared to TensoRF or KPlanes, demonstrating a universal improvement for grid-based representation.", "questions": "As stated above, it would be good to show (1) more baselines, especially baselines using multi-scale/resolution methods. (2) experiments compared to multi-resolution grids.", "limitations": "I think the limitations discussed in this paper make sense. \nI agree the current method is still preliminary and having MIP-NeRF style design would be an exciting direction.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies an interesting problem in an important direction. Extending the anti-alias idea from Mip-NeRF to explicit representation could contribute to the whole community, and the results are promising. I think it would be beneficial to the development of explicit representations.\n2. The whole method is neat, simple, and powerful. Applying CNN on the single grid and having a scale-related positional encoding is simple and intuitive. The overall framework is neat without too many tricks and complicated tweaks. Considering it, this idea could be broadly used in many situations. \n3. The paper is overall well-written and easy to follow.", "weaknesses": "1. It would be more convincing to compare several baselines with multi-resolution/multi-scale designs. For example, Instant-NGP[17] uses a multi-resolution hash. Also, the original K-Plane uses multi-resolution planes. It would be helpful to show how these method works and how the proposed method compared to these baselines.\n2. I am not fully convinced by the analysis of multi-scale and multi-resolution in Section 3.2. To me, the multi-scale one sounds like optimization multi-NeRF at different scales, while NeRFs from different scales are not related. Could you elaborate a little bit? It would be nice to show the experimental results of multi-resolution ideas. \n3. Not necessarily a weakness. It would be good to write the method names instead of baseline 1 and 2 in Figure 3.\n4. In Table 1, Mip-Kplanes's PSNR has different trends to MIP-NeRF's PSNR and mip-TensoRF's PSNR, also different from Mip-Kplanes' SSIM and LPIPS. Could you elaborate on why  PSNRs are lower with lower Res?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Clarification on multi-resolution, multi-scale grids, and multi-resolution images**\nMulti-resolution grids refer to a set of grids, each of which has a different grid resolution; instant-NGP and K-planes are two examples.\nThe opposite notion would be a single-resolution grid, such as the original TensoRF.\nTo tackle the issue of mip-mapping, we have proposed a method to generate multi-scale grids by convolving shared representations with learnable kernels, which can be applied to both single- and multi-resolution grids.\nTensoRF is an example of single-resolution-single-scale grids, and K-planes is an example of multi-resolution-single-scale grids, which has already been reported in the main text.\nSimilarly, mip-TensoRF is an example of single-resolution-multi-scale model, but the multi-scale grids are generated from the shared representations, which is different from a naive single-resolution-multi-scale model that has multiple explicit grids.\nWhat we wanted to show was that our method can be directly applied to various grid-based models without modifying other components, regardless of whether they are using single- or multi-resolution grids.\n\nAs you suggested, we trained a variant of the single-resolution-multi-scale model, dubbed multi-scale TensoRF, and reported the evaluation results in Table 1.\nAlthough multi-scale TensoRF outperforms mip-TensoRF at image resolutions of 800 and 400, it requires significantly more extra parameters (4 times that of TensoRF).\nOne interesting thing is that mip-TensoRF significantly outperforms multi-scale TensoRF at resolutions of 200 and 100, suggesting shared parameters are not only efficient but also give better representations for rendering low-resolution images.\nDue to limited resources, we were not able to include multi-resolution, multi-scale models (e.g., multiple K-planes) in the rebuttal, but we will certainly add them in the final version.\n\n**Different trends of mip-K-planes and mip-NeRF**\n\nThe differential trends between the two approaches can be traced to different experimental details. There are a number of possible causes for this phenomenon:\nTo begin, TensoRF employs single-resolution grids, while K-Planes uses multi-resolution grids. \nThere are also notable distinctions in the regularization losses implemented by TensoRF and K-Planes. \nMoreover, while K-Planes integrate mixed precision during training, TensoRF relies exclusively on 32-bit precision.\nWe had to apply kernel normalization to mitigate the occurrence of explosive gradients and parameters in K-Planes. \nWe believe this factor plays a pivotal role in contributing to the observed dissimilarity in trends."}]

Author rebuttal:
We briefly explain how the scale coordinate is defined and introduce two additional scale coordinates to show how different scale coordinates can be applied to our method.

**Continuous scale coordinates** 
The models reported in the main text use as the scale coordinate $s$ the mean of the pixel width and height in the world coordinates scaled by $2/\sqrt{12}$ [1].
Since this is a discrete value and unique for each image resolution, it may not handle all scale ambiguities, potentially making it difficult for models to render images at unseen resolution.
We introduce the continuous scale coordinate $s'$, the discrete scale coordinate scaled by the distance from a ray origin to a sampled point on a ray, to improve our method further.
Each sample point has a different scale coordinate, and the per-point scales force the kernels to output features that can be smoothly interpolated to unseen scale.

**2-dimensional scale coordinates**
Although the continuous scale coordinate can effectively guide the model to reason grid features of unseen scale, the scale ambiguity still exists.
Consider two different sample points $p_1$ and $p_2$, distanced from the ray origin by $d_1$ and $d_2$, respectively.
If each point's discrete scale coordinates are $d_2$ and $d_1$, they will have the same continuous scale coordinates $d_1 \times d_2$.
To further reduce the scale ambiguity, we use the distance from the ray origin to the sample point as the second scale coordinate.
We convolve the shared representation with 2 sets of 4 kernels, interpolate each set of features separately according to each scale coordinate, and average the feature sets to get the final representations.
The interpolated representations are decoded into densities and RGB values (using a shallow MLP) and rendered as an image.

**Additional experiments** 
We compare three variants of mip-TensoRF against baseline models on the test set of multi-scale Blender dataset [1].
TensoRF and single-scale TensoRF are the original TensoRF [4] trained on single-scale and multi-scale dataset, respectively.
Multi-scale TensoRF is a set of 4 TensoRFs, each of which learns different scales of features.
The three mip-TensoRF models use different kinds of scale coordinates: discrete scales, continuous scales, and distance coordinates (abbreviated in parentheses).
Please note that single-scale TensoRF and mip-TensoRF (disc.) are identical to TensoRF (MS) and mip-TensoRF in the main text, respectively.

Table 1 shows the evaluation results at 4 different resolutions averaged across 8 scenes.
The two mip-TensoRF models trained with discrete and continuous scale coordinates, respectively, show only a marginal difference, when evaluated at training resolutions.
A similar trend is observed in evaluations at unseen resolutions, but there is a clear difference when the image resolution is reduced from 400 to 300 (Table 2), suggesting that the continuous scale coordinates make the model generalize better.
Since our method makes multiple sets of feature grids from a shared representation, the feature sets share many similarities, which allows different features to be smoothly interpolated and rendered into visually pleasing images.
However, more information needs to be removed as the output images get smaller, resulting in relatively large differences between low-scale features.
This explains why there is a slight drop in PSNR at lower resolutions.
Moreover, the model trained with 2-dimensional scale coordinates outperforms mip-NeRF [1] by a clear margin in PSNR and SSIM.

**References**

[1] Jonathan T. Barron, et al. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.

[2] Anpei Chen, et al. Tensorf: Tensorial radiance fields. ECCV, 2022.

Meta-review: The paper presents a method for anti-aliasing neural radiance fields using a grid-based neural representation. Reviewers appreciated the timeliness and simplicity of the method, the quality of the results, and the clarity of the writing. Some concerns were raised about how the method compares to approaches based on Instant-NGP, TensoRF, and K-Planes. The authors provided additional comparisons in the rebuttal, and post-discussion, all reviewers are positive about acceptance. The AC agrees that the paper meets the bar for acceptance. The authors should include the clarifications and additional results, including the discussion of scale coordinates and the multi-scale Blender dataset experiments, in the camera ready version.
Decision: Accept (poster)