Title: Mip-Grid: Anti-aliased Grid Representations for Neural Radiance Fields
Venue: NeurIPS 2023
Accepted: yes
Keywords: Novel view synthesis

Neural radiance fields
Abstract: Despite the remarkable achievements of neural radiance fields (NeRF) in representing 3D scenes and generating novel view images, the aliasing issue, rendering 'jaggies' or 'blurry' images at varying camera distances, remains unresolved in most existing approaches. The recently proposed mip-NeRF has effectively addressed this challenge by introducing integrated positional encodings (IPE). However, it relies on MLP architecture to represent the radiance fields, missing out on the fast training speed offered by the latest grid-based methods. In this work, we present mip-Grid, a novel approach that integrates anti-aliasing techniques into grid-based representations for radiance fields, mitigating the aliasing artifacts while enjoying fast training time. Notably, the proposed method uses a single-scale shared grid representation and a single-sampling approach, which only introduces minimal additions to the model parameters and computational costs. To handle scale ambiguity, mip-Grid generates multiple grids by applying simple convolution operations over the shared grid and uses the scale-aware coordinate to retrieve the appropriate features from the generated multiple grids. To test the effectiveness, we incorporated the proposed approach into the two recent representative grid-based methods, TensoRF and K-Planes. The experimental results demonstrated that mip-Grid greatly improved the rendering performance of both methods and showed comparable performance to mip-NeRF on multi-scale datasets while achieving significantly faster training time.

Mip-Grid: Anti-aliased Grid Representations for Neural Radiance Fields

Abstract

Despite the remarkable achievements of neural radiance fields (NeRF) in representing 3D scenes and generating novel view images, the aliasing issue, rendering 'jaggies' or 'blurry' images at varying camera distances, remains unresolved in most existing approaches. The recently proposed mip-NeRF has addressed this challenge by rendering conical frustums instead of rays. However, it relies on MLP architecture to represent the radiance fields, missing out on the fast training speed offered by the latest grid-based methods. In this work, we present mip-Grid, a novel approach that integrates anti-aliasing techniques into grid-based representations for radiance fields, mitigating the aliasing artifacts while enjoying fast training time. The proposed method generates multi-scale grids by applying simple convolution operations over a shared grid representation and uses the scale-aware coordinate to retrieve features at different scales from the generated multi-scale grids. To test the effectiveness, we integrated the proposed method into the two recent representative grid-based methods, TensoRF and K-Planes. Experimental results demonstrate that mip-Grid greatly improves the rendering performance of both methods and even outperforms mip-NeRF on multi-scale datasets while achieving significantly faster training time. For code and demo videos, please see https://stnamjef.github.io/mipgrid.github.io/ .

1 Introduction

Neural radiance fields (NeRF [19]) have been remarkably successful in modeling and reconstructing 3D scenes. Since its seminal contributions to novel view synthesis, researchers have investigated potential applications of NeRF across various domains in 3D computer vision and graphics, such as dynamic scene representation [43, 24, 25, 15], multi-view image generation [30, 6, 29, 32], 3D surface reconstruction [14], 3D content creation [26, 16, 18, 17], and SLAM [33, 47, 28].

In a typical neural rendering setup, NeRF represents a particular scene, mapping input coordinates ( x, d ) , where x is the spatial coordinate, and d is the view direction, to its corresponding RGB colors and density values ( r, g, b, σ ) . Due to its one-to-one mapping, NeRF suffers from aliasing effects when rendering scenes of varying image resolutions; it is incapable of producing different values for different resolutions given the same input coordinates. Mip-NeRF [1] has addressed this issue by introducing integrated positional encoding (IPE), which takes the volume of 3D conical frustums into account. This allows MLPs to reason about the size and shape of each conical frustum, eliminating the inherent ambiguity present in the original NeRF. While this approach provides a successful multi-scale representation, using a variant of positional encoding necessitates MLP architectures for the scene representations, which often leads to a long training time, even with contemporary GPUs.

(a) A quantitative comparison

(b) Rendering samples

36

35

34

PSNR

33

32

Mip-NeRF

1/8 Resolution

TensoRF (original)

31

TensoRF (single-scale)

30

TensoRF (multi-scale)

29

-1.0

0.5

0.0

0.5

1.0

1.5

2.0

TensoRF

TensoRF

Ground Truth

Training time (hours; log scale)

(original)

(single-scale)

To accelerate the training time of NeRF, the grid-based representations with or without MLPs have been extensively studied [9, 21, 8]. These approaches allow faster retrieval of color and density values at given input coordinates, reducing training time to less than an hour while maintaining rendering performance. Furthermore, the recent hashing technique with sophisticated code optimization has demonstrated that the training can be done in a few minutes [21]. However, akin to the original NeRF, these methods still implement the one-to-one mapping between input coordinates and the radiance values, resulting in multi-scale ambiguity. In this paper, we seek to address this limitation, combining grid-based representations for faster training and multi-scale representations for anti-aliasing.

A naive approach to implementing multi-scale representations using grid-based data structures, drawing inspiration from mipmap approaches [41], could involve training separate grid representations for different scales. During the rendering process, we could look up or interpolate the appropriate grid representations. While such an approach could resolve aliasing issues, it would substantially increase the storage requirements. As NeRF has emerged as a new media paradigm, such as 3D photos and videos, large data sizes would unduly waste valuable communication and storage resources.

With only a single-scale grid representation, an effective technique for mitigating aliasing effects would be the multisampling [10] (or supersampling [40]) method. Given the spatial coordinates and scales, we can sample multiple points within the conical frustums and subsequently average features extracted from the single-scale grid representation. This method can reduce the aliasing effects by reasoning scales during the rendering process. However, this approach comes at a high computational cost, directly proportional to the number of samples. Moreover, a similar study with MLP architecture has shown it requires considerable amounts of samples to achieve comparable performance [1].

This paper proposes mip-Grid, a novel method for anti-aliased grid representations that employs a shared single-scale grid representation and a single-sampling approach. We generate multi-scale grid representations by applying convolutions to the shared single-scale grid representation. Then, the features are extracted by interpolating the generated multi-scale grid representations using an additional input coordinate that can reason about the scales. Since the multi-scale grids are generated during the rendering process, the proposed method minimizes additional storage costs. Furthermore, the convolutions to generate multi-scale grids operate on reduced dimensions (2D or 1D instead of 3D convolutions), thereby not incurring substantial computational and memory costs. Lastly, the proposed method can easily be integrated into the recently proposed grid representations, such as factorized tensor representations [7] or plane-based representations [8], with minimal modifications.

We integrated mip-Grid into two state-of-the-art grid-based NeRFs: TensoRF and K-Planes. The resulting models, mip-TensoRF and mip-K-Planes, achieved significantly better performance compared to the original TensoRF and K-Planes, and mip-TensoRF even outperformed mip-NeRF while achieving significantly faster training time (Fig. 1-(a)). We observed that the aliasing artifacts (e.g., 'jaggies') are effectively removed from images rendered with the proposed method (Fig. 1-(b)), which are visually more pleasing as well. Additionally, we conducted extensive ablation studies and visualized the learned kernels to help readers understand how the proposed method works internally.

2 Related Work

2.1 Grid-based neural radiance fields

Representing continuous radiance fields with coordinate-based neural networks, NeRF can synthesize view-consistent and photorealistic novel views. However, it requires a long training time due to the spectral bias inherent in neural network training [27] and the need to query deep neural networks thousands of times to render a single image. A line of work has addressed this issue by encoding input coordinates with Fourier features [37] or learnable positional parameters that are jointly optimized during the training. The latter approach uses various data structures, such as voxel grids [9, 34], octrees [44, 35, 38], and hash tables [21, 39], to store the learnable parameters, each of which is responsible for learning the location-aware features that are decoded into color and density values. Encoding the low-dimensional input coordinates with the learnable parameters significantly accelerates the training and reduces the rendering time, mitigating the inherent spectral bias and eliminating the need to query the deep neural networks.

The dense voxel grid is a straightforward data structure to organize the learnable parameters in 3D space. Plenoxels [9] bake the radiance fields into dense voxel grids, resulting in 100 × faster convergence compared to the NeRF. However, cubically increasing memory usage makes the model unscalable. PlenOctrees [44] reduce the memory footprint of Plenoxels at inference time by replacing the dense grids with sparse octrees after the model convergence. This allows the model to discard the empty space of a scene, but the memory requirement still increase cubically during training. TensoRF [7] and other works [8, 5, 4] decompose the dense voxel grids into a set of matrices or vectors, improving parameter efficiency and rendering quality.

The primary focus of our work lies in developing an anti-aliasing framework for decomposed grid representations. While mip-NeRF has addressed the aliasing issue by introducing integrated positional encoding, the discussion was limited to MLP-based radiance fields, and the direct application to grid representations is not straightforward. We show that our method successfully removes the aliasing artifacts in grid representations, significantly reducing the training time compared to mip-NeRF.

2.2 Anti-aliasing and mipmap

Representing a discrete signal through sampling requires a higher sampling rate than Nyquist rate [31, 22] (twice the highest frequency of a given signal), and violating this rate occur aliasing artifacts, such as jaggedness and flickering, resulting in a low-quality visualization. Anti-aliasing has been widely studied in computer graphics, and it can be divided into two categories: postfiltering and pre-filtering. Supersampling [40] is one of the most representative post-filtering methods, which renders an image at a higher resolution than the target output resolution. Though effective, supersampling requires high computational and memory costs. The pre-filtering methods [23, 42] reduce the Nyquist rate by filtering a target signal into the low-frequency region before sampling in order to meet the sampling theorem. Mipmap [41] is one of the pre-filtering methods designed for real-time multi-scale rendering. It exploits a pre-filtered hierarchy of progressively lower resolution signals, in which the appropriate level is selected based on the desired scale to render. Mipmap can also represent a scene at arbitrary scales by interpolating the different levels of filtered signals.

Two concurrent works, Zip-NeRF [3] and Tri-MipRF [11], incorporate anti-aliasing techniques into grid-based NeRFs. Zip-NeRF approximates the integration of the grid representations by using multisampling techniques [10], which requires around 50 samplings to query a point during rendering. On the other hand, Tri-MipRF introduces tri-plane mipmaps to represent multi-scale 3D scenes, where the base level mipmap is trained during the reconstruction and the other levels of mipmaps are constructed by downscaling the base level mipmap. Both methods are implemented based on the tiny-cuda-nn [20], a highly optimized deep learning framework, achieving fast training time.

Our work builds on the concept of mipmap in graphics to address the aliasing in grid representations. We share some similarities with Tri-MipRF by incorporating a hierarchy of features that can be decoded into color and density values at queried scales. However, while Tri-MipRF construct the low-level mipmap by downscaling the previous level mipmap by a factor of 2, our method learns the optimal kernels that can generate multi-scale grid representations. Moreover, unlike both Zip-NeRF and Tri-MipRF, we aim to develop a simple framework that can be easily integrated into existing grid-based NeRFs without extensive modifications and hyperparameter tuning.

Feature

(pt1)

Aggregation

Feature

pt2

pt2

(pt2)

Feature

(ptl, pt2)

pt

(a) Multi-resolution Grid

(b) Multi-scale Grid

3 Method

3.1 Background: grid representations for neural radiance fields

Grid representations have proven to be highly successful in expediting the training time of NeRF [21, 7, 9]. Combined with lightweight MLPs, typically followed by feature extraction from the grids, it reduced the training times from days to hours or minutes. Furthermore, unlike the original NeRF that relies on MLPs, grid-based approaches do not exhibit spectral bias inherent in neural networks [27]. Through the regularization and training techniques, it often surpasses the performance of the MLPsbased methods while achieving significantly faster training time [21, 7]. In a typical setup, two distinct grids represent density and RGB colors separately. Formally,

where σ ∈ [0 , ∞ ) is density, c ∈ [0 , 1] 3 is RGB colors, G σ ∈ R H × W × L × C σ is a 3D grid for the density features, and G c ∈ R H × W × L × C c is a 4D grid for appearance features. H , W , L are the resolutions of the grids along each spatial axis X , Y , Z , while C σ and C c are the number of density and appearance feature channels, respectively. Given an input coordinate, f σ interpolates the density features in G σ and applies a non-linear activation function that outputs density values. Similarly, f c extracts the appearance features from G c , and normally takes an additional directional input d , and then these inputs are passed through an MLP to generate RGB colors. The final RGB colors for a given ray are generated using differentiable volume rendering [12], and simple reconstruction loss is used to train the G σ , G c , and the parameters of the MLP θ .

3.2 Multi-resolution and multi-scale grid representations

Many recent studies have successfully utilized multi-resolution (or multi-level) grid representations to encode various signals [21, 8]. To avoid confusion, we would like to clarify that multi-resolution grids refer to the usage of grids with different resolutions, while multi-scale grids refer to grids tailored to each image scale. As illustrated in Fig. 2-(a), the multi-resolution grid representations are designed to extract features from multiple grids with different resolutions, followed by aggregating operations to combine the extracted features, such as concatenation, addition, or multiplication. This approach draws inspiration from signal processing, encouraging the low-resolution grids to capture global and smooth parts of the signal (low-frequency components), while high-resolution grids represent fine details of the signal (high-frequency components). This decomposed representation is often parameter efficient and better reconstructs the signals. In addition, the recent hashing-based technique [21] also leverages multi-resolution grids to disambiguate hash collisions.

However, due to its inherent one-to-one mapping nature, the current multi-resolution grid representations have limitations in addressing the aliasing issue. When provided with identical input coordinates, the same features are extracted regardless of the camera distances (or scales). Consequently, the grid representations tend to learn to render the average scale images when training on multi-scale images (or images from varying camera distances). This results in 'blurred' images when rendering higher resolutions than the average training resolutions and 'jaggies' when rendering at lower resolutions.

Fig. 2-(b) depicts multi-scale grid representations, where individual grids (or groups of neighboring grids) are responsible for different scale images. In this setup, features are exclusively extracted from a single grid or interpolated from neighboring grid representations. As a result, this method can effectively address scale ambiguity by introducing distinct grids for different scales. While a promising approach, it dramatically increases the space complexity, making it an impractical solution for many potential NeRF applications, excessively exhausting communication and storage resources.

3.3 Shared grid and generating multi-scale grids

Since having distinct grid representations for different scales is not a viable solution, we propose to generate multi-scale grids with minimal storage requirements. Inspired by mipmapping techniques commonly used in computer graphics [41], and also motivated by the fact that a significant amount of information is shared across different scales, we use a shared grid representation and generate multi-scale grids by applying a simple convolution over the shared grid. The modified radiance fields with the generated multi-scale grids can be written as follows:

where ˜ G σ ∈ R H × W × L × S × C σ is the generated multi-scale grids for the density features, and ˜ G c ∈ R H × W × L × S × C c is the generated multi-scale grids for the appearance features. S is the number of the generated multi-scale grids, and s is a scale-aware input coordinate.

There are several options for generating ˜ G . We can simply use Gaussian filtering, which has been widely used in conventional computer graphics. However, it may not be the optimal filter for many real-world scenarios. Alternatively, we can utilize convolution with learnable filters or leverage multilayer neural networks such as convolutional neural networks (CNNs) or Transformers. Although complex neural networks can potentially yield better multi-scale representations, there is a trade-off in terms of computational complexity and performance. This work primarily investigates a single convolutional layer within our proposed multi-scale representations.

3.4 Scale-aware input coordinates

In order to reason about the scale, mip-NeRF introduced the integrated positional encoding to augment input coordinates to disambiguate the radiance fields from different camera distances: the original NeRF: ( x, d ) → ( σ, c ) and mip-NeRF: ( x, d, σ 2 t , σ 2 r ) → ( σ, c ) , where σ 2 t ∈ R and σ 2 r ∈ R are variances of the conical frustum with respect to its points interval along the ray and radius, respectively. As a result, mip-NeRF can provide different radiance values for the same coordinates ( x, d ) , reflecting the radii of sampled points ( σ 2 t , σ 2 r ) to effectively mitigate the aliasing artifacts.

In a similar vein, we introduce additional scale-aware coordinate, which is used to extract the features from the generated multi-scale grid representations. Specifically, we introduce three types of scale coordinates: the discrete scale coordinate, the continuous scale coordinate, and the 2-dimensional scale coordinate. The discrete scale coordinate is defined as the mean of the pixel width and height in the world coordinates scaled by 2 / √ 12 [1]. Since this value is unique for each image scale, we can simply use the same value for all input coordinates (i.e., sampled points) of each image scale to extract the features, not introducing extra computation to get per-point scale coordinates.

The continuous scale coordinate is the discrete scale coordinate scaled by the distance from the ray origin to the sampled point. Training the model with the discrete scale coordinate alone can lead to overfitting to given scales, making it challenging for the model to render images at unseen scales. For instance, if a dataset includes four scales of images, the model might generate multi-scale grids that overfit to those specific scales. In contrast, the continuous scale coordinate is not unique for each image scale. Each sampled point has a distinct scale value, which allows the model to generate multi-scale grids that can be smoothly interpolated into features across a broader range of scales.

The 2-dimensional scale coordinate consists of the continuous scale coordinate and the distance from the ray origin to the sampled point. While reflecting the discrete or continuous scale coordinate can effectively resolves scale ambiguity, we introduce the third one to provide readers with an understanding of how high-dimensional scale coordinates can be integrated into our method. During the rendering process, we apply convolution to the shared grid using two sets of kernels. We then independently extract features from each multi-scale grids based on the respective scale coordinates and average the feature sets to get the final features.

Shared Grid

≈

+

+

Factorized Tensors and Convolutions

Generated Multiscale Grids

pt1

pt2

⊙

Feature (pt1)

⊙

pt1

pt2

Feature (pt2)

3.5 Mip-Grid

Current grid-based NeRFs typically involves certain versions of tensor factorization schemes. For example, TensoRF used CP decomposition for compact models and proposed a new VM (vector-matrix) factorization method for better rendering performance. K-Planes used plane-based factorization, which has been very successful not only in 3D, but also in 2D and 4D scene generation. Thus, we have integrated the proposed method into two prominent grid-based NeRFs: TensoRF and K-Planes.

3.5.1 Mip-TensoRF

In TensoRF, VM factorization for a density grid G is formulated as follows (we omitted the subscript σ for brevity):

where v X r ∈ R H denote the vector for X axis, M Y Z r ∈ R W × L is the matrix for Y and Z axes, R is the rank, and ◦ is a tensor product, hence v X r ◦ M Y Z r ∈ R H × W × L . We generate the multi-scale vectors and matrices by applying 1D or 2D depth-wise convolutions over vectors and matrices. This is equivalent to applying 3D convolutions over 3D tensors thanks to the linearity of convolution operators. The i -th scale of a multi-scale grid ˜ G ∈ R H × W × L × S can be defined as follows:

where ∗ is the convolution operator, ˜ G i ∈ R H × W × L is i -th scale of the grid, k v X i ∈ R K × R is i -th depth-wise convolution filter for the vector v X r , k v X i,r ∈ R K is the filter for the rank r , and K is the kernel size. Similarly, k M Y Z i ∈ R K × K × R is i -th depth-wise convolution filter for the matrix M Y Z r and k M Y Z i,r ∈ R K × K is the filter for the rank r . Note that we have separate learnable kernels for each rank, axis, and scale, which are more expressible than fixed Gaussian filters.

3.5.2 Mip-K-Planes

Similar to mip-TensoRF, we apply depth-wise convolutions over the plane-based grid representations. In K-Planes, a grid G k ∈ R D x × D y × D z × R can be formulated as follows:

where M Y Z ∈ R 1 × D y × D z × R , M XZ ∈ R D x × 1 × D z × R , and M XY ∈ R D x × D y × 1 × R denote three grids for 3D scene representations. ⊙ Y Z XZ R D x × D y × D z × R

denotes broadcast element-wise multiplication. For example, M ⊙ M will have the shape of . D x , D y , and D z denote the grid resolutions in X, Y, and Z axes. Similar to Eq. 4, a multi-scale grid ˜ G k i ∈ R D x × D y × D z × R for i -th scale can be written as follows:

where k Y Z i ∈ R 1 × K × K × R , k XZ i ∈ R K × 1 × K × R , and k XY i ∈ R K × K × 1 × R denote i -th depth-wise convolution filters.

1/2 Res.

Ours

GT

Original

Single-scale

4 Experiment

4.1 Experimental setup

Implementation details We implemented two variants of mip-Grid: mip-TensoRF and mip-KPlanes. Both models apply convolution to a shared grid representation using four learnable kernels of size 3, where each kernel is responsible for generating different scales of grids. We trained mipTensoRF for 40k iterations on the multi-scale Blender dataset and 25k iterations on the multi-scale LLFF dataset. All of its three baseline models were also trained for the same number of iterations. We scaled the loss of each pixel following mip-NeRF [1] to account for the scale differences. Note that we began kernel training after the grid upsampling [7] was completed. Unlike mip-TensoRF, we trained mip-K-Planes for 30k iterations and optimized the kernels from the beginning. All other training hyperparameters remained the same as the original TensoRF-VM-192 and K-Planes.

As outlined in the Sec. 3, mip-Grid introduces three kinds of scale coordinates to extract features from the generated multi-scale grids. We used the discrete scale coordinate to train mip-TensoRF and mip-K-Planes, and for mip-TensoRF, we conducted a further evaluation using the continuous scale coordinate and the 2D scale coordinate, examining the effectiveness of using more sophisticated scale coordinates. In all tables showing the results of mip-Grid integrated models, abbreviations of the scale coordinate types are provided in parentheses along with the name of each model.

Baseline models We compare our method against three baseline models: the original model, the single-scale model, and the multi-scale model. The original models are the same as the TensoRFVM-192 and K-Planes, but we test them on the multi-scale dataset. The single-scale model is trained on the multi-scale dataset with loss multipliers [1] and tested on the multi-scale dataset. The multi-scale model is a set of four independent original models, each of which is trained and tested on different scales of dataset. We include the latter two baseline models to show that naively training the single-scale representations with multi-scale dataset cannot remove aliasing artifacts effectively, and using four independent models requires four times more parameters than the original model.

Datasets All models were evaluated on the multi-scale Blender dataset [1] with three different metrics: PSNR, SSIM, and LPIPS (VGG) [46]. Mip-TensoRF and its baseline models were further evaluated on the multi-scale LLFF dataset. To convert single-scale dataset to multi-scale dataset, we bilinearly downsampled the images by a factor of 1, 2, 4, and 8, and rescaled the focal length accordingly as typically done in other multi-scale NeRF literature [1, 3].

4.2 Results

Tab. 1 shows the evaluation results on the multi-scale Blender dataset. Mip-TensoRF and mip-KPlanes outperform their baseline models, achieving the highest average PSNR. Notably, mip-TensoRF even surpasses mip-NeRF while significantly reducing the training time. The performance of the original models experiences a substantial drop as the rendering resolution decreases, leading to noticeable 'jaggies' in the images rendered at the two lowest resolutions (Fig. 4). The single-scale models achieve better performance at the two lowest resolutions compared to the original models but show worse results at other resolutions. This suggests that, to some extent, training with a multi-scale dataset and employing the loss multipliers helps the model in reasoning about different scales, but this approach does not entirely eliminate aliasing artifacts. On the other hand, the multi-scale models can effectively represent different scales of a scene, achieving the best PSNR among the three baseline models. Nevertheless, these models require four times more parameters than the original models, which is undesirable in practical applications. Unlike others, our models successfully mitigate aliasing artifacts while minimizing the additional number of parameters and computational cost.

We provide additional results of training mip-TensoRF with continuous and 2-dimensional scale coordinates in Tab. 3. The first two models show only a marginal difference when evaluated at the training resolutions. A similar trend is observed in evaluations at unseen resolutions, but there is a clear difference when the image resolution is reduced from 4/8 resolution to 3/8 resolution, suggesting that the continuous scale coordinate improves the model's generalization performance. Our method generates multiple sets of feature grids from a shared representation, and these feature sets share many similarities. This allows different features to be smoothly interpolated and rendered into visually pleasing images. However, more information needs to be removed as the output images decrease in size, leading to relatively large differences between low-scale features. This explains the marginal drop in PSNR at lower resolutions. Moreover, the model trained with 2D scale coordinate outperforms the others, indicating that the additional scale coordinate (the distance from the ray origin to the sampled point) can be beneficial for resolving the remaining scale ambiguity.

Shared grid

Learned kernels

1/2 Res.

4.3 Visualization of learned kernels and multi-scale grids

We visualize the learned kernels and the generated multi-scale grids of mip-TensoRF to illustrate how the learned patterns vary across scales. As depicted in Fig. 5, the high-scale kernels look similar to Gaussian distribution, and their shape (i.e., the standard deviation) becomes larger as the corresponding scale decreases. This observation aligns with our intuition that the kernels of low-scale features should be smoother than those of high-scale features and demonstrates that our method effectively learns optimal kernels for each scale. A similar phenomenon can be observed in the visualization of feature grids, where high-frequency details are progressively removed as the feature scale decreases. Although we do not provide the visualization of density kernels and grids due to the limited space, the learned patterns closely resemble those of the appearance kernels and grids.

4.4 Ablation studies

Fixed Gaussian kernels and learnable kernels We compare mip-TensoRF with fixed Gaussian kernels against the same model with learnable kernels. Both models were initialized with 2 sets of standard deviations {1.0, 1.0, 1.0, 1.0} and {1.0, 1.5, 2.5, 4.0}, where the leftmost value was used to set the kernel of the highest scale, and the rightmost value was used for the lowest scale. The second set is to give an inductive bias that the low-scale features should be smoothed more than high-scale features. The fixed Gaussian kernel initialized with the second set of standard deviations significantly outperforms the other (Tab. 4). This suggests that increasing standard deviations can be a beneficial inductive bias. However, these fixed Gaussian kernels may not be optimal for each scale. Instead of using heuristically selected kernels, our method fine-tunes the Gaussian kernels and finds the optimal weights during the training. Experimental results demonstrate that the learnable approach outperforms the fixed Gaussian approach, further confirming the effectiveness of our method (Tab. 4).

The number of multi-scale grids and kernel size The proposed method involves two key hyperparameters: the number of multi-scale grids and the kernel size. We provide an evaluation of mip-TensoRF with nine different combinations of these two values to give readers guidance on hyperparameter configuration. As shown in Tab. 5, the number of multi-scale grids has little effect

on the rendering quality, but variations in kernel sizes result in relatively bigger differences in the PSNR of low-scale images. Although different scales of grid representations share many similarities, the low-scale grid should be smoothed enough to produce an image without aliasing artifacts, and the kernel size is a critical factor controlling the smoothness of the grids. Using too small kernels may introduce unnecessary high-frequency components in the low-scale grid. Nevertheless, there is always a trade-off between rendering quality and speed, and users should consider their specific usage purpose when determining the kernel size.

5 Limitations and discussion

The proposed method has not been tested on scenes involving more complex camera poses, such as 360-degree scenes [2, 45, 36] or free trajectories [39]. However, addressing these scenarios would require further technical development and a thorough parameter tuning. We plan to investigate how the proposed method, or even the grid representation itself, can be extended to these scenarios.

Although the proposed method is significantly faster than mip-NeRF, it introduces additional complexity to the system. Especially in terms of training time, our method needs more computational resources to generate multi-scale grids and retrieve features. We implemented our model based on TensoRF and K-Planes with minimal modifications, which makes the training time heavily rely on the base models. Also, the implementation of the multi-scale grid generation and the feature retrieval process has not been optimized yet. We strongly believe the training time and memory requirements can be substantially improved by fusing the multi-scale grid generation and the feature retrieval process as a single CUDA kernel or using the NeRF accelerating tool [13].

6 Conclusion

We proposed mip-Grid, anti-aliased grid representations for NeRF. The proposed approach can easily be integrated into the existing grid-based NeRFs, and the two methods using our approach, mip-TensoRF and mip-K-Planes, have demonstrated that aliasing artifacts can be effectively removed. Since we generate multi-scale grids from a shared grid representation and do not rely on supersampling, the proposed method minimizes additional number of parameters and trains significantly faster than the existing MLP-based anti-aliased NeRFs. We believe our work paves the way for a new research direction toward alias-free NeRF leveraging the training efficiency of grid representations.

Acknowledgments and Disclosure of Funding

This work was supported by the Institute of Information and Communication Technology Planning Evaluation (IITP) grants (IITP-2019-0-00421, IITP-2021-0-02068) and the National Research Foundation (NRF) grants (2022R1F1A1064184, RS-2023-00245342) funded by the Ministry of Science and ICT (MSIT) of Korea.

References

Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Zip-nerf: Antialiased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. arXiv preprint arXiv:2301.09632 , 2023.

Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV) , 2022.

Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Ned Greene and Paul S Heckbert. Creating raster omnimax images from multiple perspective views using the elliptical weighted average filter. IEEE Computer Graphics and Applications , 1986.

Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang, Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: Tri-mip representation for efficient anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

James T Kajiya and Brian P Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics , 1984.

Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerfacc: A general nerf acceleration toolbox. arXiv preprint arXiv:2210.04847 , 2022.

Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and ChenHsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction of any object from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV) , 2020.

Thomas Müller. tiny-cuda-nn, 2021.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (TOG) , 2022.

Harry Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers , 1928.

Marc Olano and Dan Baker. Lean mapping. In Proceedings of the ACM SIGGRAPH symposium on Interactive 3D Graphics and Games , 2010.

Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG) , 2021.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference on Learning Representations (ICLR) , 2022.

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Learning Representations (ICLR) , 2019.

Antoni Rosinol, John J Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2023.

Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, and Deqing Sun. VQ3D: Learning a 3D-aware generative model on ImageNet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023.

Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. 2020.

Claude E Shannon. Communication in the presence of noise. 1949.

Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, and Sergey Tulyakov. 3d generation on imagenet. In International Conference on Learning Representations (ICLR) , 2022.

Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. imap: Implicit mapping and positioning in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. In ACM SIGGRAPH , 2022.

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Turner Whitted. An improved illumination model for shaded display. In ACM SIGGRAPH . 2005.

Lance Williams. Pyramidal parametrics. In Proceedings of the 10th annual conference on Computer graphics and interactive techniques , 1983.

Lifan Wu, Shuang Zhao, Ling-Qi Yan, and Ravi Ramamoorthi. Accurate appearance preserving prefiltering for rendering displacement-mapped surfaces. ACM Transactions on Graphics (TOG) , 2019.

Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 , 2020.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.

Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the anti-alias problem of grid-based static scene representation.\nThe existing anti-alias methods like Mip-NeRF still require intensive computations, while current grid/factorization-based methods don't show good anti-alias results. \nThis paper proposes a general idea to solve the anti-alias problem: it uses a single-scale grid as the representation while applying learnable convolution to generate multi-scale grids. \nCombining with scale-aware position encoding, this paper shows superior results compared to TensoRF or KPlanes, demonstrating a universal improvement for grid-based representation.\n\nSTRENGTHS:\n1. This paper studies an interesting problem in an important direction. Extending the anti-alias idea from Mip-NeRF to explicit representation could contribute to the whole community, and the results are promising. I think it would be beneficial to the development of explicit representations.\n2. The whole method is neat, simple, and powerful. Applying CNN on the single grid and having a scale-related positional encoding is simple and intuitive. The overall framework is neat without too many tricks and complicated tweaks. Considering it, this idea could be broadly used in many situations. \n3. The paper is overall well-written and easy to follow.\n\nWEAKNESSES:\n1. It would be more convincing to compare several baselines with multi-resolution/multi-scale designs. For example, Instant-NGP[17] uses a multi-resolution hash. Also, the original K-Plane uses multi-resolution planes. It would be helpful to show how these method works and how the proposed method compared to these baselines.\n2. I am not fully convinced by the analysis of multi-scale and multi-resolution in Section 3.2. To me, the multi-scale one sounds like optimization multi-NeRF at different scales, while NeRFs from different scales are not related. Could you elaborate a little bit? It would be nice to show the experimental results of multi-resolution ideas. \n3. Not necessarily a weakness. It would be good to write the method names instead of baseline 1 and 2 in Figure 3.\n4. In Table 1, Mip-Kplanes's PSNR has different trends to MIP-NeRF's PSNR and mip-TensoRF's PSNR, also different from Mip-Kplanes' SSIM and LPIPS. Could you elaborate on why  PSNRs are lower with lower Res?\n\nQUESTIONS:\nAs stated above, it would be good to show (1) more baselines, especially baselines using multi-scale/resolution methods. (2) experiments compared to multi-resolution grids.\n\nLIMITATIONS:\nI think the limitations discussed in this paper make sense. \nI agree the current method is still preliminary and having MIP-NeRF style design would be an exciting direction.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes mip-Grid to integrate anti-aliasing techniques into grid-based representations for radiance fields. Mip-Grid generates multiple grids by applying convolution operations over the shared single-scale grid representation. Then it uses the scale-aware coordinate to retrieve the appropriate features from the generated multiple grids. This anti-aliased grid representation combines grid-based representations for faster training and Mip-NeRF-like multi-scale representations for anti-aliasing. The key idea is to use a shared single-scale grid representation and a single sampling but generate multiple grid representations by applying convolutions to the shared single-scale grid representation. Then, the features are extracted by interpolating the generated multiple grid representations using an additional input coordinate.\n\nSTRENGTHS:\nThis work mitigates the aliasing artifacts when rendering images from varying camera distances, such as blurry and jaggies in the images, while enjoying fast training time.\n\nThis work uses a single-scale shared grid representation and a single-sampling approach, which only introduces minimal additions to the model parameters and computational costs.\n\nThe convolutions to generate multiple grids are lightweight and do not incur substantial computational and memory costs as it operates on reduced dimensions.\n\nThis work shows comparable performance to mip-NeRF on multi-scale datasets while achieving significantly faster training time.\n\nThis work improves the rendering performance of TensoRF and K-Planes on multi-scale datasets while introducing tolerable training time.\n\nWEAKNESSES:\nGiven the fact that Zip-NeRF is implemented on the hash-based Instant-NGP, thus it achieves fast training time despite using multisampling techniques. I will concern about the contribution of this work, as Zip-NeRF shows good performance for both anti-aliasing and training time. I would like to see the attitude of other reviewers.\n\nI believe \"unbounded\" scenes are the current research topics for Anti-Aliased NeRFs. Though I understand that addressing these scenarios would require other techniques, the lack of experiments of unbounded scenes (such as Mip-NeRF 360) will degree the rating of this work.\n\nLack of visualization videos.\n\nQUESTIONS:\nI would suggest including the comparison with Zip-NeRF since an unofficial PyTorch implementation of Zip-NeRF is available.\n\nLIMITATIONS:\nLack of experiments of unbounded scenes (such as Mip-NeRF 360).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work proposes a simple anti-aliasing method for NeRF representations based on factorized grids, e.g., TensoRF and K-planes. The key idea is to learn a lowpass filtering kernel

STRENGTHS:
1. The proposed method seems simple and easy to reproduce. Example implementation is also included in the supplementary.
2. The method is evaluated against baseline methods without the proposed anti-aliasing technique, and clear improvements in anti-aliasing can be seen both quantitatively and qualitatively.

WEAKNESSES:
1. It doesn't seem clear how the multi-scale grid is used exactly in rendering; for example, are all the scales used to get a point's feature, or only a heuristically selected scale is used?

2. One thing I'm a bit unsure is that: why does the same kernel size seem to be used across all scales? Would it makes more sense to use bigger kernel size for lower-resolution grids?

3. A visualization of the kernel weights will be helpful in understanding what filtering patterns have been learned, especially the relationship of the grid scale and learned kernel weights. 

4. In table 1 and table 2, best models should be highlighted to facilitate reading.

QUESTIONS:
See the first and second bullet points in the weakness section.

LIMITATIONS:
The proposed method doesn't seem to be easily applicable to the multi-resolution hash grid proposed by Instant-NGP. It might be better to mention this.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
The paper proposes a method to deal with the aliasing issue in factorised grid-based NeRF methods, e.g. TensoRF and K-plane, by applying convolution operations to decomposed vectors and matrices.

---
**After rebuttal**: I have read authors' rebuttal and it addresses my concerns.

STRENGTHS:
* The goal of handling aliasing in grid-based NeRF is valuable and the proposed method is novel.
* Source code is provided in supp mat.

WEAKNESSES:
About evaluation: 
* I am interested to see what is the performance difference between using convolution operations and using a simple Gaussian?

About methodology: 
* I got the impression that the proposed method can only handle a number of *discrete* scales after training, since the convolution kernels are optimised for each scale. If I understand correctly, that means a set of convolution kernels is optimised for a specific scale. If a NeRF is optimised for $4$ scales, for example, scale=$1, \frac{1}{2}, \frac{1}{4}$, and $\frac{1}{8}$, then during inference, it’s impossible to render an image for scale=$\frac{1}{3}$ and $\frac{1}{6}$ from this NeRF?
* Following the point above, if my understanding above is correct, it would be much better to be able to render images from continuous scales.

QUESTIONS:
See the weakness section above.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07