SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Seungtae Nam $^{1}$  Daniel Rho $^{2}$  Jong Hwan Ko $^{1,3}$
Eunbyung Park $^{1,3*}$
$^{1}$ Department of Artificial Intelligence, Sungkyunkwan University
$^{2}$ AI2XL, KT
$^{3}$ Department of Electrical and Computer Engineering, Sungkyunkwan University
Despite the remarkable achievements of neural radiance fields (NeRF) in representing 3D scenes and generating novel view images, the aliasing issue, rendering "jaggies" or "blurry" images at varying camera distances, remains unresolved in most existing approaches. The recently proposed mip-NeRF has addressed this challenge by rendering conical frustums instead of rays. However, it relies on MLP architecture to represent the radiance fields, missing out on the fast training speed offered by the latest grid-based methods. In this work, we present mip-Grid, a novel approach that integrates anti-aliasing techniques into grid-based representations for radiance fields, mitigating the aliasing artifacts while enjoying fast training time. The proposed method generates multi-scale grids by applying simple convolution operations over a shared grid representation and uses the scale-aware coordinate to retrieve features at different scales from the generated multi-scale grids. To test the effectiveness, we integrated the proposed method into the two recent representative grid-based methods, TensoRF and K-Planes. Experimental results demonstrate that mip-Grid greatly improves the rendering performance of both methods and even outperforms mip-NeRF on multi-scale datasets while achieving significantly faster training time. For code and demo videos, please see https://stnamjef.github.io/mipgrid.github.io/.
Neural radiance fields (NeRF [19]) have been remarkably successful in modeling and reconstructing 3D scenes. Since its seminal contributions to novel view synthesis, researchers have investigated potential applications of NeRF across various domains in 3D computer vision and graphics, such as dynamic scene representation [43, 24, 25, 15], multi-view image generation [30, 6, 29, 32], 3D surface reconstruction [14], 3D content creation [26, 16, 18, 17], and SLAM [33, 47, 28].
In a typical neural rendering setup, NeRF represents a particular scene, mapping input coordinates  $(x, d)$ , where  $x$  is the spatial coordinate, and  $d$  is the view direction, to its corresponding RGB colors and density values  $(r, g, b, \sigma)$ . Due to its one-to-one mapping, NeRF suffers from aliasing effects when rendering scenes of varying image resolutions; it is incapable of producing different values for different resolutions given the same input coordinates. Mip-NeRF [1] has addressed this issue by introducing integrated positional encoding (IPE), which takes the volume of 3D conical frustums into account. This allows MLPs to reason about the size and shape of each conical frustum, eliminating the inherent ambiguity present in the original NeRF. While this approach provides a successful multi-scale representation, using a variant of positional encoding necessitates MLP architectures for the scene representations, which often leads to a long training time, even with contemporary GPUs.
(a) A quantitative comparison
(b) Rendering samples
Figure 1: (a) A quantitative comparison of mip-TensoRF against its three baseline models and mip-NeRF (see Sec. 4 for details). (b) Rendering samples of the *lego* scene in two different scales.
To accelerate the training time of NeRF, the grid-based representations with or without MLPs have been extensively studied [9, 21, 8]. These approaches allow faster retrieval of color and density values at given input coordinates, reducing training time to less than an hour while maintaining rendering performance. Furthermore, the recent hashing technique with sophisticated code optimization has demonstrated that the training can be done in a few minutes [21]. However, akin to the original NeRF, these methods still implement the one-to-one mapping between input coordinates and the radiance values, resulting in multi-scale ambiguity. In this paper, we seek to address this limitation, combining grid-based representations for faster training and multi-scale representations for anti-aliasing.
A naive approach to implementing multi-scale representations using grid-based data structures, drawing inspiration from mipmap approaches [41], could involve training separate grid representations for different scales. During the rendering process, we could look up or interpolate the appropriate grid representations. While such an approach could resolve aliasing issues, it would substantially increase the storage requirements. As NeRF has emerged as a new media paradigm, such as 3D photos and videos, large data sizes would unduly waste valuable communication and storage resources.
With only a single-scale grid representation, an effective technique for mitigating aliasing effects would be the multisampling [10] (or supersampling [40]) method. Given the spatial coordinates and scales, we can sample multiple points within the conical frustums and subsequently average features extracted from the single-scale grid representation. This method can reduce the aliasing effects by reasoning scales during the rendering process. However, this approach comes at a high computational cost, directly proportional to the number of samples. Moreover, a similar study with MLP architecture has shown it requires considerable amounts of samples to achieve comparable performance [1].
This paper proposes mip-Grid, a novel method for anti-aliased grid representations that employs a shared single-scale grid representation and a single-sampling approach. We generate multi-scale grid representations by applying convolutions to the shared single-scale grid representation. Then, the features are extracted by interpolating the generated multi-scale grid representations using an additional input coordinate that can reason about the scales. Since the multi-scale grids are generated during the rendering process, the proposed method minimizes additional storage costs. Furthermore, the convolutions to generate multi-scale grids operate on reduced dimensions (2D or 1D instead of 3D convolutions), thereby not incurring substantial computational and memory costs. Lastly, the proposed method can easily be integrated into the recently proposed grid representations, such as factorized tensor representations [7] or plane-based representations [8], with minimal modifications.
We integrated mip-Grid into two state-of-the-art grid-based NeRFs: TensoRF and K-Planes. The resulting models, mip-TensoRF and mip-K-Planes, achieved significantly better performance compared to the original TensoRF and K-Planes, and mip-TensoRF even outperformed mip-NeRF while achieving significantly faster training time (Fig. 1-(a)). We observed that the aliasing artifacts (e.g., "jaggies") are effectively removed from images rendered with the proposed method (Fig. 1-(b)), which are visually more pleasing as well. Additionally, we conducted extensive ablation studies and visualized the learned kernels to help readers understand how the proposed method works internally.
2.1 Grid-based neural radiance fields
Representing continuous radiance fields with coordinate-based neural networks, NeRF can synthesize view-consistent and photorealistic novel views. However, it requires a long training time due to the spectral bias inherent in neural network training [27] and the need to query deep neural networks thousands of times to render a single image. A line of work has addressed this issue by encoding input coordinates with Fourier features [37] or learnable positional parameters that are jointly optimized during the training. The latter approach uses various data structures, such as voxel grids [9, 34], octrees [44, 35, 38], and hash tables [21, 39], to store the learnable parameters, each of which is responsible for learning the location-aware features that are decoded into color and density values. Encoding the low-dimensional input coordinates with the learnable parameters significantly accelerates the training and reduces the rendering time, mitigating the inherent spectral bias and eliminating the need to query the deep neural networks.
The dense voxel grid is a straightforward data structure to organize the learnable parameters in 3D space. Plenoxels [9] bake the radiance fields into dense voxel grids, resulting in  $100 \times$  faster convergence compared to the NeRF. However, cubically increasing memory usage makes the model unscalable. PlenOctrees [44] reduce the memory footprint of Plenoxels at inference time by replacing the dense grids with sparse octrees after the model convergence. This allows the model to discard the empty space of a scene, but the memory requirement still increase cubically during training. TensoRF [7] and other works [8, 5, 4] decompose the dense voxel grids into a set of matrices or vectors, improving parameter efficiency and rendering quality.
The primary focus of our work lies in developing an anti-aliasing framework for decomposed grid representations. While mip-NeRF has addressed the aliasing issue by introducing integrated positional encoding, the discussion was limited to MLP-based radiance fields, and the direct application to grid representations is not straightforward. We show that our method successfully removes the aliasing artifacts in grid representations, significantly reducing the training time compared to mip-NeRF.
2.2 Anti-aliasing and mipmap
Representing a discrete signal through sampling requires a higher sampling rate than Nyquist rate [31, 22] (twice the highest frequency of a given signal), and violating this rate occurs aliasing artifacts, such as jaggedness and flickering, resulting in a low-quality visualization. Anti-aliasing has been widely studied in computer graphics, and it can be divided into two categories: postfiltering and pre-filtering. Supersampling [40] is one of the most representative post-filtering methods, which renders an image at a higher resolution than the target output resolution. Though effective, supersampling requires high computational and memory costs. The pre-filtering methods [23, 42] reduce the Nyquist rate by filtering a target signal into the low-frequency region before sampling in order to meet the sampling theorem. Mipmap [41] is one of the pre-filtering methods designed for real-time multi-scale rendering. It exploits a pre-filtered hierarchy of progressively lower resolution signals, in which the appropriate level is selected based on the desired scale to render. Mipmap can also represent a scene at arbitrary scales by interpolating the different levels of filtered signals.
Two concurrent works, Zip-NeRF [3] and Tri-MipRF [11], incorporate anti-aliasing techniques into grid-based NeRFs. Zip-NeRF approximates the integration of the grid representations by using multisampling techniques [10], which requires around 50 samplings to query a point during rendering. On the other hand, Tri-MipRF introduces tri-plane mipmaps to represent multi-scale 3D scenes, where the base level mipmap is trained during the reconstruction and the other levels of mipmaps are constructed by downscaling the base level mipmap. Both methods are implemented based on the tiny-cuda-nn [20], a highly optimized deep learning framework, achieving fast training time.
Our work builds on the concept of mipmap in graphics to address the aliasing in grid representations. We share some similarities with Tri-MipRF by incorporating a hierarchy of features that can be decoded into color and density values at queried scales. However, while Tri-MipRF construct the low-level mipmap by downscaling the previous level mipmap by a factor of 2, our method learns the optimal kernels that can generate multi-scale grid representations. Moreover, unlike both Zip-NeRF and Tri-MipRF, we aim to develop a simple framework that can be easily integrated into existing grid-based NeRFs without extensive modifications and hyperparameter tuning.
(a) Multi-resolution Grid
(b) Multi-scale Grid
Figure 2: Multi-resolution and multi-scale grid representations: the pt1 and pt2 are the same spatial coordinates in 3D space. When rendering from varying camera distances, (a) multi-resolution grid representations still retrieve the same features for both pt1 and pt2, while (b) multi-scale grid representations have distinct grids for each pt1 and pt2, hence, effectively resolving scale ambiguity.
3.1 Background: grid representations for neural radiance fields
Grid representations have proven to be highly successful in expediting the training time of NeRF [21, 7, 9]. Combined with lightweight MLPs, typically followed by feature extraction from the grids, it reduced the training times from days to hours or minutes. Furthermore, unlike the original NeRF that relies on MLPs, grid-based approaches do not exhibit spectral bias inherent in neural networks [27]. Through the regularization and training techniques, it often surpasses the performance of the MLPs-based methods while achieving significantly faster training time [21, 7]. In a typical setup, two distinct grids represent density and RGB colors separately. Formally,
$$
\sigma = f _ {\sigma} (x; \mathcal {G} _ {\sigma}), \quad c = f _ {c} (x, d; \theta , \mathcal {G} _ {c}), \tag {1}
$$
where  $\sigma \in [0,\infty)$  is density,  $c\in [0,1]^3$  is RGB colors,  $\mathcal{G}_{\sigma}\in \mathbb{R}^{H\times W\times L\times C_{\sigma}}$  is a 3D grid for the density features, and  $\mathcal{G}_c\in \mathbb{R}^{H\times W\times L\times C_c}$  is a 4D grid for appearance features.  $H,W,L$  are the resolutions of the grids along each spatial axis  $X,Y,Z$ , while  $C_\sigma$  and  $C_c$  are the number of density and appearance feature channels, respectively. Given an input coordinate,  $f_{\sigma}$  interpolates the density features in  $\mathcal{G}_{\sigma}$  and applies a non-linear activation function that outputs density values. Similarly,  $f_{c}$  extracts the appearance features from  $\mathcal{G}_c$ , and normally takes an additional directional input  $d$ , and then these inputs are passed through an MLP to generate RGB colors. The final RGB colors for a given ray are generated using differentiable volume rendering [12], and simple reconstruction loss is used to train the  $\mathcal{G}_{\sigma},\mathcal{G}_{c}$ , and the parameters of the MLP  $\theta$ .
3.2 Multi-resolution and multi-scale grid representations
Many recent studies have successfully utilized multi-resolution (or multi-level) grid representations to encode various signals [21, 8]. To avoid confusion, we would like to clarify that multi-resolution grids refer to the usage of grids with different resolutions, while multi-scale grids refer to grids tailored to each image scale. As illustrated in Fig. 2-(a), the multi-resolution grid representations are designed to extract features from multiple grids with different resolutions, followed by aggregating operations to combine the extracted features, such as concatenation, addition, or multiplication. This approach draws inspiration from signal processing, encouraging the low-resolution grids to capture global and smooth parts of the signal (low-frequency components), while high-resolution grids represent fine details of the signal (high-frequency components). This decomposed representation is often parameter efficient and better reconstructs the signals. In addition, the recent hashing-based technique [21] also leverages multi-resolution grids to disambiguate hash collisions.
However, due to its inherent one-to-one mapping nature, the current multi-resolution grid representations have limitations in addressing the aliasing issue. When provided with identical input coordinates, the same features are extracted regardless of the camera distances (or scales). Consequently, the grid representations tend to learn to render the average scale images when training on multi-scale images (or images from varying camera distances). This results in "blurred" images when rendering higher resolutions than the average training resolutions and "jaggies" when rendering at lower resolutions.
Fig. 2-(b) depicts multi-scale grid representations, where individual grids (or groups of neighboring grids) are responsible for different scale images. In this setup, features are exclusively extracted from a single grid or interpolated from neighboring grid representations. As a result, this method can effectively address scale ambiguity by introducing distinct grids for different scales. While a promising approach, it dramatically increases the space complexity, making it an impractical solution for many potential NeRF applications, excessively exhausting communication and storage resources.
3.3 Shared grid and generating multi-scale grids
Since having distinct grid representations for different scales is not a viable solution, we propose to generate multi-scale grids with minimal storage requirements. Inspired by mipmapping techniques commonly used in computer graphics [41], and also motivated by the fact that a significant amount of information is shared across different scales, we use a shared grid representation and generate multi-scale grids by applying a simple convolution over the shared grid. The modified radiance fields with the generated multi-scale grids can be written as follows:
$$
\sigma = f _ {\sigma} (x, s; \tilde {\mathcal {G}} _ {\sigma}), \quad c = f _ {c} (x, s, d; \theta , \tilde {\mathcal {G}} _ {c}), \tag {2}
$$
where  $\tilde{\mathcal{G}}_{\sigma} \in \mathbb{R}^{H \times W \times L \times S \times C_{\sigma}}$  is the generated multi-scale grids for the density features, and  $\tilde{\mathcal{G}}_c \in \mathbb{R}^{H \times W \times L \times S \times C_c}$  is the generated multi-scale grids for the appearance features.  $S$  is the number of the generated multi-scale grids, and  $s$  is a scale-aware input coordinate.
There are several options for generating  $\tilde{\mathcal{G}}$ . We can simply use Gaussian filtering, which has been widely used in conventional computer graphics. However, it may not be the optimal filter for many real-world scenarios. Alternatively, we can utilize convolution with learnable filters or leverage multi-layer neural networks such as convolutional neural networks (CNNs) or Transformers. Although complex neural networks can potentially yield better multi-scale representations, there is a trade-off in terms of computational complexity and performance. This work primarily investigates a single convolutional layer within our proposed multi-scale representations.
3.4 Scale-aware input coordinates
In order to reason about the scale, mip-NeRF introduced the integrated positional encoding to augment input coordinates to disambiguate the radiance fields from different camera distances: the original NeRF:  $(x,d)\to (\sigma ,c)$  and mip-NeRF:  $(x,d,\sigma_t^2,\sigma_r^2)\rightarrow (\sigma ,c)$ , where  $\sigma_t^2\in \mathbb{R}$  and  $\sigma_r^2\in \mathbb{R}$  are variances of the conical frustum with respect to its points interval along the ray and radius, respectively. As a result, mip-NeRF can provide different radiance values for the same coordinates  $(x,d)$ , reflecting the radii of sampled points  $(\sigma_t^2,\sigma_r^2)$  to effectively mitigate the aliasing artifacts.
In a similar vein, we introduce additional scale-aware coordinate, which is used to extract the features from the generated multi-scale grid representations. Specifically, we introduce three types of scale coordinates: the discrete scale coordinate, the continuous scale coordinate, and the 2-dimensional scale coordinate. The discrete scale coordinate is defined as the mean of the pixel width and height in the world coordinates scaled by  $2 / \sqrt{12}$  [1]. Since this value is unique for each image scale, we can simply use the same value for all input coordinates (i.e., sampled points) of each image scale to extract the features, not introducing extra computation to get per-point scale coordinates.
The continuous scale coordinate is the discrete scale coordinate scaled by the distance from the ray origin to the sampled point. Training the model with the discrete scale coordinate alone can lead to overfitting to given scales, making it challenging for the model to render images at unseen scales. For instance, if a dataset includes four scales of images, the model might generate multi-scale grids that overfit to those specific scales. In contrast, the continuous scale coordinate is not unique for each image scale. Each sampled point has a distinct scale value, which allows the model to generate multi-scale grids that can be smoothly interpolated into features across a broader range of scales.
The 2-dimensional scale coordinate consists of the continuous scale coordinate and the distance from the ray origin to the sampled point. While reflecting the discrete or continuous scale coordinate can effectively resolves scale ambiguity, we introduce the third one to provide readers with an understanding of how high-dimensional scale coordinates can be integrated into our method. During the rendering process, we apply convolution to the shared grid using two sets of kernels. We then independently extract features from each multi-scale grids based on the respective scale coordinates and average the feature sets to get the final features.
Figure 3: The overall feature extraction pipeline in the proposed mip-TensoRF.
3.5 Mip-Grid
Current grid-based NeRFs typically involves certain versions of tensor factorization schemes. For example, TensoRF used CP decomposition for compact models and proposed a new VM (vector-matrix) factorization method for better rendering performance. K-Planes used plane-based factorization, which has been very successful not only in 3D, but also in 2D and 4D scene generation. Thus, we have integrated the proposed method into two prominent grid-based NeRFs: TensoRF and K-Planes.
3.5.1 Mip-TensoRF
In TensoRF, VM factorization for a density grid  $\mathcal{G}$  is formulated as follows (we omitted the subscript  $\sigma$  for brevity):
$$
\mathcal {G} = \sum_ {r = 1} ^ {R} v _ {r} ^ {X} \circ M _ {r} ^ {Y Z} + v _ {r} ^ {Y} \circ M _ {r} ^ {X Z} + v _ {r} ^ {Z} \circ M _ {r} ^ {X Y}, \tag {3}
$$
where  $v_{r}^{X} \in \mathbb{R}^{H}$  denote the vector for  $X$  axis,  $M_{r}^{YZ} \in \mathbb{R}^{W \times L}$  is the matrix for  $Y$  and  $Z$  axes,  $R$  is the rank, and  $\circ$  is a tensor product, hence  $v_{r}^{X} \circ M_{r}^{YZ} \in \mathbb{R}^{H \times W \times L}$ . We generate the multi-scale vectors and matrices by applying 1D or 2D depth-wise convolutions over vectors and matrices. This is equivalent to applying 3D convolutions over 3D tensors thanks to the linearity of convolution operators. The  $i$ -th scale of a multi-scale grid  $\tilde{\mathcal{G}} \in \mathbb{R}^{H \times W \times L \times S}$  can be defined as follows:
$$
\tilde {\mathcal {G}} _ {i} = \sum_ {r = 1} ^ {R} \left(v _ {r} ^ {X} * k _ {i, r} ^ {v ^ {X}}\right) \circ \left(M _ {r} ^ {Y Z} * k _ {i, r} ^ {M ^ {Y Z}}\right) + \left(v _ {r} ^ {Y} * k _ {i, r} ^ {v ^ {Y}}\right) \circ \left(M _ {r} ^ {X Z} * k _ {i, r} ^ {M ^ {X Z}}\right) + \left(v _ {r} ^ {Z} * k _ {i, r} ^ {v ^ {Z}}\right) \circ \left(M _ {r} ^ {X Y} * k _ {i, r} ^ {M ^ {X Y}}\right), \tag {4}
$$
where  $*$  is the convolution operator,  $\tilde{\mathcal{G}}_i\in \mathbb{R}^{H\times W\times L}$  is  $i$ -th scale of the grid,  $k_{i}^{v^X}\in \mathbb{R}^{K\times R}$  is  $i$ -th depth-wise convolution filter for the vector  $v_{r}^{X}$ ,  $k_{i,r}^{v^{X}}\in \mathbb{R}^{K}$  is the filter for the rank  $r$ , and  $K$  is the kernel size. Similarly,  $k_{i}^{M^{YZ}}\in \mathbb{R}^{K\times K\times R}$  is  $i$ -th depth-wise convolution filter for the matrix  $M_r^{YZ}$  and  $k_{i,r}^{M^{YZ}}\in \mathbb{R}^{K\times K}$  is the filter for the rank  $r$ . Note that we have separate learnable kernels for each rank, axis, and scale, which are more expressible than fixed Gaussian filters.
3.5.2 Mip-K-Planes
Similar to mip-TensoRF, we apply depth-wise convolutions over the plane-based grid representations. In K-Planes, a grid  $\mathcal{G}^k\in \mathbb{R}^{D_x\times D_y\times D_z\times R}$  can be formulated as follows:
$$
\mathcal {G} ^ {k} = M ^ {Y Z} \odot M ^ {X Z} \odot M ^ {X Y}, \tag {5}
$$