SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $M^{YZ} \in \mathbb{R}^{1 \times D_y \times D_z \times R}$ ,  $M^{XZ} \in \mathbb{R}^{D_x \times 1 \times D_z \times R}$ , and  $M^{XY} \in \mathbb{R}^{D_x \times D_y \times 1 \times R}$  denote three grids for 3D scene representations.  $\odot$  denotes broadcast element-wise multiplication. For example,  $M^{YZ} \odot M^{XZ}$  will have the shape of  $\mathbb{R}^{D_x \times D_y \times D_z \times R}$ .  $D_x$ ,  $D_y$ , and  $D_z$  denote the grid resolutions in X, Y, and Z axes. Similar to Eq. 4, a multi-scale grid  $\tilde{\mathcal{G}}_i^k \in \mathbb{R}^{D_x \times D_y \times D_z \times R}$  for  $i$ -th scale can be written as follows:
$$
\tilde {\mathcal {G}} _ {i} ^ {k} = \left(M ^ {Y Z} * k _ {i} ^ {Y Z}\right) \odot \left(M ^ {X Z} * k _ {i} ^ {X Z}\right) \odot \left(M ^ {X Y} * k _ {i} ^ {X Y}\right), \tag {6}
$$
where  $k_{i}^{YZ}\in \mathbb{R}^{1\times K\times K\times R}$ ,  $k_{i}^{XZ}\in \mathbb{R}^{K\times 1\times K\times R}$ , and  $k_{i}^{XY}\in \mathbb{R}^{K\times K\times 1\times R}$  denote  $i$ -th depth-wise convolution filters.
Figure 4: A qualitative comparison of mip-TensoRF against the original TensoRF and the single-scale TensoRF. The zoom-in parts of rendering samples in four different scales are shown. As resolution decreases, two baseline models exhibit aliasing artifacts. Best viewed in color and zoom-in.
Implementation details We implemented two variants of mip-Grid: mip-TensoRF and mip-K-Planes. Both models apply convolution to a shared grid representation using four learnable kernels of size 3, where each kernel is responsible for generating different scales of grids. We trained mip-TensoRF for 40k iterations on the multi-scale Blender dataset and 25k iterations on the multi-scale LLFF dataset. All of its three baseline models were also trained for the same number of iterations. We scaled the loss of each pixel following mip-NeRF [1] to account for the scale differences. Note that we began kernel training after the grid upsampling [7] was completed. Unlike mip-TensoRF, we trained mip-K-Planes for 30k iterations and optimized the kernels from the beginning. All other training hyperparameters remained the same as the original TensoRF-VM-192 and K-Planes.
As outlined in the Sec. 3, mip-Grid introduces three kinds of scale coordinates to extract features from the generated multi-scale grids. We used the discrete scale coordinate to train mip-TensoRF and mip-K-Planes, and for mip-TensoRF, we conducted a further evaluation using the continuous scale coordinate and the 2D scale coordinate, examining the effectiveness of using more sophisticated scale coordinates. In all tables showing the results of mip-Grid integrated models, abbreviations of the scale coordinate types are provided in parentheses along with the name of each model.
Baseline models We compare our method against three baseline models: the original model, the single-scale model, and the multi-scale model. The original models are the same as the TensoRF-VM-192 and K-Planes, but we test them on the multi-scale dataset. The single-scale model is trained on the multi-scale dataset with loss multipliers [1] and tested on the multi-scale dataset. The multi-scale model is a set of four independent original models, each of which is trained and tested on different scales of dataset. We include the latter two baseline models to show that naively training the single-scale representations with multi-scale dataset cannot remove aliasing artifacts effectively, and using four independent models requires four times more parameters than the original model.
Table 1: A model evaluation on the test set of multi-scale Blender dataset. The elapsed training time, evaluated using a single RTX 4090 GPU, are shown at the rightmost column. For mip-NeRF, the time elapsed for 100 iterations were measured and multiplied by 10000 to estimate the total runtime for 1 million iterations. Note that the multi-scale model has four times more parameters than the original model.
PSNR↑	SSIM↑	LPIPS↓	Time (hours)
Avg.	Full Res.	1/2 Res.	1/4 Res.	1/8 Res.	Full Res.	1/2 Res.	1/4 Res.	1/8 Res.	Full Res.	1/2 Res.	1/4 Res.
Mip-NeRF [1]	34.51	32.63	34.34	35.47	35.06	0.9579	0.9703	0.9786	0.9833	0.0469	0.0260	0.0168	>30
TensoRF (original)	30.70	33.13	33.27	30.02	26.40	0.9630	0.9704	0.9624	0.9358	0.0474	0.0337	0.0488	0.0802
TensoRF (single-scale)	30.61	28.98	32.42	32.53	28.52	0.9295	0.9618	0.9703	0.9514	0.1054	0.0608	0.0474	0.0697
TensoRF (multi-scale)	34.77	33.33	35.74	35.44	34.57	0.9637	0.9757	0.9766	0.9775	0.0456	0.0301	0.0279	0.0238
Mip-TensoRF	35.23	32.53	35.40	36.58	36.41	0.9587	0.9757	0.9828	0.9857	0.0533	0.0278	0.0200	0.0163
K-Planes (original)	29.76	32.40	31.83	28.95	25.85	0.9615	0.9663	0.9570	0.9321	0.0491	0.0369	0.0539	0.0861
K-Planes (single-scale)	30.07	32.13	32.35	29.54	26.24	0.9590	0.9667	0.9597	0.9356	0.0525	0.0361	0.0497	0.0813
K-Planes (multi-scale)	32.24	32.59	33.60	32.34	30.58	0.9605	0.9696	0.9646	0.9580	0.0508	0.0314	0.0396	0.0415
Mip-K-Planes	32.27	33.20	33.03	32.73	31.12	0.9601	0.9691	0.9724	0.9689	0.0512	0.0329	0.0291	0.0302
Table 2: A model evaluation on the test set of multi-scale LLFF dataset. Note that the multi-scale model has four times more parameters than the original model.
PSNR↑	SSIM↑	LPIPS↓
Avg.	Full Res.	1/2 Res.	1/4 Res.	1/8 Res.	Full Res.	1/2 Res.	1/4 Res.	1/8 Res.	Full Res.	1/2 Res.	1/4 Res.	1/8 Res.
TensoRF (original)	26.03	26.73	27.89	26.70	22.81	0.8386	0.8932	0.8964	0.8063	0.2044	0.1069	0.1099	0.1685
TensoRF (single-scale)	27.33	24.48	28.25	29.92	26.64	0.7457	0.8863	0.9375	0.8964	0.3114	0.1450	0.0851	0.1259
TensoRF (multi-scale)	30.18	26.75	29.58	31.33	33.06	0.8378	0.9067	0.9487	0.9731	0.1992	0.1065	0.0632	0.0339
Mip-TensoRF	30.48	26.51	29.69	32.18	33.52	0.8310	0.9165	0.9590	0.9762	0.2152	0.0946	0.0477	0.0316
Datasets All models were evaluated on the multi-scale Blender dataset [1] with three different metrics: PSNR, SSIM, and LPIPS (VGG) [46]. Mip-TensoRF and its baseline models were further evaluated on the multi-scale LLFF dataset. To convert single-scale dataset to multi-scale dataset, we bilinearly downsampled the images by a factor of 1, 2, 4, and 8, and rescaled the focal length accordingly as typically done in other multi-scale NeRF literature [1, 3].
Tab. 1 shows the evaluation results on the multi-scale Blender dataset. Mip-TensoRF and mip-K-Planes outperform their baseline models, achieving the highest average PSNR. Notably, mip-TensoRF even surpasses mip-NeRF while significantly reducing the training time. The performance of the original models experiences a substantial drop as the rendering resolution decreases, leading to noticeable "jaggies" in the images rendered at the two lowest resolutions (Fig. 4). The single-scale models achieve better performance at the two lowest resolutions compared to the original models but show worse results at other resolutions. This suggests that, to some extent, training with a multi-scale dataset and employing the loss multipliers helps the model in reasoning about different scales, but this approach does not entirely eliminate aliasing artifacts. On the other hand, the multi-scale models can effectively represent different scales of a scene, achieving the best PSNR among the three baseline models. Nevertheless, these models require four times more parameters than the original models, which is undesirable in practical applications. Unlike others, our models successfully mitigate aliasing artifacts while minimizing the additional number of parameters and computational cost.
We provide additional results of training mip-TensoRF with continuous and 2-dimensional scale coordinates in Tab. 3. The first two models show only a marginal difference when evaluated at the training resolutions. A similar trend is observed in evaluations at unseen resolutions, but there is a clear difference when the image resolution is reduced from  $4/8$  resolution to  $3/8$  resolution, suggesting that the continuous scale coordinate improves the model's generalization performance. Our method generates multiple sets of feature grids from a shared representation, and these feature sets share many similarities. This allows different features to be smoothly interpolated and rendered into visually pleasing images. However, more information needs to be removed as the output images decrease in size, leading to relatively large differences between low-scale features. This explains the marginal drop in PSNR at lower resolutions. Moreover, the model trained with 2D scale coordinate outperforms the others, indicating that the additional scale coordinate (the distance from the ray origin to the sampled point) can be beneficial for resolving the remaining scale ambiguity.
Table 3: An evaluation of mip-TensoRF trained with three different types of scale coordinates. All models were trained on the multi-scale Blender dataset (with four different resolutions; Full Res., 4/8 Res., 2/8 Res., and 1/8 Res.) and tested on the test set of Blender dataset rescaled to eight resolutions.
PSNR↑
Avg.	Full Res.	%8 Res.	%8 Res.	%8 Res.	%8 Res.	%8 Res.	%8 Res.	%8 Res.
Mip-TensoRF (disc.)	34.54	32.53	33.73	34.20	34.70	35.40	34.33	35.93	35.50
Mip-TensoRF (cont.)	34.73	32.46	33.82	34.29	34.74	35.34	35.64	35.89	35.66
Mip-TensoRF (2D)	34.88	32.56	33.94	34.43	34.90	35.53	35.77	36.09	35.80
Figure 5: Visualization of generated multi-scale grids and learned kernels of mip-TensoRF. A single channel in the  $XY$  plane of the appearance grids and the first four channels of the learned kernels are shown. The first column of the first row displays a shared grid representation, and the generated multi-scale grids are displayed from the second to the last column of the first row. Each column of the second row shows the learned kernels that is used to generate the corresponding feature grids.
4.3 Visualization of learned kernels and multi-scale grids
We visualize the learned kernels and the generated multi-scale grids of mip-TensoRF to illustrate how the learned patterns vary across scales. As depicted in Fig. 5, the high-scale kernels look similar to Gaussian distribution, and their shape (i.e., the standard deviation) becomes larger as the corresponding scale decreases. This observation aligns with our intuition that the kernels of low-scale features should be smoother than those of high-scale features and demonstrates that our method effectively learns optimal kernels for each scale. A similar phenomenon can be observed in the visualization of feature grids, where high-frequency details are progressively removed as the feature scale decreases. Although we do not provide the visualization of density kernels and grids due to the limited space, the learned patterns closely resemble those of the appearance kernels and grids.
Fixed Gaussian kernels and learnable kernels We compare mip-TensoRF with fixed Gaussian kernels against the same model with learnable kernels. Both models were initialized with 2 sets of standard deviations  $\{1.0, 1.0, 1.0, 1.0\}$  and  $\{1.0, 1.5, 2.5, 4.0\}$ , where the leftmost value was used to set the kernel of the highest scale, and the rightmost value was used for the lowest scale. The second set is to give an inductive bias that the low-scale features should be smoothed more than high-scale features. The fixed Gaussian kernel initialized with the second set of standard deviations significantly outperforms the other (Tab. 4). This suggests that increasing standard deviations can be a beneficial inductive bias. However, these fixed Gaussian kernels may not be optimal for each scale. Instead of using heuristically selected kernels, our method fine-tunes the Gaussian kernels and finds the optimal weights during the training. Experimental results demonstrate that the learnable approach outperforms the fixed Gaussian approach, further confirming the effectiveness of our method (Tab. 4).
The number of multi-scale grids and kernel size The proposed method involves two key hyperparameters: the number of multi-scale grids and the kernel size. We provide an evaluation of mip-TensoRF with nine different combinations of these two values to give readers guidance on hyperparameter configuration. As shown in Tab. 5, the number of multi-scale grids has little effect
Table 4: An evaluation of mip-TensoRF using fixed Gaussian kernels and learnable kernels, initialized with different standard deviations. Trained and tested on the multi-scale Blender dataset.
Stdev.	PSNR↑	SSIM↑	LPIPS↓
Avg.	Full Res.	1/2 Res.	1/4 Res.	1/8 Res.	Full Res.	1/2 Res.	1/4 Res.	1/8 Res.	Full Res.	1/2 Res.	1/4 Res.	1/8 Res.
Fixed	1.0; 1.0; 1.0; 1.0	30.99	28.59	32.46	33.75	29.15	0.9267	0.9627	0.9763	0.9574	0.0991	0.0494	0.0332	0.0593
1.0; 1.5; 2.5; 4.0	34.05	30.84	34.10	35.37	35.87	0.9478	0.9698	0.9787	0.9829	0.0697	0.0355	0.0255	0.0211
Learnable	1.0; 1.0; 1.0; 1.0	35.31	32.57	35.43	36.64	36.59	0.9588	0.9759	0.9829	0.9863	0.0535	0.0275	0.0196	0.0150
1.0; 1.5; 2.5; 4.0	35.40	32.62	35.47	36.70	36.81	0.9593	0.9762	0.9832	0.9868	0.0529	0.0272	0.0192	0.0141
Table 5: An evaluation of mip-TensoRF varying two hyperparameters: N - the number of multi-scale grids, K - the kernel size. Tested on the hotdog scene of the multi-scale Blender dataset.
N	K	PSNR↑	SSIM↑	LPIPS↓
Avg.	Full Res.	1/2 Res.	4/4 Res.	1/8 Res.	Full Res.	1/2 Res.	4/4 Res.	1/8 Res.	Full Res.	1/2 Res.	4/4 Res.	1/8 Res.
2	3	39.43	36.84	39.72	40.66	40.49	0.9792	0.9887	0.9917	0.9931	0.0379	0.0164	0.0104	0.0078
5	39.58	36.84	39.78	40.86	40.85	0.9790	0.9888	0.9918	0.9934	0.0382	0.0163	0.0099	0.0069
11	39.63	36.83	39.81	40.89	41.01	0.9789	0.9890	0.9920	0.9937	0.0381	0.0160	0.0096	0.0066
3	3	39.58	37.01	39.84	40.89	40.57	0.9795	0.9888	0.9918	0.9932	0.0375	0.0164	0.0102	0.0080
5	39.66	36.98	39.90	40.98	40.78	0.9796	0.9890	0.9920	0.9935	0.0367	0.0155	0.0095	0.0069
11	39.65	36.92	39.83	40.95	40.90	0.9793	0.9890	0.9920	0.9937	0.0371	0.0157	0.0093	0.0067
4	3	39.60	37.03	39.90	40.92	40.57	0.9797	0.9888	0.9919	0.9932	0.0367	0.0161	0.0099	0.0079
5	39.74	37.06	39.97	41.04	40.89	0.9798	0.9890	0.9920	0.9936	0.0365	0.0158	0.0095	0.0069
11	39.78	37.02	39.97	41.07	41.06	0.9796	0.9890	0.9921	0.9938	0.0363	0.0158	0.0093	0.0068
on the rendering quality, but variations in kernel sizes result in relatively bigger differences in the PSNR of low-scale images. Although different scales of grid representations share many similarities, the low-scale grid should be smoothed enough to produce an image without aliasing artifacts, and the kernel size is a critical factor controlling the smoothness of the grids. Using too small kernels may introduce unnecessary high-frequency components in the low-scale grid. Nevertheless, there is always a trade-off between rendering quality and speed, and users should consider their specific usage purpose when determining the kernel size.
5 Limitations and discussion
The proposed method has not been tested on scenes involving more complex camera poses, such as 360-degree scenes [2, 45, 36] or free trajectories [39]. However, addressing these scenarios would require further technical development and a thorough parameter tuning. We plan to investigate how the proposed method, or even the grid representation itself, can be extended to these scenarios.
Although the proposed method is significantly faster than mip-NeRF, it introduces additional complexity to the system. Especially in terms of training time, our method needs more computational resources to generate multi-scale grids and retrieve features. We implemented our model based on TensoRF and K-Planes with minimal modifications, which makes the training time heavily rely on the base models. Also, the implementation of the multi-scale grid generation and the feature retrieval process has not been optimized yet. We strongly believe the training time and memory requirements can be substantially improved by fusing the multi-scale grid generation and the feature retrieval process as a single CUDA kernel or using the NeRF accelerating tool [13].
We proposed mip-Grid, anti-aliased grid representations for NeRF. The proposed approach can easily be integrated into the existing grid-based NeRFs, and the two methods using our approach, mip-TensoRF and mip-K-Planes, have demonstrated that aliasing artifacts can be effectively removed. Since we generate multi-scale grids from a shared grid representation and do not rely on supersampling, the proposed method minimizes additional number of parameters and trains significantly faster than the existing MLP-based anti-aliased NeRFs. We believe our work paves the way for a new research direction toward alias-free NeRF leveraging the training efficiency of grid representations.
This work was supported by the Institute of Information and Communication Technology Planning Evaluation (IITP) grants (IITP-2019-0-00421, IITP-2021-0-02068) and the National Research Foundation (NRF) grants (2022R1F1A1064184, RS-2023-00245342) funded by the Ministry of Science and ICT (MSIT) of Korea.
[37] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[38] Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[39] Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[40] Turner Whitted. An improved illumination model for shaded display. In ACM SIGGRAPH, 2005.
[41] Lance Williams. Pyramidal parametrics. In Proceedings of the 10th annual conference on Computer graphics and interactive techniques, 1983.
[42] Lifan Wu, Shuang Zhao, Ling-Qi Yan, and Ravi Ramamoorthi. Accurate appearance preserving prefiltering for rendering displacement-mapped surfaces. ACM Transactions on Graphics (TOG), 2019.
[43] Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[44] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[45] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[46] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[47] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.