SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the models that demonstrate stable performance across multiple datasets and can train within one hour when running on a conventional single GPU. Moreover, we evaluate the compression methods that require post-optimization steps (Re:NeRF [34] and VQRF [36]) and masked wavelet transform (DWT-NeRF [37]) to compress existing voxel-based models. Furthermore, we also compare with representative implicit radiance field models (NeRF [1] and mip-NeRF [24]).
Implementation details We implement our feature grid using hash encoding [20]. We use 16 levels of multi-resolution 3D feature grids with resolutions from 16 to 1024, while each grid includes up to  $T_{\mathrm{3D}}$  feature vectors. We also utilize four levels of multi-resolution 2D feature grids with resolutions from 64 to 512, while each grid includes up to  $T_{\mathrm{2D}}$  feature vectors. We use  $\{\log_2(T_{\mathrm{2D}}), \log_2(T_{\mathrm{3D}})\} = \{15, 17\}$  for our small model denoting Ours-S and  $\{\log_2(T_{\mathrm{2D}}), \log_2(T_{\mathrm{3D}})\} = \{17, 19\}$  for our base model denoting Ours-B. Our model variants are scaled by the dimension of feature vectors per level of 2, 4, and 8. For instance, Ours-B2 denotes a base variant with feature dimension  $F = 2$ . We exploit two types of MLPs with 128-channel hidden layers and rectified linear unit activation on them, one for density prediction with one hidden layer and another for color prediction with two hidden layers. The total storage capacity of MLPs varies from 0.06 to 0.11 MB, depending on the input feature dimension. The spherical harmonics basis function is used to encode the directional information. Our implementation is based on Instant-NGP [20] using the occupancy grid implemented in NerfAcc [42]. We optimize all our models for 20K iterations on a single GPU (NVIDIA RTX A6000). It takes approximately 6, 9, and 14 min of average time to train scenes of the Synthetic-NeRF dataset for Ours-B2 to Ours-B8 models, respectively. We use the Adam [43] optimizer with an initial learning rate of 0.01, which we decay at 15K and 18K iterations by a factor of 0.33. Furthermore, we adopt a warm-up stage during the first 1K iterations to ensure stable optimization.
5.2 Comparison
We measured the storage size of each method and evaluated the reconstruction quality (PSNR, SSIM) on various datasets. Note that we present the average scores of all scenes in each dataset; scene-wise full scores are reported in the appendix. Fig. 2 summarizes quantitative evaluations of our radiance field representation, compared to baseline data structure-based, compression-based methods, and implicit-based methods.
Data structure-based approaches Fig. 3 shows the qualitative evaluations of our model, compared to data structure-based methods. Our method successfully demonstrates superior reconstruction performance against SOTA data structure-based methods with comparable training time and remarkably small storage size; especially Ours-S2 requires only within  $0.5\mathrm{MB}$ . While baseline approaches
Ours-S2
Ours-B2
Ours-B4
Ours-B8
CCNeRF-CP
TensoRF-CP
Figure 3: Qualitative comparison of reconstruction quality using the mic scene of the NeRF-Synthetic dataset. For each subfigure, PSNR and storage size are shown on the right upper.
Instant-NGP
TensoRF-VM
require large storage space to achieve high performance, all our models are sufficient to exceed them by less than  $6\mathrm{MB}$ . In particular, Ours-B2 outperforms the reconstruction quality of almost baselines with a much smaller storage size of within  $1.5\mathrm{MB}$ . It takes  $6.1\mathrm{min}$  to accomplish this without any temporal burden to train our model in a compact manner. Although our small models are enough to accomplish outstanding performance, we adopt larger models to attain higher quality. As a result, both Ours-B4 and Ours-B8 jump to excessive reconstruction quality, outperforming state-of-the-art models. Accordingly, the results indicate that our model contains the most storage-efficient data structure for radiance field representation, which also does not require much computational cost.
Compression-based approaches We also compare our models with SOTA compression-based methods. Although these approaches make the existing data structure-based models (e.g., DVGO, Plenoxels, and TensoRF-VM) highly compact by compressing the optimized model, our binary feature encoding model outperforms highly compressed data structure-based models in terms of reconstruction performance and storage usage. Specifically, while compressed TensoRF-VM models (Re:TensoRF-High [34], VQ-TensoRF [36], and DWT-TensoRF [37]) preserve most high performance among these compression works, Ours-B4 model has superior reconstruction quality only with 2.8 MB of storage capacity, which is smaller than the compressed TensoRF-VM models. Consequently, the results verify that our binary radiance field models accomplish higher compactness than compression-based models.
Implicit-based approaches In addition, we evaluate implicit representations with a few number of parameters due to the efficient structure of MLPs. Although their simple structure results in a small storage size, our models achieve significantly higher performance with a similar storage size.
Benefits of binary radiance field Note that our approach differs from post quantization-based approaches [14, 33, 34] that demonstrate bounded performance from original features and necessitate additional pre-/post-processing. It is noteworthy that our method learns binary features during training without sacrificing the rendering quality. Our approach also differs from compression-based approaches [34, 36, 37] that require additional computations to compress or decompress the radiance field. Instead, our learned binary feature can be directly interpolated to render realistic scenes.
Feature grid design We analyze the impact of the feature grid designs for scene representation and verify that our 2D-3D hybrid feature grid effectively improves the reconstruction performance. We compare three architectures with similar sizes: (a) tri-plane representation, (b) voxel representation, and (c) our 2D-3D hybrid representation. Table 1 demonstrates the effectiveness of our 2D-3D hybrid representation. Compared to the tri-plane and voxel grid, our hybrid feature grid enhances the radiance field reconstruction quality with a similar number of parameters. As a result, this confirms that our architectural choice for the feature grid allows us to achieve more compact feature encoding.
Table 1: Ablation study on the feature grid design. Results are averaged over all scenes of the Synthetic-NeRF dataset. We highlight the best scores in bold.
Design	# Params	PSNR↑	SSIM↑
Tri-plane (only 2D)	13.7 M	31.44	0.949
Voxel (only 3D)	13.6 M	32.35	0.956
Hybrid (2D + 3D)	13.2 M	32.64	0.958
Table 2: Ablation study on the use of sparsity loss. We report train time and inference speed with reconstruction quality. Results are averaged over all scenes of the Synthetic-NeRF dataset.
Train↓	Inference↑	PSNR↑	SSIM↑
Ours-B2 (w/o Lsparsity)	6.4 m	3.3 fps	32.69	0.958
Ours-B2 (w/ Lsparsity)	6.1 m	3.8 fps	32.64	0.958
Ours-B8 (w/o Lsparsity)	14.7 m	2.2 fps	33.60	0.964
Ours-B8 (w/ Lsparsity)	13.9 m	2.7 fps	33.59	0.964
(a) w/o sparsity loss
(b) w/ sparsity loss
Figure 4: Visualization of a 2D slice of occupancy grid for chair scene of the Synthetic-NeRF dataset according to the use of sparsity loss  $\mathcal{L}_{\mathrm{sparsity}}$
Figure 5: Ablation study on the hash table size. Results are averaged over all scenes of the Synthetic-NeRF dataset.
Sparsity loss We investigate the effectiveness of sparsity loss that accelerates the rendering speed by regularizing the radiance fields more sparsely. We evaluate the train time and inference speed of our radiance field model, according to the use of sparsity loss. As shown in Table 2, we can improve the rendering time in both training and inference with a minor decrease in reconstruction quality. In particular, we can accelerate the  $23\%$  of rendering speed for Ours-B8 model by adopting sparsity loss. The visualization of the occupancy grid, including the bitmap for the empty or non-empty area, also demonstrates that our model is trained sparsely due to sparsity loss, as shown in Fig. 4.
Hash table size We use hash encoding to construct our 2D/3D feature grids, which are restricted in their scale by the hash table size and the number of feature vectors per level. In other words, we can further reduce the storage size for compactness or increase the storage size to improve performance by scaling the number of feature vectors in the hash table. As shown in Fig. 5, we evaluate the storage size and the reconstruction performance of different hash table sizes  $\{\log_2(T_{\mathrm{2D}}),\log_2(T_{\mathrm{3D}})\}$ , where  $T_{\mathrm{2D}}$  and  $T_{\mathrm{3D}}$  denote the hash table size of the 2D and 3D grid, respectively.
Computational cost for binarization We evaluate the training time and memory requirement of a real-valued feature grid and a binary feature grid. The binarization procedure in our model adds only a small portion of training time compared to the real-valued feature grid, as shown in Table 3. Also, there is no noticeable increase in memory usage due to binarization, as shown in Table 4. This is because we binarize only several grid parameters corresponding to a ray, not a whole grid. Furthermore, there are some cases where the binary grid converges faster or requires less memory than the real-valued grid since the sparsity of the optimized scene affects the computational cost.
Table 3: Evaluation on the training time for 20K iterations of a real-valued feature grid and a binary feature grid.
Method	Synthetic-NeRF	Synthetic-NSVF	Tanks and Temples
Ours-B2	Ours-B4	Ours-B8	Ours-B2	Ours-B4	Ours-B8	Ours-B2	Ours-B4	Ours-B8
Real-valued Binary	6.02 min	8.42 min	13.31 min	6.23 min	8.77 min	14.08 min	5.93 min	8.41 min	13.46 min
6.10 min	8.66 min	13.86 min	6.22 min	8.93 min	14.53 min	6.00 min	8.59 min	14.04 min
Table 4: Evaluation on the memory requirement of a real-valued feature grid and a binary feature grid.
Method	Synthetic-NeRF	Synthetic-NSVF	Tanks and Temples
Ours-B2	Ours-B4	Ours-B8	Ours-B2	Ours-B4	Ours-B8	Ours-B2	Ours-B4	Ours-B8
Real-valued Binary	5.46 GB	6.70 GB	10.04 GB	5.47 GB	6.02 GB	10.04 GB	6.52 GB	7.09 GB	11.10 GB
5.45 GB	6.58 GB	10.06 GB	5.47 GB	6.30 GB	10.02 GB	6.51 GB	7.92 GB	11.10 GB
Table 5: Quantitative evaluations of dynamic scene reconstruction using binary feature encoding. Results are averaged over all scenes of D-NeRF dataset [6] and HyperNeRF dataset [8].  $\dagger$  denotes that binary feature encoding is applied.
Method	D-NeRF [6] (synthetic)
Size (MB)↓	PSNR↑	SSIM↑
TiNeuVox-S	7.63	31.03	0.957
TiNeuVox-S†	0.59	28.68	0.938
TiNeuVox-B	47.51	33.02	0.972
TiNeuVox-B†	3.88	31.23	0.961
Method	HyperNeRF [8] (real)
Size (MB)↓	PSNR↑	MS-SSIM↑
TiNeuVox-B	47.85	24.20	0.835
TiNeuVox-B†	3.90	23.87	0.826
Figure 6: Qualitative evaluations of dynamic scene reconstruction on the use of binary feature encoding. Original (up) and binary feature encoding (bottom). PSNR and storage size are shown on the right upper.
5.4 Applications
Dynamic scene reconstruction We finally adopted our binarization strategy in dynamic scene reconstruction. We employ TiNeuVox [22], as our base model, one of the most efficient approaches, which utilizes time-aware voxel features. Instead of conventional feature encoding for the voxel grid, we apply the binary feature encoding on the time-aware voxel features and this leads to a highly compact reconstruction of dynamic scenes, even in  $0.6\mathrm{MB}$  of storage space for the synthetic jumpingjacks scene. Table 5 and Fig. 6 demonstrate that our approach is easily applicable to various feature encoding tasks, and it enables remarkably efficient representation in terms of storage size.
In this work, we have introduced binary radiance fields, a storage-efficient radiance field representation that significantly reduces the storage capacity by adopting binary feature encoding. Our experiments have verified that our approach is the most storage-efficient representation outperforming recent data structure-based models, compression-based models, and implicit-based models in terms of reconstruction quality and storage consumption. This capability enables us to access numerous radiance fields without a storage burden and leads to the expansion of the applications, such as dynamic scenes.
Limitations Despite the binarization of learnable parameters, any bit-wise operations in multilayered perception would lead to more efficient computation, requiring fewer GPU resources. We expect our approach can be boosted for the rendering by adopting the baking process.
7 Acknowledgments and Disclosure of Funding
The authors acknowledge the support provided by Samsung Advanced Institute of Technology (SAIT). This work was supported by IITP grants (RS-2023-00227993: Detailed 3D reconstruction for urban areas from unstructured images, No.2022-0-00290: Visual Intelligence for Space-Time Understanding and Generation based on Multi-layered Visual Common Sense, No.2019-0-01906: AI Graduate School Program at POSTECH, No.2021-0-01343: AI Graduate School Program at Seoul National University) funded by Korea government (MSIT).
[37] Daniel Rho, Byeonghyeon Lee, Seungtae Nam, Joo Chan Lee, Jong Hwan Ko, and Eunbyung Park. Masked wavelet representation for compact neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20680-20690, 2023.
[38] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.
[39] Subin Kim, Sihyun Yu, Jaeho Lee, and Jinwoo Shin. Scalable neural video representations with learnable positional features. arXiv preprint arXiv:2210.06823, 2022.
[40] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875-5884, 2021.
[41] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017.
[42] Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerfacc: A general nef acceleration toolbox. arXiv preprint arXiv:2210.04847, 2022.
[43] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.