Binary Radiance Fields
Authors: Seungjoo Shin
Jaesik Park
Venue: NeurIPS 2023 poster (2023)
Keywords: neural radiance fields
inverse rendering
binarization
TL;DR: A storage-efficient radiance field representation with binary feature encoding.

Abstract: In this paper, we propose \textit{binary radiance fields} (BiRF), a storage-efficient radiance field representation employing binary feature encoding in a format of either $+1$ or $-1$. This binarization strategy lets us represent the feature grid with highly compact feature encoding and a dramatic reduction in storage size. Furthermore, our 2D-3D hybrid feature grid design enhances the compactness of feature encoding as the 3D grid includes main components while 2D grids capture details. In our experiments, binary radiance field representation successfully outperforms the reconstruction performance of state-of-the-art (SOTA) storage-efficient radiance field models with lower storage allocation. In particular, our model achieves impressive results in static scene reconstruction, with a PSNR of 32.03 dB for Synthetic-NeRF scenes, 34.48 dB for Synthetic-NSVF scenes, 28.20 dB for Tanks and Temples scenes while only utilizing 0.5 MB of storage space, respectively. We hope the proposed binary radiance field representation will make radiance fields more accessible without a storage bottleneck.

Binary Radiance Fields
Seungjoo Shin GSAI, POSTECH seungjoo. hwwggioo...ii@ppstech........
Jaesik Park
CSE IPAL, Scoul National University
jaesik. parkésnu. ac kr
https: https://seumgoooshhin www.ooshw..nnn..n...o...ooo wwt.gg..tttt..i....... 1o/BiRF
Abstract
In this paper, we propose binary radiance fields (BiRF), storage-efficiett radiance field representation employing binary feature encoding in a format of either +1 or -1. This binarization strategy lets us represent the feature grid with highly compact feature encoding and a dramatic reduction in storage size. Furthermore, our 2D-3D hybrid feature grid design enhances the compactness of feature encoding as the 3D grid includes main components while 2D grids capture details. In our experiments, binary radiance field representation successfully outperforms the reconstruction performance of state-of-the-art (SOTA) storage-efficiest radiance field models with lower storage allocation. In particular, our model achieves impressive results static scene reconstruction, with PSNR of 32.03 dB for Synthetic-No NeRF scenes, 34.48 dB for Synthetic-NSVF scenes, 28.20 dB for Tanks and Temples scenes while only utilizing 0.5 MB of storage space, respectively. We hope the proposed binary radiance field representation will make radiance fields more accessible without storage bottleneck.
Introduction
In recent years, the emergence of Neural Radiance Fields (NeRF) [1] has greatly impacted 3D) scene modeling and novel-view synthesis. The methodology models complex volumetric scene as an implicit function that maps positional and directional information of sampled points to the corresponding color and density values, enabling the rendering of photo-realistic novel views from any desired viewpoints. Subsequent advancements [2, 3, 4, 5, 6, 7, 8, 9, 10] have demonstrated their ability reconstruct various 3D scenes using images and corresponding camera poses, which opens radiance fields as a promising approach for representing the real 3D world.
Despite the significant progress, the computational burden of utilizing large-scale multi-layer percep- trons (MI.ss) remains a critical challenge, leading to speed issue in both training and rendering radiance fields. To tackle this issue, auxiliary explicit voxel grid has been utilized for encoding local features, denoting a voxel-based method. While the implicit radiance field representations should update all shared learnable parameters in MLPs, parametric encoding calculates only a small local part of encoded features leading to less computational cost. The voxel-based feature encoding has been implemented various data structures, such as dense grids [11, 12], octrees [13, 14], sparse voxel grids [15], decomposed grids [16, 17, 18, 19], and hash tables [20]. These representations succeed in efficiently reducing the time required for convergence and inference. Nonetheless, explicit feature encoding methods have significant disadvantage: their excessive storage usage. Now, we are facing a new bottleneck that restricts accessibility. Consequently, the desire for new radiance field representation to implement realistic 3D scene with little storage has been raised.
This paper introduces new binary feature encoding to represent storage-efficient radiance fields with binary feature grids, referred to as binary radiance fields (BiRF). Here, we focus on embedding sufficient feature information restricted in binary format. We achieve this by adopting a binarization-
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

aware training scheme, binary feature encoding, that constraints feature encoding parameters to either +1 or -1 and update them during optimization, inspired by Binarized Neural Networks (BNNs) [21]. Accordingly, our radiance field representation can successfully reconstruct complicated 3D scenes with binary encoding parameters that can represented using compact data resulting in a storage- efficient radiance field model. Furthermore, we extend the modern multi- musti-resolution hash encoding architecture to a 3D voxel grid and three orthogonal 2D plane grids. This hybrid structure allows for more efficient feature capture in more compact manner.
As a result, our radiance field model achieves superior reconstruction performance compared to prior efficient and lightweight methods. To be specific, our model attains outstanding performance in static scene reconstruction, with PSNR of 32.03 dB for Synthetic-Neee scenes, 34.48 dB for Synthetic- NSV scenes, 28.20 dB for Tanks and Temples scenes while only utilizing 0.5 MB of storage space, respectively.
We summarize our contributions as follows:
We propose binary radiance fields (BiRF), a binary radiance field representation that concisely encodes the binary feature values of either +1 or -1 in 2D-3D hybrid feature grid to represent storage-effiiiet radiance fields.
Our binarization-ave training scheme, binary feature encoding, allows effectively encode the feature information with binary parameters and update these parameters during optimization.
We demonstrate that our representation achieves superior performance despite requiring only minimal storage space, even in 0.5 MB for Synthetic-NeRF scenes.
2 Related Work
Neural Radiance Fields Neural radiance fields (NeRF) [1] is a leading method for novel-view synthesis by reconstructing high-quality 3D scenes. To achieve scene representation, it optimizes coordinate-based multi-layer perceptrons (MLPs) to estimate the color and density values of the 3D scene via differentiable volume rendering.
Improving radiance field representations begins with embracing diverse scenarios where the scene is intricate. The sampling strategy used in the original NeRF assumes that the entire scene can fit within a bounded volume, which limits its ability to capture background elements in an unbounded scene. To address this issue, several works [2, 3] have separately modeled foreground and background by re- re-parameterizing the 3D space. These parameterizations have been primarily applied in unbounded 360" view captured scenes. Additionally, due insufficient capacity, NeRF's lighting components have limitations in dealing with glossy surfaces. To address this challenge, transmitted and reflected radiance are optimized separately [4, 5]. Furthermore, there are approaches extend to dynamic domain with object movements [6, 7, 8, 22, 23].
Despite performing impressive results, it has limitations, including slow training and rendering speed. relies solely on utilizing implicit functions for 3D scene representation, which may lead to computational inefficiencies [20].
Radiance Fields Representations NeRF methods can be categorized into three types: implicit [1, 3, 24, 25, 26, 27], explicit [14, 15, 18, 28], and hybrid representations [11, 12, 13, 17, 20, 23, 29, 30, 31], depending on how the approach represent the scenes.
Implicit representations extensively use neural networks to represent radiance fields, as done in the pioneering method [1]. The network has a simple structure and can render photo-realistic images with few parameters. However, they take a lot of time to converge and require significant inference time because they share the entire weight and bias parameters of the MLPs for an arbitrary input coordinate, resulting in significant computational cost. Therefore, recent studies have proposed explicit and hybrid radiance field representations to overcome the slow speed by incorporating explicit data structures (such as 2D/3D grids 3D points) for local feature encoding.
Explicit representations directly encode view-dependent color and opacity values with basis functions (e.g., spherical harmonics). For instance, PlenOctrees [14] bakes implicit radiance fields into an octree structure for rendering speed acceleration. Plenoxels [15] uses a sparse voxel structure, and the approach by Zhang et al. [28] utilize a point cloud. Similarly, CCNeRF [18] employs low-rank tensor

---

grids for 3D scene manipulation. On the other hand, hybrid representations utilize encoded local features as imput for the MLPs. NSVF [13] achieves fast rendering speed thanks to the octree structure. To store local features, Point-NeRF [29] uses point cloud, DVGO [11, 12] employs two dense voxel grids, and TensoRF [17] makes use of a factorized tensor grid. More recently, Instant-NGP [20] introduces a multi-resolution hash encoding technique that has demonstrated exceptional effectiveness in terms of convergence and rendering speed, achieving superior performance. Additionally, Zhan et al. [32] introduce a learning method of the gauge transformation in radiance fields.
While explicit and hybrid representations boost training time and rendering speed, they inevitably suffer from the critical disadvantage of large storage consumption due to excessive local features. In this study, we construct a multi- reloluiosoo 2D-3D hybrid grid by combining 2D planes and 3D grids and binarizing their encoding parameters to fully leverage their minimal information.
Radiance Fields Compression Despite the acceleration of training and rendering, the explicit and the hybrid radiance fields have difficulties utilized in various applications due to their large storage. Consequently, there are several attempts to reduce the storage of the models.
For instance, PlenOctrees [14] and Plenoxels [15] filter the voxels using weight-based thresholding for leaving only the set of sparse voxels, which sufficient represent the scene. The distortion loss in DVGO-v2 [12] enables it to achieve better quality and greater resolution compactness. Other approaches apply bit quantization after the training. PlenOctrees [14]. PeRFception [33], and Re:NeRF [34] all apply low-bit quantization of trained local features. VQAD [35] compresses the feature grid parameters into a small codebook with learned indices. Re:NeRF [34] and VQRF [36] have proposed methods for compressing existing explicit or hybrid radiance field models, including post-optimization processes. Also, Rho et al. [37] achieve high storage efficiency by applying wavelet transform on hybrid radiance field models with learnable masks.
Although the post-processing approaches above successfully reduce the storage requirements of the radiance field models, they have several disadvantages. Firstly, they require additional optimization steps for compression, which can be time-consuming. Also, their performance is bounded by the performance of the pre-trained models. In contrast, our approach performs binarization during training with the efficient 2D/3D feature grid representation. Our approach does not require any post-optimization processes and shows better rendering quality with even smaller storage space.
3 Preliminaries
The methodology of NeRF [1] optimizes a 5D function, as a radiance field representation, to model a continuous volumetric scene with a view-dependent effect. The implicit 5D function, which consists of MLPs, maps a 3D coordinate x E (ar, y; =) and a 2D viewing direction = (0, Φ) to an emitted color (r, g, b) and volume density a:
(c, MLPe(x, d).
Due to the use of an implicit function, updating all the weight and bias parameters of MLPs is necessary to train single point. Consequently, this leads to slow convergence, requiring more than day to optimize a scene.
For volume rendering, the colors }c;} and densities { of sampled points along a ray r(t) = o + td are accumulated obtain the color of the ray:
im1
i-1
Ö(r)= = Σ Tiaci T, = [I a5) a; = 111-exxp-o00o)
j=1
where Tp and a, represent accumulated transmittance and alpha of i-th sampled point, respectively. ò, 1 t; denotes the distance between adjacent points. Recently, an occupancy grid [20] is adopted to skip non-empty space for efficient ray sampling, leading to an advance in rendering speed.
To accelerate the rendering process, the hybrid representations [11, 17, 20] have been developed to encode local features in explicit data structures & (e.g., 2D/3D grid). These features are then linearly interpolated and used as inputs for small MLPs to predict colors and densities:
f = interp(x, 26). (c, MLPe d),
(3)
3

---

0.54
+0.49
Real-valued parameters
sign
interp
Binary parameters
Input coordiante
Real-valued grid
Binary grid
Interpolation
(b)
level
Density
Color
(x,),2)
6xyz
Multi-resolution 3D Feature Grid
levol ns
Exy fxz
ML.Ss
Volumetric scene
Fyz
y(x)
Positional
Directional
Multi-resolution Feature Grids
encoding
information
Figure I: Illustration of overall framework: (a) binary feature encoding, and (b) binary radiance field representation. (a) Our binary feature encoding begins with applying binarization operation to the real-valued grid. Next, linearly interpolate the binary parameters to obtain feature values. (b) Our radiance field representation comprises 3D and three 2D feature grids. Given a 3D coordinate, the corresponding feature values are computed from each grid using binary feature encoding. The concatenated feature values are then fed as input to shallow density MI.P with positional encoded coordinates. Then we can obtain the density value and embedding features that are further provided as input to shallow color MLPs to acquire color value.
where interp means linear interpolation operator, and f denotes interpolated feature. The choice of explicit data structure & significantly affects the number of learnable parameters and hence decides the total storage size radiance field representation. Therefore, we need to consider an efficient data structure.
4 Method
this section, we introduce binary radiance fields that require only a small storage space by adopting binary feature grids. Fig. shows the overall scheme of our radiance field reconstruction. We first introduce how to encode the binary parameters present 2D-3D hybrid feature grid, which enhances the feature encoding by leveraging the strengths of both 2D plane and 3D voxel.
feature grid during optimization. Furthermore,
4.1 Binarization
Binarization of learnable parameter The binarization procedure of real-valued variables is achieved using a deterministic binarization operator [21], the sign function, which transforms real-valued variable into either +1 or -1:
+1 if 2 0,
0' sign(() = -1 otherwise,
where / denotes the real-valued variable, and 0' represents binary variable.
Since the derivative of the sign function zero almost everywhere, we cannot use traditional backpropagation to compute gradients for real-valued parameters. Instead, the straight-

---

through estimator (STE) [38], which is simple but effective technique for backpropagating through threshold functions (e.g.. the sign function):
ac
ac
30 10111
(5)
00
where is the loss function. This strategy allows us to maintain the gradient flow and ensure the differentiability of our framework. During training, we maintain and update real-valued parameters rather than directly learning binary parameters. We stop propagating gradients when a large value. This also constrains the value to the range {-1, 1} preventing the divergence of the scale.
Binary feature encoding We propose binary feature encoding, feature encoding scheme with binary parameters. Instead of real-valued parameters, binary parameters are used for encoding local features in a specific data structure, which are then employed to represent the radiance fields. While real-valued parameters require expensive floating-point pringg data for representation, binary parameters can be expressed in a single bit (1-bit), leading to a significant reduction in storage overhead. Here, we describe how to implement this strategy in the feature grid Dg. We first adopt binarization operation, described in Eq. 4, to the real-valued grid parameter 8:
8' = sign(O)
2o sign(@o))
(6)
where B' denotes the binary grid parameter of binary feature grid Pgr. Next, we linearly interpolate these binary parameters depending the given coordinate
f interp(x, ) = interp(x, (igg((D)))
(7)
where x is an imput coordinate, and denotes the encoded feature value. We utilize the STE [38] to propagate the gradients into the feature grid and update the grid parameters, as described in Eq. 5.
Now, our system is capable of training the feature encoding using binary parameters. This enables us to represent the feature grid with compact data, instead using expensive floating-poir data (16-bit or 32-bit). As result, there is tremendous reduction in the total storage size, making our radiance field representation storage-efficient.
4.2 Radiance Field Representation
For effective binary feature encoding, we design our multi-resolution feature grids with a 3D voxel grid & fo... and three additional multi-resolution 2D planes L L L designated capture features along z-, y-, and æ-axis respectively, inspired by NVP [39]. Upon this architectural design, we efficiently encode the local features and use them as inputs of MLPs to predict color and density that represent the radiance fields via volumetric rendering. Note that all feature grids are implemented using a hash encoding [20] for efficiency.
We intend two types of feature grids to contain the feature information in different manners. Despite the effectiveness of the 3D feature grid, the 3D grid is more severely impacted by hash collisions at higher resolutions [20], which leads to limited performance. Thus, we still need to supplement finegrrined components to improve the performance further and incorporate 2D feature grids that alleviate hash collision impact. Since the 3D feature grid encodes the main components, the 2D feature grids efficiently reinforce the feature information with fewer parameters.
2D-3D hybrid multi-resolution feature grid
Feature evaluation Here, we describe the details of binary feature encoding processes to derive the
local features, based on an input 3D coordinate x = (zr,y, y, 2).
For the 3D feature grid, we compute the local feature by tri-linearly interpolating binary parameters for each level of resolutions, and concatenating them:
Fzyz = (interp(x, sign V1 8 ))
(8)
where feyz is the computed feature from the 3D grid, I denotes the grid level and L indicates the number of grid levels.
In a different way, we perform the binary feature encoding across each axis for the 2D feature grids. Firstly, we project the 3D coordinate along each axis to obtain the projected 2D coordinates
5

---

Xay (z. v), Xxz (z, and Xyz (v, Next, we adopt bi-linear interpolation to extract features from these three projected 2D coordinates. Then, we acquire the feature value for each level of resolutions and concatenate them:
Fyy (interp(Xxy, sign( }
(9)
where fay is the computed feature from the 2D grid across z-axis, denotes the grid level, and M indicates the number of grid levels. The feature encoding steps for other features fzz and fyz operate in the similar manner.
Network architecture Finally, all these features are concatenated as f and fed into MLPs to predict the color and density values a. We utilize two MLPs each for density prediction and color prediction:
f= [Gez, fay, faz, fzz), (a,
MooPe (e, d), dePdiesity ( ()) c=
(10)
where e presents embedded feature and 7xx) is the sinusoidal positional encoding [1].
4.3 Loss
Reconstruction loss According to the volumetric rendering process described in Sec. 3, render the RGB pixel values along the sampled rays and optimize them through the color and density values in Eq. 10:
Lrecon
Σ - FER
(11)
where r denotes the sampled ray encouraged by the occupancy grid. The efficient ray sampling through the occupancy grid allows us focus on non-empty space.
Sparsity loss sparsely to skip the ray sampling in the empty area using the occupancy grid. Thus, we regularize the sparsity with Cauchy loss [15, 40]:
For accelerating the rendering speed, it is important to model the volumetric scene
Cspariity
Σ log(1+ (rx 1,ke
(12)
The overall training loss for our radiance field model is defined as = Freeom + Xxparsity fapurityy where Asparsay is the syper-parameter for sparsity loss. We set Asparsity 2.0 x 10-5 in this work.
Experiments
We conduct experiments various benchmark datasets verify the compactness of our radiance field representation. We evaluate the quantitative and qualitative results against prior works and analyze different architectural design choices.
5.1 Experimental Settings
Datasets We adopt representative benchmark novel-view synthesis datasets. We use two synthetic datasets: the Synthetic-NeRF dataset [1] and the Synthetic-NSVF dataset [13]. Both datasets consist of eight object scenes rendered with 100 training views and 200 test views at resolution of 800 x 800 pix. We also employ the Tanks and Temples dataset [41] that consists of five real-world captured scenes whose backgrounds are masked [13]. The training and test views are rendered at a resolution of 1920 1080 pix.
Baselines We have three types of baseline methods, (a) data structure-basse methods, (b)
compression-batt based methods, and (c) implicit-based methods. Firstly, we compare our radiance fields with state-of-the-art radiance field representations adopting efficient data structures (DVGO [11, 12).
Plenoxels [15], TensoRF [17], CCNeRF [18], Instant-NGP [20], and K-Planes [19]), which succeed to reconstruct high-quality 3D scenes with fast convergence speed. Specifically, we are interested in

---

Synthetic-NeFF Dataset
Synthetic-NSFF Dalaset
Tanks and Temples Dataset
Ou
Ours- B8
Ours-Ba
Durs-B4
TA
Oure-
Ours- B4
TonsoFF
TenadiI--M
20.6
Ours-Sä
TensuRF-WW VM
Oure-B2
28.4
Ours-82
FOurs-B
DWT Torsuncy
Mipbaar
ReTTersRFF
ovao
PSNR (dB)
Ourg
X
TenesRE-CP
Planooks
DVG0
PSNR (dB)
TeneoRi-CP #VDDV50
RvD0
CONNY
DVGO
PSNA (dB)
26.2
Ours S2V0 ToURF
v.
Pkmoozts
28.0
TonsuRF-OF
K-Panos
OCNART-AT
VD Pinoous
ReDvOg
DVDD
Picoress
27.8
DWT--TeeeRF
hahr
CCNPNFOPP OP
COMPTT-CP
31
30F
10' 10a 10' 10P 107 10' 10
10'
367
10'
10
Size in log (MB)
Size log (MB)
Size in log (MB)
Figure 2: Comparison with baseline radiance field models on the Synthetic-NeRR dataset, Synthetic- NSVF dataset, and Tanks and Temples dataset. We utilize different dot shapes depending on the model categories: squares ) for our models, circles for data structure- based models, crosses (x) for compression-bast based models, and triangles for implicit-bass based models.
the models that demonstrate stable performance across multiple datasets and can train within one hour when running on conventional single GPU. Moreover, we evaluate the compression methods that require post-optimization steps (Re:NeRF [34] and VQRF [36]) and masked wavelet transform (DWT-NeRFF [37]) to compress existing voxel-based models. Furthermore, also compare with representative implicit radiance field models (NeRF [1] and mip-NeRF [24]).
Implementation details We implement our feature grid using hash encoding [20]. We 16 levels of multi-resolution 3D feature grids with resolutions from 16 to 1024, while each grid includes up to TiD feature vectors. We also utilize four levels of multi-resolution 2D feature grids with resolutions from 64 to 512, while each grid includes up to Thp feature vectors. We use (log (Tap), loga (TaD)} logg(Ta = (15, 17} for our small model denoting Ours-S and {log,(T20) logz (T = {17, 19) for our base model denoting Ours-B. -B Our model variants are scaled by the dimension of feature vectors per level of 2, 4. and 8. For instance, Ours-B2 denotes base variant with feature dimension F' = 2. We exploit two types of MLPs with 128-channe hidden layers and rectified linear unit activation on them, one for density prediction with one hidden layer and another for color prediction with two hidden layers. The total storage capacity of MLPs varies from 0.06 to 0. MB, depending on the input feature dimension. The spherical harmonics basis function used to encode the directional information. Our implementation is based on Instant-NGP [20] using the occupancy grid implemented in NerfAcc [42]. We optimize all our models for 20K iterations on single GPU (NVIDIA RTX A6000). It takes approximately 6, 9, and 14 min of average time to train scenes of the Centhetic-NeRY dataset for Ours-B2 to Ours-B8 models, respectively. We use the Adam [43] optimizer with an initial Icarning rate of 0.01, which we decay at 15K and 18K iterations by factor of 0.33. Furthermore, adopt warm-up stage during the first IK iterations to ensure stable optimization.
5.2 Comparison
We measured the storage size of each method and evaluated the reconstruction quality (PSNR, SSIM) on various datasets. Note that we present the average scores of scenes in each dataset; scene-wise full scores are reported in the appendix. Fig. 2 summarizes quantitative evaluations of our radiance field representation, compared to baseline data cructure-basede compression-ban based methods, and implicit-basee based methods.
Data structure-based approaches Fig. shows the qualitative evaluations of our model, compared to data structure-based methods. Our method successfully demonstrates superior reconstruction performance against SOTA data structure-based methods with comparable training time and remark- ably small storage size; especially Ours- S2 requires only within 0.5 MB. While baseline approaches
7

---

35.06 0.4 MB
36.01 1.3 MB
36.91 2.7 MB
37.44/ 5.6 MB
Ours-S2
Ours-R2
Ours-B4
Ours-R8
Ground truth
31.30 / 1.4 MB
33.81/ MB
35.77 24.5 MB
34.92 66.8
mic
CCNeRF-CP
TensoRF-CP
Instant-NGP
TeusoRF-VA.
Figure 3:
Qualitative comparison of reconstruction quality using the mic scene of the NeRF-Synthetic dataset. For each subfigure, PSNR and storage size are shown on the right upper.
require large storage space to achieve high performance, all our models are sufficient to exceed them by less than 6 MB. In particular, Ours-B2 B2 outperforms the reconstruction quality of almost baselines with much smaller storage size of within 1.5 MB. It takes 6.1 min to accomplish this without any temporal burden to train our model a compact manner. Although our small models are enough to accomplish outstanding performance, both Ours-B4 and Ours-B8 jump to excessive reconstruction quality, outperforming state-of-the-at models. Accordingly, the results indicate that model contains the most storage-efficiet data structure for radiance field representation, which also does not require much computational cost.
adopt larger models attain higher quality. As result,
Compression-basee approaches We also compare our models with SOTA compression-basee methods. Although these approaches make the existing data structure-based models (e.g., DVGO, Plenoxels, and TensoRF-VM) highly compact by compressing the optimized model, our binary feature encoding model outperforms highly compressed data structure-basee models in terms of reconstruction performance and storage usage. Specificallyy, while compressed TensoRF-VM models (Re:TensoRF-Hips [34], VQ-TensoRF [36], and NWWT-TensRRE [37]) preserve most high performance among these compression works, Ours- B4 model has superior reconstruction quality only with 2.8 MB of storage capacity, which is smaller than the compressed TensoRF- VM models. Consequently, the results verify that our binary radiance field models accomplish higher compactness than compression- based models.
Implicit-based approaches In addition, we evaluate implicit representations with a few number of parameters due to the efficient structure of MLPs. Although their simple structure results in a small storage size, our models achieve significantly higher performance with a similar storage size.
Benefits of binary radiance field Note that our approach differs from post quantization based approaches [14, 33, 34] that demonstrate bounded performance from original features and necessitate additional ptrppoosttproessingg..m training without sacrificing the rendering quality. Our approach also differs from compression- based approaches [34, 36, 37] that require additional computations to compress or decompress the radiance field. Instead, our learned binary feature can be directly interpolated to render realistic scenes.
is noteworthy that our method learns binary features during
5.3 Ablation Study
Feature grid design We analyze the impact of the feature grid designs for scene representation and verify that our 2D-3D hybrid feature grid effectively improves the reconstruction performance. We compare three architectures with similar sizes: (a) tri-plane representation. (b) voxel representation, and (c) our 2D-3D hybrid representation. Table demonstrates the effectiveness our 2D-3D hybrid representation. Compared to the tri-plane and voxel grid, our hybrid feature grid enhances the radiance field reconstruction quality with a similar number of parameters. As result, this confirms that our architectural choice for the feature grid allows to achieve more compact feature encoding.

---

Table 1: Ablation study on the feature grid de- sign. Results are averaged over all scenes of the Synthetie-Nele dataset. We highlight the best scores in bold.
Table Ablation study on the use of sparsity loss. We report train time and inference speed with re- construction quality. Results are averaged over all scenes of the SyntheticNNRRF dataset.
Design
# Params
PSNR
SSIM
Train]
Inferenect
PSNR1
SSIM)
Tri-plane (only 2D)
31.44
0.949
Ours-B2 B2 (wlo Lquenty)
6.4
fps
32.69
0.958
Voxel (only 3D)
13.6
32.35
0.956
Ours-B2 (wl Lyparaty)
6.1 m
3.8 fps
32.64
0.958
Hybrid (2D 3D)
13.2
32.64
0.958
(wlo Expasty)
fps
33.60
0.964
Ours-B8 B8 (wl Eepreay))
13.9
2.7 lps
33.59
0.964
Synthetic-Nert
B8
PSNR (dB)
B2
32
B1
(13, 15)
31
(15, 17)
(17. 19)
30
(19, 21)
(21. 23)
29
100
10a
(a) w/o sparsity loss
(b) w/ sparsity loss
Size in log (MB)
Figure 4: Visualization of 2D slice of occupancy grid for chair scene the Synthetic-Nede dataset according to the use of sparsity loss Sspuriity
Figure 5: Ablation study the hash table size. Results are averaged over all scenes of the Synthetic-NeRF dataset.
Sparsity loss We investigate the effectiveness of sparsity loss that accelerates the rendering speed by regularizing the radiance fields more sparsely. We evaluate the train time and inference speed of our radiance field model, according to the use of sparsity loss. As shown in Table 2, we can improve the rendering time in both training and inference with minor decrease in reconstruction quality. In particular, we can accelerate the 23% of rendering speed for Ours-Bi B8 model by adopting sparsity loss. The visualization the occupancy grid, including the bitmap for the empty or non-empty area, also demonstrates that our model is trained sparsely due to sparsity loss, as shown in Fig. 4.
Hash table size We use hash encoding to construct 2D/3D feature grids, which restricted in their scale by the hash table size and the number of feature vectors per level. In other words, we can further reduce the storage size for compactness increase the storage size improve performance by scaling the number of feature vectors in the hash table. As shown in Fig. 5, we evaluate the storage size and the reconstruction performance of different hash table sizes {log>(Tı)) log2 (Tap)) where Thp and Tap denote the hash table size of the 2D and 3D grid, respectively.
Computational cost for binarization We evaluate the training time and memory requirement of a real-valued feature grid and a binary feature grid. The binarization procedure in our model adds only small portion of training time compared the real-valued feature grid, shown in Table 3. Also, there is no noticeable increase in memory usage due to binarization, as shown in Table 4. This is because binarize only several grid parameters corresponding Furthermore, there are some cases where the binary grid converges faster or requires less memory than the real-valued grid since the sparsity of the optimized scene affects the computational cost.
ray, not whole grid.
Table 3: Evaluation on the training time for 20K iterations of real-valued feature grid and a binary feature grid.
Synthetic- NeRR
Synthetic-SSVI
Tunks and Temples
Method
Ours-B2
Ours-B4
Ours-B8
Ours-B2
Ours-Be
Ours-B8
Ours-B2
Ours-B4 Ours-B8
Reul-valued
6.02 min
8.42 min
13.31 min
6.23 min
8.77 min
14.08 min
5.93 min
8.41 min
13.46 min
Binary
6.10 min
8.66 min
13.86 min
6.22 min
8.93 min
14.53 min
6.00 min
8.59 min
14.04 min

---

Table 4: Evaluation the memory requirement of real-valued feature grid and binary feature
grid.
Syathetic-NeRF
Synthetic-SSVE
Tanks and Temples
Method
Ours-B2
Ours-B4
Ours-B8
Ours-B2
Ours-B4
Ours-B8
Ours-B2
Ours-B4
Ours-B8
Real-valued
5.46 GB
6.70 GB
10.04 GB
5.47 GB
6.02 GB
10.04 GB
6.52 GB
GB
11.10
Binary
5,45 GB
6,58 GB
10.06 GB
5.47 GB
6.30 GB
10.02 GB
6.51 GB
7.92 GB
11.10 GB
Table 5: Quantitative evaluations of dynamic scene reconstruction using binary feature encoding. Re- sults are averaged over all scenes of D-NeRF dataset [6] and HyperNeRR dataset [8]. f denotes that binary feature encoding is applied.
te binary feamre enc. wel batarry Sesture enc.
34.00
28.78
30 M6
27 27.551444 M4
D-NeRF [6] (synthetic)
Method
Size (MB)
PSNR1
SSIM
TiNeu
7.63
31.03
0.957
TiNeu Vox-St
0.59
28.68
0.938
-ox-B
47.51
33.02
0.972
Voxe (synchtti))
TiNeaVoouB (synthenic)
ToNeeVox--
TiNeu Vox-B*
3.88
31.23
0.961
(rai))
HyperNeRF [8] (real)
Figure 6: Qualitative evaluations of dynamic scene reconstruction on the use of binary fea- ture encoding. Original (up) and binary feature encoding (bottom). PSNR and storage size are shown on the right upper.
Method
Size (MB)
PSNR
MS-SSIM
ToNeu
47.85
24.20
0.835
TiNeu -oxeuVox-Bi
3.90
23.87
0.826
5.4 Applications
Dynamic scene reconstruction We finally adopted our binarization strategy in dynamic scene reconstruction. We employ ToNeuVox [22], base model, one of the most efficient approaches, which utilizes time-aware voxel features. Instead of conventional feature encoding for the voxel grid, apply the binary feature encoding the time-aware voxel features and this leads highly compact reconstruction of dynamic scenes, even 0.6 MB storage space for the synthetic jumpingjacks scene. Table and Fig. demonstrate that our approach easily applicable to various feature encoding tasks, and it enables remarkably efficient representation in terms of storage size.
Conclusion
In this work, we have introduced binary radiance fields, a storage-efficient radiance field repre- sentation that significantly reduces the storage capacity by adopting binary feature encoding. Our experiments have verified that our approach is the most storage-efficien representation outperforming recent data structure- based models, compression-bas of reconstruction quality and storage consumption. This capability enables to access numerous radiance fields without storage burden and leads to the expansion of the applications, such as dynamic scenes.
models, and implicit- baseid models in terms
Limitations Despite the binarization of learnable parameters, any bit-wise operations in multi- layered perception would lead to more efficient computation, requiring fewer GPU resources. We expect our approach can be boosted for the rendering by adopting the baking process.
7 Acknowledgments and Disclosure of Funding
The authors acknowledge the support provided by Samsung Advanced Institute of Technology (SAIT). This work was supported by IITP grants (RS-2023-00227999) Detailed 3D reconstruction for urban areas from unstructured images, No. N0.2022-0000290: Visual Intelligence for Space-Time Understanding and Generation based on Multi-layered Visual Common Sense, N012001-0001900 Al Graduate School Program at POSTECH, No. No.2021-0-01343 AI Graduate School Program at Seoul National University) funded by Korea government (MSIT).
10

---

References
[1] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[2] Kai Zhang, Germot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint 2rXiv:2010.07492, 2020.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[4] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Structured view deweneeend appearance for neural radiance fields. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages
IEEE, 2022.
[5] Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Hong-Hai Zhang. Nerfren: Neural radiance fields with reflections. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 18409-18418, 2022.
[6] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Nopee Norener D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[7] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IFFF/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[8] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ri- cardo Martin Brualla, and Steven M Seitz. Hypemerf: a niihee-diiensione. representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG), 40(6): 12, 2021.
[9] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[10] Michael Niemeyer. Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5480-5490, 2022.
[11] Cheng Sun, Min Sun, and Ttwann-Tzone Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[12] Cheng Sun, Min Sun, and Hwann-Tzone Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprint arXiv:2206.0508: 05085, 2022.
[13] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33: 15651 1/651-116633 15663, 2020.
[14] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-tima rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752-5761, 2021.
[15] Sara Fridovich Keil, Alex Yu. Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5501-5510, 2022.
[16] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Samch Khamis, al. Efficient geometry-aware 3d generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123-16133, 2022.
[17] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Computer Misio-EECCV ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 27, 2022, Proceedings, Part XXXII. pages 333- 350- Springer, 2022.

---

[18] Jiaxiang Tang, Xiaokang Chen, Jingbo Wang, and Gang Zeng. Compressibl-ccompsabbe nerf via rank- residual decomposition. arXiv preprint arXiv:2205.14877 2022.
[19] Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, and Angjoo Kanazawa. K- planes: Explicit radiance fields space, time, and appearance. arXiv preprint arXiv:2301. 10241, 2023.
[20] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1-15, 2022.
[21] Matthicu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI- El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1, arXiv preprint arXiv: 028800000222...... 2016.
[22] Jiemin Fang. Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias NieBner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[23] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. arXiv preprint arXiv:2210.1540 2022.
[24] Jonathan Barron. Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3855-5864, 2021.
[25] Ricardo Martin-Bruallly Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2219, 2021.
[26] Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07066 2021.
[27] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335-14345, 2021.
[28] Qiang Zhang, Seung- Hwan Back, Szymon Rusinkiewicz, and Felix Heide. Differentiable point-based radiance fields for efficient view synthesis. arXiv preprint arXiv:2205. 14330, 2022.
[29] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-baseed neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5438-5448, 2022.
[30] Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes. arXiv preprint arXiv:2.01.0965 2023.
[31] Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, and Andreas Geiger. Factor fields: A unified framework for neural fields and beyond. arXiv preprint arXiv:2302.01222 2023.
[32] Fangneng Zhan, Lingjie Liu, Adam Kortylewski, and Christian Theobalt. General neural gauge fields. arXiv preprint arXiv:2305.0346.. 03463200..36..... 2023.
[33] Yoonwoo Jeong, Scungjoo Shin, Junha Lee, Chris Choy, Anima Anandkumar, Minsu Cho, and Jacsik Park. Perfception: Perception using radiance fields. Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmurks Track, 2022.
[34] Chenxi Lola Deng and Enzo Tartaglione. Compressing explicit voxel grid representations: fast nerfs become also small. In Proceedings of the IEEF/CVF Winter Conference Applications of Computer Vision, pages 1236-1245, 2023.
[35] Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-9, 2022.
[36] Lingzhi Li, Zhen Shen, Zhongshu Wang. Li Shen, and Liefeng Bo. Compressing volumetric radiance fields to mb. arXiv preprint xrXiv22211.16386 2022.
12

---

[37] Daniel Rho, Byeonghyeon Lee, Seungtae Nam, Joo Chan Lee, Jong Hwan Ko, and Eunbyung Park. Masked wavelet representation for compact neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20680-20690, 2023.
[38] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons conditional computation. arXiv preprint arXiv:1308.3432 2013.
[39] Subin Kim, Sihyun Yu, Jaeho Lee, and Jinwoo Shin. Scalable neural video representations with learnable positional features. arXiv preprint arXiv:2210.00 22.22000..02, 2022.
[40] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875-5884, 2021.
[41] Arno Knapitsch, Jaesik Park. Qian- Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):11131 2017
[42] Ruilong Li, Matthew Tancik, Angjoo Kanazawa. Nerfacc: general nerf accleration toolbox. arXiv preprint arXiv:2210.048444 2022.
[43] Diederik Kingma and Jimmy Adam: method stochastic optimization. arXiv preprint
arXiv:1412.6980, 2014.
13


Reviews:
[{"review_id": "ujgou6n3KS", "reviewer": "Reviewer_EsYg", "summary": "In this paper, the authors are proposing BiRF (BInary Radiance Fields), a storage-efficient representation for neural radiance fields. The technique relies on a hybrid representation that leverages explicit feature grids (both one 3D and three 2D, each at multiple resolutions) combined with density and color MLPs. To achieve high storage efficiency, the feature grids are binarized following the technique from Binarized Neural Networks. The reconstruction loss includes a sparsity inducing loss (similarly to SNeRG and Plenoxels).\n\nThe main technical contributions of the paper are this specific storage-efficient representation of neural radiance fields with a matching training scheme and an array of comparisons against other methods.\n\nThe authors demonstrate results through a number of quantitative evaluations on the Synthetic-NeRF, Synthetic-NSVF and Tanks&Temples datasets and against multiple baselines (fast ones: DVGO, Plenoxels, TensoRF, CCNeRF, Instant-NGP and K-Planes and compact ones: Re:NeRF, VQRF):\n- the proposed representation is indeed very storage efficient, with < 1 MB at acceptable quality,\n- it generally delivers quality reconstructions with lower storage requirements compared to either efficient or compressed alternatives,\n- training time remains reasonable (only behind Instant-NGP, DVGO and Plenoxels depending on the operating point).\n\nLastly, the storage efficiency of the technique is showcased to be relevant for an application on dynamic scenes.", "questions": "Following the above, would the authors be able to include a thorough comparison of rendering speed against the considered baselines?\n\nBinarized Neural Networks, which this paper is following, was using in some of their experiments stochastic binarization (as a form of regularization), is this something that has been considered?\n\nTable 3 in the supplementary material suggests increasing, have the authors tried going beyond {2^19,2^21}? Are the results plateau-ing?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Storage efficiency is an important aspect of making neural radiance fields more practical for concrete applications. While there are several neural radiance fields approaches focusing on trading rendering (and training) speed at the expense of storage efficiency and other approaches tackling storage efficiency separately by compressing a trained representation, the proposed approach addresses storage efficiency directly without necessitating a post-processing step after training and also without sacrificing quality.\n\nThe approach offers competitive operating points in terms of quality v.s. storage, while offering reasonable training time.\n\nThe method is conceptually simple and the paper does a solid job at presenting it, demonstrating its value through both quantitative and qualitative experiments, including a large number a baselines to compare against. The ablations are also thorough.", "weaknesses": "Novelty is limited as this is essentially an application of Binarized Neural Networks to neural radiance fields approaches.\n\nAs quality is one of the claims of the approach, a comparison against a few non-compressed and non-optimized baselines (e.g. original NeRF, mip-NeRF) would have made sense.\n\nThe approach is exploring with good results another trade-off compared to the efficiency-oriented techniques that sacrifice storage. However, the only mention of rendering speed (at test time) are in the ablation on the sparsity loss and in the limitations section. The apparently unoptimized implementation is another weak point of the submission and a comparison of rendering speed, especially against the considered baselines would have also made sense.\n\nMinor corrections:\n- l.104 Pleoxels -> Plenoxels\n- l.135 consider the -> consider an", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments. We have additionally performed more experiments following your comments: inference speed (Q1 in global response), more baselines (Q1), and extended ablation study (Q3). Also, we positively consider adopting stochastic binarization for further improvement of binary feature encoding (Q2). The detailed responses to your comments are as below.\n\n---\n\n**Q1. Evaluation on original NeRF and mip-NeRF.**\n\nWe provide additional results for original NeRF [1] and mip-NeRF [2] in our comparison. We have followed the default architectural setting of the original paper using the code implemented in NeRF-Factory [3]. We optimize both NeRF and mip-NeRF for 300K iterations with a batch size of 4,096 on a single NVIDIA RTX A6000. It takes about 24 hours to converge for a single scene.\nAs shown in the below table, both NeRF and mip-NeRF show less reconstruction quality with a larger storage size compared to Ours-F2. Despite the lightweight network architecture of implicit NeRFs, we have proved that our model (Ours-F2) demonstrates higher performance in a more compact storage size than NeRF and mip-NeRF.\n\n**Experiment 1) Quantitative evaluation of original NeRF and mip-NeRF.** \n\n(We reported the averaged score of each dataset.)\n\n||Size (MB)|PSNR|SSIM|LPIPS|\n|:---|:---:|:---:|:---:|:---:|\n|***Synthetic-NeRF***|\n|NeRF [1]    |4.6|31.69|0.951|0.065|\n|Mip-NeRF [2]|2.3|32.20|0.955|0.062|\n|**Ours-F2**|**1.4**|**32.64**|**0.959**|**0.049**|\n|\n|***Synthetic-NSVF***|\n|NeRF [1]    |4.6|34.46|0.967|0.044|\n|Mip-NeRF [2]|6.1|35.33|0.971|0.039|\n|**Ours-F2**|**1.5**|**35.40**|**0.976**|**0.024**|\n|\n|***Tanks and Temples***|\n|NeRF [1]    |4.6|27.58|0.902|0.171|\n|Mip-NeRF [2]|6.1|27.77|0.901|0.171|\n|**Ours-F2**|**1.5**|**28.44**|**0.916**|**0.122**|\n\n---\n\n**Q2. Consideration of stochastic binarization.**\n\nThank you for suggesting stochastic binarization [4]. We agree that stochastic binarization might yield an interesting regularization effect. Despite the strengths, our current approach does not consider stochastic binarization. The reason is that we wanted to optimize a 3D scene as fast as possible. Thus, it is beneficial to use simple and effective deterministic binarization rather than stochastic binarization that generates random numbers. \n\nWe have compared the performance of stochastic binarization and deterministic binarization (ours), as shown in the below table. As a result, applying stochastic strategy directly tends to be unstable during optimization. Compared to deterministic binarization, stochastic binarization takes a longer time and shows lower rendering quality. However, properly combining stochastic strategy with deterministic binarization may lead to better performance. Consequently, we consider employing regularization via stochastic binarization for further improvement in our future work.\n\n**Experiment 2) Comparison of binarization using Ours-F2.**\n\n(We reported the averaged score of each dataset.)\n\n||Train time (min)|PSNR|SSIM|LPIPS|\n|:---|:---:|:---:|:---:|:---:|\n|***Synthetic-NeRF***|\n|Stochastic   |14.16|29.81|0.924|0.093|\n|Deterministic| 6.10|32.64|0.959|0.049|\n|***Synthetic-NSVF***|\n|Stochastic   |11.53|27.64|0.913|0.108|\n|Deterministic| 6.22|35.40|0.976|0.024|\n|***Tanks and Temples***|\n|Stochastic   |9.13|27.77|0.888|0.174|\n|Deterministic|6.00|28.44|0.916|0.122|\n\n---\n\n**Q3. Further ablation on the hash table size described in Table 3 in the supplementary material. [Fig. 2 in the attached PDF]**\n\nThank you for noticing the extensive ablation on the hash table size {$\\log{T_{2D}}$, $\\log{T_{3D}}$}, where $T_{2D}$ and $T_{3D}$ denote the hash table size of the 2D and 3D grid, respectively. We have performed additional ablations on the smaller and larger hash table size, {13, 15} and {21, 23}. As shown in Fig. 2 in the attached PDF and the below table, it is no longer beneficial to use a larger hash table {21, 23} because there is only a slight improvement in reconstruction quality while the storage size significantly increases. In contrast, a significant drop in rendering quality is observed when we use a smaller size of the hash table {13, 15} due to severe hash collisions. The results show that hash collision does not considerably affect the performance from the size {17, 19}, so we choose {17, 19} as the default setting of our model.\n\n**Experiment 3) Additional ablation on the hash table size {$\\log{T_{2D}}$, $\\log{T_{3D}}$}. Results are averaged over all scenes of the Synthetic-NeRF dataset.**\n\n||Ours-F1||Ours-F2||Ours-F4||Ours-F8||\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||**Size (MB)**|**PSNR**|**Size (MB)**|**PSNR**|**Size (MB)**|**PSNR**|**Size (MB)**|**PSNR**\n|{13, 15}|0.12|29.20|0.18|30.66|0.31|32.00|0.58|32.90|\n|{15, 17}|0.27|30.66|0.46|32.03|0.87|32.99|1.73|33.46|\n|{17, 19}|0.72|31.53|1.14|32.64|2.83|33.26|5.76|33.59|\n|{19, 21}|1.94|31.61|3.99|32.71|7.84|33.20|16.61|33.51|\n|{21, 23}|7.05|31.93|14.70|32.89|29.65|33.31|61.35|33.55|\n\n---\n\n**Reference**\n\n[1] Mildenhall et al., Nerf: Representing scenes as neural radiance fields for view synthesis, ECCV 2020\n\n[2] Barron et al., Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields, ICCV 2021\n\n[3] Kakao Brain, Nerf-factory: an awesome pytorch nerf collection, https://github.com/kakaobrain/nerf-factory\n\n[4] Hubara et al., Binarized neural networks, NeurIPS 2016"}, {"review_id": "VsUb2kkoZa", "reviewer": "Reviewer_Bcw2", "summary": "This paper proposes a novel binary radiance fields (BiRF) which binarized the feature encoding to save memory usage of NeRF. In the experiments, the binary radiance field representation demonstrates superior reconstruction performance compared to state-of-the-art efficient radiance field models, all while requiring lower storage allocation. Notably, the proposed model achieves remarkable results in reconstructing static scenes, achieving PSNR values of 31.53 dB for Synthetic-NeRF scenes, 34.26 dB for Synthetic-NSVF scenes, and 28.02 dB for Tanks and Temples scenes. These impressive outcomes are attained using minimal storage space, with only 0.7 MB, 0.8 MB, and 0.8 MB utilized, respectively. The intention behind introducing the binary radiance field representation is to eliminate storage bottlenecks and make radiance fields more accessible for various applications.\n\n\n\n\n", "questions": "See weakness. ", "limitations": "Yes. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-organized\n2. Experiments are convincing and extensive. ", "weaknesses": "1. Only binarizing the feature encoding to save the memory usage is limited, since the acceleration rate is also import in network quantization and the inference of NeRF. \n\n2. The proposed binarization of learnable parameter is limited of novelty. The analysis or discuss about bottleneck of NeRF quantization or binarization is lack, which is crucial. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments. We have found that your concerns are mainly from two sources: the limited contribution of binary feature encoding (Q1) and the analysis of the bottleneck of NeRF quantization (Q2). Thus, we focus on addressing these concerns in this rebuttal. The detailed responses to your comments are as follows.\n\n---\n\n**Q1. Only binarizing the feature encoding to save the memory usage is limited, since the acceleration rate is also import in network quantization and the inference of NeRF.**\n\nWe agree that acceleration is one of the critical issues for more practical NeRF. However, we aim to address the issue of large storage size, which restricts the accessibility of recent radiance fields. Since this storage problem yields a bottleneck in NeRF advancement, several works have also been concerned with the seriousness and handled it with post-quantization [1, 2, 3] and post-optimization [3, 4]. Nonetheless, there is a limitation, as we described in Sec. 2 in the manuscript, so we propose our BiRF to solve the large storage problem by adopting binary feature encoding. \n\nAlthough acceleration is not our main target, we have considered it. First, we employ a hash grid of instant-NGP [5], one of the SOTA methods having both fast convergence and inference time, as the base grid implementation of the proposed BiRF. Moreover, we regularize the sparsity of the scene to achieve further improvement in rendering speed, as described in Sec. 4.3 & Sec. 5.3 in the manuscript. There is room for further improvement of inference speed since our binary feature encoding can be easily applied in other feature encoding models, as described in Sec. 5.4 in the manuscript.\n\n---\n\n**Q2. The proposed binarization of learnable parameter is limited of novelty. The analysis or discuss about bottleneck of NeRF quantization or binarization is lack, which is crucial. [Fig. 6 in the attached PDF]**\n\nAlthough we introduced several works (PlenOctrees [1], PeRFception [2], Re:NeRF [3]) that employ quantization in L107-108, it might be insufficient to fully explain the previous NeRF quantization approaches.\n\nPrevious methods [1, 2, 3] apply 8-bit quantization for their learned feature values after optimization to reduce the final storage size of the radiance field model. However, there is a severe degradation of rendering quality when we try to quantize the feature encoding parameters to a lower bit, as shown in the below table and Fig. 6 in the attached PDF. This is because post-quantization yields information loss, existing as a bottleneck of NeRF quantization. \n\nTo quantify the loss from the post-quantization, we have evaluated the rendering quality of the quantized models (1-, 2-, 3-, 4-, 8-bit) using post-quantization methods following previous methods [1, 2, 3]. To be specific, we first optimize the models using a binary feature grid (w/ binary feature encoding) denoting “ours (1-bit)” and a real-valued feature grid (w/o binary feature encoding) denoting “base (16-bit)”. After then, we quantize the optimized real-valued feature grid from 16-bit to n-bit, denoting “post-quant (n-bit)”. As shown in Fig. 6 in the attached PDF and the below table, there is a significant drop in rendering quality as the parameters are quantized to a lower bit. Binarization (or 1-bit quantization) especially leads to severe information loss, so we can not observe the texture of the target scene. In contrast, our model still has high rendering quality despite using 1-bit data since we update the binary parameters during optimization. Therefore, we expect that our binarization strategy solves the existing bottleneck of NeRF quantization and can be considered an important contribution to our work.\n\n\n**Experiment 1) Comparison of the reconstruction performance according to the post-quantization. \"Post-quant. (n-bit)\" denotes post-quantization to n-bit data.**\n\n||PSNR|SSIM|LPIPS|\n|:---|:---:|:---:|:---:|\n|Post-quant. (1-bit)|16.85|0.797|0.219|\n|Post-quant. (2-bit)|17.85|0.933|0.078|\n|Post-quant. (3-bit)|25.33|0.958|0.048|\n|Post-quant. (4-bit)|31.61|0.962|0.041|\n|Post-quant. (8-bit)|33.68|0.963|0.039|\n|Ours (1-bit)|32.64|0.959|0.049|\n|Base (16-bit)|33.69|0.963|0.039|\n\n---\n\n**Reference**\n\n[1] Yu et al., Plenoctrees for real-time rendering of neural radiance fields, ICCV 2021\n\n[2] Jeong et al., PerfCeption: perception using radiance fields, NeurIPS 2022\n\n[3] Deng et al., Compressing explicit voxel grid representations: fast nerfs become also small, WACV 2023\n\n[4] Li et al., Compressing volumetric radiance fields to 1 mb, CVPR 2023"}, {"review_id": "9N1GrAsolz", "reviewer": "Reviewer_ptCH", "summary": "The paper proposes a novel approach called binary radiance fields (BiRF) for efficient storage and representation of radiance fields. BiRF utilizes binary feature encoding, where local features are encoded using binary parameters of +1 or -1. This compact encoding significantly reduces storage size and computational costs. The authors introduce a binarization-aware training scheme and extend the multi-resolution hash encoding architecture to a 3D voxel grid and orthogonal 2D plane grids.\nThe contributions of the paper are 1. The introduction of binary radiance fields (BiRF) as a storage-efficient representation that encodes features using binary parameters. 2. A binarization-aware training scheme that effectively captures feature information and updates binary parameters during optimization. 3. The demonstration of superior reconstruction performance with minimal storage space usage, achieving impressive results in static scene reconstruction.", "questions": "1. I would suggest the author to provide an evaluation of the inference speed with other relevant methods such as TensoRF and Instant-NGP, and analyze the effect of 3D&2D feature grid and estimated gradient.\n2. It would be helpful to discuss the potential reasons for this performance gap of K-plans and provide an analysis of the inferior results. \n3. How does the proposed BiRF model handle hash collision compared to Instant-NGP?\n4. It would be beneficial to provide a more thorough discussion of these related works and explain the specific connections and differences between the proposed BiRF approach and the existing literature.\n5. Streamlining the mentioned sections to enhance the clarity of the paper.\n", "limitations": "The limitation is discussed in this work.\nThere is no concern of potential societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel approach called binary radiance fields (BiRF) for representing radiance fields using binary feature encoding. This idea of employing binary parameters to represent local features in radiance fields is innovative and distinguishes it from traditional methods. The application of binarization-aware training and the extension of multi-resolution hash encoding to a hybrid structure further contribute to the originality of the approach.\n\n2. The paper demonstrates high-quality research through rigorous experimentation and evaluation. The proposed BiRF representation outperforms state-of-the-art methods in terms of reconstruction performance while utilizing significantly less storage space. The experiments conducted on various scene datasets provide compelling evidence of the effectiveness and efficiency of the proposed approach.\n\n3. The paper is well-written and effectively communicates the concepts and methodologies to the readers. The authors provide clear explanations of the key ideas, including the binary feature encoding, binarization-aware training scheme, and the hybrid structure of the feature grid. The organization of the paper enables easy comprehension of the research objectives, methodology, and experimental results.\n\n4. The paper's contributions have significant implications for the field of radiance fields and 3D scene modeling. By introducing the BiRF representation, the authors address the critical challenge of storage efficiency in radiance field models, which can greatly impact practical applications. The superior reconstruction performance achieved by BiRF, coupled with its minimal storage requirements, opens up new possibilities for real-world implementation and broader accessibility of radiance fields.", "weaknesses": "While the paper demonstrates several strengths, there are also a few areas where it could be improved:\n\nExperimental Evaluation: While the paper presents compelling results by reporting the model size and psnr, the paper lacks a quantitative evaluation and comparison of the inference speed with other relevant methods such as TensoRF and Instant-NGP. To what extent does the hybrid 3D and 2D feature grid architecture impact the inference speed & training speed of the proposed model? As the backward gradient to the grid is estimated & approximated, will the binary design affect the convergence speed?\nWhat is more, it seems the reported training speed of TensoRF is slower than the original paper.\n\nResults: In Figure 2, I find the the results of K-Planes on Synthetic NSVF dataset are significant worse than other methods, certain analysis of the inferior results should be discussed. It is crucial to consider and provide information about the training time and inference speed, as these factors play a significant role in assessing the effectiveness and practicality of the proposed NeRF model.\n\nHash Collision: In original Instant-NGP, the hash collision is explicit handled as the largest gradients—those most relevant to the loss function—will dominate the optimization, the multi-scale will also alleviate it as collision is statistically unlikely to occur simultaneously at every level for a given pair of points. For this proposed BiRF, how about the situation of hash collision compared with instant-ngp? With binary code, it seems the multi-scale design will be less effective to prevent hash collision.\n\nBinarization of learnable parameter: The binarization with straight-through-estimator is a special case vector quantization or discrete representation, which has been explored in some recent NeRF research [1,2]. The paper could benefit from a more thorough discussion of these related works. Besides, there appears to be some duplication of content between lines 165-168 and lines 148-151. Streamlining these sections would improve the clarity of the paper.\n\nReference:\n\n[1] Variable bitrate neural fields\n\n[2] General neural gauge fields\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful comments. Following your comments, we have performed more experiments on inference speed (Q1 in global response, Q1) and convergence speed (Q1, Q2). Also, we have clearly explained several questionable parts (Q3, Q4) and the strategy for hash collision (Q5). Furthermore, we will cite the additional related works (Q2 in global response). The detailed responses to your comments are as follows.\n\n---\n\n**Q1. Impact of 2D-3D hybrid feature grid architecture on the inference and training speed.**\n\nWe have further evaluated the inference speed (FPS) and training time (min) of different grid designs: (a) tri-plane representation, (b) voxel representation, and (c) our 2D-3D hybrid representation. Note that the number of parameters of the three representations is similar. \nAs shown in the below table, the inference and training speed of 2D-3D hybrid representation is faster than tri-plane (2D), while slower than voxel grid (3D). This is because the tri-plane representation needs more computations to get local feature values. Specifically, local features in the voxel grid are computed by linear interpolation of 9 nearest points. In contrast, the tri-plane needs to perform linear interpolation of 4 nearest points for three 2D planes each, a total of 12 points. Thus, a more computational cost of the tri-plane leads to a slower speed. In conclusion, the training and rendering speed of our 2D-3D hybrid representation is positioned in the middle of the tri-plane and voxel representation as ours combines them.\n\n**Experiment 1) Comparison of inference speed (up) and training time (bottom) according to the different grid design.**\n\n||Synthetic-NeRF|Synthetic-NSVF|Tanks and Temples|\n|:---|:---:|:---:|:---:|\n|***Inference speed$\\uparrow$ (fps)***|\n|Tri-plane (only 2D)|3.56|3.98|0.71|\n|Voxel (only 3D)    |4.46|4.79|0.82|\n|Tri-plane (only 2D)|3.83|4.64|0.91|\n|***Train time$\\downarrow$  (min)***|\n|Tri-plane (only 2D)|8.69|8.34|8.30|\n|Voxel (only 3D)    |5.37|5.51|5.28|\n|Tri-plane (only 2D)|6.10|6.22|6.00|\n\n---\n\n**Q2. Impact of binary feature encoding on the convergence speed. [Fig. 3 in the attached PDF]**\n\nFollowing your suggestion, we have explored the impact of binary feature encoding on the convergence speed by comparing a real-valued feature grid (w/o binary feature encoding) and a binary feature grid (w/ binary feature encoding). As shown in Fig. 3 in the attached PDF, we have evaluated the rendering quality of these two different feature grids according to the training time. We have validated five test views for each scene to reduce time-consuming. In the early stages, we observe that the binary grid converges faster than the real-valued grid. However, the real-valued grid outperforms within 1 min, and both grids reach the fully converged performance at a similar time. Therefore, there is no degradation of convergence speed due to binarization, despite the different rendering quality.\n\n---\n\n**Q3. Slow training speed of TensoRF reported in the manuscript compared to the original paper.**\n\nWe have followed the official code of TensoRF [1] with the default configuration. Different environmental settings, such as hardware might cause the inconsistency. We use a single NVIDIA RTX A6000, while the authors of TensoRF use a single NVIDIA Tesla V100. As we have performed all experiments, including baselines, in the same environmental setting, the fairness of our comparison is ensured.\n\n**Experiment 3) Comparison of training time for TensoRF models.**\n\n||TensoRF-CP-384|TensoRF-VM-192|\n|:---|:---:|:---:|\n|Original paper|25.2 min|17.4 min|\n|Ours|24.7 min|21.5 min|\n\n---\n\n**Q4. Performance gap of K-Planes on the Synthetic NSVF dataset in Fig. 2.**\n\nThank you for your detailed concern about the results. As shown in Table 7 in the appendix, the score of K-Planes for the *Lifestyle* scene is noticeably lower than other models. In the *Lifestyle* scene, we have found a severe artifact that causes low performance and does not disappear by changing the random seed number. This result might yield confusion, as pointed out, so we will add comments for the failure case. If reviewers indicate that the K-Planes results for the Synthetic-NSVF dataset are not reasonable, we are willing to exclude them.\n\n---\n\n**Q5. How about the situation of hash collision compared with instant-ngp? [Fig. 5 in the attached PDF]**\n\nThe multi-scale design also helps to prevent the hash collision. Still, the key idea mitigating the hash collision of our model is the 2D-3D hybrid grid representation alleviating the hash collision explicitly.\n\nIn Instant-NGP [2], the main idea of a hash grid is to represent a large number of grid points with only a small size of the hash table. Thus, we cannot avoid the hash collision, which means more than two grid points are mapped to the same index in the hash table. To reduce the frequency of hash collisions, we need to decrease the number of grid points we represent. Therefore, we consider a 2D hash grid of $O(N^2)$ whose number of grid points is less than a 3D hash grid of $O(N^3)$ to alleviate the hash collision, where N is the resolution of the grid.\n\nAs shown in Fig. 5 in the attached PDF, we have quantified the average frequencies of hash collision according to different grid designs, 2D grid and 3D grid. There are severe hash collisions in the 3D grid at higher resolutions, while the 2D grid yields fewer hash collisions even at higher resolutions. Therefore, we combine 2D hash grids, which have less hash collision frequency, with a 3D hash grid to improve the restricted performance due to severe hash collisions. As a result, this 2D-3D feature grid can obtain higher rendering quality, as shown in Table 1 of the manuscript.\n\n---\n\n**Reference**\n\n[1] Chen et al., Tensorf: Tensorial radiance fields, ECCV 2022\n\n[2] Müller et al., Instant neural graphics primitives with a multiresolution hash encoding, SIGGRAPH 2022"}, {"review_id": "cpnLORIEJc", "reviewer": "Reviewer_fKiM", "summary": "This paper proposes a new representation, binary radiance fields (BiRF), for memory-efficient novel view synthesis tasks. The representation is inspired by the binary neural network. BiRF is built upon Instant-NGP. The critical component of this representation is the binarization of real-valued feature grids, such that the resulting feature grids can store bitwise feature grids, which highly reduces the storage of NeRF models. BiRF also enhances the 3D voxel grid with three 2D plane grids, where the 2D features are incorporated to alleviate the hash collision. The training loss composes of the RGB loss and the sparsity loss. Experiments are conducted on the NeRF-synthetic dataset and NSVF dataset. Compared with state-of-the-art NeRF methods (data structure-based and compression-based methods), BiRF can obtain comparable or even higher reconstruction quality while requiring less storage. The ablation study also shows the effectiveness of introducing the 2D planes and sparsity loss. ", "questions": "The reconstruction quality of BiRF outperforms other methods by a large margin in terms of PSNR. But the reason seems not very clear since BiRF replaced the real-valued feature grids with the binary feature grids. In other words, the performance should drop (slightly) compared to the non-binary version (Instant-NGP). I think the performance gain is from the incorporation of 2D plane features. However, the authors did not provide ablations on with and without the binarization of their network architecture. I would definitely rate my score if the authors provide that.\n", "limitations": "Some minor typos are that l and L are not explained for Eq.(8), though it is obvious they're the number of grid levels. (the same issue for Eq (9)).", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written. The idea is simple but very effective and easy to implement. The insight of combining 3D feature grids and 2D plane grids is really cool. Overall, the method proposed in this paper is critical to the computer vision community (both the industrial and academia). For example, memory storage is an issue if we are reconstructing very large-scale scenes; there is also a demand for deploying NeRF models to mobile devices. ", "weaknesses": "The paper also mentioned a limitation is that it requires a longer time to train BiRF compared to its Instant-NGP counterpart, due to the binarization operation on real-valued feature grids. Moreover, I think the memory requirement during training can be higher than the non-binarized version since BiRF needs to maintain the temporal real-valued feature grids in addition to the binary feature grids.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful comments. Following your comments, we have additionally performed more experiments on the binary feature encoding for training time (Q1), memory requirement (Q2), and reconstruction quality (Q3). The detailed responses to your comments are as follows. We have compared a real-valued (or un-binarized) feature grid and a binary feature grid. Please note that “real-valued” refers to a real-valued feature grid that does not apply binary feature encoding, and “binary” refers to a binary feature grid proposed in this work.  The detailed responses to your comments are as follows.\n\n---\n\n**Q1. Longer training time of the proposed BiRF during training.**\n\nWe have evaluated the training time for 20K iterations of a real-valued feature grid and a binary feature grid, as shown in the below table. Although the increase in training time is the limitation of our model, it is not critical since we utilize a simple binarization -- a sign function. Also, the binarization procedure takes only a small portion of the whole pipeline so that such simple computation does not significantly impact the total training time. Furthermore, there are some cases where the binary grid converges faster than the real-valued grid since the sparsity of the optimized scene affects the training time, as described in Sec. 5.3 in the manuscript. \n\n**Experiment 1) Evaluation of the training time for 20K iterations of a real-valued feature grid and a binary feature grid.**\n\n||Ours-F1|Ours-F2|Ours-F4|Ours-F8|\n|:---|:---:|:---:|:---:|:---:|\n|***Synthetic-NeRF***|||||\n|Real-valued|5.15 min|6.02 min|8.42 min|13.31 min|\n|Binary|5.12 min|6.10 min|8.66 min|13.86 min|\n|***Synthetic-NSVF***|||||\n|Real-valued|5.31 min|6.23 min|8.77 min|14.08 min|\n|Binary|5.17 min|6.22 min|8.93 min|14.53 min|\n|***Tanks and Temples***|||||\n|Real-valued|5.04 min|5.93 min|8.41 min|13.46 min|\n|Binary|5.01 min|6.00 min|8.59 min|14.04 min|\n\n---\n\n**Q2. Higher memory requirement of the proposed BiRF during training.**\n\nThe memory requirement during training can be higher than the non-binarized version since BiRF needs to maintain the temporal real-valued feature grids and the binary feature grids.\n\nWe have evaluated the memory usage of a real-valued feature grid and a binary feature grid. As shown in the below table, there is *no noticeable increase in memory usage* due to binarization. \nThis is because we binarize only several grid parameters corresponding to a ray, *not a whole grid*. Thus, the additional memory usage from the binarization is not as much as we need to concern.\nAlso, the number of samples per ray is an important factor for memory usage. As we use an occupancy grid for efficient ray sampling, the memory usage is also affected by the sparsity of the optimized scene. This leads to less memory requirement of the binary grid for several scenes, despite the additional computation of binary feature encoding.\n\n**Experiment 2) Evaluation of the memory requirement of a real-valued feature grid and a binary feature grid.**\n\n||Ours-F1|Ours-F2|Ours-F4|Ours-F8|\n|:---|:---:|:---:|:---:|:---:|\n|***Synthetic-NeRF***|||||\n|Real-valued|4.39 GB|5.46 GB|6.70 GB|10.04 GB|\n|Binary|4.40 GB|5.45 GB|6.58 GB|10.06 GB|\n|***Synthetic-NSVF***|||||\n|Real-valued|4.36 GB|5.47 GB|6.02 GB|10.04 GB|\n|Binary|4.37 GB|5.47 GB|6.30 GB|10.02 GB|\n|***Tanks and Temples***|||||\n|Real-valued|5.42 GB|6.52 GB|7.09 GB|11.10 GB|\n|Binary|5.41 GB|6.51 GB|7.92 GB|11.10 GB|\n\n---\n\n**Q3. Can you provide ablations on the binary feature encoding? [Fig.4 in the attached PDF]**\n\nWe have additionally performed ablations on the binary feature encoding. Specifically, we have compared the rendering quality of a real-valued feature grid denoting “real-valued” and a binary feature grid denoting “binary.” As shown in Fig.4 in the attached PDF, there is a drop in rendering quality when we use the binary grid rather than the real-valued grid with the same resolution setting. Nonetheless, we need to focus on the fact that the binary grid achieves a highly compact model size with an impressive compression rate. Also, please note that our binarized grid with multi-bit is comparable to the similar or smaller storage size of a real-valued grid (e.g., F1 of real-valued vs. F8 of binary)."}, {"review_id": "gHHQ7H2JwZ", "reviewer": "Reviewer_VKML", "summary": "This paper introduces a new voxel grid radiance field representation in which the feature vectors are restricted to contain binary values.  The motivation for this is to greatly reduce the storage requirements of voxel grid radiance fields.  They use the straight-through estimator to allow backpropagation through the binary values.  The representation combines a 3D hash grid with a triplane hash grid and uses trilinear or bilinear interpolation on the binary feature vectors.  They conduct experiments on the Synthetic-NeRF, Synthetic-NSVF, and Tanks and Templs object datasets and compare against many other voxel grid radiance field methods.  They show that they are able to achieve competitive rendering quality while using an order of magnitude less storage than the best-performing competitor.", "questions": "Is the model actually being stored as binary vectors during training and inference, or are you actually using 8-bit or larger integers (for example due to the way TinyCudaNN is implemented)?   I wondered if it is actually feasible to represent binary vectors in this way using PyTorch, or if that is only a hypothetical currently.", "limitations": "The technical limitations appear reasonable; there is no mention of broader social impacts.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The idea is straightforward but novel to the best of my knowledge.  They are able to achieve high-quality renders with a large reduction in storage requirements.  It is surprising to see that a radiance field built only with binary feature vectors could perform so well.   They also outperform other methods in which the radiance field is compressed as a post-process.  This is probably because they are able to train the quantized model in an end-to-end manner, which is interesting to see.\n\nThere is also some innovation in the idea of combining a 3D voxel grid plus a triplane representation as usually only one or the other is used, as far as I know.", "weaknesses": "They are missing a reference to Variable Bitrate Neural Fields:\n\nTakikawa, T., Evans, A., Tremblay, J., Müller, T., McGuire, M., Jacobson, A., & Fidler, S. (2022, July). Variable bitrate neural fields. In ACM SIGGRAPH 2022 Conference Proceedings (pp. 1-9).\n\nHowever they include other compression-based methods in the evaluation, and VBNF did not provide results on the standard benchmark datasets, so I can understand why they didn't include it in the evaluation.\n\nIt would have been nice to see videos in the supplemental material, which would helpful for appreciating the visual quality of the results, especially for dynamic scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful comments. Following your comments, we will cite the missing related work in the manuscript (Q1, Q2 in global response) and supplement per-scene videos for the results in the supplementary material (Q2). Also, we have elaborately explained several questionable parts of the detailed implementation of binary data (Q3). The detailed responses to your comments are as follows.\n\n---\n\n**Q1. However they include other compression-based methods in the evaluation, and VBNF did not provide results on the standard benchmark datasets, so I can understand why they didn't include it in the evaluation.**\n\nThank you for understanding the exclusion of VBNF (referred to as VQAD) [1] as our baseline model in evaluation. As the reviewer mentioned, it does not contain results for standard benchmark datasets that we use. Please refer to the Q2 in the global response for a more detailed comment. \n\n---\n\n**Q2. Addition of video for checking visual quality of the results, especially for dynamic scenes.**\n\nThank you for suggesting the supplement to the video. We will add per-scene videos for each dataset, including dynamic scenes. Furthermore, we plan to show these videos on the project page when available.\n\n---\n\n**Q3. Is the model actually being stored as binary vectors during training and inference, or are you actually using 8-bit or larger integers (for example due to the way TinyCudaNN is implemented)? I wondered if it is actually feasible to represent binary vectors in this way using PyTorch, or if that is only a hypothetical currently.**\n\nDuring training and inference, the grid parameters are stored in floating-point data type since we implement our feature grid using tiny-cuda-nn [2] that only supports floating-point data type (16-bit or 32-bit). \nAlso, we currently cannot implement actual binary vectors because most common frameworks, including PyTorch, do not support data types of a single byte. Thus, it is only available to use 8-bit (1-byte) or larger data types to represent binary parameters. Accordingly, we still need to cast n (≥8) binary parameters and store them in the n-bit data type. Our optimized binary parameters are also stored in 8-bit integers due to such limitations. \nAs we mentioned in Sec.6 in the manuscript, we acknowledge that the un-optimized implementation is a limitation of our work. We expect it to be refined by implementing an optimal binary feature grid.\n\n---\n\n**Reference**\n\n[1] Takikawa et al., Variable bitrate neural fields, SIGGRAPH 2022\n\n[2] Müller, Tiny cuda neural networks, https://github.com/NVlabs/tiny-cuda-nn"}]

Author rebuttal:
**Global response**

Dear reviewers,

We thank all reviewers for their insightful feedback. As highlighted by reviewers, our paper proposes concise (EsYg, fKiM, VKML) and innovative (ptCH, VKML) ideas and is well-written (Bcw2, ptCH, fKiM). Also, we are delighted that the reviewers thoroughly agree on the importance of storage efficiency for NeRF models (EsYg, ptCH, fKiM). As additional verification and explanation are required by the reviewers, we have tried to respond to all these valuable concerns within a limited rebuttal period. Typos and additional results will be updated in our revision. Please refer to the attached PDF containing figures for global response and response to each reviewer.

---

**Shared response for inference speed**

**Q1. Evaluation on the inference speed (EsYg, ptCH). [Fig.1 in the attached PDF]**

We have evaluated the inference speed of our models and baselines, as shown in Fig.1 in the attached PDF and the below table. Despite the highly compact storage size, our models demonstrate a comparable inference speed, especially Ours-F2 shows 3.83 fps, 4.64 fps, and 0.82 fps for each dataset, respectively. However, several models (DVGO [1], Plenoxels [2], Instant-NGP [3]) still show faster rendering speed, so we need to consider adopting an acceleration method, such as an efficient ray sampling or binary operation, as our future work. There are no significant obstacles since our binarization strategy can be applied to any feature encoding method as a plug-in. Additionally, the slow speed of Ours-F1 is from the inefficiency of atomic half-precision accumulation implemented in tiny-cuda-nn [4], a library for our hash grid implementation.

**Experiment 1) Comparison of inference speed (FPS).**

We reported the average FPS of each dataset.
||Synthetic-NeRF|Synthetic-NSVF|Tanks and Temples|
|:---|:---:|:---:|:---:|
|DVGO|5.90|6.69|1.29|
|Plenoxels|10.35|7.52|2.31|
|TensoRF-CP|0.71|0.77|0.16|
|TensoRF-VM|1.20|1.25|0.28|
|CCNeRF-CP|1.16|1.22|0.24|
|CCNeRF-HY|1.01|1.07|0.22|
|Instant-NGP|3.90|4.23|1.36|
|K-Planes-explicit|0.88|0.91|0.24|
|K-Planes-hybrid|0.75|0.91|0.29|
|Ours-F1|3.70|4.45|0.80|
|Ours-F2|3.83|4.64|0.82|
|Ours-F4|3.41|4.13|0.64|
|Ours-F8|2.72|3.45|0.45|

---

**Shared response for related work**

**Q2. Discussion on additional related work (ptCH, VKML).**

Thank you for noticing more relevant works. We will cite the mentioned papers [5, 6] in related work. In particular, VQAD [5] successfully compresses the feature grid parameters into a small codebook with learned indices rather than using a hash function. Nonetheless, expensive data is still needed for each grid point to represent the learned indices of the codebook, while ours needs only binary information. Moreover, as commented by reviewer VKML, it does not directly fit for representative NeRF benchmark we use. Specifically, it may require depth information for pre-processing, but the datasets we used do not contain depth maps. Please understand excluding VQAD in our comparison, despite its strengths.

---

**Shared response for the manuscript**

**Q3. Modification of the manuscript on typos, duplicated contents, and missing explanations for variables (EsYg, ptCH, fKiM).**

Thank you for noticing the parts that need to be modified in the manuscript. We will correct the minor typos in L104 & L135 and add explanations for variables in L191 & L197. Moreover, we have found that L148-151 and L165-168 contain similar contents, as reviewer ptCH mentioned. We will modify to briefly introduce the use of STE in L165-168 because it is already described in L148-151. When it is available, we will update all of them in the revised manuscript.

---

**Reference**

[1] Sun et al., Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction, CVPR 2022

[2] Fridovich et al., Plenoxels: Radiance fields without neural networks, CVPR 2022

[3] Müller et al., Instant neural graphics primitives with a multiresolution hash encoding, SIGGRAPH 2022

[4] Müller, Tiny cuda neural networks, https://github.com/NVlabs/tiny-cuda-nn

[5] Takikawa et al., Variable bitrate neural fields, SIGGRAPH 2022

[6] Zhan et al., General neural gauge fields, ICLR 2023

Meta-review: The final scores of this paper are SA, A, WA, WA, BR. Although one reviewer express a concern that proposed binarization of learnable parameter is limited of novelty, all other reviewers think that this paper has made important technical contributions. The rebuttal has done an excellent job in providing additional experiments and explanations. The AC has checked the paper, the review comments and the discussions, and agree with majority of the reviewers. Please incorporate all the additional materials in rebuttal into final version of this paper.
Decision: Accept (poster)