SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Seungjoo Shin
GSAI, POSTECH
seungjoo.shin@postech.ac.kr
Jaesik Park
CSE & IPAI, Seoul National University
jaesik.park@snu.ac.kr
https://seungjooshin.github.io/BiRF
In this paper, we propose binary radiance fields (BiRF), a storage-efficient radiance field representation employing binary feature encoding in a format of either  $+1$  or  $-1$ . This binarization strategy lets us represent the feature grid with highly compact feature encoding and a dramatic reduction in storage size. Furthermore, our 2D-3D hybrid feature grid design enhances the compactness of feature encoding as the 3D grid includes main components while 2D grids capture details. In our experiments, binary radiance field representation successfully outperforms the reconstruction performance of state-of-the-art (SOTA) storage-efficient radiance field models with lower storage allocation. In particular, our model achieves impressive results in static scene reconstruction, with a PSNR of  $32.03\mathrm{dB}$  for Synthetic-NeRF scenes,  $34.48\mathrm{dB}$  for Synthetic-NSVF scenes,  $28.20\mathrm{dB}$  for Tanks and Temples scenes while only utilizing  $0.5\mathrm{MB}$  of storage space, respectively. We hope the proposed binary radiance field representation will make radiance fields more accessible without a storage bottleneck.
In recent years, the emergence of Neural Radiance Fields (NeRF) [1] has greatly impacted 3D scene modeling and novel-view synthesis. The methodology models a complex volumetric scene as an implicit function that maps positional and directional information of sampled points to the corresponding color and density values, enabling the rendering of photo-realistic novel views from any desired viewpoints. Subsequent advancements [2, 3, 4, 5, 6, 7, 8, 9, 10] have demonstrated their ability to reconstruct various 3D scenes using images and corresponding camera poses, which opens radiance fields as a promising approach for representing the real 3D world.
Despite the significant progress, the computational burden of utilizing large-scale multi-layer perceptrons (MLPs) remains a critical challenge, leading to a speed issue in both training and rendering radiance fields. To tackle this issue, an auxiliary explicit voxel grid has been utilized for encoding local features, denoting a voxel-based method. While the implicit radiance field representations should update all shared learnable parameters in MLPs, parametric encoding calculates only a small local part of encoded features leading to less computational cost. The voxel-based feature encoding has been implemented in various data structures, such as dense grids [11, 12], octrees [13, 14], sparse voxel grids [15], decomposed grids [16, 17, 18, 19], and hash tables [20]. These representations succeed in efficiently reducing the time required for convergence and inference. Nonetheless, explicit feature encoding methods have a significant disadvantage: their excessive storage usage. Now, we are facing a new bottleneck that restricts accessibility. Consequently, the desire for a new radiance field representation to implement a realistic 3D scene with little storage has been raised.
This paper introduces a new binary feature encoding to represent storage-efficient radiance fields with binary feature grids, referred to as binary radiance fields (BiRF). Here, we focus on embedding sufficient feature information restricted in binary format. We achieve this by adopting a binarization-
aware training scheme, binary feature encoding, that constraints feature encoding parameters to either  $+1$  or  $-1$  and update them during optimization, inspired by Binarized Neural Networks (BNNs) [21]. Accordingly, our radiance field representation can successfully reconstruct complicated 3D scenes with binary encoding parameters that can be represented using compact data resulting in a storage-efficient radiance field model. Furthermore, we extend the modern multi-resolution hash encoding architecture to a 3D voxel grid and three orthogonal 2D plane grids. This hybrid structure allows for more efficient feature capture in a more compact manner.
As a result, our radiance field model achieves superior reconstruction performance compared to prior efficient and lightweight methods. To be specific, our model attains outstanding performance in static scene reconstruction, with a PSNR of 32.03 dB for Synthetic-NeRF scenes, 34.48 dB for Synthetic-NSVF scenes, 28.20 dB for Tanks and Temples scenes while only utilizing 0.5 MB of storage space, respectively.
We summarize our contributions as follows:
- We propose binary radiance fields (BiRF), a binary radiance field representation that concisely encodes the binary feature values of either  $+1$  or  $-1$  in 2D-3D hybrid feature grid to represent storage-efficient radiance fields.
- Our binarization-aware training scheme, binary feature encoding, allows us to effectively encode the feature information with binary parameters and update these parameters during optimization.
- We demonstrate that our representation achieves superior performance despite requiring only minimal storage space, even in  $0.5\mathrm{MB}$  for Synthetic-NeRF scenes.
Neural Radiance Fields Neural radiance fields (NeRF) [1] is a leading method for novel-view synthesis by reconstructing high-quality 3D scenes. To achieve scene representation, it optimizes coordinate-based multi-layer perceptrons (MLPs) to estimate the color and density values of the 3D scene via differentiable volume rendering.
Improving radiance field representations begins with embracing diverse scenarios where the scene is intricate. The sampling strategy used in the original NeRF assumes that the entire scene can fit within a bounded volume, which limits its ability to capture background elements in an unbounded scene. To address this issue, several works [2, 3] have separately modeled foreground and background by re-parameterizing the 3D space. These parameterizations have been primarily applied in unbounded  $360^{\circ}$  view captured scenes. Additionally, due to insufficient capacity, NeRF's lighting components have limitations in dealing with glossy surfaces. To address this challenge, transmitted and reflected radiance are optimized separately [4, 5]. Furthermore, there are approaches to extend to a dynamic domain with object movements [6, 7, 8, 22, 23].
Despite performing impressive results, it has limitations, including slow training and rendering speed. It relies solely on utilizing implicit functions for 3D scene representation, which may lead to computational inefficiencies [20].
Radiance Fields Representations NeRF methods can be categorized into three types: implicit [1, 2, 3, 24, 25, 26, 27], explicit [14, 15, 18, 28], and hybrid representations [11, 12, 13, 17, 20, 23, 29, 30, 31], depending on how the approach represents the scenes.
Implicit representations extensively use neural networks to represent radiance fields, as done in the pioneering method [1]. The network has a simple structure and can render photo-realistic images with few parameters. However, they take a lot of time to converge and require significant inference time because they share the entire weight and bias parameters of the MLPs for an arbitrary input coordinate, resulting in a significant computational cost. Therefore, recent studies have proposed explicit and hybrid radiance field representations to overcome the slow speed by incorporating explicit data structures (such as 2D/3D grids or 3D points) for local feature encoding.
Explicit representations directly encode view-dependent color and opacity values with basis functions (e.g., spherical harmonics). For instance, PlenOctrees [14] bakes implicit radiance fields into an octree structure for rendering speed acceleration. Plenoxels [15] uses a sparse voxel structure, and the approach by Zhang et al. [28] utilize a point cloud. Similarly, CCNeRF [18] employs low-rank tensor
grids for 3D scene manipulation. On the other hand, hybrid representations utilize encoded local features as input for the MLPs. NSVF [13] achieves fast rendering speed thanks to the octree structure. To store local features, Point-NeRF [29] uses a point cloud, DVGO [11, 12] employs two dense voxel grids, and TensoRF [17] makes use of a factorized tensor grid. More recently, Instant-NGP [20] introduces a multi-resolution hash encoding technique that has demonstrated exceptional effectiveness in terms of convergence and rendering speed, achieving superior performance. Additionally, Zhan et al. [32] introduce a learning method of the gauge transformation in radiance fields.
While explicit and hybrid representations boost training time and rendering speed, they inevitably suffer from the critical disadvantage of large storage consumption due to excessive local features. In this study, we construct a multi-resolution 2D-3D hybrid grid by combining 2D planes and 3D grids and binarizing their encoding parameters to fully leverage their minimal information.
Radiance Fields Compression Despite the acceleration of training and rendering, the explicit and the hybrid radiance fields have difficulties utilized in various applications due to their large storage. Consequently, there are several attempts to reduce the storage of the models.
For instance, PlenOctrees [14] and Plenoxels [15] filter the voxels using weight-based thresholding for leaving only the set of sparse voxels, which are sufficient to represent the scene. The distortion loss in DVGO-v2 [12] enables it to achieve better quality and greater resolution compactness. Other approaches apply bit quantization after the training. PlenOctrees [14], PeRFception [33], and Re:NeRF [34] all apply low-bit quantization of trained local features. VQAD [35] compresses the feature grid parameters into a small codebook with learned indices. Re:NeRF [34] and VQRF [36] have proposed methods for compressing existing explicit or hybrid radiance field models, including post-optimization processes. Also, Rho et al. [37] achieve high storage efficiency by applying wavelet transform on hybrid radiance field models with learnable masks.
Although the post-processing approaches above successfully reduce the storage requirements of the radiance field models, they have several disadvantages. Firstly, they require additional optimization steps for compression, which can be time-consuming. Also, their performance is bounded by the performance of the pre-trained models. In contrast, our approach performs binarization during training with the efficient 2D/3D feature grid representation. Our approach does not require any post-optimization processes and shows better rendering quality with even smaller storage space.
The methodology of NeRF [1] optimizes a 5D function, as a radiance field representation, to model a continuous volumetric scene with a view-dependent effect. The implicit 5D function, which consists of MLPs, maps a 3D coordinate  $\mathbf{x} \in (x,y,z)$  and a 2D viewing direction  $\mathbf{d} = (\theta ,\phi)$  to an emitted color  $\mathbf{c} = (r,g,b)$  and a volume density  $\sigma$ :
$$
(\mathbf {c}, \sigma) = \operatorname {M L P} _ {\Theta} (\mathbf {x}, \mathbf {d}). \tag {1}
$$
Due to the use of an implicit function, updating all the weight and bias parameters of MLPs is necessary to train a single point. Consequently, this leads to slow convergence, requiring more than a day to optimize a scene.
For volume rendering, the colors  $\{\mathbf{c}_i\}$  and densities  $\{\sigma_i\}$  of sampled points along a ray  $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$  are accumulated to obtain the color of the ray:
$$
\hat {C} (\mathbf {r}) = \sum_ {N} ^ {i = 1} T _ {i} \alpha_ {i} \mathbf {c} _ {i}, \quad T _ {i} = \prod_ {j = 1} ^ {i - 1} (1 - \alpha_ {j}), \quad \alpha_ {i} = 1 - \exp (- \sigma_ {i} \delta_ {i}), \tag {2}
$$
where  $T_{i}$  and  $\alpha_{i}$  represent accumulated transmittance and alpha of  $i$ -th sampled point, respectively.  $\delta_{i} = t_{i + 1} - t_{i}$  denotes the distance between adjacent points. Recently, an occupancy grid [20] is adopted to skip non-empty space for efficient ray sampling, leading to an advance in rendering speed.
To accelerate the rendering process, the hybrid representations [11, 17, 20] have been developed to encode local features in explicit data structures  $\Phi_{\theta}$  (e.g., 2D/3D grid). These features are then linearly interpolated and used as inputs for small MLPs to predict colors and densities:
$$
\mathbf {f} = \operatorname {i n t e r p} (\mathbf {x}, \Phi_ {\theta}), \quad (\mathbf {c}, \sigma) = \mathrm {M L P} _ {\Theta} (\mathbf {f}, \mathbf {d}), \tag {3}
$$
Real-valued grid
Binary grid
(a)
(b)
Figure 1: Illustration of overall framework: (a) binary feature encoding, and (b) binary radiance field representation. (a) Our binary feature encoding begins with applying binarization operation to the real-valued grid. Next, we linearly interpolate the binary parameters to obtain feature values. (b) Our radiance field representation comprises a 3D and three 2D feature grids. Given a 3D coordinate, the corresponding feature values are computed from each grid using binary feature encoding. The concatenated feature values are then fed as input to shallow density MLP with positional encoded coordinates. Then we can obtain the density value and embedding features that are further provided as input to shallow color MLPs to acquire color value.
where interp means a linear interpolation operator, and  $\mathbf{f}$  denotes a interpolated feature. The choice of explicit data structure  $\Phi_{\theta}$  significantly affects the number of learnable parameters and hence decides the total storage size of the radiance field representation. Therefore, we need to consider an efficient data structure.
In this section, we introduce binary radiance fields that require only a small storage space by adopting binary feature grids. Fig. 1 shows the overall scheme of our radiance field reconstruction. We first introduce how to encode the binary parameters in the feature grid during optimization. Furthermore, we present our 2D-3D hybrid feature grid, which enhances the feature encoding by leveraging the strengths of both 2D plane and 3D voxel.
4.1 Binarization
Binarization of learnable parameter The binarization procedure of real-valued variables is achieved using a deterministic binarization operator [21], the sign function, which transforms a real-valued variable into either  $+1$  or  $-1$ :
$$
\theta^ {\prime} = \operatorname {s i g n} (\theta) = \left\{ \begin{array}{r l} + 1 & \text {i f} \theta \geq 0, \\ - 1 & \text {o t h e r w i s e}, \end{array} \right. \tag {4}
$$
where  $\theta$  denotes the real-valued variable, and  $\theta^{\prime}$  represents binary variable.
Since the derivative of the sign function is zero almost everywhere, we cannot use traditional backpropagation to compute gradients for real-valued parameters. Instead, we use the straight-
through estimator (STE) [38], which is a simple but effective technique for backpropagating through threshold functions (e.g., the sign function):
$$
\frac {\partial \mathcal {L}}{\partial \theta} = \frac {\partial \mathcal {L}}{\partial \theta^ {\prime}} \mathbb {1} _ {| \theta | \leq 1}, \tag {5}
$$
where  $\mathcal{L}$  is the loss function. This strategy allows us to maintain the gradient flow and ensure the differentiability of our framework. During training, we maintain and update real-valued parameters rather than directly learning binary parameters. We stop propagating gradients when  $\theta$  is a large value. This also constrains the value to the range  $\{-1, 1\}$  preventing the divergence of the scale.
Binary feature encoding We propose binary feature encoding, a feature encoding scheme with binary parameters. Instead of real-valued parameters, binary parameters are used for encoding local features in a specific data structure, which are then employed to represent the radiance fields. While real-valued parameters require expensive floating-point data for representation, binary parameters can be expressed in a single bit (1-bit), leading to a significant reduction in storage overhead. Here, we describe how to implement this strategy in the feature grid  $\Phi_{\theta}$ . We first adopt binarization operation, described in Eq. 4, to the real-valued grid parameter  $\theta$ :
$$
\theta^ {\prime} = \operatorname {s i g n} (\theta) \quad \rightarrow \quad \Phi_ {\theta^ {\prime}} = \operatorname {s i g n} \left(\Phi_ {\theta}\right), \tag {6}
$$
where  $\theta^{\prime}$  denotes the binary grid parameter of binary feature grid  $\Phi_{\theta^{\prime}}$ . Next, we linearly interpolate these binary parameters depending on the given coordinate  $\mathbf{x}$ :
$$
\mathbf {f} = \operatorname {i n t e r p} \left(\mathbf {x}, \Phi_ {\theta^ {\prime}}\right) = \operatorname {i n t e r p} \left(\mathbf {x}, \operatorname {s i g n} \left(\Phi_ {\theta}\right)\right), \tag {7}
$$
where  $\mathbf{x}$  is an input coordinate, and  $\mathbf{f}$  denotes the encoded feature value. We utilize the STE [38] to propagate the gradients into the feature grid and update the grid parameters, as described in Eq. 5.
Now, our system is capable of training the feature encoding using binary parameters. This enables us to represent the feature grid with compact data, instead of using expensive floating-point data (16-bit or 32-bit). As a result, there is a tremendous reduction in the total storage size, making our radiance field representation storage-efficient.
4.2 Radiance Field Representation
For effective binary feature encoding, we design our multi-resolution feature grids with a 3D voxel grid  $\Phi_{\theta_{xyz}}$ , and three additional multi-resolution 2D planes  $\Phi_{\theta_{xy}}, \Phi_{\theta_{xz}}, \Phi_{\theta_{yz}}$ , designated to capture features along  $z-$ ,  $y-$ , and  $x$ -axis respectively, inspired by NVP [39]. Upon this architectural design, we efficiently encode the local features and use them as inputs of MLPs to predict color and density that represent the radiance fields via volumetric rendering. Note that all feature grids are implemented using a hash encoding [20] for efficiency.
2D-3D hybrid multi-resolution feature grid We intend two types of feature grids to contain the feature information in different manners. Despite the effectiveness of the 3D feature grid, the 3D grid is more severely impacted by hash collisions at higher resolutions [20], which leads to limited performance. Thus, we still need to supplement fine-grained components to improve the performance further and incorporate 2D feature grids that alleviate hash collision impact. Since the 3D feature grid encodes the main components, the 2D feature grids efficiently reinforce the feature information with fewer parameters.
Feature evaluation Here, we describe the details of binary feature encoding processes to derive the local features, based on an input 3D coordinate  $\mathbf{x} = (x,y,z)$ .
For the 3D feature grid, we compute the local feature by tri-linearly interpolating binary parameters for each level of resolutions, and concatenating them:
$$
\mathbf {f} _ {\text {x y z}} = \left\{\operatorname {i n t e r p} \left(\mathrm {x}, \operatorname {s i g n} \left(\Phi_ {\theta_ {\text {x y z}}} ^ {l}\right)\right) \right\} _ {l = 1} ^ {L}, \tag {8}
$$
where  $\mathbf{f}_{\mathrm{xyz}}$  is the computed feature from the 3D grid,  $l$  denotes the grid level and  $L$  indicates the number of grid levels.
In a different way, we perform the binary feature encoding across each axis for the 2D feature grids. Firstly, we project the 3D coordinate  $\mathbf{x}$  along each axis to obtain the projected 2D coordinates
$\mathbf{x}_{\mathrm{xy}} = (x,y)$ ,  $\mathbf{x}_{\mathrm{xz}} = (x,z)$ , and  $\mathbf{x}_{\mathrm{yz}} = (y,z)$ . Next, we adopt bi-linear interpolation to extract features from these three projected 2D coordinates. Then, we acquire the feature value for each level of resolutions and concatenate them:
$$
\mathbf {f} _ {\mathrm {x y}} = \left\{\operatorname {i n t e r p} \left(\mathbf {x} _ {\mathrm {x y}}, \operatorname {s i g n} \left(\Phi_ {\theta_ {\mathrm {x y}}} ^ {m}\right)\right) \right\} _ {m = 1} ^ {M}, \tag {9}
$$
where  $\mathbf{f}_{\mathrm{xy}}$  is the computed feature from the 2D grid across  $z$ -axis,  $m$  denotes the grid level, and  $M$  indicates the number of grid levels. The feature encoding steps for other features  $\mathbf{f}_{\mathrm{xz}}$  and  $\mathbf{f}_{\mathrm{yz}}$  operate in the similar manner.
Network architecture Finally, all these features are concatenated as  $\mathbf{f}$  and fed into MLPs to predict the color c and density values  $\sigma$ . We utilize two MLPs each for density prediction and color prediction:
$$
\mathbf {f} = \left\{\mathbf {f} _ {\mathrm {x y z}}, \mathbf {f} _ {\mathrm {x y}}, \mathbf {f} _ {\mathrm {x z}}, \mathbf {f} _ {\mathrm {y z}} \right\}, \quad (\sigma , \mathbf {e}) = \operatorname {M L P} _ {\text {d e n s i t y}} (\gamma (\mathbf {x}), \mathbf {f}), \quad \mathbf {c} = \operatorname {M L P} _ {\text {c o l o r}} (\mathbf {e}, \mathbf {d}), \tag {10}
$$
where  $\mathbf{e}$  presents embedded feature and  $\gamma (\mathbf{x})$  is the sinusoidal positional encoding [1].
4.3 Loss
Reconstruction loss According to the volumetric rendering process described in Sec. 3, we can render the RGB pixel values along the sampled rays and optimize them through the color and density values in Eq. 10:
$$
\mathcal {L} _ {\text {r e c o n}} = \sum_ {\mathbf {r} \in \mathcal {R}} | | \hat {C} (\mathbf {r}) - C (\mathbf {r}) | | _ {2} ^ {2}, \tag {11}
$$
where  $\mathbf{r}$  denotes the sampled ray encouraged by the occupancy grid. The efficient ray sampling through the occupancy grid allows us to focus on non-empty space.
Sparsity loss For accelerating the rendering speed, it is important to model the volumetric scene sparsely to skip the ray sampling in the empty area using the occupancy grid. Thus, we regularize the sparsity with Cauchy loss [15, 40]:
$$
\mathcal {L} _ {\text {s p a r s i t y}} = \sum_ {i, k} \log \left(1 + 2 \sigma \left(\mathbf {r} _ {i} \left(t _ {k}\right)\right) ^ {2}\right). \tag {12}
$$
The overall training loss for our radiance field model is defined as  $\mathcal{L} = \mathcal{L}_{\mathrm{recon}} + \lambda_{\mathrm{sparsity}}\mathcal{L}_{\mathrm{sparsity}}$ , where  $\lambda_{\mathrm{sparsity}}$  is the hyper-parameter for sparsity loss. We set  $\lambda_{\mathrm{sparsity}} = 2.0\times 10^{-5}$  in this work.
We conduct experiments on various benchmark datasets to verify the compactness of our radiance field representation. We evaluate the quantitative and qualitative results against prior works and analyze different architectural design choices.
5.1 Experimental Settings
Datasets We adopt representative benchmark novel-view synthesis datasets. We use two synthetic datasets: the Synthetic-NeRF dataset [1] and the Synthetic-NSVF dataset [13]. Both datasets consist of eight object scenes rendered with 100 training views and 200 test views at a resolution of  $800 \times 800$  pix. We also employ the Tanks and Temples dataset [41] that consists of five real-world captured scenes whose backgrounds are masked [13]. The training and test views are rendered at a resolution of  $1920 \times 1080$  pix.
Baselines We have three types of baseline methods, (a) data structure-based methods, (b) compression-based methods, and (c) implicit-based methods. Firstly, we compare our radiance fields with state-of-the-art radiance field representations adopting efficient data structures (DVGO [11, 12], Plenoxels [15], TensoRF [17], CCNeRF [18], Instant-NGP [20], and K-Planes [19]), which succeed to reconstruct high-quality 3D scenes with fast convergence speed. Specifically, we are interested in
Figure 2: Comparison with baseline radiance field models on the Synthetic-NeRF dataset, SyntheticNSVF dataset, and Tanks and Temples dataset. We utilize different dot shapes depending on the model categories: squares (■) for our models, circles (●) for data structure-based models, crosses (×) for compression-based models, and triangles (▼) for implicit-based models.