Title: Binary Radiance Fields
Venue: NeurIPS 2023
Accepted: yes
Keywords: neural radiance fields

inverse rendering

binarization
Abstract: In this paper, we propose \textit{binary radiance fields} (BiRF), a storage-efficient radiance field representation employing binary feature encoding in a format of either $+1$ or $-1$. This binarization strategy lets us represent the feature grid with highly compact feature encoding and a dramatic reduction in storage size. Furthermore, our 2D-3D hybrid feature grid design enhances the compactness of feature encoding as the 3D grid includes main components while 2D grids capture details. In our experiments, binary radiance field representation successfully outperforms the reconstruction performance of state-of-the-art (SOTA) storage-efficient radiance field models with lower storage allocation. In particular, our model achieves impressive results in static scene reconstruction, with a PSNR of 32.03 dB for Synthetic-NeRF scenes, 34.48 dB for Synthetic-NSVF scenes, 28.20 dB for Tanks and Temples scenes while only utilizing 0.5 MB of storage space, respectively. We hope the proposed binary radiance field representation will make radiance fields more accessible without a storage bottleneck.

Binary Radiance Fields

Abstract

In this paper, we propose binary radiance fields (BiRF), a storage-efficient radiance field representation employing binary feature encoding in a format of either +1 or -1 . This binarization strategy lets us represent the feature grid with highly compact feature encoding and a dramatic reduction in storage size. Furthermore, our 2D-3D hybrid feature grid design enhances the compactness of feature encoding as the 3D grid includes main components while 2D grids capture details. In our experiments, binary radiance field representation successfully outperforms the reconstruction performance of state-of-the-art (SOTA) storage-efficient radiance field models with lower storage allocation. In particular, our model achieves impressive results in static scene reconstruction, with a PSNR of 32.03 dB for Synthetic-NeRF scenes, 34.48 dB for Synthetic-NSVF scenes, 28.20 dB for Tanks and Temples scenes while only utilizing 0.5 MB of storage space, respectively. We hope the proposed binary radiance field representation will make radiance fields more accessible without a storage bottleneck.

1 Introduction

In recent years, the emergence of Neural Radiance Fields (NeRF) [1] has greatly impacted 3D scene modeling and novel-view synthesis. The methodology models a complex volumetric scene as an implicit function that maps positional and directional information of sampled points to the corresponding color and density values, enabling the rendering of photo-realistic novel views from any desired viewpoints. Subsequent advancements [2, 3, 4, 5, 6, 7, 8, 9, 10] have demonstrated their ability to reconstruct various 3D scenes using images and corresponding camera poses, which opens radiance fields as a promising approach for representing the real 3D world.

Despite the significant progress, the computational burden of utilizing large-scale multi-layer perceptrons (MLPs) remains a critical challenge, leading to a speed issue in both training and rendering radiance fields. To tackle this issue, an auxiliary explicit voxel grid has been utilized for encoding local features, denoting a voxel-based method. While the implicit radiance field representations should update all shared learnable parameters in MLPs, parametric encoding calculates only a small local part of encoded features leading to less computational cost. The voxel-based feature encoding has been implemented in various data structures, such as dense grids [11, 12], octrees [13, 14], sparse voxel grids [15], decomposed grids [16, 17, 18, 19], and hash tables [20]. These representations succeed in efficiently reducing the time required for convergence and inference. Nonetheless, explicit feature encoding methods have a significant disadvantage: their excessive storage usage. Now, we are facing a new bottleneck that restricts accessibility. Consequently, the desire for a new radiance field representation to implement a realistic 3D scene with little storage has been raised.

This paper introduces a new binary feature encoding to represent storage-efficient radiance fields with binary feature grids, referred to as binary radiance fields (BiRF). Here, we focus on embedding sufficient feature information restricted in binary format. We achieve this by adopting a binarization-

aware training scheme, binary feature encoding, that constraints feature encoding parameters to either +1 or -1 and update them during optimization, inspired by Binarized Neural Networks (BNNs) [21]. Accordingly, our radiance field representation can successfully reconstruct complicated 3D scenes with binary encoding parameters that can be represented using compact data resulting in a storageefficient radiance field model. Furthermore, we extend the modern multi-resolution hash encoding architecture to a 3D voxel grid and three orthogonal 2D plane grids. This hybrid structure allows for more efficient feature capture in a more compact manner.

As a result, our radiance field model achieves superior reconstruction performance compared to prior efficient and lightweight methods. To be specific, our model attains outstanding performance in static scene reconstruction, with a PSNR of 32.03 dB for Synthetic-NeRF scenes, 34.48 dB for Synthetic-NSVF scenes, 28.20 dB for Tanks and Temples scenes while only utilizing 0.5 MB of storage space, respectively.

We summarize our contributions as follows:

We propose binary radiance fields (BiRF), a binary radiance field representation that concisely encodes the binary feature values of either +1 or -1 in 2D-3D hybrid feature grid to represent storage-efficient radiance fields.

Our binarization-aware training scheme, binary feature encoding, allows us to effectively encode the feature information with binary parameters and update these parameters during optimization.

We demonstrate that our representation achieves superior performance despite requiring only minimal storage space, even in 0.5 MB for Synthetic-NeRF scenes.

2 Related Work

Neural Radiance Fields Neural radiance fields (NeRF) [1] is a leading method for novel-view synthesis by reconstructing high-quality 3D scenes. To achieve scene representation, it optimizes coordinate-based multi-layer perceptrons (MLPs) to estimate the color and density values of the 3D scene via differentiable volume rendering.

Improving radiance field representations begins with embracing diverse scenarios where the scene is intricate. The sampling strategy used in the original NeRF assumes that the entire scene can fit within a bounded volume, which limits its ability to capture background elements in an unbounded scene. To address this issue, several works [2, 3] have separately modeled foreground and background by re-parameterizing the 3D space. These parameterizations have been primarily applied in unbounded 360 ◦ view captured scenes. Additionally, due to insufficient capacity, NeRF's lighting components have limitations in dealing with glossy surfaces. To address this challenge, transmitted and reflected radiance are optimized separately [4, 5]. Furthermore, there are approaches to extend to a dynamic domain with object movements [6, 7, 8, 22, 23].

Despite performing impressive results, it has limitations, including slow training and rendering speed. It relies solely on utilizing implicit functions for 3D scene representation, which may lead to computational inefficiencies [20].

Radiance Fields Representations NeRF methods can be categorized into three types: implicit [1, 2, 3, 24, 25, 26, 27], explicit [14, 15, 18, 28], and hybrid representations [11, 12, 13, 17, 20, 23, 29, 30, 31], depending on how the approach represent the scenes.

Implicit representations extensively use neural networks to represent radiance fields, as done in the pioneering method [1]. The network has a simple structure and can render photo-realistic images with few parameters. However, they take a lot of time to converge and require significant inference time because they share the entire weight and bias parameters of the MLPs for an arbitrary input coordinate, resulting in a significant computational cost. Therefore, recent studies have proposed explicit and hybrid radiance field representations to overcome the slow speed by incorporating explicit data structures (such as 2D/3D grids or 3D points) for local feature encoding.

Explicit representations directly encode view-dependent color and opacity values with basis functions (e.g., spherical harmonics). For instance, PlenOctrees [14] bakes implicit radiance fields into an octree structure for rendering speed acceleration. Plenoxels [15] uses a sparse voxel structure, and the approach by Zhang et al. [28] utilize a point cloud. Similarly, CCNeRF [18] employs low-rank tensor

grids for 3D scene manipulation. On the other hand, hybrid representations utilize encoded local features as input for the MLPs. NSVF [13] achieves fast rendering speed thanks to the octree structure. To store local features, Point-NeRF [29] uses a point cloud, DVGO [11, 12] employs two dense voxel grids, and TensoRF [17] makes use of a factorized tensor grid. More recently, Instant-NGP [20] introduces a multi-resolution hash encoding technique that has demonstrated exceptional effectiveness in terms of convergence and rendering speed, achieving superior performance. Additionally, Zhan et al. [32] introduce a learning method of the gauge transformation in radiance fields.

While explicit and hybrid representations boost training time and rendering speed, they inevitably suffer from the critical disadvantage of large storage consumption due to excessive local features. In this study, we construct a multi-resolution 2D-3D hybrid grid by combining 2D planes and 3D grids and binarizing their encoding parameters to fully leverage their minimal information.

Radiance Fields Compression Despite the acceleration of training and rendering, the explicit and the hybrid radiance fields have difficulties utilized in various applications due to their large storage. Consequently, there are several attempts to reduce the storage of the models.

For instance, PlenOctrees [14] and Plenoxels [15] filter the voxels using weight-based thresholding for leaving only the set of sparse voxels, which are sufficient to represent the scene. The distortion loss in DVGO-v2 [12] enables it to achieve better quality and greater resolution compactness. Other approaches apply bit quantization after the training. PlenOctrees [14], PeRFception [33], and Re:NeRF [34] all apply low-bit quantization of trained local features. VQAD [35] compresses the feature grid parameters into a small codebook with learned indices. Re:NeRF [34] and VQRF [36] have proposed methods for compressing existing explicit or hybrid radiance field models, including post-optimization processes. Also, Rho et al. [37] achieve high storage efficiency by applying wavelet transform on hybrid radiance field models with learnable masks.

Although the post-processing approaches above successfully reduce the storage requirements of the radiance field models, they have several disadvantages. Firstly, they require additional optimization steps for compression, which can be time-consuming. Also, their performance is bounded by the performance of the pre-trained models. In contrast, our approach performs binarization during training with the efficient 2D/3D feature grid representation. Our approach does not require any post-optimization processes and shows better rendering quality with even smaller storage space.

3 Preliminaries

The methodology of NeRF [1] optimizes a 5D function, as a radiance field representation, to model a continuous volumetric scene with a view-dependent effect. The implicit 5D function, which consists of MLPs, maps a 3D coordinate x ∈ ( x, y, z ) and a 2D viewing direction d = ( θ, ϕ ) to an emitted color c = ( r, g, b ) and a volume density σ :

Due to the use of an implicit function, updating all the weight and bias parameters of MLPs is necessary to train a single point. Consequently, this leads to slow convergence, requiring more than a day to optimize a scene.

For volume rendering, the colors { c i } and densities { σ i } of sampled points along a ray r ( t ) = o + t d are accumulated to obtain the color of the ray:

where T i and α i represent accumulated transmittance and alpha of i -th sampled point, respectively. δ i = t i +1 -t i denotes the distance between adjacent points. Recently, an occupancy grid [20] is adopted to skip non-empty space for efficient ray sampling, leading to an advance in rendering speed.

To accelerate the rendering process, the hybrid representations [11, 17, 20] have been developed to encode local features in explicit data structures Φ θ (e.g., 2D/3D grid). These features are then linearly interpolated and used as inputs for small MLPs to predict colors and densities:

𝐱

Multi-resolution 3D Feature Grid

Volumetric scene

𝐱 = (𝑥, 𝑦, 𝑧)

Multi-resolution 2D Feature Grids

𝐟

!"#

𝐟

!"

𝐟

!#

𝐟

"#

⊕

MLPs

-0.54

+0.49

-0.2

3

+0.72

-1

+1

-1

+1

-1

+1

-1

+1

𝐟

sign

interp

Real-valued grid

Binary grid

Interpolation

: Real-valued parameters

: Binary parameters

: Input coordiante

(a)

(b)

𝛾(𝐱)

𝐝

Positional

encoding

Directional

information

σ

𝐜

Density

Color

level 𝑚 = 1, … , 𝑀

level 𝑙 = 1, … , 𝐿

where interp means a linear interpolation operator, and f denotes a interpolated feature. The choice of explicit data structure Φ θ significantly affects the number of learnable parameters and hence decides the total storage size of the radiance field representation. Therefore, we need to consider an efficient data structure.

4 Method

In this section, we introduce binary radiance fields that require only a small storage space by adopting binary feature grids. Fig. 1 shows the overall scheme of our radiance field reconstruction. We first introduce how to encode the binary parameters in the feature grid during optimization. Furthermore, we present our 2D-3D hybrid feature grid, which enhances the feature encoding by leveraging the strengths of both 2D plane and 3D voxel.

4.1 Binarization

Binarization of learnable parameter The binarization procedure of real-valued variables is achieved using a deterministic binarization operator [21], the sign function, which transforms a real-valued variable into either +1 or -1 :

where θ denotes the real-valued variable, and θ ′ represents binary variable.

Since the derivative of the sign function is zero almost everywhere, we cannot use traditional backpropagation to compute gradients for real-valued parameters. Instead, we use the straight-

through estimator (STE) [38], which is a simple but effective technique for backpropagating through threshold functions (e.g., the sign function):

✶ where L is the loss function. This strategy allows us to maintain the gradient flow and ensure the differentiability of our framework. During training, we maintain and update real-valued parameters rather than directly learning binary parameters. We stop propagating gradients when θ is a large value. This also constrains the value to the range {-1 , 1 } preventing the divergence of the scale.

Binary feature encoding We propose binary feature encoding, a feature encoding scheme with binary parameters. Instead of real-valued parameters, binary parameters are used for encoding local features in a specific data structure, which are then employed to represent the radiance fields. While real-valued parameters require expensive floating-point data for representation, binary parameters can be expressed in a single bit (1-bit), leading to a significant reduction in storage overhead. Here, we describe how to implement this strategy in the feature grid Φ θ . We first adopt binarization operation, described in Eq. 4, to the real-valued grid parameter θ :

where θ ′ denotes the binary grid parameter of binary feature grid Φ θ ′ . Next, we linearly interpolate these binary parameters depending on the given coordinate x :

where x is an input coordinate, and f denotes the encoded feature value. We utilize the STE [38] to propagate the gradients into the feature grid and update the grid parameters, as described in Eq. 5.

Now, our system is capable of training the feature encoding using binary parameters. This enables us to represent the feature grid with compact data, instead of using expensive floating-point data (16-bit or 32-bit). As a result, there is a tremendous reduction in the total storage size, making our radiance field representation storage-efficient.

4.2 Radiance Field Representation

For effective binary feature encoding, we design our multi-resolution feature grids with a 3D voxel grid Φ θ xyz , and three additional multi-resolution 2D planes Φ θ xy , Φ θ xz , Φ θ yz , designated to capture features along z -, y -, and x -axis respectively, inspired by NVP [39]. Upon this architectural design, we efficiently encode the local features and use them as inputs of MLPs to predict color and density that represent the radiance fields via volumetric rendering. Note that all feature grids are implemented using a hash encoding [20] for efficiency.

2D-3D hybrid multi-resolution feature grid We intend two types of feature grids to contain the feature information in different manners. Despite the effectiveness of the 3D feature grid, the 3D grid is more severely impacted by hash collisions at higher resolutions [20], which leads to limited performance. Thus, we still need to supplement fine-grained components to improve the performance further and incorporate 2D feature grids that alleviate hash collision impact. Since the 3D feature grid encodes the main components, the 2D feature grids efficiently reinforce the feature information with fewer parameters.

Feature evaluation Here, we describe the details of binary feature encoding processes to derive the local features, based on an input 3D coordinate x = ( x, y, z ) .

For the 3D feature grid, we compute the local feature by tri-linearly interpolating binary parameters for each level of resolutions, and concatenating them:

where f xyz is the computed feature from the 3D grid, l denotes the grid level and L indicates the number of grid levels.

In a different way, we perform the binary feature encoding across each axis for the 2D feature grids. Firstly, we project the 3D coordinate x along each axis to obtain the projected 2D coordinates

x xy = ( x, y ) , x xz = ( x, z ) , and x yz = ( y, z ) . Next, we adopt bi-linear interpolation to extract features from these three projected 2D coordinates. Then, we acquire the feature value for each level of resolutions and concatenate them:

where f xy is the computed feature from the 2D grid across z -axis, m denotes the grid level, and M indicates the number of grid levels. The feature encoding steps for other features f xz and f yz operate in the similar manner.

Network architecture Finally, all these features are concatenated as f and fed into MLPs to predict the color c and density values σ . We utilize two MLPs each for density prediction and color prediction:

where e presents embedded feature and γ ( x ) is the sinusoidal positional encoding [1].

4.3 Loss

Reconstruction loss According to the volumetric rendering process described in Sec. 3, we can render the RGB pixel values along the sampled rays and optimize them through the color and density values in Eq. 10:

where r denotes the sampled ray encouraged by the occupancy grid. The efficient ray sampling through the occupancy grid allows us to focus on non-empty space.

Sparsity loss For accelerating the rendering speed, it is important to model the volumetric scene sparsely to skip the ray sampling in the empty area using the occupancy grid. Thus, we regularize the sparsity with Cauchy loss [15, 40]:

The overall training loss for our radiance field model is defined as L = L recon + λ sparsity L sparsity, where λ sparsity is the hyper-parameter for sparsity loss. We set λ sparsity = 2 . 0 × 10 -5 in this work.

5 Experiments

We conduct experiments on various benchmark datasets to verify the compactness of our radiance field representation. We evaluate the quantitative and qualitative results against prior works and analyze different architectural design choices.

5.1 Experimental Settings

Datasets We adopt representative benchmark novel-view synthesis datasets. We use two synthetic datasets: the Synthetic-NeRF dataset [1] and the Synthetic-NSVF dataset [13]. Both datasets consist of eight object scenes rendered with 100 training views and 200 test views at a resolution of 800 × 800 pix. We also employ the Tanks and Temples dataset [41] that consists of five real-world captured scenes whose backgrounds are masked [13]. The training and test views are rendered at a resolution of 1920 × 1080 pix.

Baselines We have three types of baseline methods, (a) data structure-based methods, (b) compression-based methods, and (c) implicit-based methods. Firstly, we compare our radiance fields with state-of-the-art radiance field representations adopting efficient data structures (DVGO [11, 12], Plenoxels [15], TensoRF [17], CCNeRF [18], Instant-NGP [20], and K-Planes [19]), which succeed to reconstruct high-quality 3D scenes with fast convergence speed. Specifically, we are interested in

VQ-DVGO

Ours-S8

Ours-B8

VQ-TensoRF

TensoRF-VM

CCNeRF-CP

Instant-NGP

TensoRF-CP

Plenoxels

DVGO

Ours

-S2

Ours-S2

Ours-S8

Ours-B8

VQ-DVGO

Re:DVGO

VQ-TensoRF

Re:TensoRF

TensoRF-VM

Plenoxels

DVGO

CCNeRF-CP

Instant-NGP

K-Planes

NeRF

Ours-B2

Ours-S8

Ours-B8

VQ-DVGO

Re:DVGO

VQ-TensoRF

Re:TensoRF

TensoRF-CP

Instant-NGP

TensoRF-VM

DVGO

K-Planes

K-Planes-hybrid

K-Planes-explicit

Re:Pleonxels

Re:DVGO

VQ-Plenoxels

Re:TensoRF

VQ-Plenoxels

Re:Plenoxels

NeRF

Mip-NeRF

NeRF

TensoRF-CP

Mip-NeRF

Ours-B2

DWT-TensoRF

Ours-B2

DWT-TensoRF

DWT-TensoRF

Ours-S2

Ours-B4

Ours-B4

Synthetic-NeRF Dataset

Synthetic-NSVF Dataset

Tanks and Temples Dataset

Size in log (MB)

Size in log (MB)

Size in log (MB)

PSNR (dB)

PSNR (dB)

PSNR (dB)

Mip-NeRF

37

28.6

33

32

口

.

36

35

28.4

28.2

#

31

34

28.0

27.8

33

27.6

30

32

27.4

31

0

100

101

102

103

0

100

101

102

103

0

100

101

102

103

the models that demonstrate stable performance across multiple datasets and can train within one hour when running on a conventional single GPU. Moreover, we evaluate the compression methods that require post-optimization steps (Re:NeRF [34] and VQRF [36]) and masked wavelet transform (DWT-NeRF [37]) to compress existing voxel-based models. Furthermore, we also compare with representative implicit radiance field models (NeRF [1] and mip-NeRF [24]).

Implementation details We implement our feature grid using hash encoding [20]. We use 16 levels of multi-resolution 3D feature grids with resolutions from 16 to 1024, while each grid includes up to T 3D feature vectors. We also utilize four levels of multi-resolution 2D feature grids with resolutions from 64 to 512, while each grid includes up to T 2D feature vectors. We use { log 2 ( T 2D ) , log 2 ( T 3D ) } = { 15 , 17 } for our small model denoting Ours-S and { log 2 ( T 2D ) , log 2 ( T 3D ) } = { 17 , 19 } for our base model denoting Ours-B. Our model variants are scaled by the dimension of feature vectors per level of 2, 4, and 8. For instance, Ours-B2 denotes a base variant with feature dimension F = 2 . We exploit two types of MLPs with 128-channel hidden layers and rectified linear unit activation on them, one for density prediction with one hidden layer and another for color prediction with two hidden layers. The total storage capacity of MLPs varies from 0.06 to 0.11 MB, depending on the input feature dimension. The spherical harmonics basis function is used to encode the directional information. Our implementation is based on Instant-NGP [20] using the occupancy grid implemented in NerfAcc [42]. We optimize all our models for 20K iterations on a single GPU (NVIDIA RTX A6000). It takes approximately 6, 9, and 14 min of average time to train scenes of the Synthetic-NeRF dataset for Ours-B2 to Ours-B8 models, respectively. We use the Adam [43] optimizer with an initial learning rate of 0.01, which we decay at 15K and 18K iterations by a factor of 0.33. Furthermore, we adopt a warm-up stage during the first 1K iterations to ensure stable optimization.

5.2 Comparison

We measured the storage size of each method and evaluated the reconstruction quality (PSNR, SSIM) on various datasets. Note that we present the average scores of all scenes in each dataset; scene-wise full scores are reported in the appendix. Fig. 2 summarizes quantitative evaluations of our radiance field representation, compared to baseline data structure-based, compression-based methods, and implicit-based methods.

Data structure-based approaches Fig. 3 shows the qualitative evaluations of our model, compared to data structure-based methods. Our method successfully demonstrates superior reconstruction performance against SOTA data structure-based methods with comparable training time and remarkably small storage size; especially Ours-S2 requires only within 0.5 MB. While baseline approaches

CCNeRF-CP

TensoRF-CP

Instant-NGP

TensoRF-VM

Ours-S2

Ours-B2

Ours-B4

Ours-B8

Ground truth

35.06 /  0.4 MB

36.01 /  1.3 MB

36.91 /  2.7 MB

37.44 /  5.6 MB

31.30 /  1.4 MB

33.81 /  3.7 MB

35.77 / 24.5 MB

34.92 / 66.8 MB

mic

require large storage space to achieve high performance, all our models are sufficient to exceed them by less than 6 MB. In particular, Ours-B2 outperforms the reconstruction quality of almost baselines with a much smaller storage size of within 1.5 MB. It takes 6.1 min to accomplish this without any temporal burden to train our model in a compact manner. Although our small models are enough to accomplish outstanding performance, we adopt larger models to attain higher quality. As a result, both Ours-B4 and Ours-B8 jump to excessive reconstruction quality, outperforming state-of-the-art models. Accordingly, the results indicate that our model contains the most storage-efficient data structure for radiance field representation, which also does not require much computational cost.

Compression-based approaches We also compare our models with SOTA compression-based methods. Although these approaches make the existing data structure-based models (e.g., DVGO, Plenoxels, and TensoRF-VM) highly compact by compressing the optimized model, our binary feature encoding model outperforms highly compressed data structure-based models in terms of reconstruction performance and storage usage. Specifically, while compressed TensoRF-VM models (Re:TensoRF-High [34], VQ-TensoRF [36], and DWT-TensoRF [37]) preserve most high performance among these compression works, Ours-B4 model has superior reconstruction quality only with 2.8 MB of storage capacity, which is smaller than the compressed TensoRF-VM models. Consequently, the results verify that our binary radiance field models accomplish higher compactness than compressionbased models.

Implicit-based approaches In addition, we evaluate implicit representations with a few number of parameters due to the efficient structure of MLPs. Although their simple structure results in a small storage size, our models achieve significantly higher performance with a similar storage size.

Benefits of binary radiance field Note that our approach differs from post quantization-based approaches [14, 33, 34] that demonstrate bounded performance from original features and necessitate additional pre-/post-processing. It is noteworthy that our method learns binary features during training without sacrificing the rendering quality. Our approach also differs from compression-based approaches [34, 36, 37] that require additional computations to compress or decompress the radiance field. Instead, our learned binary feature can be directly interpolated to render realistic scenes.

5.3 Ablation Study

Feature grid design We analyze the impact of the feature grid designs for scene representation and verify that our 2D-3D hybrid feature grid effectively improves the reconstruction performance. We compare three architectures with similar sizes: (a) tri-plane representation, (b) voxel representation, and (c) our 2D-3D hybrid representation. Table 1 demonstrates the effectiveness of our 2D-3D hybrid representation. Compared to the tri-plane and voxel grid, our hybrid feature grid enhances the radiance field reconstruction quality with a similar number of parameters. As a result, this confirms that our architectural choice for the feature grid allows us to achieve more compact feature encoding.

(a) w/o sparsity loss

(b) w/ sparsity loss

B1

B2

B4

B8

S1

S2

S8

Synthetic-NeRF

33

PSNR (dB)

32

{13, 15}

31

{15,17}

{17,19}

30

{19,21}

{21,23}

29

0

100

101

Sizeinlog(MB)

Sparsity loss We investigate the effectiveness of sparsity loss that accelerates the rendering speed by regularizing the radiance fields more sparsely. We evaluate the train time and inference speed of our radiance field model, according to the use of sparsity loss. As shown in Table 2, we can improve the rendering time in both training and inference with a minor decrease in reconstruction quality. In particular, we can accelerate the 23% of rendering speed for Ours-B8 model by adopting sparsity loss. The visualization of the occupancy grid, including the bitmap for the empty or non-empty area, also demonstrates that our model is trained sparsely due to sparsity loss, as shown in Fig. 4.

Hash table size We use hash encoding to construct our 2D/3D feature grids, which are restricted in their scale by the hash table size and the number of feature vectors per level. In other words, we can further reduce the storage size for compactness or increase the storage size to improve performance by scaling the number of feature vectors in the hash table. As shown in Fig. 5, we evaluate the storage size and the reconstruction performance of different hash table sizes { log 2 ( T 2D ) , log 2 ( T 3D ) } , where T 2D and T 3D denote the hash table size of the 2D and 3D grid, respectively.

Computational cost for binarization We evaluate the training time and memory requirement of a real-valued feature grid and a binary feature grid. The binarization procedure in our model adds only a small portion of training time compared to the real-valued feature grid, as shown in Table 3. Also, there is no noticeable increase in memory usage due to binarization, as shown in Table 4. This is because we binarize only several grid parameters corresponding to a ray, not a whole grid. Furthermore, there are some cases where the binary grid converges faster or requires less memory than the real-valued grid since the sparsity of the optimized scene affects the computational cost.

5.4 Applications

Dynamic scene reconstruction We finally adopted our binarization strategy in dynamic scene reconstruction. We employ TiNeuVox [22], as our base model, one of the most efficient approaches, which utilizes time-aware voxel features. Instead of conventional feature encoding for the voxel grid, we apply the binary feature encoding on the time-aware voxel features and this leads to a highly compact reconstruction of dynamic scenes, even in 0.6 MB of storage space for the synthetic jumpingjacks scene. Table 5 and Fig. 6 demonstrate that our approach is easily applicable to various feature encoding tasks, and it enables remarkably efficient representation in terms of storage size.

6 Conclusion

In this work, we have introduced binary radiance fields , a storage-efficient radiance field representation that significantly reduces the storage capacity by adopting binary feature encoding. Our experiments have verified that our approach is the most storage-efficient representation outperforming recent data structure-based models, compression-based models, and implicit-based models in terms of reconstruction quality and storage consumption. This capability enables us to access numerous radiance fields without a storage burden and leads to the expansion of the applications, such as dynamic scenes.

Limitations Despite the binarization of learnable parameters, any bit-wise operations in multilayered perception would lead to more efficient computation, requiring fewer GPU resources. We expect our approach can be boosted for the rendering by adopting the baking process.

7 Acknowledgments and Disclosure of Funding

The authors acknowledge the support provided by Samsung Advanced Institute of Technology (SAIT). This work was supported by IITP grants (RS-2023-00227993: Detailed 3D reconstruction for urban areas from unstructured images, No.2022-0-00290: Visual Intelligence for Space-Time Understanding and Generation based on Multi-layered Visual Common Sense, No.2019-0-01906: AI Graduate School Program at POSTECH, No.2021-0-01343: AI Graduate School Program at Seoul National University) funded by Korea government (MSIT).

w/o binary feature enc.

w/ binary feature enc.

(a) TiNeuVox-S

(synthetic)

(b) TiNeuVox-B

(synthetic)

(c) TiNeuVox-B

(real)

32.95 / 7.6 MB

28.28 / 47.8 MB

34.80 / 47.5 MB

33.24 / 3.9 MB

30.08 / 0.6 MB

27.75 / 3.9 MB

References

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 , 2020.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5481-5490. IEEE, 2022.

Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Zhang. Nerfren: Neural radiance fields with reflections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18409-18418, 2022.

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10318-10327, 2021.

Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5865-5874, 2021.

Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: a higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG) , 40(6):1-12, 2021.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4578-4587, 2021.

Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5480-5490, 2022.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5459-5469, 2022.

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Improved direct voxel grid optimization for radiance fields reconstruction. arXiv preprint arXiv:2206.05085 , 2022.

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems , 33:15651-15663, 2020.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5752-5761, 2021.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5501-5510, 2022.

Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16123-16133, 2022.

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII , pages 333-350. Springer, 2022.

Jiaxiang Tang, Xiaokang Chen, Jingbo Wang, and Gang Zeng. Compressible-composable nerf via rankresidual decomposition. arXiv preprint arXiv:2205.14870 , 2022.

Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, and Angjoo Kanazawa. Kplanes: Explicit radiance fields in space, time, and appearance. arXiv preprint arXiv:2301.10241 , 2023.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) , 41(4):1-15, 2022.

Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 , 2016.

Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers , pages 1-9, 2022.

Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. arXiv preprint arXiv:2210.15947 , 2022.

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021.

Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7210-7219, 2021.

Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 , 2021.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14335-14345, 2021.

Qiang Zhang, Seung-Hwan Baek, Szymon Rusinkiewicz, and Felix Heide. Differentiable point-based radiance fields for efficient view synthesis. arXiv preprint arXiv:2205.14330 , 2022.

Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5438-5448, 2022.

Ang Cao and Justin Johnson. Hexplane: a fast representation for dynamic scenes. arXiv preprint arXiv:2301.09632 , 2023.

Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, and Andreas Geiger. Factor fields: A unified framework for neural fields and beyond. arXiv preprint arXiv:2302.01226 , 2023.

Fangneng Zhan, Lingjie Liu, Adam Kortylewski, and Christian Theobalt. General neural gauge fields. arXiv preprint arXiv:2305.03462 , 2023.

Yoonwoo Jeong, Seungjoo Shin, Junha Lee, Chris Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park. Perfception: Perception using radiance fields. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Chenxi Lola Deng and Enzo Tartaglione. Compressing explicit voxel grid representations: fast nerfs become also small. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1236-1245, 2023.

Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. In ACM SIGGRAPH 2022 Conference Proceedings , pages 1-9, 2022.

Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Liefeng Bo. Compressing volumetric radiance fields to 1 mb. arXiv preprint arXiv:2211.16386 , 2022.

Daniel Rho, Byeonghyeon Lee, Seungtae Nam, Joo Chan Lee, Jong Hwan Ko, and Eunbyung Park. Masked wavelet representation for compact neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20680-20690, 2023.

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 , 2013.

Subin Kim, Sihyun Yu, Jaeho Lee, and Jinwoo Shin. Scalable neural video representations with learnable positional features. arXiv preprint arXiv:2210.06823 , 2022.

Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5875-5884, 2021.

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG) , 36(4):1-13, 2017.

Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerfacc: A general nerf accleration toolbox. arXiv preprint arXiv:2210.04847 , 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new voxel grid radiance field representation in which the feature vectors are restricted to contain binary values.  The motivation for this is to greatly reduce the storage requirements of voxel grid radiance fields.  They use the straight-through estimator to allow backpropagation through the binary values.  The representation combines a 3D hash grid with a triplane hash grid and uses trilinear or bilinear interpolation on the binary feature vectors.  They conduct experiments on the Synthetic-NeRF, Synthetic-NSVF, and Tanks and Templs object datasets and compare against many other voxel grid radiance field methods.  They show that they are able to achieve competitive rendering quality while using an order of magnitude less storage than the best-performing competitor.\n\nSTRENGTHS:\nThe idea is straightforward but novel to the best of my knowledge.  They are able to achieve high-quality renders with a large reduction in storage requirements.  It is surprising to see that a radiance field built only with binary feature vectors could perform so well.   They also outperform other methods in which the radiance field is compressed as a post-process.  This is probably because they are able to train the quantized model in an end-to-end manner, which is interesting to see.\n\nThere is also some innovation in the idea of combining a 3D voxel grid plus a triplane representation as usually only one or the other is used, as far as I know.\n\nWEAKNESSES:\nThey are missing a reference to Variable Bitrate Neural Fields:\n\nTakikawa, T., Evans, A., Tremblay, J., Müller, T., McGuire, M., Jacobson, A., & Fidler, S. (2022, July). Variable bitrate neural fields. In ACM SIGGRAPH 2022 Conference Proceedings (pp. 1-9).\n\nHowever they include other compression-based methods in the evaluation, and VBNF did not provide results on the standard benchmark datasets, so I can understand why they didn't include it in the evaluation.\n\nIt would have been nice to see videos in the supplemental material, which would helpful for appreciating the visual quality of the results, especially for dynamic scenes.\n\nQUESTIONS:\nIs the model actually being stored as binary vectors during training and inference, or are you actually using 8-bit or larger integers (for example due to the way TinyCudaNN is implemented)?   I wondered if it is actually feasible to represent binary vectors in this way using PyTorch, or if that is only a hypothetical currently.\n\nLIMITATIONS:\nThe technical limitations appear reasonable; there is no mention of broader social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new representation, binary radiance fields (BiRF), for memory-efficient novel view synthesis tasks. The representation is inspired by the binary neural network. BiRF is built upon Instant-NGP. The critical component of this representation is the binarization of real-valued feature grids, such that the resulting feature grids can store bitwise feature grids, which highly reduces the storage of NeRF models. BiRF also enhances the 3D voxel grid with three 2D plane grids, where the 2D features are incorporated to alleviate the hash collision. The training loss composes of the RGB loss and the sparsity loss. Experiments are conducted on the NeRF-synthetic dataset and NSVF dataset. Compared with state-of-the-art NeRF methods (data structure-based and compression-based methods), BiRF can obtain comparable or even higher reconstruction quality while requiring less storage. The ablation study also shows the effectiveness of introducing the 2D planes and sparsity loss.\n\nSTRENGTHS:\nThe paper is well written. The idea is simple but very effective and easy to implement. The insight of combining 3D feature grids and 2D plane grids is really cool. Overall, the method proposed in this paper is critical to the computer vision community (both the industrial and academia). For example, memory storage is an issue if we are reconstructing very large-scale scenes; there is also a demand for deploying NeRF models to mobile devices.\n\nWEAKNESSES:\nThe paper also mentioned a limitation is that it requires a longer time to train BiRF compared to its Instant-NGP counterpart, due to the binarization operation on real-valued feature grids. Moreover, I think the memory requirement during training can be higher than the non-binarized version since BiRF needs to maintain the temporal real-valued feature grids in addition to the binary feature grids.\n\nQUESTIONS:\nThe reconstruction quality of BiRF outperforms other methods by a large margin in terms of PSNR. But the reason seems not very clear since BiRF replaced the real-valued feature grids with the binary feature grids. In other words, the performance should drop (slightly) compared to the non-binary version (Instant-NGP). I think the performance gain is from the incorporation of 2D plane features. However, the authors did not provide ablations on with and without the binarization of their network architecture. I would definitely rate my score if the authors provide that.\n\nLIMITATIONS:\nSome minor typos are that l and L are not explained for Eq.(8), though it is obvious they're the number of grid levels. (the same issue for Eq (9)).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach called binary radiance fields (BiRF) for efficient storage and representation of radiance fields. BiRF utilizes binary feature encoding, where local features are encoded using binary parameters of +1 or -1. This compact encoding significantly reduces storage size and computational costs. The authors introduce a binarization-aware training scheme and extend the multi-resolution hash encoding architecture to a 3D voxel grid and orthogonal 2D plane grids.\nThe contributions of the paper are 1. The introduction of binary radiance fields (BiRF) as a storage-efficient representation that encodes features using binary parameters. 2. A binarization-aware training scheme that effectively captures feature information and updates binary parameters during optimization. 3. The demonstration of superior reconstruction performance with minimal storage space usage, achieving impressive results in static scene reconstruction.\n\nSTRENGTHS:\n1. The paper introduces a novel approach called binary radiance fields (BiRF) for representing radiance fields using binary feature encoding. This idea of employing binary parameters to represent local features in radiance fields is innovative and distinguishes it from traditional methods. The application of binarization-aware training and the extension of multi-resolution hash encoding to a hybrid structure further contribute to the originality of the approach.\n\n2. The paper demonstrates high-quality research through rigorous experimentation and evaluation. The proposed BiRF representation outperforms state-of-the-art methods in terms of reconstruction performance while utilizing significantly less storage space. The experiments conducted on various scene datasets provide compelling evidence of the effectiveness and efficiency of the proposed approach.\n\n3. The paper is well-written and effectively communicates the concepts and methodologies to the readers. The authors provide clear explanations of the key ideas, including the binary feature encoding, binarization-aware training scheme, and the hybrid structure of the feature grid. The organization of the paper enables easy comprehension of the research objectives, methodology, and experimental results.\n\n4. The paper's contributions have significant implications for the field of radiance fields and 3D scene modeling. By introducing the BiRF representation, the authors address the critical challenge of storage efficiency in radiance field models, which can greatly impact practical applications. The superior reconstruction performance achieved by BiRF, coupled with its minimal storage requirements, opens up new possibilities for real-world implementation and broader accessibility of radiance fields.\n\nWEAKNESSES:\nWhile the paper demonstrates several strengths, there are also a few areas where it could be improved:\n\nExperimental Evaluation: While the paper presents compelling results by reporting the model size and psnr, the paper lacks a quantitative evaluation and comparison of the inference speed with other relevant methods such as TensoRF and Instant-NGP. To what extent does the hybrid 3D and 2D feature grid architecture impact the inference speed & training speed of the proposed model? As the backward gradient to the grid is estimated & approximated, will the binary design affect the convergence speed?\nWhat is more, it seems the reported training speed of TensoRF is slower than the original paper.\n\nResults: In Figure 2, I find the the results of K-Planes on Synthetic NSVF dataset are significant worse than other methods, certain analysis of the inferior results should be discussed. It is crucial to consider and provide information about the training time and inference speed, as these factors play a significant role in assessing the effectiveness and practicality of the proposed NeRF model.\n\nHash Collision: In original Instant-NGP, the hash collision is explicit handled as the largest gradients—those most relevant to the loss function—will dominate the optimization, the multi-scale will also alleviate it as collision is statistically unlikely to occur simultaneously at every level for a given pair of points. For this proposed BiRF, how about the situation of hash collision compared with instant-ngp? With binary code, it seems the multi-scale design will be less effective to prevent hash collision.\n\nBinarization of learnable parameter: The binarization with straight-through-estimator is a special case vector quantization or discrete representation, which has been explored in some recent NeRF research [1,2]. The paper could benefit from a more thorough discussion of these related works. Besides, there appears to be some duplication of content between lines 165-168 and lines 148-151. Streamlining these sections would improve the clarity of the paper.\n\nReference:\n\n[1] Variable bitrate neural fields\n\n[2] General neural gauge fields\n\nQUESTIONS:\n1. I would suggest the author to provide an evaluation of the inference speed with other relevant methods such as TensoRF and Instant-NGP, and analyze the effect of 3D&2D feature grid and estimated gradient.\n2. It would be helpful to discuss the potential reasons for this performance gap of K-plans and provide an analysis of the inferior results. \n3. How does the proposed BiRF model handle hash collision compared to Instant-NGP?\n4. It would be beneficial to provide a more thorough discussion of these related works and explain the specific connections and differences between the proposed BiRF approach and the existing literature.\n5. Streamlining the mentioned sections to enhance the clarity of the paper.\n\nLIMITATIONS:\nThe limitation is discussed in this work.\nThere is no concern of potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
This paper proposes a novel binary radiance fields (BiRF) which binarized the feature encoding to save memory usage of NeRF. In the experiments, the binary radiance field representation demonstrates superior reconstruction performance compared to state-of-the-art efficient radiance field models, all while requiring lower storage allocation. Notably, the proposed model achieves remarkable results in reconstructing static scenes, achieving PSNR values of 31.53 dB for Synthetic-NeRF scenes, 34.26 dB for Synthetic-NSVF scenes, and 28.02 dB for Tanks and Temples scenes. These impressive outcomes are attained using minimal storage space, with only 0.7 MB, 0.8 MB, and 0.8 MB utilized, respectively. The intention behind introducing the binary radiance field representation is to eliminate storage bottlenecks and make radiance fields more accessible for various applications.

STRENGTHS:
1. This paper is well-organized
2. Experiments are convincing and extensive.

WEAKNESSES:
1. Only binarizing the feature encoding to save the memory usage is limited, since the acceleration rate is also import in network quantization and the inference of NeRF. 

2. The proposed binarization of learnable parameter is limited of novelty. The analysis or discuss about bottleneck of NeRF quantization or binarization is lack, which is crucial.

QUESTIONS:
See weakness.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors are proposing BiRF (BInary Radiance Fields), a storage-efficient representation for neural radiance fields. The technique relies on a hybrid representation that leverages explicit feature grids (both one 3D and three 2D, each at multiple resolutions) combined with density and color MLPs. To achieve high storage efficiency, the feature grids are binarized following the technique from Binarized Neural Networks. The reconstruction loss includes a sparsity inducing loss (similarly to SNeRG and Plenoxels).\n\nThe main technical contributions of the paper are this specific storage-efficient representation of neural radiance fields with a matching training scheme and an array of comparisons against other methods.\n\nThe authors demonstrate results through a number of quantitative evaluations on the Synthetic-NeRF, Synthetic-NSVF and Tanks&Temples datasets and against multiple baselines (fast ones: DVGO, Plenoxels, TensoRF, CCNeRF, Instant-NGP and K-Planes and compact ones: Re:NeRF, VQRF):\n- the proposed representation is indeed very storage efficient, with < 1 MB at acceptable quality,\n- it generally delivers quality reconstructions with lower storage requirements compared to either efficient or compressed alternatives,\n- training time remains reasonable (only behind Instant-NGP, DVGO and Plenoxels depending on the operating point).\n\nLastly, the storage efficiency of the technique is showcased to be relevant for an application on dynamic scenes.\n\nSTRENGTHS:\nStorage efficiency is an important aspect of making neural radiance fields more practical for concrete applications. While there are several neural radiance fields approaches focusing on trading rendering (and training) speed at the expense of storage efficiency and other approaches tackling storage efficiency separately by compressing a trained representation, the proposed approach addresses storage efficiency directly without necessitating a post-processing step after training and also without sacrificing quality.\n\nThe approach offers competitive operating points in terms of quality v.s. storage, while offering reasonable training time.\n\nThe method is conceptually simple and the paper does a solid job at presenting it, demonstrating its value through both quantitative and qualitative experiments, including a large number a baselines to compare against. The ablations are also thorough.\n\nWEAKNESSES:\nNovelty is limited as this is essentially an application of Binarized Neural Networks to neural radiance fields approaches.\n\nAs quality is one of the claims of the approach, a comparison against a few non-compressed and non-optimized baselines (e.g. original NeRF, mip-NeRF) would have made sense.\n\nThe approach is exploring with good results another trade-off compared to the efficiency-oriented techniques that sacrifice storage. However, the only mention of rendering speed (at test time) are in the ablation on the sparsity loss and in the limitations section. The apparently unoptimized implementation is another weak point of the submission and a comparison of rendering speed, especially against the considered baselines would have also made sense.\n\nMinor corrections:\n- l.104 Pleoxels -> Plenoxels\n- l.135 consider the -> consider an\n\nQUESTIONS:\nFollowing the above, would the authors be able to include a thorough comparison of rendering speed against the considered baselines?\n\nBinarized Neural Networks, which this paper is following, was using in some of their experiments stochastic binarization (as a form of regularization), is this something that has been considered?\n\nTable 3 in the supplementary material suggests increasing, have the authors tried going beyond {2^19,2^21}? Are the results plateau-ing?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}