Compressible-composable NeRF via Rank-residual Decomposition
Authors: Jiaxiang Tang
Xiaokang Chen
Jingbo Wang
Gang Zeng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural Radiace Fields
Tensor Rank Decomposition
Compression
Composition.

Abstract: Neural Radiance Field (NeRF) has emerged as a compelling method to represent 3D objects and scenes for photo-realistic rendering. 
However, its implicit representation causes difficulty in manipulating the models like the explicit mesh representation.
Several recent advances in NeRF manipulation are usually restricted by a shared renderer network, or suffer from large model size. 
To circumvent the hurdle, in this paper, we present a neural field representation that enables efficient and convenient manipulation of models.
To achieve this goal, we learn a hybrid tensor rank decomposition of the scene without neural networks. 
Motivated by the low-rank approximation property of the SVD algorithm, we propose a rank-residual learning strategy to encourage the preservation of primary information in lower ranks. 
The model size can then be dynamically adjusted by rank truncation to control the levels of detail, achieving near-optimal compression without extra optimization.
Furthermore, different models can be arbitrarily transformed and composed into one scene by concatenating along the rank dimension.
The growth of storage cost can also be mitigated by compressing the unimportant objects in the composed scene. 
We demonstrate that our method is able to achieve comparable rendering quality to state-of-the-art methods, while enabling extra capability of compression and composition.
Code is available at https://github.com/ashawkey/CCNeRF.

Compresiiblecomppsable NeRF via Rank-residual Decomposition
Jiaxiang Tang', Xiaokang Chen', Jingho Wang", Gang Zeng 1.3
School of Intelligence Science and Technology, Peking University Chinese University of Hong Kong Inteeliggent Terminal Key Laboratory of SiChuan Province {tjx, pkucxk)@pku edu. en, xj020@ie. cuhk cubkk.. edu. hk,, zengêpku. mng@pku.edu.com edu.
Abstract
Neural Radiance Field (NeRF) has emerged as a compelling method to represent 3D objects and scenes for photo-realistic rendering. However, its implicit represen- tation causes difficulty in manipulating the models like the explicit mesh represen- tation. Several recent advances in NeRF manipulation are usually restricted by a shared renderer network, or suffer from large model size. eircumvent the hurdle, in this paper, we present neural field representation that enables efficient and con- venient manipulation of models. To achieve this goal, we learn hybrid tensor rank decomposition of the scene without neural networks. Motivated by the low-rank approximation property of the SVD algorithm, we propose rank-residual learning strategy to encourage the preservation of primary information in lower ranks. The model size can then be dynamically adjusted by rank truncation to control the levels of detail, achieving near-optimal compression without extra optimization. Fur- thermore, different models can be arbitrarily transformed and composed into one scene by concatenating along the rank dimension. The growth of storage cost also be mitigated by compressing the unimportant objects in the composed scene. We demonstrate that method is able to achieve comparable rendering quality to seat-ooftth--an methods, while enabling extra capability of compression and composition. Code available https: //gi thub. com/ wwwwwaay/aaaaa/a CCN.R..
1 Introduction
Photo-realistic rendering and manipulation of 3D scenes have been long standing problems with numerous real-world applications, such as VR/AR, computer games, and video creation. Recently, the volumetric Neural Radiance Field (NeRF) representations [26, 1, 8, 27] show impressive progress in rendering photo-realistic images with rich details. However, due to this implicit representation of geometry and appearance, manipulating the underlying scenes encoded by NeRF still remains challenging problem. To solve this problem, some works [21, 45, 19] introduce scene-specific features and scene agnostic rendering network, that scenes trained with shared rendering network can be composed together. However, the constrained and biased capability of these rendering networks causes difficulty in extending to various objects scenes. New objects have to be trained with a fixed rendering network to be compatible with the old objects. Other works [36] discard the rendering network and adopt an no-neural-network NeRF representation, which is more convenient to manipulate the reconstructed scenes and is still able to render high-quality images. Nevertheless, the large storage requirement for each single model is detrimental composing complex scenes with lots of objects.
We present a novel approach that allows efficient and convenient manipulation of scenes represented with our model. Two aspects should be fulfilled to achieve this goal. The first that we can dynamically adjust the model size to support different levels of detail (LOD) in different scenarios.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

36.0:
PSNR: 33.62
79MB
1.6MB
dynamic compression
transformation & composition
Figure 1: Compressibility and Composability of our method. We present tensor rank decomposi- tion based neural field representation, which supports model compression through rank truncation, and arbitrary composition between different models through rank concatenation. Both of these operations require no extra optimization, or any constraints in training (e.g., s,, shared renderer).
This functions similar mipmaps in graphics and requires extra optimization step. The second is that all models can be transformed and composed arbitrarily for manipulation with no constraints in training. This promises that models are always reusable, and support the most basic operations in a 3D editor like blender [9]. We name these two properties as compressibility and composability.
For the compressibility, we are motivated by the properties of Singular Value Decomposition (SVD) and High-order SVD (HOSVD) [10]. Our aim is to learn the decomposition of a 3D scene from only 2D observations like TensoRF [8], and further preserve the near-optimal low-rank approximation property. We propose a simple and flexible tensor rank decomposition based neural radiance field, and rank-residual learning strategy. Each 3D scene is modeled by 4D feature volume, which can be described with a set of rank components and a matrix storing the weights for each feature channel. The rank components either vector- matrix- -baeed corresponding to the CANDECOMP/PARAEAS (CP) decomposition [14, 4] and a less compact triple plane variant. We introduce a residul learning strategy to encourage the lower ranks to preserve more important information of the whole scene. Combined with an empirical cort-and-truncalt strategy. the proposed method achieves near- optimal low-rank approximation at any targeted rank. Different LODs represented with different low-rank truncations of the model, allowing dynamic trade-off between model size and rendering quality without retraining. Besides, our model contains neural networks and thus naturally supports composability. Since there are no MLP renderers in our model, we can compose different objects by simply concatenating their rank components. A transformation matrix is recorded for each object to control its position and orientation in the scene.
As demonstrated in Figure 1, we able to control each model's LOD and size in flexible range, and perform arbitrary transformation and composition of different models. Furthermore, these two properties connected together through the underlying concept of rank, and can be combined in practical use. For example, we can mitigate the growth of model size of a complex scene composed of multiple objects, by compressing the less important objects. Our contributions can be summarized as follows: (1) We propose a simple radiance field representation based on two types of tensor rank decomposition, which allows flexible control of model size and naturally supports transformation and composition of different models. (2) We design a rank-residual learning strategy to enable near-optimal low-rank approximation. After training, our model can be dynamically adjusted to trade off between performance and model size without retraining. (3) The proposed method reaches comparable rendering quality with state-of-the-ate while additionally enabling both compressibility and composability.
2 Related Work
2.1 Scene Representation with NeRF
3D scenes can be represented with various forms, including volumes, point clouds, meshes, and implicit representations [33, 34, 7, 30, 38, 25, 23, 28]. NeRF [26] proposes use 5D function

---

to represent the scene and applies volumetric rendering for novel view synthesis, achieving photo- realistic results and detailed geometry reconstruction. This powerful representation quickly receives attention and is extensively studied and applied in various fields [50, 24], such as generative settings [6, 37, 29, 5], dynamic scenes [20, 31], and texture mapping [44]. In particular, we categorize recent progress by the design of the underlying functions into three classes: neural network- based, hybrid and no-neural-netwond function to encode 3D scenes. The original NeRF [26] and most following works [1, 2, 21,, 43, 46, 35] choose this representation for its simplicity. However, the training and inference speed of such a network is generally slow due to the relatively expensive MLP computation. Therefore, hybrid representations try to reduce the size of the MLP, by storing the 3D features in an explicit data structure. Since dense 3D representation is unaffordable, different methods are explored. For example, NSVF [21] adopts sparse voxel grids, PlenOctrees [49] adopts octrees, instant-ngp [27] adopts multi-scale hashmap, and TensoRF [8] factorizes the scene into lower-rank components. Querying such hybrid representation is much faster, thus reducing training and inference time and even reaching interactive FPS. Lastly, no-neural-neer network representations attempt model the 31) scene without neural networks. Plenoxels [36] shows that only the explicit sparse voxels representation is enough to model complex 3D scenes. Our method also belongs to this representation, sharing the similar tensor rank decomposition idea to TensoRF [8]. but we focus on two additional capabilities, i.e., compressibility and composability, which are important yet usually absent previous work.
neural network- based representations typically apply an MLP, as the implicit
2.2 Tensor Decomposition and Low-rank Approximation
Decomposition of high-order tensors [18] can be considered as the generalizations of matrix singular value decomposition. The Tucker decomposition [40] decomposes a tensor into a core tensor multiplied by a matrix along each mode. The CANDECOMPPARA (CP) decomposition [14, 4]
factorizes a tensor into a sum of component rank-one tensors, and can be viewed as a special case of Tucker where the core tensor is superdiagonal. The high-order singular value decomposition [10] provides a method to compute a specific Tucker decomposition with an all-orthogonal core tensor. Low-rank approximation is a common problem that applies tensor decomposition, and has found various applications such as image compression. Although the truncated HOSVD does not hold the optimal property contrary to the truncated SVD, it still results in quasi-optimal solution [10, 41, 12), which is enough to yield a sufficiently good solution in practical uses. Tensor rank decomposition and its variants [10, 11] has been used in various vision and learning tasks [47, 8, 48]. Specifically, TensoRF [8] first leverages the CP decomposition and a Vertex-Matris (VM) decomposition to factorize neural radiance fields, but its other designs (e.g., use of MLP) disturbs the property of tensor rank decomposition and prevents it from achieving compression or composition. Instead, we focus on modeling neural radiance fields only with tensor rank decomposition, and aim to preserve the low-rank approximation property, enabling the compression of learned neural radiance field similar to the SVD compression of an image.
2.3 Manipulation and Composition of NeRF
Manipulation and Composition important for a 3D representation's practical usage. Explicit 3D representations, e.g., meshes, are natively editable and composable. However, neural network-based implicit representations like vanilla NeRF is difficult to perform such operations. NSVF [21] can composite separate objects together. but these objects have to be trained together using shared MLP, which limits its flexibility and potential usage. Later works [45, 29, 13, 19] learn boject-compositional NeRF, but are usually scene- speeific and do not allow cross-scene composition without retraining. Geometry and appearance editing [22, 42] of neural fields also requires an extra optimization step to modify the neural network- representation. With the explicit sparse voxel representation, Plenoxels [36] naturally supports direct composition of different objects, but suffers from the large storage on the dense index matrix. Our method also supports arbitrary affine transformations and compositions without extra optimization. Further, we can efficiently mitigate the model size growth due to the compact tensor rank decomposition and the compressibility.
3

---

Rvec + Rmat
Rvc
Rmat
11
8
S
U SSUUPUU U
Wx.22 UK2 uy
Figure 2: Model structure. Our model is composed of a matrix storing rank weights for different fea- ture channels, and a set of decomposed rank components. Each rank component can be either vector- matrix- based, and the ratio can be controlled to trade off between model size and performance. To query any 3D coordinate, we first project it to the decomposed vectors or matrices as denoted by the black lines, and then perform weighted interpolation. denotes concatenation along the rank dimension.
3 Methodology
3.1 Preliminaries on Neural Radiance Fields
Neural Radiance Fields (NeRF) [26] represents 3D volumetric scene with 5D) function fe that maps a 3D coordinate
(z, y, 2) and 2D viewing direction d = (0 into a volume density a and emitted color с (r.g, b). Given ray originating at o with direction d, query fe points x; o + 4,d sequentially sampled along the ray to get densities {ai} and colors {c: The color of the pixel corresponding to the ray then estimated by numerical quadrature:
Ō = Σ Tiacc
II aa),)))) = exp(-o;0;), = f,+1 - t jCi
(1)
where 6; is the step size, o; is the opacity, and T; is the transmittance. Since this volume rendering process is differentiable, NeRF can be optimized only from 2D image supervision by minimizing the L2 difference between each pixel's predicted color C(r) and the ground truth color C(r) from the image:
LNERF Σ [(((r)) (
3.2 Preliminaries on Tensor Decomposition
For 3D tensor T #xXDD, each element Trjj. E I can be represented via the Tucker decomposition [40] by:
P R
Trj.k =
Spa.
U U U
(3)
p=1 1=11=1
where E PxQxR is the core tensor, P.Q. are the number of components along each axis,
and U E #x U E I W U E DxR are the factor matrices. The CP decomposition can viewed as a special case of Tucker when P Q = and is superdiagonal [18] (i.e., Si.j + + ==j k):
Taj.k Σ s. S. 1 U
(4)
where diag(S) E R is the reduced core tensor (or rank weights). Although s usually absorbed into the factor matrices, write it out for the convenience of later discussion.

---

3.3 Decompose NeRF without MLP
Hybrid Feature Volume Decomposition. We are interested in multi- volumetric encoded as a 4D tensor T E RC HXWXD xD, where C' is the feature dimension (e.g., density, RGB values, or other features), and (H, W,D) D) the spatial resolution (usually C < min(H, W. D)). A straightforward way is to perform C' independent decompositions for each channel. However, different feature channels such as the RGB values are highly correlated in real 3D scenes. A more compact way is to share the factorized matrices like TensoRF [8], and only use different rank weights s for different channels. Therefore, we propose to represent through the CP decomposition by:
Tz.jk = S (U; UU + +
where S E RCxR is the matrix of rank weights for C' channels, and * denotes the Hadamard product. Since the above decomposition relies on rank-one vectors (ID tensors), it may require very high ranks to represent complex 3D scenes, which leads to expensive computation at each location. A less compact but more computation-frient alternative is to adopt matrices (2D tensors) to factorize the 3D scene. This variant the form of CP decomposition given by:
To.j.k S (2 U (,,)))
(6)
where U=.y E RR ,18 R WXXXX u E H are the factorized matrices along three planes, each containing R components. This variant can be comprehended by first slicing and tiling the original 3D space along each axis, and then learning CP decomposition on (r'" ITD Although this representation is less compact and takes more storage, recent works [8. 5] have shown that it is able to represent scenes with smaller R and better quality. We denote this variant as the Triple Plane (TP) decomposition. Further, we notice that for each individual rank, the underlying vector- or matrix- -bseed decomposition can be selected independently. Therefore, hybrid variant (HY) that combines the above CP and TP decomposition is proposed. We can flexibly adjust the ratio of two decompositions by R- Rvee + Rmma to trade off between model size and performance. The model structure is illustrated in Figure 2.
Learning the Decomposition via Differentiable Rendering. For simplicity, we take the CP decom- position as an example. Our model only consists of four tensors to optimize, i.e., S, U", U U To represent scenes through neural radiance fields, we need to learn the volume density o and color at each location. As the volume density is only dependent on the 3D coordinate x, we can represent it with one feature channel Trlemalty E R. However, the color is dependent on both the 3D location x and the 2D viewing direction d, which is a 51) function in total. To represent it within the 3D feature volumes, we adopt the spherical harmonics (SH) functions to approximate the additional 2D viewing directions dependency [49, 36]. In particular, for spherical harmonics of maximum degree maxx takes (Emax + 1)² SH coefficients to model the view-dependent color per channel. We use Tii E [f 1)² R E {r. g, b} to represent these coefficients. The density and color at each location (coordinate indices are omitted for simplicity) can then be represented with:
Eau
の donsity c"
Σ
Σ
Yim (())(((d) (d)), E g),
E=0
where cb.) is the density activation, (-) is the color activation, d is the viewing direction, and Y!" are the SH functions. Our model can then be optimized through the standard RGB loss in Equation 2.
3.4 Rank-residual Learning for Compressibility
The idea of low-rank approximation is to only keep the most important rank components, where the importance of each rank component can be represented by the singular values in the SVD algorithm. Similarly, we can define the importance of each rank component in our decomposition by the rank weights S averaged on all feature channels, and multiplied with the magnitude of three factor matrices along the rank dimension. However, the CP decomposition doesn't 't hold this low-rank approximation property [18]. If we directly sort the rank components by the rank importance and truncate the model for compression, the rendering quality drops sharply compared to the optimal model (i.e., model retrained with the same parameters), illustrated by the baseline method in Figure 3.
We propose a rank-residual learning strategy to close this performance gap and achieve near-optimal compression results. This strategy aims to simulate SVD's low-rank approximation property, where
5

---

34
Ranks
Ranks
32
50-
Channels
28
Optimal Baseline Proposed
24
22
50
Size (MB)
Figure 4: Visualization of rank importance. Ranks are sorted column-wisely based on the averaged rank importance. The rank importance is more concen- trated in the proposed method (right) compared to the baseline (left), which crucial for truncation- based compression. We use the HY model on the LEGO dataset as an example, and the dashed lines indicate where we apply rank-residun supervision.
Figure 3: Compression at any rank. Com- bined with the empirical nort-and-truncan strategy, the proposed model achieves near- optimal compression at any rank. We use the HY-S model on the LEGO dataset as an exam- ple, and the dashed lines indicate where apply rank-residual supervision.
the lower rank components contain more information and contribute more to the approximation. Since the rank components in our method can be flexibly adjusted, a direct solution train in progressive way. Suppose the total number of ranks is R, and the number of training stages is M. The total R rank components can be sequentially divided into M non-empty groups, and we denote the accumulated number of ranks for each group by Rm where E {1, 2, .M.. We start from training the first R, rank components, and after its convergence, we fix them and append the next Ra - Ri rank components to train in new stage. In the last stage, all R ranks are involved. However, this is inefficient since we need make sure each stage is fully converged before increasing the number of ranks. A more efficient way is train all stages in parallel. In particular, we simultaneously supervise the sequentially accumulated outputs from all M groups This can viewed as supervising the outputs from M truncations of the decomposition with rank-residual loss:
Cresidual
IIC(r) C. (r)
(8)
PER
he where (r) is RGB values calculated from the truncated decomposition S., U Uv U which only keeps the first Rn, rank components (taking the CP decomposition as an example). During training. each group is able to learn the residual error from previous groups, and eventually leads to the desired low-rank approximation property. Ideally, choosing M = R groups assures this low-rank approximation property to hold at any targeted rank, but usually computationally unaffordable with a large R. In practice we choose M < R, so the property only holds at those dividing ranks {Rm). For any other rank R', assuming R., < R' < Rm+1, we first keep the fully covered R,m rank components, and then the empirical sort-and- tonnncata strategy select the remaining R' R. ranks in the last group { {Rm + 1, Ry, + 2... Rm,+1 J. By selecting the top rank components with the largest average importance, it enough to keep the near-optimal low-rank approximation property any targeted ranks, as shown by the proposed method in Figure 3.
The decomposition model trained with our rank-residust learning allows dynamic adjustment of model size and rendering quality with no extra optimization. practice, we usually need different LODs to adapt to different cases, such as the texture mipmaps. While the other NeRF representations need to retrain for different LODs and store every model separately, we just train once and get a unified model for all LODs. Given targeted storage upper bound performance lower bound, only need to select the targeted rank and truncate the decomposition with the simple slicing operation.
3.5
Composability without Constraints
As illustrated in Figure any 3D object or scene represented by our model can be composed without the constraints in previous work [21, 45]. Since our model describes each 3D scene with set of rank components, composability is naturally accomplished by concatenating along the rank dimension and summing up the number of ranks = > Rn, where n denotes the object index. In practice, this is implemented by appending new rank components to parameter list, so that models with different

---

68 x 612 (MB)
1.5 x 9 13.5 (MB)
Figure 5: Compressing a scene composed of multiple objects. For scene composed of lots of different objects, can compress the less important objects to achieve better efficiency and less storage with a little sacrifice of rendering quality.
resolution and decomposition forms can be composed together. For each object, record its number of ranks so we can still distinguish it from the whole scene. Arbitrary affine transformations of individual objects which includes the translation t, E R2, rotation R,, E SO(3) and scale 8n E RR3 We warp the ray into each object's coordinate system before querying the density {or,n) } and color (ci,)) the sampled points:
supported by recording transformation matrix T,, RAx4 for each object,
iin Ci, (T,,,, R,
Here we assume the coordinate x; is homogeneous, and the viewing direction is represented by unit vector. correctly handle the occlusion between different objects, we still sample one ray per pixel, and perform the composition at each sample point by:
a; = 1- exp( -5, E Tim)) c; c= N I (o((o..))....
(10)
PN
We sum up the density from all objects to calculate opacity, and weight the color by the density after the softmax function pNN. The rendering formula in Equation can then be applied to calculate the pixel color. Although the model size and rendering time grows linearly with the total number of ranks (complexity of the scene), we show in experiments that the compression property can be applied to mitigate the growth and improve efficiency.
4 Experiments
4.1
Implementation Details
The model is implemented with the PyTorch framework [32]. The degree for SH coefficients is 3, which equals 48 channels for the color feature. We use the Adam optimizer [16] with initial learning rate of 0.02 for the factorized matrices, and 0.001 for the singular values. All the experiments performed on one NVIDIA V100 GPU. The resolution of the feature grid is determined by the total number voxels and the bounding box, where is increased from 128 to 300 for HY models and 5003 for CP models in early training steps. To accelerate rendering, we adopt the binary occupancy mask pruning technique as in [8, 27], and use separate rank components for density and color to avoid unnecessary querying in empty space. This occupancy mask is also used to shrink the initial bounding box for more precise modeling. We mainly carry out experiments on the -ynthsytt dataset [26] (CC BY 3.0 license) and the Tanks and Temples dataset [17] (CC BY BY-NC-SA 3.0 license). Please check the supplementary materials for more details.
4.2 Compression Results
Firstly, We evaluate the compressibility of our model. Since the color components cost most of the total storage, we mainly focus on compressing the color components, and keep the density components fixed. Given the number of ranks R, first train two models with the original loss
7

---

compress
Bazeinn
Proposed
Propooed
Figure 7: Visualization of composition. We
Figure 6: Visualization of compression. The baseline method deteriorates signifi- cantly, while our proposed method remains high rendering quality.
show composition between different models with method. Our method can successfully handle occlu- sion and remain high-quality rendering.
Table 1: Compression Results. We report the PSNR for different compression strategies. CP, HY and HY-S denotes three model settings with different range of ranks. Ranks are denoted by Ricessty R -Rpeler / /Rallor We emphasize how the proposed method improves the truncate baseline, especially at highly compressed conditions.
Model
Ranks
Resolution
Size (MB)
Optimal
Baseline
Proposed
96/0-384/0
500
4.4
30.78
30.78
30.55 (-0.23)
96/ 96/0-288/0
500
3.8
30.68
28.78
30.46 (+1.68)
CP
96/ 06/0-192/0
500
3.2
30.38
26.95
30.15 (+33 20)
96 06/0-96/0
500
2.7
29.78
24.97
29.53 (4-4.56)
96/0-9666a
300
68.9
31.54
31.54
31.22 (-0.32)
96/0-96/32
300
35.2
31.36
27.57
31.09 (+3.552)
96/0-96/16
31.04
25.66
30.83 (+5..17)
96/0-96/4
300
5.7
30.40
24.24
30.13 (+5.89)
96/0-96/0
300
1.5
29.49
23.54
29.30 (+5.776)
64/ 64/16-256/
300
88.0
32.43
32.43
32.36 (-0.07)
61/16-192/15
300
70.8
32.42
30.63
32.35 (+1.72)
61/16-128/32
300
53.7
32.31
28.50
32.29 (+4.09)
64/16-64/16
300
36.5
31.96
26.30
31.94 (+5.64)
LNERF in Equation 2 and our rank-residual loss Cresidual in Equation 8. We denote them as Mn and any targeted rank r S R to compress, we design three strategies verify whether the proposed compression is netr-optimal: (1) Retrain a model M, with LNERF at the given rank. This requires a retraining from scratch, and can be viewed as the optimal compression result at the given rank. (2) Sort and truncate the rank of the original model M to M which can be viewed as the baseline compression result. (3) Sort and truncate the rank of the proposed model Mours to Mours We denote these three settings 'Optimal', 'Baseline", and 'Proposed' respectively.
M respectively.
The quantitative results are listed in Table We find that the performance of the baseline method degrades significantly compared to the optimal method, whereas our proposed method is comparable the optimal method at all targeted ranks. The visualization in Figure demonstrates how the baseline model gradually deteriorates compared to the proposed model. Note that at the the right-most column where the baseline model not compressed and the same as the optimal model, the rendering quality of the proposed model is hard to discern from the optimal model. In Figure 3, we show that even though our model only supervised at 5 discrete ranks, can remain good compression quality at any other ranks. Figure 3 provides explanation for the compressibility of our model. With the rank-residual learning, the rank importance more concentrated to the lower ranks, which benefits the low-rank approximation.

---

Table 2:
Comparison with recent methods. Our method achieves comparable results while enabling both capability of compression and composition.
Capability
Synthetic FeettettieNeRRFFeee
Temkses
Method
Size (MB)
Composable
Compressible
PSNR
SSIM
PSNRT
SSIM
SRN [38]
X
X
22.26
0.846
21.10
0.847
NeRF [26]
5.0
X
31.01
0.947
25.78
0.864
NSVF [21]
X
31.75
0.953
28.48
0.901
SNeRG [15]
1771.5
X
X
30.38
0.950
PlenOctrees [49]
1976.3
V
X 31.71
27.99
0.917
Plenoxels [36]
778.1
X 31.71
27.43
0.906
DVGO [39]
612.1
X
X 31.95
0.975
28.41
0.911
TensoRF-CPP ConsoRF-CP-384 [8]
3.9
X
X
31.56
0.949
27.55
0.897
T [8]
71.8
X
x
33.14
0.963
28,
0.920
Instant-NGP [27]
63.3
X
X
33.18
Ours-CP
4.4
30.55
0.935
27.01
0.878
Ours-HY-S
68.9
/
31.22
0.947
27.5
0.900
Ours-HY
32.37
0.955
28.08
0.913
4.3
Composition Results
In Figure we demonstrate the composability of our method. Without extra optimization, we are able to perform affine transformation and composition of different models like composing meshes in a 3D editor. Note that our model can correctly handle the occlusion and collision between different objects. In Figure 5, combine the compression capability of our model in scene composition. In complex scene composed of lots of objects, we can compress the less important objects to make a trade-off between the model size and rendering quality.
4.4 Comparisons and Discussion
Rendering Quality. We compare our method with some recent works in Table 2. We focus on the extra capabilities facilitate practical applications, but boosting the rendering quality over the previous state-of-the-arts, since the vanilla NeRF [26] already reaches photo-realistic rendering in most cases. Although the performance of the proposed method is not the best, the simple model design with the extra compressibility and composability is unique and enables various applications.
MLP Renderers. As discussed in [8, 27], the absence of small MLP renderer network generally leads to worse performance, especially for the specular details. However, these renderers are trained separately and cannot be shared across scenes unless explicitly restricted, which causes inconvenience or limits the potency composition. We therefore discard the MLP at the cost slightly worse rendering quality, but facilitates the composition and compression.
Limitations. Although our method can correctly compose the geometry multiple objects, we don't consider the lighting conditions. Our model bakes lighting conditions into the color like the vanilla NeRF, and cannot perform re-lighting to achieve consistent lighting effect after composition. A future direction is to integrate the reflectance models [3, 52]. Besides, only support modeling bounded objects for now. A background model [51] can be combined to simulate unbounded scenes.
Conclusion
In this work, we present novel compressible and composable neural radiance field representation. Our model is designed to be simple and flexible, yet still being effective enough to render photo- realistic images. A rank-reste residual learning strategy is proposed to enable near-optimal low-rank approximation, which allows dynamic adjustment of the model size to support different LODs in different scenarios. All models represented with our method can be arbitrarily transformed and composed together like in a 3D editor. Powered by these properties, we are able to efficiently and conveniently manipulate complex scenes with multiple objects. We believe our method will further facilitate the NeRF-based scene representation in real-world applications.

---

Acknowledgements. This work is supported by the National Key Research and Development Program of China (2000YFB1788002) National Natural Science Foundation of China (61632003, 61375022, 61403005), Grant Province, Beijing Advanced Innovation Center for Intelligent Robots and Systems (2018IRS1) 1), and PEK-SenseTimme Joint Laboratory of Machine Vision.
of Intelligent Terminal Key Laboratory of SiChuan
References
[1] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brually and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields, 2021.
[2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. arXiv preprint arXiv:21 2021.
[3] Mark Boss, Raphael Braun, Varun Jampani, Jonathan Barron, Ce Liu, and Hendrik P.A. Lensch. Nerd: Neural reflectance decomposition from image collections. In ICCV, 2021.
[4] Douglas Carroll and Jih-Jie Chang. Analysis of individual differences in multidimensional scaling via an n-way generalization of "eckart-young" decomposition. Psychometrika, 35(3):283-
319, 1970.
[5] Eric R Chan, Connor Z. Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry- aware 3d generative adversarial networks. arXiv preprint arXiv:2112.07945 2021.
[6] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, pages 5799-5809, 2021.
[7] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, pages 5939-5948, 2019.
[8] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. arXiv preprint arXiv:2203.09517, 2022.
[9] Blender Online Community. Blender 3DD modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
[10] Lieven De Lathauwer, Bart De Moor, and Joos Vandewalle. A multilinear singular value decomposition. SIAM journal on Matrix Analysis and Applications, 21(4):1253-1278 2000.
[11] Lieven De Lathauwer. Decompositions of higher-order tensor in block terms-part ii: Defi- nitions and uniqueness. SIAM Journal on Matrix Analysis and Applications, 30(3):103--1066 2008.
[12] Lars Grasedyck. Hierarchical singular value decomposition of tensors. SIAM journal on matrix analysis and applications, 3 31(4):202922544, 2010.
[13] Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503. 2020.
[14] Richard A Harshman et al. Foundations of the parafac procedure: Models and conditions for an" explanatory" multimodal factor analysis. 1970.
[15] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV, pages 5875 5875-5884, 2021.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[17] Arno Knapitsch, Jaesik Park, Qian-Yi Yi Zhou, and Vladlen Koltun. Tanks and temples: Bench- marking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017.
10

---

[18] Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Rev., 51:455-500, 2009.
[19] Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editable feature volumes for scene rendering and manipulation. arXiv preprini arXiv:2204. 10850, 2022.
[20] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, pages 1498-6508, 2021.
[21] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NeurIPS, 2020.
[22] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In ICCV, pages 5773-5783, 2021.
[23] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images, arXiv preprint 2019.
[24] Ricardo Martin-Bruall. Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Doso- vitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, pages 7210-7219, 2021.
[25] Ben Mildenhall, Pratul Srinivasan, Rodrigo Ortiz-Cayon Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
[26] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[27] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. w:Xi::2001.05989 January 2022.
[28] Jacob Munkberg. Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. arXiv preprint arXiv:2111. 2021.
[29] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In CVPR, pages 11453-1 1464, 2021.
[30] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, pages 165-174, 2019.
[31] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. In ICCV, pages 5865-5874, 2021.
[32] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 32, 2019.
[33] Charles R Qi, Hao Su. Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. Volumetric and multi-view cnns for object classification on 3d data. In CVPR, pages 5648-5651 2016.
[34] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, pages 652-660, 2017.
[35] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In ICCV, pages 14335-14345, 2021.
11

---

[36] Sara Fridovich- and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[37] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. NewrIPS, 18:20154-20166 2020.
[38] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awar neural scene representations. In NeurlPS, 2019.
[39] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. arXiv preprint arXiv:2111.11221 2021.
[40] Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika. 31(3):279-11, 1966.
[41] Nick Vannieuwenhoven, Raf Vandebril, and Karl Meerbergen. A new truncation strategy for the higher-order singular value decomposition. SIAM Journal on Scientific Computing, 34(2):A1027- A1052, 2012.
[42] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and- image driven manipulation of neural radiance fields. arXiv preprint arXiv:21 2021.
[43] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint Ar0iv:2106. 10689, 2021.
[44] Fanbo Xiang, Zexiang Xu, Milos Hasan, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Hao Su. Neutex: Neural texture mapping for volumetric neural rendering. In CVPR, pages 7119-7128,
2021.
[45] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning coject-compssitional neural radiance field for editable scene render- ing. In ICCV, October 2021.
[46] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. NewrIPS, 34, 2021.
[47] Jinmian Ye, Linnan Wang. Guangxi Li, Di Chen, Shandian Zhe, Xinqi Chu, and Zenglin Xu. Learning compact recurrent neural networks with block-term tensor decomposition. CVPR. pages 9378-998888 008099377220 2018.
[48] Miao Yin, Yang Sui, Siyu Liao, and Bo Yuan. Towards efficient tensor déccompoiition-basee dnn model compression with optimization framework. CVPR, pages 10674-10683, 2021.
[49] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
[50] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In CVPR, pages 4578-4587, 2021.
[51] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[52] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG), 40(6):1-18, 2021.
12


Reviews:
[{"review_id": "xf4TatTslIS", "reviewer": "Reviewer_i1vi", "summary": "This paper proposes a rank-residual learning strategy to obtain a radiance field representation that supports compressibility and compositionality with an acceptable diminishment of rendering quality.\n", "questions": "- What’s the training and inference time of the proposed method?\n", "limitations": "No.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n*Originality*: \n- I think this work is a nice combination of decomposition and Neural Radiance Fields (NeRFs). \n- The capability column of Table 2 clearly illustrates the difference between this work and existing works, demonstrating the new capabilities enabled by the proposed method.\n*Clarity*\n- The submission is well-written and easy to follow.\n- Related works are adequately cited and compared. For example, Line 103 discusses the similarities and differences between this work and TensoRF clearly.\n*Quality*: \n- The proposed approach is technically sound.\n- The claims are well supported by the experimental results. \n- The authors are careful and honest with the limitations (e.g., the model size and rendering time grow linearly with the total number of ranks, baked lighting, and bounded scene).\n\n### Weaknesses\n \n*Clarity*\n- Line 70-90, I think categorizing methods into a) neural network-based, b) hybrid, and c) no neural network is more accurate than a) implicit, b) hybrid, and c) implicit. Some 3D representations are called **implicit** for years even if they are stored in voxels, e.g., TSDF, where the signed distance function is actually an implicit shape function.\n- I think Figure 2 needs more annotations. For example, what’s the meaning of dotted lines in the middle sub-figure? What’s the meaning of black lines in the middle and right sub-figures? I assume the meanings of those lines are weighted interpolations. It would be nice to confirm that.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ljk2nMqWNM5", "reviewer": "Reviewer_rjyH", "summary": "\nThis paper proposes an MLP-free NeRF representation that supports both compression and composability. The representation allows for efficient and convenient manipulation of the scene of interest and resembles the Level of Detail (LOD) concept in computer graphics. The ability to compose multiple scenes or objects comes naturally from this representation.\n\nWith no neural network involved, the model represents the radiance field of a scene using tensors that are the tensor rank decomposition of a full tensor explicitly expressing the radiance field. The authors study two types of tensor decomposition allowing the user to control model sizes. A rank-residual learning strategy is used to support an easy trade-off between model size and quality, without any re-training. \n\nThe authors use a hybrid feature volume decomposition that mixes the more expensive CP decomposition and the more compact TP decomposition. The mixing ratio controls model size vs. quality. \n\nCompared with spatial locations XYZs, viewing directions are less “native” to the voxel grid (or tensor) representation, so the authors represent them using the Spherical Harmonics (SH) functions.\n\nBecause the CP decomposition does not come with the property of “rank importance,” the authors propose a rank-residual learning strategy to learn the coefficient residuals either sequentially or in parallel. \n\n====== POST-REBUTTAL UPDATE ======\n\nI read the authors' rebuttal that addresses my concerns reasonably. I appreciate the extra experiments, too. Overall, I'm willing to raise my rating to Weak Accept, to the best of my non-expert knowledge. This of course is conditioned on that the authors will add these new experiments to their final version and add necessary clarifications.", "questions": "\n\nI understand how the rank residual learning works for the sequential case but want the authors to clarify how the parallel computation works for this case. In other words, would a parallel computation lead to exactly the same set of coefficients as a sequential computation does? \n\nWhich of CP and TP decompositions does a better job of modeling specularities?\n\nAre there special decomposition constraints that can be applied to the decomposition such that view-dependent effects are prioritized and preserved? \n\nAlso related to view-dependent effects, I think the higher degree you use for SH, the more accurate view-dependent effects you get since the view directions are more “concentrated” and hence more accurate. But with an SH degree of 4, I don’t think you will be able to get smoothly moving specularities. Have the authors tried higher degrees (I’m aware of the squared growth of the number of SH coefficients)? This sounds like a meaningful ablation study to do.\n\n\n", "limitations": "Yes, the authors mention baked-in lighting as a limitation. ", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe paper offers a fresh perspective of NeRF, similar to Plenoxels’: radiance fields are not necessarily represented by a neural network. A full voxel grid representation is either low-resolution or expensive, but using tensor decomposition ameliorates this problem. \n\nThe major weakness of this paper is that it has not studied view-dependent effects such as specular highlights. As the figure and video of the “drums” scene show, the compressed model struggles to produce photorealistic or smoothly varying specularities. Since the main use of NeRF is view synthesis where view-dependent effects are the first-class citizen, studying how the tensor decomposition methods affect modeling view-dependent effects. For instance, which of CP and TP decompositions does a better job of modeling specularities? Are there special decomposition constraints that can be applied to the decomposition such that view-dependent effects are prioritized and preserved? Given the authors consider just NeRF as the application, I don’t think this can be neglected by this paper.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DLds5Sz5WMH", "reviewer": "Reviewer_x5VX", "summary": "This paper aims to compress neural field representations. To this end, the authors first propose a hybrid tensor decomposition and learn the decomposition via differentiable rendering. To encourage the primary information be learned in lowewr ranks, they introduce a novel training strategy that gradually increase the rank components to approximate the scene content.", "questions": "Please fix the problems in the weaknesses, which can improve the paper quality.\n\nThe property of this approach that impresses me is enabling the explicit controll of model capacity. Analysis on this property will make me feel better to this method.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Overall, I feel the proposed approach is novel and sound. Compared with NeRF, this approach can explicitly add rank components to increase the model capacity, enabling them to dynamically adjust the model size during training.  This is a nice property.\n- The paper is well-written and clearly presented.\n- The comparison experiments and ablation studies are sufficient.\n\nWeaknesses\n\n1. Experiments\n\n- Current experiments do not show that the proposed model has obvious advantages than NeRF or Mip-NeRF 360. What is the training time of the proposed approach.\n- How to better analyze the property that you can gradually increase the rank components to improve the model capacity. An ablation study on this property will significanly bring readers more insights.\n\n2. Writting\n\n- Table 2 says that NeRF is not composable. I do not think so. As shown in [1, 2], object-centric NeRFs can be composed. A single NeRF model can even be explicitly manipulated, as shown in [3].\n\n[1] Object-Centric Neural Scene Rendering  \n[2] Neural Scene Graphs for Dynamic Scenes  \n[3] NeRF-Editing: Geometry Editing of Neural Radiance Fields  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "7640BB9fdA6", "reviewer": "Reviewer_46de", "summary": "This paper presents a compressible NeRF model which also supports the composition of different NeRFs to form a new scene. The proposed method uses a mixture of CANDECOMP/PARAFAC (CP) decomposition and Triple Plane (TP) decomposition, which are vector-based and matrix-based respectively, contributing to a hybrid decomposition method. At the same time, in order to ensure that the selected rank components can achieve near-optimal compression results, the authors propose a rank-residual learning strategy. The entire model does not use the MLP network, and different LODs can be achieved by selecting different numbers of rank components. The rank components of different scenes are concatenated together to achieve the composition of different NeRFs.", "questions": "-- As for composition, combining different NeRF models does not pose a particular technical challenge. One can obtain a rough geometric bounding box after the training is completed, and then combine the bounding boxes to render combined images. So in Table 2, why can’t some methods, such as PlenOctree, achieve composition?\n--What is the difference between the composition that this method can achieve compared to other methods such as NSVF and Plenoxels?\n--In Table 2, why only the proposed method can compress the NeRF model? Due to it can change the compression ratio? Similar method, TensoRF, can also achieve a small model size. What’s the advantage of the proposed method?\n-- TensoRF uses the MLP network to solve the problem of worse performance caused by Sphere Harmonics (SH). The proposed method also uses the SH functions, but does not use MLP. How does the proposed method solve the problem of worse performance?\n", "limitations": "The authors have discussed the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n--The decomposition method is novel, although in some aspects, such as CP decomposition, are the same as the existing work TensoRF. But this work is based on low-rank approximation, the proposed hybrid decomposition strategy can adjust the ratio of vector- and matrix-based rank components, which is something that TensoRF can't do.\n--The method can adjust the model size, and level of detail.\n--The paper is clear and easy to understand.\n\nWeaknesses\n-- Although in terms of method, the authors discuss the differences with TensoRF in the supplementary material, it is undeniable that the motivation of the two works is very similar (both introduce vector-based or matrix-based decomposition into NeRF). And in the comparison of TensoRF, the proposed method does not show superiority. In Table 2, the model size of method ‘TensoRF-CP-384’ is 3.9MB, which is smaller than ‘Ours-CP’ (4.4MB), while the quality is also better than ‘Ours-CP’. What’s more, the quality of ‘TensoRF-CP-384’ (3.9MB) is even slightly better than ‘Ours-HY-S’ whose model size is 68.9MB. Therefore, it is hard to illustrate the superiority of the proposed method in terms of both compression and rendering quality.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept