Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This paper proposes an MLP-free NeRF representation that supports both compression and composability. The representation allows for efficient and convenient manipulation of the scene of interest and resembles the Level of Detail (LOD) concept in computer graphics. The ability to compose multiple scenes or objects comes naturally from this representation.

With no neural network involved, the model represents the radiance field of a scene using tensors that are the tensor rank decomposition of a full tensor explicitly expressing the radiance field. The authors study two types of tensor decomposition allowing the user to control model sizes. A rank-residual learning strategy is used to support an easy trade-off between model size and quality, without any re-training. 

The authors use a hybrid feature volume decomposition that mixes the more expensive CP decomposition and the more compact TP decomposition. The mixing ratio controls model size vs. quality. 

Compared with spatial locations XYZs, viewing directions are less “native” to the voxel grid (or tensor) representation, so the authors represent them using the Spherical Harmonics (SH) functions.

Because the CP decomposition does not come with the property of “rank importance,” the authors propose a rank-residual learning strategy to learn the coefficient residuals either sequentially or in parallel. 

====== POST-REBUTTAL UPDATE ======

I read the authors' rebuttal that addresses my concerns reasonably. I appreciate the extra experiments, too. Overall, I'm willing to raise my rating to Weak Accept, to the best of my non-expert knowledge. This of course is conditioned on that the authors will add these new experiments to their final version and add necessary clarifications.
strengths_and_weaknesses: 
The paper offers a fresh perspective of NeRF, similar to Plenoxels’: radiance fields are not necessarily represented by a neural network. A full voxel grid representation is either low-resolution or expensive, but using tensor decomposition ameliorates this problem. 

The major weakness of this paper is that it has not studied view-dependent effects such as specular highlights. As the figure and video of the “drums” scene show, the compressed model struggles to produce photorealistic or smoothly varying specularities. Since the main use of NeRF is view synthesis where view-dependent effects are the first-class citizen, studying how the tensor decomposition methods affect modeling view-dependent effects. For instance, which of CP and TP decompositions does a better job of modeling specularities? Are there special decomposition constraints that can be applied to the decomposition such that view-dependent effects are prioritized and preserved? Given the authors consider just NeRF as the application, I don’t think this can be neglected by this paper.


questions: 

I understand how the rank residual learning works for the sequential case but want the authors to clarify how the parallel computation works for this case. In other words, would a parallel computation lead to exactly the same set of coefficients as a sequential computation does? 

Which of CP and TP decompositions does a better job of modeling specularities?

Are there special decomposition constraints that can be applied to the decomposition such that view-dependent effects are prioritized and preserved? 

Also related to view-dependent effects, I think the higher degree you use for SH, the more accurate view-dependent effects you get since the view directions are more “concentrated” and hence more accurate. But with an SH degree of 4, I don’t think you will be able to get smoothly moving specularities. Have the authors tried higher degrees (I’m aware of the squared growth of the number of SH coefficients)? This sounds like a meaningful ablation study to do.



limitations: Yes, the authors mention baked-in lighting as a limitation.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents a compressible NeRF model which also supports the composition of different NeRFs to form a new scene. The proposed method uses a mixture of CANDECOMP/PARAFAC (CP) decomposition and Triple Plane (TP) decomposition, which are vector-based and matrix-based respectively, contributing to a hybrid decomposition method. At the same time, in order to ensure that the selected rank components can achieve near-optimal compression results, the authors propose a rank-residual learning strategy. The entire model does not use the MLP network, and different LODs can be achieved by selecting different numbers of rank components. The rank components of different scenes are concatenated together to achieve the composition of different NeRFs.
strengths_and_weaknesses: Strengths
--The decomposition method is novel, although in some aspects, such as CP decomposition, are the same as the existing work TensoRF. But this work is based on low-rank approximation, the proposed hybrid decomposition strategy can adjust the ratio of vector- and matrix-based rank components, which is something that TensoRF can't do.
--The method can adjust the model size, and level of detail.
--The paper is clear and easy to understand.

Weaknesses
-- Although in terms of method, the authors discuss the differences with TensoRF in the supplementary material, it is undeniable that the motivation of the two works is very similar (both introduce vector-based or matrix-based decomposition into NeRF). And in the comparison of TensoRF, the proposed method does not show superiority. In Table 2, the model size of method ‘TensoRF-CP-384’ is 3.9MB, which is smaller than ‘Ours-CP’ (4.4MB), while the quality is also better than ‘Ours-CP’. What’s more, the quality of ‘TensoRF-CP-384’ (3.9MB) is even slightly better than ‘Ours-HY-S’ whose model size is 68.9MB. Therefore, it is hard to illustrate the superiority of the proposed method in terms of both compression and rendering quality.

questions: -- As for composition, combining different NeRF models does not pose a particular technical challenge. One can obtain a rough geometric bounding box after the training is completed, and then combine the bounding boxes to render combined images. So in Table 2, why can’t some methods, such as PlenOctree, achieve composition?
--What is the difference between the composition that this method can achieve compared to other methods such as NSVF and Plenoxels?
--In Table 2, why only the proposed method can compress the NeRF model? Due to it can change the compression ratio? Similar method, TensoRF, can also achieve a small model size. What’s the advantage of the proposed method?
-- TensoRF uses the MLP network to solve the problem of worse performance caused by Sphere Harmonics (SH). The proposed method also uses the SH functions, but does not use MLP. How does the proposed method solve the problem of worse performance?

limitations: The authors have discussed the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a rank-residual learning strategy to obtain a radiance field representation that supports compressibility and compositionality with an acceptable diminishment of rendering quality.

strengths_and_weaknesses: ### Strengths

*Originality*: 
- I think this work is a nice combination of decomposition and Neural Radiance Fields (NeRFs). 
- The capability column of Table 2 clearly illustrates the difference between this work and existing works, demonstrating the new capabilities enabled by the proposed method.
*Clarity*
- The submission is well-written and easy to follow.
- Related works are adequately cited and compared. For example, Line 103 discusses the similarities and differences between this work and TensoRF clearly.
*Quality*: 
- The proposed approach is technically sound.
- The claims are well supported by the experimental results. 
- The authors are careful and honest with the limitations (e.g., the model size and rendering time grow linearly with the total number of ranks, baked lighting, and bounded scene).

### Weaknesses
 
*Clarity*
- Line 70-90, I think categorizing methods into a) neural network-based, b) hybrid, and c) no neural network is more accurate than a) implicit, b) hybrid, and c) implicit. Some 3D representations are called **implicit** for years even if they are stored in voxels, e.g., TSDF, where the signed distance function is actually an implicit shape function.
- I think Figure 2 needs more annotations. For example, what’s the meaning of dotted lines in the middle sub-figure? What’s the meaning of black lines in the middle and right sub-figures? I assume the meanings of those lines are weighted interpolations. It would be nice to confirm that.


questions: - What’s the training and inference time of the proposed method?

limitations: No.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to compress neural field representations. To this end, the authors first propose a hybrid tensor decomposition and learn the decomposition via differentiable rendering. To encourage the primary information be learned in lowewr ranks, they introduce a novel training strategy that gradually increase the rank components to approximate the scene content.
strengths_and_weaknesses: Strengths

- Overall, I feel the proposed approach is novel and sound. Compared with NeRF, this approach can explicitly add rank components to increase the model capacity, enabling them to dynamically adjust the model size during training.  This is a nice property.
- The paper is well-written and clearly presented.
- The comparison experiments and ablation studies are sufficient.

Weaknesses

1. Experiments

- Current experiments do not show that the proposed model has obvious advantages than NeRF or Mip-NeRF 360. What is the training time of the proposed approach.
- How to better analyze the property that you can gradually increase the rank components to improve the model capacity. An ablation study on this property will significanly bring readers more insights.

2. Writting

- Table 2 says that NeRF is not composable. I do not think so. As shown in [1, 2], object-centric NeRFs can be composed. A single NeRF model can even be explicitly manipulated, as shown in [3].

[1] Object-Centric Neural Scene Rendering  
[2] Neural Scene Graphs for Dynamic Scenes  
[3] NeRF-Editing: Geometry Editing of Neural Radiance Fields  
questions: Please fix the problems in the weaknesses, which can improve the paper quality.

The property of this approach that impresses me is enabling the explicit controll of model capacity. Analysis on this property will make me feel better to this method.
limitations: Yes.
review_title: null

Meta-review: metareview: This paper presents a new NeRF method based on tensor decomposition. The method supports both compression and composability, while achieving similar results compared to standard NeRF models. The method does not use a neural network. Several reviewers found the paper easy to follow, the method novel & sound, and the comparisons comprehensive. Two reviewers mentioned the similarity between the proposed work and TensoRF. The rebuttal addressed most concerns and highlighted the differences between the two works. As TensoRF is a concurrent ECCV submission, the existence of TensoRF should not be used against the proposed work. The AC agreed with most of the reviewers and recommended accepting the paper. 

confidence: Certain
Decision: Accept