Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a method of compressing neural radience fields (NeRF's) by learning mappings from latent codes to model parameters such that both distortion/reconstruction quality and the rate get minimized. While maintaining the same level of quality, this method is able to compress NeRF models for more efficient sender-to-receiver transmission.

The strong aspects of this paper include:
(1) it addresses the valid problem of compressing NeRF models, which is particularly valid given that one usually trains one NeRF per scene. Without compression, the storage of NeRF's will grow dramatically as more scenes are considered;
(2) it achieves the same level of rendering quality, while compressing the model by 8-9x;
(3) it might have implications to future work that attempts to render NeRF's in real time or under a sender-receiver setup.

In terms of drawbacks, this paper can be made stronger by addressing the following points:
(1) while model sizes are compressed significantly at almost no cost of rendering quality, there is no improvement in rendering speed. In fact, because of the additional decompression, rendering novel views might very well take longer than the original NeRF. Since improving the NeRF rendering speed is vital to eventually achieving real-time rendering, I consider this a major drawback of this work;
(2) another major drawback in my opinion is that the work does not demonstrate its usefulness under the multi-scene setup. As the original NeRF is per-scene, and people desire a multi-scene variant of NeRF, the work would be much more impactful if it provides significant compression while achieving the same quality in such setups. Unfortunately, Fig. 5 shows having separate NeRF's have higher rendering quality at comparable sizes as the compressed models;
(3) it would be interesting to explain how this is related to meta-learning. One can imagine having a meta-NeRF that quickly adapts to different scenes. This also achieves similar compression effects, and may work better for the multi-scene setup. Maybe comparisons can be made to such meta-learning methods; and
(4) the work heavily relies on prior works in the compression aspect, and it's unclear to me what the novelty is in that regard.
review_title: Compressing NeRF's for single or multiple scenes

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Paper Summary**:

Compressing 3D scene is an interesting problem to explore. The paper proposes to add entropy penalized reparametrization (Oktay et al. (2020)) technique into Nerf (Mildenhall et al. (2020)) and compress the neural network in Nerf. Experiments also show some compressing rates improvement with the proposed baseline, which I have some concerns in the weakness below.  

**Strength**:
1. The problem this paper is attacking is interesting, compressing 3D scene plays an important role in real-time applications. 
2. The paper also showed the approach to compress multiple scenes in one network (though I have some concerns for the experiments in weakness below).

**Weakness**:
1. Lack of novelty. The main technique from this paper is merging two methods together (Oktay et al. (2020), and Mildenhall et al. (2020)),  with some improvements by extending the nerf to multiple scenes. However, the multiple scenes experiments are not performed well to demonstrate the effectiveness of the proposed method (see below), so the overall novelty is not enough. 

2. Experiments are not great enough to show the effectiveness of the full pipeline. 

   a) In single scene experiment, I think the main reason why HEVC+LLFF has lower PSNR is that LLFF is interpolating multiple training views, and this is obvious and has already been shown in the original Nerf paper (Table 1, Fig 5 in  Mildenhall et al. (2020)). Therefore, this is not a valid experiment to show the pipeline in this paper is better in compression. A more valid baseline should be: receiver receives the training images compressed by HEVC -> decode the images -> receiver train a new Nerf on the decoded images -> run the trained model on the test set. 

  b) In multiple scenes experiment. The paper only showed the experiments on compressing two scenes, it's not clear how will the performance be and conclusion generalize to more scenes, e.g. train only one model on all 8 synthetic scenes together. 

3. The paper also didn't consider the efficiency problem, training a Nerf model takes a long time, and also the author did not report the running time when doing inference on a trained nerf model, and do not have a comparison with HEVC+LLFF baseline. In literature, Nerf model takes more than one day to converge, while HEVC is very fast for both encoding and decoding.

Overall, considering the novelty and the experiment section in this paper, I don't think they are enough to reach the bar of ICLR, so I vote for rejection. 

review_title: Interesting problem, but not enough novelty and the not well-conducted experiements

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper proposes to compress nerf models with entropy loss, where instead of directly training nerf model parameters, it trains a new function F which takes some compressed information and decodes to the nerf models. Then it did the same things as nerf, which render scenes in novel views. The authors show that the function F could largely compress the original nerf models while keeping similar PSNR.

Comments:
The paper combines network compression and neural renderings, which is pretty interesting. However, I have several concerns :

Novelty:  The paper seems to combine two methods together, where in network compression, it adopts Oktay et al. (2020) while in neural renderings, it relies on nerf. Though it introduces compression to neural rendering, such a combination seems to be not very creative.

Generalization. While one can train such a network for nerf scenes, the network may get overfit to these scenes since the whole dataset has no more than 20 scenes.  It would be great to validate on shapenet datasets, or some dataset has at least 100 models.

Comaprison. The authors compare their methods with HEVC+LLFF method, which is pretty unfair. Since HEVS is a traditional video compression method while LIFF is also a traditional blending view rendering method. The authors are encouraged to compare with some neural network compression methods + neural rendering methods.

Conclusion: Overall, I think this paper proposes an interesting idea and shows good results. However, due to lack of creativity and unfair comparison, I rate it below the acceptance bar.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: I think the paper is well written, and explains the details of the method well. I addition I think choices made in the method are intelligent and well justified. 

My concerns with the paper lie in two area. The first is that I am not convinced that the problem this paper seeks to solve, ie  compression of a 3D scene, is so relevant to the ML research community that is justifies the lack of novelty in the method. From reading this paper it feel like 2 approaches have been found , ie, NeRF and model compression using entropy penalty, which happen to work well together, but the actual degree of novel research contribution seems low.  If 3D scene compression was an established area of research in the ML research community, with many previous works proposing high performing solutions,  this approach may be justified as sufficiently better across various metrics. However, given the lack of this prior work, the solution feels far more like an engineering solution to allow NeRF to work well on cellphones then machine learning research. 

My second concern is the lack of comparison to other approaches to scene compression. From the way I view the problem you have proposed, you are assuming practically explicit 3D scene information as input and attempting to transfer this information between devices in as small a size as possible such that it images of the scene can be sampled on the new device in as high a quality as possible.  There are other ways of describing a 3D scene then with multiple images. The experiment I really want to see here is if this method is better then if the explicit scene information is compressed and rendered on the second device. If you can provide this in new experiments I will be happy to raise my score. Examples of this would be compressing  a 3D mesh, or surface defined by voxels with lighting parameters and textures.  My main concern here is that for simple scenes I am very sure that compressing or even transferring directly the 3D scene explicitly is a far superior, though there is probably some level of scene complexity at which your method overtakes it. I think it is important to establish this, and understand where your method is applicable. 
review_title: 3D Scene Compression through Entropy Penalized Neural Representation Functions

Meta-review: Description:
The paper presents a method for encoding a compressed version of an implicit 3D scene, from given images from arbitrary view points. This is achieved via a function, learning with a NeRF model, that maps spatial coordinates to a radiance vector field and is optimized for high compressibility and low reconstruction error. Results shows better compression, higher reconstruction quality and lower bitrates compared to other STOA.

Strengths:
- Method for significantly compressing NerF models, which is very useful since such models are often trained for every new scene
- Retain reconstruction quality after compression by an order of magnitude
 
Weaknesses:
- The need for decompressing the model before rendering can be done means reduced rendering speed. This also requires longer training times.
- Experiments against other scene compression + neural rendering technique will have further strengthened the papers’s claims 
- The techniques used are well established, and thus there is not as much technical novelty.

Decision: Reject