Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper combines a mixture of experts approach with NeRFs for large-scale neural scene rendering. The key idea is to learn a gating network which uses the positional embedding to choose which NeRF model will be queried for the density and color of a given point. All NeRFs share the same head. The gating network needs to be regularized, similar to other MoE models. 

strength_and_weaknesses: Using MoE here makes sense. The idea appears to be well implemented. The experiments are very thorough, and answer my main curiosities about the model details. 

There are a few issues with the writing/clarity, which I list next, but overall the paper is good. 

clarity,_quality,_novelty_and_reproducibility: > "the different sub-networks are typically optimized independently, and thus the inconsistency among them cannot be effectively handled during the optimization"

I am not sure about this. I think BlockNerf handles the inconsistency between NeRFS quite effectively, through the compositing/blending technique. That was one of the contributions of the work. 


The three "limitations" of BlockNerf and MegaNerf described in the introduction are all essentially the same limitation -- the partitioning is handcrafted rather than learned.  

> "Another problem is that common MoE implementations (Hwang et al., 2022) define a capacity factor to limit the number of tokens dispatched to each expert"

Why do you believe this is a problem? I think this is an intentional choice of that work, to help the experts specialize. 



> "It is because in NeRF ..." "It is reasonable ..."

I think it's bad to start a sentence like this. (The word "It" is referring to nothing here.)


> "Multiple gating operations will remarkably influence training and testing speed." 

"Remarkably" is usually good, so this sentence sounds like you are saying multiple gating operations will improve things, but I think you mean the opposite. 


The section before 3.1 describe the use of a top-1 operation, but then 3.1 describes the use of top-k and averaging between their outputs. Later it says k is either 2 or 1, and later still it says k=1. It would be great to eliminate all of the ambiguity here. 


> "We can put the gating network to the deeper layers"

What does this mean?

> "However, the layer numbers of expert networks will shrink and the sparsity of the whole network will be limited"

What does this mean? I don't think it's possible for the number of layers to shrink unless the implementor changes some code. 

> "we put the gating network at the beginning of Switch-NeRF to maximize the layer numbers of the expert networks"

Again, this sounds very unusual. Is the gating network adjusting the number of layers in the expert NeRFs? Perhaps this is similar to differentiable neural architecture search, but I did not see any related work cited on this subject.



> "The random decomposition randomly dispatches 3D points into sub-networks. "

I am not sure that I understand this, because the method actually produces pretty good results according to Table 2. It seems like two points that are infinitesimally close to each other will be dispatched to different networks at random, and no specialization can occur. Why does this produce results so close to distance-based decomposition, and so close to Mega-NeRF (i.e., within 1 PSNR point)? 
summary_of_the_review: I think this is a good paper, using MoE as a natural learnable alternative to handcrafted assignment strategies like those in BlockNerf and MegaNerf. The experiments answer a variety of interesting questions. The writing could be improved, but overall things make sense.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies large-scale neural rendering. To tackle the challenges of lacking universal scene decomposition, not learnable decomposition procedure, and independent sub-networks optimization, the paper proposes Switch-NeRF, a new end-to-end large-scale NeRF with learning-based scene decomposition. It designs a Sparsely Gated Mixture of Experts network to dispatch 3D points to different NeRF sub-networks. The gating network can be optimized together with the NeRF sub-networks for different scene partitions. The method achieves state-of-the-art performances on several large-scale datasets Mill 19 and UrbanScene3D.
strength_and_weaknesses: Strength:
- Overall, I appreciate the idea of applying a MOE network to large-scale neural rendering. Some detailed designs in this paper make it to be not so trivial (Not a simple "A+B" approach)
- The paper shows its approach in detail, and the structure is clear.
- The experiments are tested on different datasets, and the ablation studies are rich.


Weaknesses:
Although I am interested in the claims made by the authors, I have some questions and concerns. It will be better if the authors can provide more insights to the claims and experiments.

- For existing methods, “the different sub-networks are typically trained separately”. How do the expert networks in this paper connect with each other? The separate training in Block-NeRF helps it achieve fine-grained results at each block. Does Switch-NeRF achieve better results than Block-NeRF. What are the benefits of training together? Or how does one expert affect the other?

- It will be better if the paper can show some comparison and analysis with Block-NeRF and BungeeNeRF/CityNeRF[1]. The evaluation datasets Mill 19 and UrbanScene3D may not well prove the effectiveness of this paper in the large-scale setting. It might be better if the paper can provide some results on the same setting of Block-NeRF and BungeeNeRF/CityNeRF, e.g., Google Map/Earth.

- The qualitative results are not convincing enough. It would be better if the authors could provide a supplementary video.

- As shown in Figure 6, how do different experts in Switch-NeRF learn different semantic parts? What will different experts learn if Switch-NeRF deals with building-level cases? It is also interesting to see how different numbers of experts behave and affect performance.

- Is there any failure case of Switch-NeRF? From Table 1, the improvement on campus/rubble is lower than sci-art/building. How to understand this? It is meaningful to see how Switch-NeRF behaves on different scales, which can help to inspire the community.

- The paper designs “a deeper gating network to guarantee enough parameters to learn robust scene decomposition”. How robust is it?

- Although the paper claims that “Multiple gating operations will remarkably influence training and testing speed.” It would be meaningful if the authors could provide some insights on what will happen if there is two gating operations. How slow will it be, and how will the performance change?


Reference:
[1] Xiangli, Yuanbo, et al. "BungeeNeRF: Progressive neural radiance field for extreme multi-scale scene rendering." The European Conference on Computer Vision (ECCV). Vol. 2. 2022, former version CityNeRF: Building NeRF at City Scale
clarity,_quality,_novelty_and_reproducibility: Please refer to the above for clarity, quality, and novelty. For reproducibility, I think a graduate student can reproduce it.


summary_of_the_review: Considering the novelty, quality and some concerns of this paper. I vote for borderline now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a framework for joint learning scene decomposition and NeRF, targeting NeRF for large-scale scene. It is well-recognized that decomposing NeRF is needed for large-scale scene, i.e., a standalone NeRF model for a specific region, because of the model can be extremely large without decomposition and thus cannot be optimized with existing solutions. However, most large-scale NeRF use heuristic hand-crafted scene decomposition, i.e., decomposing into different blocks or balls in the 3D space, which maybe sub-optimal. Thus, this work propose to learn such decomposition configurations together to the NeRF model. Specifically, it inserts a gating function before querying each points' embeddings in MLP and such a gating function will assign points to different NeRF model experts. The experiments show that such a learnt decomposition can achieve better PSNR than heuristic hand-crafted  solutions.
strength_and_weaknesses: # Strength
> + Target an interesting problem: while most large-scale NeRF use heuristic hand-crafted scene decomposition, how to make the decomposition learnable and different for different datasets is an interesting direction to explore.
> + Comprehensive experiments: The experiments, especially ablation studies are sufficient to support the proposed Switch-NeRF is effective to boost the PSNR of NeRF on large-scale scenes.
> + Well written paper: the paper is well written and easy to follow.

# Weaknesses
> + Expandability to new environment: One advantage of NeRF with hand-crafted scene decomposition is that the independence of different blocks or decomposed areas are explicit, which enables the ability to expand the environment with additional NeRF models or update blocks without retraining the entire environment. It is unclear how the proposed Switch-NeRF can be expanded to new environment without retraining the entire environment because the decomposition assignment to previous environment maybe changed when new environment is involved in.
> + Ability of "baking" for fast rendering: Mega-NeRF has shown its ability to be converted to Plenoctree or KiloNeRF-based models for interactive rendering, thus it may not be fair to compare with  Mega-NeRF  without such conversion in Table 3. Also, because the proposed Switch-NeRF has a gating function for the decomposition assignment and such assignment is implicit, it is not clear how Switch-NeRF can be converted to other representations for fast rendering like Mega-NeRF. If it can, not sure whether it will suffer from larger PSNR drop as compare to Mega-NeRF with explicit decomposition. Since speeding the rendering to real-time is an important feature for interactive view as mentioned in Mega-NeRF, it would be better if the authors can add more discussion on it.
> + Need more analysis on the efficiency: as claimed by the authors, "As Switch-NeRF is trained end-to-end, it only uses one background NeRF and one version of AE.", the training cost  should not be larger than baselines, e.g., Mega-NeRF. However, it cost more than Mega-NeRF (42 vs. 30 h) as shown in Table 5. The corresponding explanation, "Since it has a gating network and is trained end-to-end, it is reasonable that it requires more floating point operations (FLOPs) for each point and costs slightly more memory and time for training", is not clear enough. What is main cause of more training cost? More difficult to optimize because of the gating function or larger models to forward in each iteration?
clarity,_quality,_novelty_and_reproducibility: The quality, clarity, and originality is on of above the average of ICLR accepted papers.

The reproducibility will be sounder if the authors can provide the code of it because optimizing a gating function together with NeRF model may be tricky thus checking the whole training process including all hyper-parameters may be important to determine the reproducibility.
summary_of_the_review: Overall, this paper provides some interesting insights to the community. It is nice to see learnable decomposition can surpass heuristic hand-crafted scene decomposition although it should be true intuitively.
However, I do have some concerns on its efficiency, expandability to new environment, and ability of "baking" for fast rendering. I will raise my scores if the authors can solve these concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to use a sparsely gated mixture of experts (MoE) model for NeRF models trained on large-scale scenes. The paper evaluates a variety of design choices and hyperparameters and compares results to Mega-NeRF, which also uses multiple NeRFs to model a large scene but simply uses the distance to a centroid to determine which NeRF to use.
strength_and_weaknesses: Strengths:
- Demonstrates good results, improving on rendering quality relative to Mega-NeRF while keeping parameters and FLOPs roughly constant.
- Motivates the approach well and visualizes the result of the gating network by showing that different experts learn to handle different parts of the scene.
- Includes a good about of ablations and comparisons to quantitatively evaluate the impact of their choices.

Weaknesses:
- (Minor) The use of the composition operator to describe the series of linear layers, softmaxes, ReLUs, etc, is non-conventional. Could be presented more clearly with a visual depiction or simply function application F(G(x)).
- The presentation is general in the k (for Top-k) but in practice k=1. When k=1, it seems confusing why you would multiply the network output by G(x), since that's just a linear scaling factor. It also seems unclear why this would help the gating network train, although it is true that it makes it differentiable with respect to the output. If k > 1, then the gradient for a sample would favor the experts which provide a better input for that sample, but with k=1 this is less intuitive. I think the clarity of the paper could be improved either by elucidating this point, or perhaps by just rewriting this with k fixed to 1 instead of general.
- (Minor) The authors observe that "we observe that dropping sample points can significantly decrease the test accuracy." It is unclear to me what happens when a sample is dropped. Is the 2nd ranked expert used? Or is the value fed to the uniform head simply Linear(x)? In either case,  it seems inappropriate to drop samples, but would be helpful to understand what the alternative is.
- (Minor) Missing parentheses in "A capacity factor Cf of 2.0 and 4.0 with Batch Prioritized Routing Riquelme et al"
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is detailed and clear. The main novel contribution is the usage of a Sparsely Gated MoE for large-scale NeRF training. Given that sparse MoEs are not common for this or similar uses, this contribution is novel, and a significant amount of exploration is done to get good performance out of the approach. The hyperparameters and implementation are described in detail for reproducibility.
summary_of_the_review: This paper does one thing – sparse MoEs for large-scale NeRFs – and does it well, exploring many different choices and ablations and showing what is needed to get this approach to work. The approach is motivated well in the discussion and by the results. In summary, this is a strong contribution to the growing field of large-scale NeRF modeling.
review_title: null

Decision: Accept: poster