Switch-NeRF: Learning Scene Decomposition with Mixture of Experts for Large-scale Neural Radiance Fields
Authors: Zhenxing MI
Dan Xu
Venue: ICLR 2023 poster (2023)
Keywords: Neural Radiance Fields
Mixture of Experts
Large-scale scene
Novel view synthesis
Sparse network

Abstract: The Neural Radiance Fields (NeRF) have been recently applied to reconstruct building-scale and even city-scale scenes. To model a large-scale scene efficiently, a dominant strategy is to employ a divide-and-conquer paradigm via performing scene decomposition, which decomposes a complex scene into parts that are further processed by different sub-networks. Existing large-scale NeRFs mainly use heuristic hand-crafted scene decomposition, with regular 3D-distance-based or physical-street-block-based schemes. Although achieving promising results, the hand-crafted schemes limit the capabilities of NeRF in large-scale scene modeling in several aspects. Manually designing a universal scene decomposition rule for different complex scenes is challenging, leading to adaptation issues for different scenarios. The decomposition procedure is not learnable, hindering the network from jointly optimizing the scene decomposition and the radiance fields in an end-to-end manner.  The different sub-networks are typically optimized independently, and thus hand-crafted rules are required to composite them to achieve a better consistency.  To tackle these issues, we propose Switch-NeRF, a novel end-to-end large-scale NeRF with learning-based scene decomposition. We design a gating network to dispatch 3D points to different NeRF sub-networks. The gating network can be optimized together with the NeRF sub-networks for different scene partitions, by a design with the Sparsely Gated Mixture of Experts (MoE). The outputs from different sub-networks can also be fused in a learnable way in the unified framework to effectively guarantee the consistency of the whole scene. Furthermore, the proposed MoE-based Switch-NeRF model is carefully implemented and optimized to achieve both high-fidelity scene reconstruction and efficient computation. Our method establishes clear state-of-the-art performances on several large-scale datasets. To the best of our knowledge, we are the first to propose an applicable end-to-end sparse NeRF network  with learning-based decomposition for large-scale scenes. Codes are released at https://github.com/MiZhenxing/Switch-NeRF.

Published as a conference paper at ICLR 2023
SWITCH-NERF: LEARNING SCENE DECOMPOSITION WITH MIXTURE OF EXPERTS FOR LARGE-SCALE NEU- RAL RADIANCE FIELDS
Zhenxing Mi Dan Xu
Department of Computer Science and Engineering
The Hong Kong University of Science and Technology (HKUST)
Clear Water Bay, Kowloon, Hong Kong zmi aaê connect. ust.hk, sanxu@cse.ust...
ABSTRACT
The Neural Radiance Fields (NeRF) have been recently applied to reconstruct building-scanl scale and even city-scale scenes. To model large-scale scene efficiently, a dominant strategy is to employ a divide-and-conquer paradigm via performing scene decomposition, which decomposes complex scene into parts that fur- ther processed by different sub-networks. Existing large-scale NeRFs mainly use heuristic hand-crafted scene decomposition, with regular 3D-distance-based thyyscal-ssrred srreet-block-bee block-based schemes. Although achieving promising results, the hand-crafted schemes limit the capabilities of NeRF in large-scale scene model- ing in several aspects. Manually designing universal scene decomposition rule for different complex scenes is challenging, leading to adaptation issues for dif- ferent scenarios. The decomposition procedure is not learnable, hindering the network from jointly optimizing the scene decomposition and the radiance fields in end-to-end manner. The different sub-networks are typically optimized inde- pendently, and thus hand-crafted rules are required to composite them to achieve better consistency. To tackle these issues, propose Switch-NeRF, novel end-to-end large-scale NeRF with learning-based scene decomposition. We de- sign a gating network to dispatch 3D points different NeRF sub-networks. The gating network can be optimized together with the NeRF sub-networks for dif- ferent scene partitions. by a design with the Sparsely Gated Mixture of Experts (MoE). The outputs from different sub-networks can also be fused in learn- able way in the unified framework to effectively guarantee the consistency the whole scene. Furthermore, the proposed MoE-based Switch-NeRF model is carefully implemented and optimized to achieve both high-fidelity scene re- construction and efficient computation. Our method establishes clear state-of- the-art performances on several large-scale datasets. To the best of our knowl- edge, the first to propose applicable end-to-end sparse NeRF network with learning decomposition for large-scale scenes. Codes are released at https: thu.. com Mii Zhenx ng/ Swi
INTRODUCTION
The Neural Radiance Fields (NeRF) method (Mildenhall et al.) 2020) has gathered wide popularity in novel-view synthesis and 3D reconstruction due to its high quality and simplicity. It encodes a 3D scene from multiple 2D posed images. The original NeRF typically targets small scenes or objects, while in real-world applications such autonomous driving and augmented reality (AR) virtual reality (VR), building NeRF models to effectively handle large-scale scenes is critically important.
The problem of large-scale NeRF is that more data typically requires higher network capacity (number of network parameters). A naive solution is to densely increase the network width and depth. However, this will also greatly increase the computation for each sample and is harder to optimize. A more applicable network should have a large capacity while maintaining almost con- stant computational cost for each sample. Therefore, building an applicable large-scale NeRF can be

---

Published as a conference paper at ICLR 2023
Loss
Loss
Loss
Loss
Loss
Loss
Loss
NeRF
NeRF
NeRF
NeRF
NeRF
NeRF
Head
Expert
Expert
Expert
Inactive
Active
Distance
Gradient
clustering
Blocks
Trainable gate
3D pointe
(a) Learning after distance-based decomposition (egg. Mega-NeRF)
(b) Learning after physical-distriuutee based decomposition (e.g. Block-NeRF)
(c) Learning with scene
decomposition (Ours)
Figure I: Different kinds of decomposition methods. The dot lines mean non-differentiable opera- tions. The solid lines mean differentiable operations that can be trained by back puekkpropagation. The Mega-NeRF (Turki et al., (002)) clusters pixels by 3D sampling distances to centroids in training. The Block-NeRF (Tancik et al.) 2022) clusters images by dividing the whole scene according to street blocks. The sub-netwon nebboss in both methods are trained separately. Our Switch-NeRF learns to decompose the 3D points by a trainable gating network and the whole network is trained end-to-end.
considered building sparse neural network. The core of the design is to select different network parameters (i.e. sub-networks) for different inputs. This procedure can be formulated as a scene decomposition problem in the NeRF task. Each sub-network handles different part of the scene.
Along the scene decomposition and learning a sparse neural network, recent Mega-NeRF (Turki et al. 2022) and Block-NeRF (Tancik al., 2022) have extended NeRF to building-scale and city-scale scenes based on heuristic hand-crafted scene decomposition. As depicted in Fig. [ the Mega-NeRF and Block-NeRF simply use 3D sampling distances street blocks to decompose the scene and train different NeRF models separately. With promising results on large-scale scenes, their hand-crafted scene decomposition methods still lead several issues. The large-scale scenes are essentially complex and irregular. Designing a universal scene decomposition rule for different scenes extremely challenging in a hand-crafted way. This accordingly brings adaptation issues for distinct scenarios in the real world. Hand-crafted rules require rich priors of the target scene, such as the structure of the scene, to deploy the partition centroids as Mega-NeRF and the physical distribution of the scene images in Block-NeRF. These priors may not be available in practical applications. The hand-crafted decomposition is not learnable, hindering the network from jointly optimizing the scene decomposition and the radiance fields in an end-to-end manner. The gaps between the decomposition, composition and NeRF optimization may lead to sub-optimal results. Besides, the different sub-networks are typically trained separately, leading to possible inconsistency among different sub-networks. To handle this problem, they usually set overlapping among adjacent partitions in training and use hand-crafted rules in inference composite results from different sub-networks. (Tancik et al. 2022. Turki et al. (022)..
To address above-mentioned issues, in this paper, we make the following contributions.
An end-to-end framework for joint learning of seene decomposition and NeRF. We present Switch-NeRF, an end-to-end sparse neural network framework, which jointly learns the scene de- composition and NeRF. As shown in Fig. De. we propose a learnable gating network for scene decomposition. dynamically selects and sparsely activates a sub-network for each 3D point. The overall network is trained end-to-end without any heuristic intervention. We do not require any pri- ors of the 3D scene shape or the distribution of scene images, leading to a generic framework for large-scale scenes. Since the selection of sub-networks in training is a discrete operation, criti- cal problem is how to back- proppgate gradients into the gating network. We use the strategy from the Sparsely- -ated Mixture-of-Fe Experts (MoE) (Shazeer al. (017)) to deal with this problem. We structure our sub-networks as NeRF experts for different scene partitions. 3D points are dispatched into different NeRF experts based on the gating network. Besides the gating network, also de-
2

---

Published as a conference paper at ICLR 2023
sign head to unify the predictions of multiple NeRF experts, which aligns the high-level implicit features from different NeRF experts to effectively address the inconsistency problem.
Efficient network design and implementation. With the framework design of Switch-Neds NeRR, how- ever, optimizing and implementing it efficiently and stably is not trivial. In NeRF, the number of 3D points in a forward pass is orders of magnitude larger than that of the input tokens in other NLP and vision tasks within an MoE framework. Dispatching samples to different NeRF experts inevitably introduces large computation and memory usage. Therefore, we consider dispatching 3D points only once with an effective gating network, and we design a deeper gating network to guarantee enough parameters to boost the accuracy of the scene rendering. Another common design in MoE implementations (Hwang et al. (2022) is to define a capacity factor to limit the number of tokens dis- patched to each expert. This dynamically drops overflow 3D points for our NeRF experts. It works well when training the network but brings large influence on the testing accuracy. To address this issue, we implement full dispatch operation by CUDA based on Tutel (Hwang et al., (022) to significantly improve the testing performance, while avoiding unnecessary memory allocation.
High-quality results. Extensive experiments are conducted on challenging benchmarks with large- scale scenes. Qualitative results demonstrate that our network can learn reasonable decomposition of large-scale complex scenes. Our model also establishes state-of-the-art performances. It shows clearly more superior results with much less network parameters compared to those hand-crafted decomposition counterparts.
RELATED WORK
Neural Radiance Field. The Neural Radiance Field (NeRF) is proposed by Mildenhall et al. (2020) to use volumetric rendering for novel view synthesis from posed images. It encodes 3D scene into a multilayer perceptron (MLP), which is simple and requires very limited priors of the scene. Due to the success of NeRF in high-quality rendering and 3D reasoning, many works have been proposed
to improve its efficiency (Reiser et al. 2021; Yu et al. 2021 Müller et al,, (022). accuracy (Barron
et al. 2021 Verbin al.] 2022) and apply to challenging scenes (Zhang al. 2020 Martin-
Brualla et al. 2021] Xiangli et al. 2022 and 3D reconstruction tasks (Wang et al. 2021).
We pay more attention the closely related works, i.e. NeRF methods for large-scale scenes. As shown in Fig. D.. Mega-NeRF (Turki et al., 2022) proposes to use a simple 3D distance-based method to cluster training pixels into parts that can be trained separately by different NeRF models. It samples centroids uniformly in the 3D scene and groups 3D points in testing. Block-NeRF (Tancik) et al. 2022) proposes to scale NeRF to city-level scenes. As shown in Fig. [p. it divides the whole scene based on physical distribution of the scene images, i.e. partitioning through street blocks. The scene images in different street blocks are trained separately by sub-networks. In the testing, both methods consider offline fusion of prediction results from different sub-networks. In contrast to Mega-NeRF and Block-NeRF which use hand-craft scene decomposition, our Switch-NeRFF jointly learns the scene decomposition and a large-scale NeRF in end-to-er manner. Our method is not dependent on any priors of the 3D shape and the physical image distribution of a target scene. Therefore, our method is more generic for arbitrary large-scale scenes.
Mixture of Experts (MoE). Modern MoE methods mainly follow the work of Shazeer et al. (017). It proposes Sparsely-Gated of work (i.e. FFN or MLP) in language model. It designs vanilla Top-k gating network to dispatch samples into A experts, and proposes an auxiliary loss for balancing the training of different ex-
layer in place of the feed- forward net-
perts. The MoE has been widely used in Natural Language Processing (NLP) (Lepikhin et al. 202T al.) 2021). Switch Transformer (Fedus al. 2022)
Fedus al. 2022) and Vision (Riquelme
suggests that Top-2 gating is not necessary. It trains a network of high quality with Top-1 gating to largely reduce the dispatch computation and communication. Besides, different gating mechanisms are also proposed, such as Hash Routing (Roller et al. 202T) and BASE (Lewis et al. (021). There have been popular implementations of MoE in Mesh- TensorFlow (Shazeer et al. 2018), Deepspeed (Rajbhandari et al. (022)) and Tutel (Hwang et al., 2022). which focus on improving large-scale distributed training. To the best of our knowledge, none of the existing works considers develop- ing -f-ire--o-NeRF-EE-p
(MoNE) for large-scale scenes. We design effective structure of MoNE for jointly learning scene decomposition and NeRF, and also improve the efficiency of MoNE by handling the issue of dispatching large-scale 3D points different NeRF experts.
3

---

Published as a conference paper at ICLR 2023
Gate value of Expert 2
PE(x)
Expert
Shared MLP
Positional encoding
Softmax
Active expert
Inactive expert
Gate
Multiply
Appeurance embedding.
Top-1
Expert 2
(x, yi z)
Dispatch to Expert 2
Prediction
RGBo
Expert
x
Rendering
Rendering
Loss
Expert 4
Sampling 3D puints
PE(d) AE
Figure 2: The framework our Switch-NeRF. A 3D point will first go through gating network and then be dispatched to only one expert according to the gating network output. The expert output is multiplied by the corresponding gate value and sent to a head for density and color prediction with direction d and appearance embedding. The rendering loss is used for supervision. The images on the left of each expert are the visualization of 3D radiance fields handled by different experts.
SWITCH-NERF
Our Switch-NeRF is Sparse Neural Radiance Field network targeting large-scale scenes. A frame- work overview of Switch-eeRFis depicted in Fig. 2 We represent large-scale 3D scene as a sparse 5D radiance function Fe: (x, d) - (c. o), where (x, d) are 3D point and its view direction, (c, o) are predicted color and density, θ represents network parameters. Fe sparsely activates only a part of its parameters for an input each time. The overall structure of our network mainly consists of gating network G, set of n experts [E,] and shared prediction head H for generating a and The actual inputs of our network are the positional encoding (PE(x), PE(d)) for (x, d) as in the
vanilla NeRF (Mildenhall et al. 2020). We omit PE(-) in our following equations for simplicity.
Given input (x, d), we first send into the gating network and obtain the gate values G(x). Then, we apply a Top-1 operation on G(x) to determine which expert should be activated. As shown in Fig. 2 only one expert (i.e. Expert 2) is activated. The point will be dispatched to this selected expert. Other experts do not participate the processing of x. The output feature of the expert Exx) will be multiplied by the gate value corresponding this expert. This makes the gating network be trained jointly with the expert networks. After that, the feature is used to predict the density a and color together with d and the appearance embedding AE. In the next, we first introduce details of the trainable gating and our gating network architecture in Sec. 3.1] Then we introduce our expert and head network architectures in Sec. 3.2 We discuss the capacity factor and full dispatch in Sec.
3.3
We finally formulate the rendering procedure and our loss functions in Sec. 3.4
3.1
SPARSE GATING IN SWITCH-NERF
The sparse gating network plays an important role in our Switch-NeRF because determines the optimization routes of different NeRF experts. In Switch-NeRF, we only consider one gating net- work because there is very large number of 3D points that require gating and dispatching in NeRF optimization. Multiple gating operations will largely decrease the training and testing speed.
Trainable gating Switch-NeRF. The trainable gating in our network follows mechanism in the MoE method Switch Transformer (Shazeer et al.] 2017). The gate values G(x) is vector of n-dimensions normalized via Softmax, in which G(x), represents the probability of selecting the i-th NeRF expert. We apply Top-1 function on G(x) to sparsely select only 1 expert E, from set of NeRF experts, i.e. ( for each 3D point. The input x will be dispatched into the selected expert E, and obtain an output (x). The final output Exx the output of E, multiplied by the corresponding gate value:
(l) As the predicted gate values are multiplied to the corresponding outputs of NeRF experts, the gating network can be optimized together with the NeRF experts in the backward pass. This
É(x) - G(x),
4

---

Published as a conference paper at ICLR 2023
makes the network able to directly learn scene decomposition during network training. Our net- work structure is highly sparse because we use Top-1 to select only one experts for each sample.
In previous MoE meth- ods (Shazeer et al. 2017; Lepikhin et al. 2021). the gating net-
Gating network architecture.
Laeerr & Sigmoi
work is typically simple linear mapping. This choice is reason-
able as they usually have multiple MoE layers. The input of gat- ing networks in deep MoE layers can be high-level features from previous layers. Therefore, their gating networks share more in- formation from the main network. Let the original input be the 3D point x. Typically will first go through a sub-network S with several layers and learn an internal feature S(x). The real gating operation of the deep MoE layers can be written as:
Head
Linse
Concut d
inner Suftpha
Linear ReLU
Ral
Litear RaLt
G(x) = Softmax S6ottmax(Linear( (Linear(S(x)))
(2)
Softmax
Linear ReLA
Linus
Therefore, the gating networks in these methods actually share an network from the main network.
Lincr & LayerNion
iine Rel
Linzar & Rol.
ince & R
Considering that we only use one gate network in our Switch- NeRF for efficiency, allocating more parameters for our gating network is necessary to learn powerful gating. We can put the gating network closer to the prediction head. It will share more parameters with the main network. However, this will also make each sample point share more layers before the gating network. In this case, the capacity of the whole network shrinks, network capacity is controlled by the number of layers of each unshared expert. Therefore, as shown in the Fig. B] we put the gating network at the beginning of Switch-NeRF to maximize the layer numbers of the expert networks. In order to allocate more parameters to the gating network, we use a shallow MLP instead of linear mapping. Our gating network G(x) consists of 4 Linear layers and LayerNorm as shown Fig. 3 Our design balances the efficiency, network sparsity and the number of parameters in the gating network.
Gate
Licer
Expert
PE(x)
Figure 3: The architectures of our gating, one expert and the head networks. It shows the for- ward pass of a point. It goes through the gating network and a selected expert, and is passed to a shared prediction head.
the
3.2 EXPERT AND HEAD NETWORKS
NeRF Expert Network. The set of n NeRF experts [ in Switch-NeRF contains most of the network parameters. As shown in Fig. 3 each expert in Switch-NeRF is deep MLP with skip connection. The structure and depth of the experts are aligned with the main structure of vanilla NeRF. Each expert only processes part of the 3D points, determined by the gating network. The output of the selected expert is multiplied by the corresponding gate value to obtain a feature vector Exx). This feature vector is then used to predict density a and the direction-depentee increasing the number of NeRF experts n, can easily scale the network's capacity.
color e. By
Unified NeRF Head. The head network H is designed for the final predictions of each input sample. is shared for all the samples, as shown in Fig. 3. After obtaining the expert output E(x). we use a Linear layer with a Softplus activation (Zheng al. 2015) predict a. The use of Softplus follows Mip-NeRF (Barron al. 2021) for stable prediction of Then, (x goes through Linear layer and is concatenated with PE(d) and an appearance embedding AE. The color e is predicted from the concatenated feature by an MLP. The appearance embedding AE is a trainable vector to capture image-level photometric and environmental variations (Martin-Brualla et al. 2021).
3.3 CAPACITY FACTOR AND FULL DISPATCH
Capacity factor for training. In our Switch-NeRF, to efficiently dispatch 3D points different ex- perts is important. Einops-based dispatch (Lepikhin et al. 2021) causes memory overflow due to the large number of 3D points. In our training. we use the CUDA-based fast dispatch in Tutel (Hwang et al. 2022). Following previous MoE methods (Lepikhin et al.) 202T). we set a capacity factor for each NeRF expert the training. caps the number of sample points dispatched to each NeRF expert, leading to uniform tensor shapes, balanced computation and communication. Let B be the batch size of the whole network, C₁ be a capacity factor and n be the number of NeRF experts.
Then the maximum number of sample points going into each NeRF expert is B. = ceil( LBCI). ))
5

---

Published as a conference paper at ICLR 2023
The dispatch with capacity factor can be called uniform dispatch. Fig. shows the uniform dispatch with Cy = 1.0. Overflow points are dropped. If the capacity is not fully used, will be zero-padded.
larger capacity factor decreases the dropping ratio but increases the memory and computa- tion. In our network, we set the capacity factor to 1.0 without requiring extra memory. We use the Batch Prioritized Rout- ing (Riquelme et al. 2021) to improve the training with lower expert capacity.
Pad
Unitfor dispatch
Full dispatch
Full dispatch for testing. Previous MoE methods usually use the uniform dispatch for both the training and testing (Lepikhin et al., 2021 Fedus et al. (2022). which inevitably drops sample points. In our Switch-NeRI: although the uniform dispatch works well in the training, we observe that dropping sample points can significantly decrease the test accuracy. A possible reason is that we do not use stacked MoE layers and skip nections between them as previous MoE methods usually do, to maintain the network efficiency. To improve the testing curacy, we implement an efficient full dispatch strategy based on Pytorch, CUDA, and Tutel. It can dispatch all the points to their corresponding expert with only slight memory increase. A description of the strategy is depicted in Fig. 4
20 Cy
Full disputch Ground truhh
Figure 4: In the uniform dispatch, each expert has the same tensor shapes. will drop overflow to- kens and perform padding. The full dispatch makes sure each point will be processed by an expert. The images rendered by uniform dispatch have apparent artifacts.
With this efficient full dispatch, we can largely improve the test accuracy.
3.4 VOLUME RENDERING AND LOSSES
Our volume rendering procedure follows the vanilla NeRF (Mildenhall et al. 2020 The training data of Switch-NeRF consists of multiple posed images. For each pixel in the training images, we back-project camera ray r into 3D space. A set of N samples are sampled along the ray. For each sample (x, d), our network Fe predicts volume density and color e = (r, g, b). Let 6, be the distance between adjacent points. The expected color ((r) of this pixel is synthesized along the ray by the volume rendering function in NeRF (Mildenhall et al. 2020
N Σ (011-xxp(- where T; exp( og,,),
Č(r)
Rendering loss. The main loss of Switch-NeRF is the rendering loss Ly. After rendering the color C(r) of a ray from our network through Equation 3 we compute I,, for supervision:
L ===3 II TER
(4)
where R is the set of sampled rays. C(r) is the ground truth color of ray in the training images.
Auxiliary loss tackling imbalanced optimization. One problem of training MoE-based networks that the gating network can favor only few experts (Shazeer et al. 2017). The optimization and utilization of NeRF experts will thus be imbalanced. This could even cause several experts not trained, and then the whole network converges to a sub-optimal solution and cannot scale well through the training since some of the network capacities are not fully utilized. Following [Shazeer al. (2017): Lepikkhn et al. (202T), we use an auxiliary loss L, to regularize the gating network and balance the utilization of NeRF experts. Specifically, following the differentiable load balancing loss proposed in GShard (Lepikhin et al. 202T), we define our auxiliary loss as follows. Given n NeRF experts and batch 1 with N sample points, let c, be the number of points dispatched to each expert E, by Top-1. We first compute the gating values m, distributed to E, with m, = Exes G(x), The auxiliary loss La can be computed as Lg = N2 Σr c,m,- This auxiliary loss encourages balanced gating because it is minimized when the dispatching is ideally balanced. Under the balanced gating, and my are both expected to be N. n. Then >> e,m; will be N The loss L, will be 1.
a
The total loss L is the weighted sum of the above-mentioned two losses:
L= Tap + XLa
(5)
where À is the weight for auxiliary loss. We set A 5 x 10-4 4 for all main results and it is sufficient to balance the utilization of experts.

---

Published as a conference paper at ICLR 2023
Building
Sci-Art
Residence
Rubbie
Campus
Swithh Doicc-NoRfR Mege-Nolt Ground Trub.
3
Figure 5: The comparison of the rendered images from Mega-NeRF and our Switch-NeRF. Our method renders more details and tiny structures than Mega-NeRF, Please zoom in see the details.
Table 1: The testing results of our Switch-Nete NeRF on large-scale datasets. Our method gets state-of-
the-art accuracy compared to the dense NeRF, NeRF++ and the sparse Mega-NeRF.
Dataset
Metrics
NeRF
NeRF++
SVS
Deep
Mega-NeRF
Switch-NeRF
PSNR1
19.54
19.48
12.59
13.28
20.93
21.54
Building
SSIMT
0.525
0.520
0.299
0.295
0.547
0.579
LPIPS]
0.512
0.514
0.778
0.751
0,504
0.474
PSNRT
21.14
20.90
13.97
14.47
24.06
24.31
Rubble
SSIM
0.522
0.519
0.323
0.310
0,553
LPIPS
0.546
0.548
0.788
0.734
0.516
0.496
PSNRT
19.01
18.99
16.55
13.07
22.08
22.57
Residence
SSIM
0.593
0.586
0.388
0.313
0.628
0.654
I.PPS
0.488
0.493
0.704
0.767
0.489
0.457
PSNR
20.70
20.83
15.05
12.22
25.60
26.52
Sci-Art
SSIM
0.727
0.755
0.493
0.454
0.770
0.795
LPIPS
0.418
0.393
0.716
0,831
0,390
0.360
PSNR
21.83
21.81
13.45
13.77
23.42
23.62
Campus
SSIM
0.521
0.520
0.356
0.351
0.537
0.541
LPIPS
0.630
0.630
0.773
0.764
0.618
0.609
4 EXPERIMENTS
4.1 DATASETS, METRICS AND VISUALIZATION
Datasets. We use the Building, Rubble datasets from Mill 19 Turki et al. 2022) and Residence, Sci-Art, Campus datasets from UrbanScene3D (Liu al. 2021 evaluate our Switch-NeRR Each scene contains thousands high-resolution images. The camera parameters for all the images are the same Mega-NeRF (Turki et al. 2022). These datasets cover large enough areas while they can still be handled by consumer-level workstations with commonly used GPUs.
Metrics. We the PSNR, SSIM
(Wang et al., 2004) (both higher better), and VGG implementa- tion of LPIPS (Zhang et al. 2018) (lower is better) to quantitatively evaluate our results on the novel view synthesis. The PSNR space. The SSIM focuses more on structural similarity. The LPIPS measures perceptual similarity.
measure the mean squared error between two images logarithmic
Visualization. Besides the rendered images, visualize the 3D radiance fields. We sample 3D points along rays and use the = 1 - exp(-a,6;) as the opacity of each 3D point. We use Point Cloud Library (Rusu Cousins 2011 to show the color and opacity of each 3D point.
4.2 SETTING
Similar to NeRF++ (Zhang al. 2020) and Mega-NeRF, the 3D scene space is spilt into fore- ground and background. We use experts with Top-1 gating and C₁ = 1.0 for the foreground to end-to-end learn scene decomposition, and one original NeRF the background. This differs from Mega-NeRF using a foreground and background NeRF for each of its 8 sub-networks requiring more network parameters. Each of our NeRF experts contains layers with each layer 256 channels. We sample 256 coarse and 512 fine points per ray in the main network and 128/256 samples in the background network. We use 8 NVIDIA RTX 3090 GPUs for distributed data-parallel training and sample 1024 rays for each GPU. We Adam optimizer (Kingma & Ba 2015) and learning rate decaying exponentially from x 10-4 to 5 10 and use b in traiiing and float in testing reduce memory and time. We train 500k iterations for each dataset and test validation images.

---

Published as a conference paper at ICLR 2023
Mega-Nole Nolt
All
Centroid
Centroid 3
Centroid 4
Centroid 5
Switch-NoRe
All
Export
Expert
Expert 3
Expert 5
Figure 6: The visualization of 3D radiance fields handled by experts in Switch-NeRF and sub- networks Mega-NeRF. The Mega-NeRF just regularly decomposes the scene. Our experts are roughly specialized different semantic parts of the scene such as buildings, trees and grounds. 4.3 BENCHMARK PERFORMANCE
The main quantitative results reported in Table and qualitative results in Fig. The statisties of NeRF (Mildenhall et al. 2020). NeRF++ (Zhang et al. (020). SVS [Reeller Koltun), Deep- View (Flynn al. 2019), and Mega-NeRF and NeRF++ is both set to 2048 to obtain a similar capacity between Mega-NeRF and Switch-NeRF.
quoted from Mega-NeRF. The MLP width of NeRF
As shown in Table 1 our Switch-NeRF achieves state-of-the-art accuracy all the datasets in terms of PSNR, SSIM, and LPIPS. produces much better accuracy than NeRF and NeRF++. This con- firms the effectiveness of scaling NeRF with sparse Mixture-of-Experte instead of densely expanding the network. Our method also outperforms the sparse Mega-NeRF, showing the advantage of learn- ing decomposition instead of the cand-crafted scene decomposition. Fig. 5 presents the rendered images of Mega-NeRF and Switch-NeRE: Our method can render more tiny structures and details compared to Mega-NeRF. These main results demonstrate the overall performance of our method.
4.4 MODEL ANALYSIS
Besides the main results, we perform ablation ex- periments analyze different aspects our model. The Mill 19 Building scene is used by default. Decomposition. We analyze our learning- based decomposition compared to other decomposition methods. The random decomposition randomly dis- patches 3D points into sub-networks. The distance decomposition dispatches 3D points based on the 3D distances in training and testing. The network de- sign and the training setup for these variants are the same as Switch-NeRE We also include the results of Mega-NeRI, which uses distances to cluster pixels training and 3D points in testing. The results are shown Table 2 The random decomposition cannot fully utilize parameters in the network. The distance decomposition performs better and similar to Mega-NeRF. Our Switch-NeRF with learned decomposition achieves the best accuracy. NeRF expert specialization. We visualize the 3D radiance fields of the UrbanScceeeID-Scii -cii-AArt for Mega-NeRF and Switch-NeRF in Fig. 6 The Mega-NeRF partitions the scene into regular parts. In Switch-NeRF, the experts have been roughly specialized to different semantic parts the scene. Experts 1 and focus different buildings. Expert 3 focuses on the green grasses and trees the ground. Expert specializes on other parts of the ground. Our Switch-NeRF can achieve more reasonable and semantically meaningful scene decomposition in learning way.
Table 2:
The ablation results on decompo- sition, gating network design. auxiliary loss La and the unified head in Switch-No NeRF.
PSNR
SSIMT
LPIPS
Randon
20.06
0.455
0.575
Mega-NeRt NeRF
20.93
0.547
0.504
Lineur
20.75
0.532
0.524
Gate
w/o Norm
21.39
0.577
0.474
w/o fi
15.32
0.363
0.769
w/o unified head
19.72
0.472
0.561
Switch-NeRtE
21.54
0.579
0.474
Table 3: The accuracy, test time, test mem- ory and parameters numbers of Switch-NeRF and Mega-NeRF with different expert or sub- network numbers. From to 16 experts, our network scales much better than Mega-NeRF.
The accuracy, training memory and training time with Top- Top-2 and capacity factors. Increasing Cý or using T2p-2 in train- ing can improve the accuracy while increasing the training time and memory.
Table 4:
Model
PSNR1
SSIM
LPIPS
Mem.
Timel
Param..
Swiscb-4
21.00
0.547
0.504
5825M
106s
2.78M
Top-k
Capacity
PSNRI 21.54
SSIM!
LPIPSI
Mom
Tine]
0.579
0.474
10182M
42.5h
Swiich-8
21.54
0.474
5847M
110.
4.53M
0.594
12271M
Mega-16
21.47
0.590
04462
8042M
101s
21.M
2
21.77
0.590
0.165
1451MM
58.7h
Switch- 16
22.49
0.625
0.429
5876M
118
805M
8

---

Published as a conference paper at ICLR 2023
Table Comparisons of parameter number, time, mem- ory and FLOPs. Although with more training time and memory, our network uses less testing memory and has much less parameters while achieving better accuracy.
Table 6: The accuracy and testing time of uniform dispatch with C = 2.0 and 4.0 and full dispatch. The full dispatch clearly perform better with less time usage.
Method
Test
Train
Param.J
FLOPs)
Dispatch
PSNR1
SSIM
LPIPS]
Time (8)
Mem.
Time!
Mem.
Timel
Uniferm 2.0
17.82
0.410
0.563
131
Mcga-NeRF
6935M
87.9s
5124M
30.7h
10.8M
0.79M
Uniform 4.0
20.07
0.521
0.507
182
Switch-NcRF
5847M
110s
10182M
42.5h
4,53M
0.99M
Pull
21.54
0.579
0,474
110
Gating network. We analyze different designs of our gating network. As shown in Table 2 the Linear gating just feeds the PE(x) into a trainable linear layer. The w/o Norm version does not have a LayerNorm. We also evaluate the gating without the auxiliary loss Lu. The Linear gating with fewer parameters does not generate satisfactory results. The MLP+Norm can boost the performance of our whole network. The network without auxiliary loss I., does not even converge. This shows La is vital for the success of MoE. methods, consistent with the observations in (Sazzerr et al. (211))
Unified head. We remove the shared unified head and add head separately for each NeRF expert. Table 2 shows that without the unified head the accuracy significantly decreases. We also observe that in this case, the network is not robust enough to the mixed-pree -eexcedpreisson training and testing. A possible reason is that the gating value is directly multiplied to the predictions rather than the high- level features, which makes the gating and prediction unstable in training and testing. This verifies motivation to design unified head for multiple NeRF experts.
Scalability. Table 3 shows the accuracy and efficiency of Switch-NeRF with different numbers of experts. With only 4 experts,
already achieve similar accuracy to Mega-NeRF using 8 sub- networks. Compared to the model with 8 experts, the model with 16 experts increases the perfor- mance remarkably, while without a large increase of the memory and time in testing. Compared to Mega-NeRF with 16 sub-networks, our Switch-NeRF with 16 experts scales much better with a higher accuracy, fewer number of the parameter and less memory footprint in testing. All of these prove the scalability of our Switch-NeRR It can obtain much better results by increasing network capacity while maintaining almost constant computational and memory costs.
Effect of top-2 and capacity factor. Table 4 studies using Top-2 and C5 when training Switch- NeRF. Increasing Cj from 1.0 to 1.5 or using Top-2 in training can improve the accuracy while increasing the training time and memory, especially for Top-2. With our design of Switch-NeRF, Top-1 with C2 = 1.0 already obtains good results with acceptable efficiency, suggesting that in- creasing Cy instead of k is better because C' in training does not affect testing with the full dispatch.
Efficiency. In Table 5 we compare the efficiency of Switch-NeRF with Mega-NeRF. Our model achieves better accuracy using only half of the parameters of Mega-NeRF. Mega-NeRF uses back- ground NeRF and different versions of appearance embeddings (AE) for each sub-network, leading to much more parameters. As Switch-NeRF is trained end-to-end, it only uses one background NeRF and one version of AE. Our network is more parameter efficient and with better accuracy. Since has gating network and is trained end-to-end, it reasonably requires more floating point operations (FLOPs) for each point and costs slightly more memory and time for training. Notably, compared to Mega-NeRF, it uses around 20% less testing memory, and only a very minor increase in testing time. Our Switch-NeRF achieves good efficiency together with a better accuracy.
Full dispatch. We test our trained model with a uniform dispatch and our full dispatch discussed in Sec. 3.3 A capacity factor Cj of 2.0 and 4.0 with Batch Prioritized Routing (Riquelme et al.) 2021) is used in the uniform dispatch. Table 6 shows that the results of the uniform dispatch with C2 2.0 largely decreases. Although the results with Cj 4.0 improve, they are still far lower than the full dispatch. Moreover, it costs more time than the full dispatch because of a large Cj with zero padding. Fig. ssoos that images rendered with the uniform dispatch contain much more artifacts than the full dispatch, proving the effectiveness of the full dispatch in Switch-NeRF.
5 CONCLUSION
In this paper, we present Switch-NeRE. To the best of our knowledge, it is the first sparse large-scale NeRF with learnable scene decomposition. We propose an end-to-end Moxture-f-NeeRT-experr framework to learn scene decomposition jointly with NeRF. We further design and implement an efficient network architecture and full dispatch strategy to boost accuracy. Extensive experiments demonstrate that our network can learn more reasonable scene decomposition and shows state-of- the-art accuracy on the scene synthesis compared to hand-crafted decomposition methods.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research is supported in part by HKUST-SAIL. research funding, the Early Career Scheme of the Research Grants Council (RGC) of the Hong Kong SAR under grant No. 26202321 and HKUST Startup Fund No. R9253.
REFERENCES
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Bruall and Pratul P Srinivasan. Mip-nerf: multiscale representation for anti aliasing neural radiance fields. In ICCV, 2021.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR, 2022.
John Flynn, Michael Broxton, Paul Debevec, Matthew Du Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In CVPR, 2019.
Changho Hwang. Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of ofxture-of-experts at scale. arXiv preprint arXiv:2206.088 03388, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In ICI.R, 2021.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In ICML, pp. 6265-6274. PMLR, 2021.
Yilin Liu, Fuyou Xue, and Hui Huang. Urbanscene3d: A large scale urban scene dataset and simu- lator. arXiv preprint 1rXiv22177002200 2021.
Ricardo Martin Bruuala Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovit- skiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ToG, 2022.
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-moe: Advancing mixture-of-experrs inference and training to power next-generation Al scale. In ICMI., 2022.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural
radiance fields with thousands of tiny mlps. Proceedings of the IEEE/CVF International Con- ference Computer Vision, pp. 14335-14345, 2021.
Gernot Riegler and Vladlen Koltun. Stable view synthesis. In CVPR, pp. 12216-12225.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto. Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In NeurrPS, 2021.
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models.
NeurIPS, 2021.
Radu Bogdan Rusu and Steve Cousins, 3d is here: Point cloud library (pcl). In ICRA. IEBE, 201 1.
10

---

Published as a conference paper at ICLR 2023
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The spansely- gated mixture-of-experts layer. In ICLR, 2017.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. NeurIPS, 2018.
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srini- vasan, Jonathan T. Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. CVPR, 2022.
Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In CVPR, 2022.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler. Jonathan T Barron, and Pratul P Srini- vasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5491-5500, 2022.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 2004.
Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In ECCV, 2022.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 5752-5761, 2021.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Hao Zheng, Zhanlei Yang, Wenju Liu, Jizhong Liang, and Yanpeng Li. Improving deep neural networks using softplus units. IJCN, 2015.

---

Published as a conference paper at ICLR 2023
APPENDIX
A BUNGEENERF DATASET
We compare Switch-NeRF with BungeeNeRF (Xiangli et al,, 2022) on the 561 and Transamerica datasets provided from BungeeNeRF. We first analyze the different motivations and focuses between the BungeeNeRE and our Switch-NeRF. The BungeeNeRF focuses on modeling scenes with multi-scales and different levels of details, while Switch-NeRR targets large-scale scene modeling with large data volumes. BungeeNeRF has several special designs for training. It decom- poses the training data by predefined scales and uses multi-level supervision for its middle blocks. It designs special data feeding scheme and model growing training. We conducted experiments on the BungeeNeRF dataset, and show the performance comparison Table It is worth noting that the data volume of BungeeNeRF (around 450 images with 640x360 resolution for each scene) is similar the vanilla NeRF but is significantly smaller compared to that of Mega-NeRF (the Build- ing dataset with 1940 images of 4608x3456 resolution). Thus, we use less experts for BungeeNeRF dataset. As seen in Table 7 our Switch-NeRF with 4 experts can outperform the BungeeNeRF in all scales.
Table 7: Comparisons of PSNR on the 56neod and Transamerica scenes in BungeeNeRF. Switch- NeRF with 4 experts outperforms BungeeNeRF in all scales.
Scene
Method
Scale 1
Scale 2
Scale 3
Scale 4
Avg.
BungeeNeRF
24.120
24.345
25.382
25.112
24.513
56Leonard
Switch-NeRF-2
23.666
24.329
24.676
24.475
24.162
Switch-NeRF-4
24.624
25.263
25.844
25.697
25.196
BungeeNeRF
24.608
24.350
24.357
24.608
24.415
Transamerica
Switch-NeRF-2
24.468
24.233
24.410
23.495
24.220
Switch-NeRF-44
25.242
25.276
25.504
24.672
25.226
B BLOCK-NERF DATASET
We train a Switch-NeRF on the San Francisco Mission Bay Dataset of Block-NeRF (Tancik et al. 2022). It consists of 12,000 images with an image resolution of around 1200x900. Although the data volume is similar to the datasets used by Mega-NeRF and Switch-NeRF, the training setting of Block-NeRFis quite different from ours. The Block-NeRF uses much larger computing and memory resources its training setting. It trains each scene block with 32 TPU cores combining offer 512 GB memory, which is not easy to be followed by the common computer workstations. The batch- size of Block-NeRF for each block is 16384, significantly larger than the averaging batch-size 1024 for each sub-network in Mega-NeRF and Switch-NeRF. The networks of BlockNeRF with full precision while ours with 6 precision. Besides, an important part of the training data, i.c., the masks for dynamic objects in the scenes used by Block-NeRF, are also not released in the available training data. The Block-NeRF does not open-source its code, which makes us difficult align its syper-parameters on their training data.
trained
Due to our lack of comparable computing resources and the unavailability of the same training data used in Block-NeRF, is not feasible for us to train Switch-NeRF based on the same setting as Block-NeRF, to allow direct and fair comparison. We thus train our network on the Block-NeRF dataset with a similar setting used in our paper. We use 8 NVIDIA RTX 3090 GPUs to train a Switch- NeRF model with 8 experts, and sample 1 6664 rays for each GPU, resulting in a batch-size of 1664 on average for each expert. This far less than the batch-size of 16384 for each Block in Block-NeRF. The width of each layer is set as 512 in Block-NeRF. We also use the positional encoding of Mip-NeRF as used by Block-NeRF. We sample 256 points each ray for the coarse and fine networks in training, and 512 points each ray for testing. We the validation dataset of Block-NeRF to evaluate the performance with the PSNR, SSIM and LPIPS metrics. The experimental results are shown in Table
12

---

Published as a conference paper at ICLR 2023
Table Accuracy Switch-NeRF on the San Francisco Mission Bay Dataset proposed by Block- NeRF. We train the Switch-NeRF with limited computing resources, half float precision, and much smaller batch-size compared Block-NeRE Besides, Switch-NeRF does not utilize the important dynamic object masks that are used by Block-NeRI, they are not available in the released training dataset.
Method
PSNR
SSIM
LPIPS!
Switch-NeRF (8 experts)
23.86
0.762
0.489
C TWo GATING OPERATIONS
We add an additional ablation study on two gating operations. We train a Switch-NeRF with 2 gating networks, each acting as a gating operation. The first gating network is added after the 1-st linear layer of the main network, and the second gating network is added after 4-th linear layer of the main network. The total number of expert layers is the gating network to make them have similar network capacity. The results terms of both accuracy clearly see that two gating operations
as that of the Switch-NeRF using
and efficiency shown in Table From the results, cost more training time, training memory, and testing time, while achieving a similar accuracy to the critical
model with one gating operation. This suggests that the number of gating operations for the model performance, and one gating operation already shows sufficient capabilities in the dispatch of points.
Table 9: Accuracy and efficiency with different numbers of gating operations. The Switch-NeRF with two gating operations and the same number of expert layers, does not improve the rendering accuracy over the model with one gating operation, while consuming more memory and time for training and testing.
Train
Test
Gating number
PSNRT
SSIM
LPIPS
Mem.
Time
Mem.
Time
21.54
0.579
0.474
10182M
42.5h
5847M
110s
2
21.55
0.574
0.477
15315M
56.2h
5838M
152s
D MORE EXPERTS
The expert number directly controls the network capacity of Switch-NeRF, With large-scale scenes we may need to increase the expert number to get better results. In this section we add more ex- periments on the expert number. In Table TO the results show that more experts can consistently produce better results for larger-scale dataset.
We additionally visualize 3D radiance fields of Switch-NeRF trained with experts on UrbanScene3D-Sci-i dataset in Figure 7 Compared with the Figure b in the main paper, we can see that our network can still roughly specialized to different fine-grained semantic parts of the scene.
Table 10: The accuracy of different expert numbers. improve the accuracy.
Increasing expert number consistently
Building SSIM
Sci-Art SSIM
Campus SSIM
Method
PSNR1
LPIPS
PSNR
LPIPS
PSNR1
LPIPS
Switch-NeRF-8
21.54
0.579
0.474
26.52
0.795 0.819
0.360
23.62
0.541
0.609
Switch-NeRF-16
22.49
0.625
0.429
27.17
0.327
24.40
0.578
0,553
13

---

Published as a conference paper at ICLR 2023
All
Expert
Expert 8
Export 3
Expert II
Expert
Expert
Expert 13
Export 4
Expert 12
Figure The visualization of 3D radiance fields handled by experts in Switch-NeRF with 16 experts on the UrbanScene3D-Sen -ctiii-rt dataset.
E FAST RENDERING
We explore the capability of Switch-NeRF to be integrated into fast rendering techniques based octrees similar to the interactive exploration used in Mega-NeRF-Dynamic Turki et al. (2022). As the detailed evaluation protocol for the faster rendering is not provided by Mega-NeRI-Dynaniin
we thus define protocol to evaluate for both methods, on the validation images of each dataset. Specifically, we first convert the trained model into coarse octree. Given view point, we render coarse image directly from the coarse octree. Then, we follow the dynamic octree refinement strategy proposed by Mega-NeRF-Dynamii, to refine the octree for the current view point for several rounds by querying the trained model. Here, consider 16 rounds. We finally render refined image from the refined octree. We evaluate both Mega-NeRF model and Switch-NeRF model on the same fast-rendering protocol we just described, and report the PSNR and average refinement time octree Table I1 and images Figure 8 As shown Table [T] since the octrees before refinement very coarse, the coarse images for both Mega-NeRF and Switch-NeRF are of low quality. After the dynamic octree refinement, the quality of images improves. Since the resolution of refined octrees is still limited, the quality is not comparable the original model. However, our Switch-NeRF can obtain similar or better results on the validation datasets compared to Mega- NeRF It should be noted that the octrees do not handle the background, and thus the rendered images may have black regions in both Mega-NeRF and Switch-NeRF, which decreases the PSNR values. However, as shown Figure 8 our method can render good quality refined images in the foreground. This shows our Switch-NeRF can be flexibly and effectively integrated into existing faster rendering techniques.
The PSNRs of images rendered from the coarse octrees and refined octrees of Switch- NeRF and Mega-NeRF, and the average time used for dynamic octree refinement. Switch-NeRF can get similar or better results on the validation datasets compared to Mega-NeRF, Note that the octrees do not handles background the rendered images may have black regions in both Mega-NeRF and Switch-NeRF, which will decrease the PSNR values.
Table 11:
Building
Rubble
Residence
Sci-Art
Campus
Time
Coarse
14.66
14.99
12.15
10.86
14.95
Mega-NeRF
Refined
16.30
16.93
13.85
13.50
18.30
6.3s
Coarse
14.29
15.16
12.05
10.74
16.05
Switch-NeRF
Refined
16.29
17.04
13.96
13.61
18.66
13.0s
14

---

Published as a conference paper at ICLR 2023
Building
Sci-Art
Residence
Rubble
Campus
Ground Truth
Switch- Refined Mega- Refineefieddd Switch-- Switch-CCarse Coa-CCCasee Mega- Mega-Coarse Coarse
Figure 8: Images of Switch-NeRF and Mega-Ne NeRF rendered by coarse octrees and refined octrees. Note that the octrees do not handles background so the rendered images may have black regions in both Nega-NeRF and Switch-NeRF This will decrease the PSNR values. However, from the visu- alization, we can see that Our method can render good quality of refined images in the foreground.
15


Reviews:
[{"review_id": "t_ahf41bex", "reviewer": "Reviewer_QZyS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies large-scale neural rendering. To tackle the challenges of lacking universal scene decomposition, not learnable decomposition procedure, and independent sub-networks optimization, the paper proposes Switch-NeRF, a new end-to-end large-scale NeRF with learning-based scene decomposition. It designs a Sparsely Gated Mixture of Experts network to dispatch 3D points to different NeRF sub-networks. The gating network can be optimized together with the NeRF sub-networks for different scene partitions. The method achieves state-of-the-art performances on several large-scale datasets Mill 19 and UrbanScene3D.", "summary_of_the_review": "Considering the novelty, quality and some concerns of this paper. I vote for borderline now.", "strength_and_weaknesses": "Strength:\n- Overall, I appreciate the idea of applying a MOE network to large-scale neural rendering. Some detailed designs in this paper make it to be not so trivial (Not a simple \"A+B\" approach)\n- The paper shows its approach in detail, and the structure is clear.\n- The experiments are tested on different datasets, and the ablation studies are rich.\n\n\nWeaknesses:\nAlthough I am interested in the claims made by the authors, I have some questions and concerns. It will be better if the authors can provide more insights to the claims and experiments.\n\n- For existing methods, “the different sub-networks are typically trained separately”. How do the expert networks in this paper connect with each other? The separate training in Block-NeRF helps it achieve fine-grained results at each block. Does Switch-NeRF achieve better results than Block-NeRF. What are the benefits of training together? Or how does one expert affect the other?\n\n- It will be better if the paper can show some comparison and analysis with Block-NeRF and BungeeNeRF/CityNeRF[1]. The evaluation datasets Mill 19 and UrbanScene3D may not well prove the effectiveness of this paper in the large-scale setting. It might be better if the paper can provide some results on the same setting of Block-NeRF and BungeeNeRF/CityNeRF, e.g., Google Map/Earth.\n\n- The qualitative results are not convincing enough. It would be better if the authors could provide a supplementary video.\n\n- As shown in Figure 6, how do different experts in Switch-NeRF learn different semantic parts? What will different experts learn if Switch-NeRF deals with building-level cases? It is also interesting to see how different numbers of experts behave and affect performance.\n\n- Is there any failure case of Switch-NeRF? From Table 1, the improvement on campus/rubble is lower than sci-art/building. How to understand this? It is meaningful to see how Switch-NeRF behaves on different scales, which can help to inspire the community.\n\n- The paper designs “a deeper gating network to guarantee enough parameters to learn robust scene decomposition”. How robust is it?\n\n- Although the paper claims that “Multiple gating operations will remarkably influence training and testing speed.” It would be meaningful if the authors could provide some insights on what will happen if there is two gating operations. How slow will it be, and how will the performance change?\n\n\nReference:\n[1] Xiangli, Yuanbo, et al. \"BungeeNeRF: Progressive neural radiance field for extreme multi-scale scene rendering.\" The European Conference on Computer Vision (ECCV). Vol. 2. 2022, former version CityNeRF: Building NeRF at City Scale", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mFs1NkjEWA", "reviewer": "Reviewer_7KWF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a framework for joint learning scene decomposition and NeRF, targeting NeRF for large-scale scene. It is well-recognized that decomposing NeRF is needed for large-scale scene, i.e., a standalone NeRF model for a specific region, because of the model can be extremely large without decomposition and thus cannot be optimized with existing solutions. However, most large-scale NeRF use heuristic hand-crafted scene decomposition, i.e., decomposing into different blocks or balls in the 3D space, which maybe sub-optimal. Thus, this work propose to learn such decomposition configurations together to the NeRF model. Specifically, it inserts a gating function before querying each points' embeddings in MLP and such a gating function will assign points to different NeRF model experts. The experiments show that such a learnt decomposition can achieve better PSNR than heuristic hand-crafted  solutions.", "summary_of_the_review": "Overall, this paper provides some interesting insights to the community. It is nice to see learnable decomposition can surpass heuristic hand-crafted scene decomposition although it should be true intuitively.\nHowever, I do have some concerns on its efficiency, expandability to new environment, and ability of \"baking\" for fast rendering. I will raise my scores if the authors can solve these concerns. ", "strength_and_weaknesses": "# Strength\n> + Target an interesting problem: while most large-scale NeRF use heuristic hand-crafted scene decomposition, how to make the decomposition learnable and different for different datasets is an interesting direction to explore.\n> + Comprehensive experiments: The experiments, especially ablation studies are sufficient to support the proposed Switch-NeRF is effective to boost the PSNR of NeRF on large-scale scenes.\n> + Well written paper: the paper is well written and easy to follow.\n\n# Weaknesses\n> + Expandability to new environment: One advantage of NeRF with hand-crafted scene decomposition is that the independence of different blocks or decomposed areas are explicit, which enables the ability to expand the environment with additional NeRF models or update blocks without retraining the entire environment. It is unclear how the proposed Switch-NeRF can be expanded to new environment without retraining the entire environment because the decomposition assignment to previous environment maybe changed when new environment is involved in.\n> + Ability of \"baking\" for fast rendering: Mega-NeRF has shown its ability to be converted to Plenoctree or KiloNeRF-based models for interactive rendering, thus it may not be fair to compare with  Mega-NeRF  without such conversion in Table 3. Also, because the proposed Switch-NeRF has a gating function for the decomposition assignment and such assignment is implicit, it is not clear how Switch-NeRF can be converted to other representations for fast rendering like Mega-NeRF. If it can, not sure whether it will suffer from larger PSNR drop as compare to Mega-NeRF with explicit decomposition. Since speeding the rendering to real-time is an important feature for interactive view as mentioned in Mega-NeRF, it would be better if the authors can add more discussion on it.\n> + Need more analysis on the efficiency: as claimed by the authors, \"As Switch-NeRF is trained end-to-end, it only uses one background NeRF and one version of AE.\", the training cost  should not be larger than baselines, e.g., Mega-NeRF. However, it cost more than Mega-NeRF (42 vs. 30 h) as shown in Table 5. The corresponding explanation, \"Since it has a gating network and is trained end-to-end, it is reasonable that it requires more floating point operations (FLOPs) for each point and costs slightly more memory and time for training\", is not clear enough. What is main cause of more training cost? More difficult to optimize because of the gating function or larger models to forward in each iteration?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jvXkqSi7QS", "reviewer": "Reviewer_YY1K", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use a sparsely gated mixture of experts (MoE) model for NeRF models trained on large-scale scenes. The paper evaluates a variety of design choices and hyperparameters and compares results to Mega-NeRF, which also uses multiple NeRFs to model a large scene but simply uses the distance to a centroid to determine which NeRF to use.", "summary_of_the_review": "This paper does one thing – sparse MoEs for large-scale NeRFs – and does it well, exploring many different choices and ablations and showing what is needed to get this approach to work. The approach is motivated well in the discussion and by the results. In summary, this is a strong contribution to the growing field of large-scale NeRF modeling.", "strength_and_weaknesses": "Strengths:\n- Demonstrates good results, improving on rendering quality relative to Mega-NeRF while keeping parameters and FLOPs roughly constant.\n- Motivates the approach well and visualizes the result of the gating network by showing that different experts learn to handle different parts of the scene.\n- Includes a good about of ablations and comparisons to quantitatively evaluate the impact of their choices.\n\nWeaknesses:\n- (Minor) The use of the composition operator to describe the series of linear layers, softmaxes, ReLUs, etc, is non-conventional. Could be presented more clearly with a visual depiction or simply function application F(G(x)).\n- The presentation is general in the k (for Top-k) but in practice k=1. When k=1, it seems confusing why you would multiply the network output by G(x), since that's just a linear scaling factor. It also seems unclear why this would help the gating network train, although it is true that it makes it differentiable with respect to the output. If k > 1, then the gradient for a sample would favor the experts which provide a better input for that sample, but with k=1 this is less intuitive. I think the clarity of the paper could be improved either by elucidating this point, or perhaps by just rewriting this with k fixed to 1 instead of general.\n- (Minor) The authors observe that \"we observe that dropping sample points can significantly decrease the test accuracy.\" It is unclear to me what happens when a sample is dropped. Is the 2nd ranked expert used? Or is the value fed to the uniform head simply Linear(x)? In either case,  it seems inappropriate to drop samples, but would be helpful to understand what the alternative is.\n- (Minor) Missing parentheses in \"A capacity factor Cf of 2.0 and 4.0 with Batch Prioritized Routing Riquelme et al\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4smGAXGOyq", "reviewer": "Reviewer_qZQg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines a mixture of experts approach with NeRFs for large-scale neural scene rendering. The key idea is to learn a gating network which uses the positional embedding to choose which NeRF model will be queried for the density and color of a given point. All NeRFs share the same head. The gating network needs to be regularized, similar to other MoE models. \n", "summary_of_the_review": "I think this is a good paper, using MoE as a natural learnable alternative to handcrafted assignment strategies like those in BlockNerf and MegaNerf. The experiments answer a variety of interesting questions. The writing could be improved, but overall things make sense. ", "strength_and_weaknesses": "Using MoE here makes sense. The idea appears to be well implemented. The experiments are very thorough, and answer my main curiosities about the model details. \n\nThere are a few issues with the writing/clarity, which I list next, but overall the paper is good. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster