Rad-NeRF: Ray-decoupled Training of Neural Radiance Field
Authors: Lidong Guo
Xuefei Ning
Yonggan Fu
Tianchen Zhao
Zhuoliang Kang
Jincheng Yu
Yingyan Celine Lin
Yu Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural rendering field
Mutual learning
Novel view synthesis
Soft gate module
Complex scenes with occlusions
Primary area: machine_vision

Abstract: Although the neural radiance field (NeRF) exhibits high-fidelity visualization on the rendering task, it still suffers from rendering defects, especially in complex scenes. In this paper, we delve into the reason for the unsatisfactory performance and conjecture that it comes from interference in the training process. Due to occlusions in complex scenes, a 3D point may be invisible to some rays. On such a point, training with those rays that do not contain valid information about the point might interfere with the NeRF training. Based on the above intuition, we decouple the training process of NeRF in the ray dimension softly and propose a Ray-decoupled Training Framework for neural rendering (Rad-NeRF). Specifically, we construct an ensemble of sub-NeRFs and train a soft gate module to assign the gating scores to these sub-NeRFs based on specific rays. The gate module is jointly optimized with the sub-NeRF ensemble to learn the preference of sub-NeRFs for different rays automatically. Furthermore, we introduce depth-based mutual learning to enhance the rendering consistency among multiple sub-NeRFs and mitigate the depth ambiguity. Experiments on five datasets demonstrate that Rad-NeRF can enhance the rendering performance across a wide range of scene types compared with existing single-NeRF and multi-NeRF methods. With only 0.2% extra parameters, Rad-NeRF improves rendering performance by up to 1.5dB. Code is available at https://github.com/thu-nics/Rad-NeRF.

Rad-NeRF: Ray-decoupled Training of Neural Radiance Field
Lidong Guol+
Xuefei Ning I+f
Yonggan Fu²
Tianchen Zhao'
Zhuoliang Kang²
Jincheng Yu!
Yingyan (Celine) Lin
Yu Wang
TTiinguua University
Georgia Institute of Technology
Meituan
Abstract
Although the neural radiance field (NeRF) exhibits high-fidelity visualization on the rendering task, it still suffers from rendering defects, especially in complex scenes. In this paper, we delve into the reason for the unsatisfactory performance and conjecture that it comes from interference in the training process. Due occlusions in complex scenes, 3D point may be invisible to some rays. On such a point, training with those rays that do contain valid information about the point might interfere with the NeRF training. Based on the above intuition, we decouple the training process of NeRF in the ray dimension softly and propose Ray- decoupled Training Framework for neural rendering (Rad-NeRF). Specifically, we construct an ensemble of sub-NeRFs and train soft gate module assign the gating scores to these sub-NeRFss based on specific rays. The gate module is jointly optimized with the sub-NeRF ensemble to learn the preference sub- NeRFs for different rays automatically. Furthermore, we introduce depth-based mutual learning enhance the rendering consistency among multiple Sub-NeRFs and mitigate the depth ambiguity. Experiments on five datasets demonstrate that Rad-NeRF can enhance the rendering performance across a wide range scene types compared with existing single-NeRF and multi-NeRF methods. With only 0.2% extra parameters, Rad-NeRF improves rendering performance by up to 1. 1.5dB. Code is available at htttps://gthhub.com. wttp:gitwwwuooooooouoooo////////
Introduction
Novel view synthesis is an important task within the domains of computer vision and computer graphics, playing an essential role in variety of applications, such autonomous driving, augmented reality, and so on. Recently, Neural Radiance Field (NeRF) [17] has emerged as a promising solution, achieving high-fidelity visualizations on the novel view synthesis task. It implicitly encodes 3D scenes through neural networks and trains the networks using volume rendering.
Despite NeRF's excellent scene representation ability, it still suffers from rendering defects when dealing with complex scenes, such as 360-degree unbounded scenes [37] and large scenes with free shooting trajectories [30] 27 26). One of the main reasons is the limited model capacity. However, directly increasing the network's size yields marginal performance improvement (18).
Our fundamental intuition is that the training interference from invisible rays affects NeRF's performance. Let consider a simple case of 360-degree unbounded scene with central object
"Both authors contribute equally to this work. CCoresponding authors: Xuefei Ning (foxdoraame gom.l.commmmm Yu Wang (yu-wang Cooinngtssigggn (wwiingua.eeuu.cnn..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Sucaprised
Avkyosst
OPEN Class
Regioedd
Pryion
Ray elory
Trawning interference
(8) training ray (Ours)
sompling
Figure 1: A case in 360-degree unhounded scenes (bird-eye view). (a) For the distant object, invisible ray-3 interferes with 12y-1/2 training. (b) The ray-based multi-NeRF framework considers variable visibility of objects to different rays and decouples training in the ray dimension. (c) Compared to the theoretical weight distribution, the sampling along ray-3 inaccurate incurring training interference.
(truck) and distant object (car), As illustrated in Figure Ta). 3D point located on the distant object can be observed from ray-1 and ray-2, but is invisible to ray-3 due to the occlusion presented by the central object. Although NeRF models transmittance in its volume rendering formula, exhibits low geometric modeling accuracy and inaccurate sampling distribution in complex scenes, especially at the start of training. as Figure [c) shows. So, 3D points on the distant object might be sampled by the ray-3, and the model is trained on these points by the ray-3 color. However, ray-3 does not contain any meaningful information about the distant object, potentially interfering with the NeRI's training. In contrast, considering the different visibility of the object to different rays, our intuition is that rather than using one NeRF, assigning the rays terminating at the distant object to NeRF-1 and the rays terminating at the central object to NeRF-2 could be better, shown in Figure [
To verify the above intuition, we manually select two sets of images the TAT dataset 13 One set contains 80 images of the train's 's front side, while the other set includes the former set and 80 backside images. We train two NeRFs using these two sets respectively. As shown in Figure 2 the model trained on the mixed set performs worse the front side, which matches our intuition.
To mitigate the training interference caused by invisible rays, the intuition solution is decouple the training of the rays terminating at different regions. To this end, we propose a ray-decoupled training framework for neural rendering (Rad-NeRF). Within the Rad-NeRF framework, an ensemble of sub-NeRFs has different preferences for different rays through gate module. With the help of the gate module, sub-NeRFs' outputs are fused by nost-volume-rendering fusion yield final rendering results. Notably, the gate module is jointly optimized with NeRF, allowing it to automatically learn the preference of each sub-NeRF for various rays in end-to-ent manner. This learnable gating design makes Rad-NeRF generally applicable to diverse scenes, which stands in contrast to prior multi-NeRI methods 27 26 that rely on manually defined allocation rules.
Additionally, we design depth-basee mutual learning method for the multi-NeRt framework to ensure the rendering consistency among multiple sub-NeRFs. In addition learning colors, sub- NeRFs teach each other with their rendered depths. Traditional NeRF methods may struggle with generalization novel views despite accurately rendering training views, as they often fail to capture precise geometry [ 37 In contrast, our depth-based mutual learning approach serves as a form of geometric regularization, alleviating the depth ambiguity and avoiding overfitting.
212
Train
Train
Region B
Reglon
Region
PSN 18 GB9
PS4R-13 188--18.488
*Thatning decoupting
visuely
20
regintt provdc sybonooioe larget Acskatts traviing
infomsties whon cauree unooeetts mraving intefvencce
4 Comera Pose
Frost A
Block side
Witbout unining interfoccoee
Decoupling mitipates trunnigg interference
Figure 2: Oracle experiment: Training interference from invisible rays affects NeRF's performance.

---

To verify the effectiveness of Rad-NeRF, we conduct extensive experiments various types of datasets. The results show that Rad-NeRF can exhibit anti-aliasing effects and obtain supertor geometry modeling. thus consistently improving the rendering quality of novel views. In addition, Rad-NeRF is parameter-efficent and super simple to implement. With only 0.2% extra parameters, Rad-NeRF can increase rendering performance by up to 1.5dB compared to Instant-NGP. By scaling the number of sub-NeRFss through ray-wise decoupling, Rad-NeRF achieves better performance-to- parameter scalability than scaling other dimensions, such as the MLP width or the feature grid.
2
Related Work
2.1 Neural Radiance Field
Neural Radiance Field (NeRF) 17 has received much attention since it was proposed. uses MLPs
to implicitly represent 3D objects scenes, achieving realistic rendering results. There have been
intensive studies on NeRF's extension, including increasing NeRF's training/inference efficiency [36
2, 1851, I. applying NeRF to specific scenes hunge//mbouunddd/ooo--eetrrr
[13]
37
2 31 26
27 38), applying NeRF to other tasks (surface reconstructionnsccnne editing) [35] 20 29
14 33 32),
increasing NeRF rendering quality in few-shot setting [10 12 19 77. In this work, we aim to increase
NeRF's rendering quality in complex scenes, and propose a multi-NeRF training framework, which can leverage the techniques proposed by these single-NeRF researches.
2.2 Multi-NeRF Representation
Due to the limited model capacity, the multi-NeRF method is widely adopted to improve the rendering quality, which can be categorized into point- and ray-based multi-NeRF methods.
Point-based multi-NeRF method. These methods divide the 3D space in the point-dimension 30 37
38
3D points in different regions are computed by different sub-NeRFs. For example, NeRF++ proposes the sphere inversion transformation to map an infinite space to bounded sphere first, and it uses two NeRFs to model the foreground and background regions respectively. Switch-NeRF (38 also partitions the scenes in the point-dimension. These methods do not consider the different visibility of a target region to different views and cause training interference on complex scenes with many occlusions. For example, the front side of an object is not visible when it is observed from the back view or blocked by an occlusion. Training the sub-NeRF with rays that do not contain any valid information about the target region might interfere with the training.
Ray-based multi-NeRF methods. These methods allocate training rays to different sub-NeRFs and train sub-NeRFs independently. Block-NeRF (6 and Mega-NeRF (27] perform the ray allocation in the image-granularis granularity and sxve--raauularity respectively. Both of them need manually defined allocation rule, which requires prior scene knowledge and cannot be easily adapted to other types of scenes. The former work trains sub-NeRF's in large-scale road scenes with prior knowledge of the image shooting position distribution, and the latter one trains sub-NeRFs in open drone scenes and allocates the rays based on the ray intersecting positions with a horizontal plane. However, defining a ray allocation rule for complex scenes lacking prior scene-specific knowledge remains challenging. Another related work is NID (28), which proposes a mixture-of-expert NeRF for generalizable scene modeling. In this work, different experts serve as the basis to construct the implicit field of different scenes and the gating module takes in the new scene's 's image as the input (i.e., image-gras granularity).
In this work, we propose gate-guided multi-NeRF mutual learning framework, performing the allocation and decoupling the training in the ray dimension softly. Compared to other multi-NeRt methods, Rad-NeRF boosts the rendering quality without the need for prior scene knowledge.
3 Preliminary
uses neural networks to represent 3D scenes implicitly. Two MLPs model the density and color of spatial points respectively. The input of density MI.P F, is the 3D point coordinate The input of color MI.P F. includes view direction 0 and feature f output by density MIP. NeRF proposes the volume rendering method to render each pixel of an image. It samples N points along
NeRF 17
3

---

Randering
Gate-guided Nelti--eeRFFae fusion
Depth-based based mutual learning
Gating
Rendered color & degtt of Relf
Fused color & depth of the nay
inpouve rendering
Aort Volt--Vlume--ee weweenn---dddaagggooao Aurize guidedd module
Geomenite Regutebattin::
0,003
Rey
ΣΣ (6,00
0,09
Plaeed eapra
ocur
5,62
Color Sapervision
1, 2 1000-coy
Figure 3: The overview of Rad-NeRF. We construct a multi-NeRF framework based on the hybrid representation, where the feature grid shared for all sub-NeRFs and the MLP decoders are indepen- dent. (Left) Given ray, the soft gate module encodes the ray's data and outputs soft score. Then, guided by the gating score, sub-NeRFs' outputs are fused after the volume rendering process. (Right) The fused rendered depth of the ray is used to regularize each sub- RebFNs geometric encoding.
the ray and renders the pixel's color C(r) by discretely summing density o; and color of each point i, which approximates the integral ((r) as follows:
C(r)= ' w(t)c(t)dtt
C(r) = Σ W
(1)
T exp
w. = (1- -.ai) O5
(2)
where f is the distance between i-th sample's position and the starting point of the ray, 6; 44+1 t, the distance between adjacent samples and T; represents the probability that the ray travels from the start to point i without hitting. The NeRF optimization is based on color supervision.
Rad-NeRF
NeRF faces the challenge of limited model capacity when rendering complex scenes 37 30, 38
However, directly increasing the number of model parameters yields marginal improvement in the rendering quality (18). posing an important research question: "how to effectively scale up the capacity of NeRF". While the multi-NeRF methods have been proposed as an effective technique in response this question, they still face limitations handling complex scenes (with many occlusions and arbitrary shooting trajectories) due to training interference among invisible rays. In this work, we propose a ray-decoupled training framework (Rad-NeRF), effectively scaling up model's capacity by decoupling training in the ray dimension in a learnable way. Figure gives an overview of Rad-NeRF.
4.1 Gate-guided Multi-NeRF Fusion
Motivated by the intuition and oracle experiment discussed in Section I rather than using a single NeRF model, designing multi-NeRF structure that considers different visibility of the region to different rays and decouple NeRF's training in the ray-dimension could be better. We design a ray-based multi-NeRF model structure and introduce soft gate module to learn the preference of each sub-NeRF for various rays in a learnable way.
Multi-NeRF Structure. As shown in Figure D we employ a shared feature grid among sub-NeRFa and keep MLP decoders independent for the multi-NeRF structure. As different rays may pass through the same region of 3D space, weight sharing for the feature grid helps training, owing to feature grid's responsibility for encoding features of 3D spatial points. As validated by the Oracle experiment, training with regions A and B facilitates the training of the shared feature grid

---

and improves the rendering quality(PSNR 18.685 vs 18.488). Meanwhile, as the MLP decoder is designed to encode view information, constructing an ensemble of independent MLP decoders helps to decouple the training in the ray dimension, and thus maintains the preference of sub-Nett aub-NeRFs for
various rays. Additionally, such structure is a multi-model extension of Instant-NGPP [8]. helping to avoid a significant increase in the number of parameters and training complexity. The hybrid representation also maintains high training efficiency.
Soft Gate Module. We incorporate a soft gate module to assign gating scores to the sub- NeRIs for each ray. The soft gate module is jointly optimized with NeRF, enabling learn the preference of each sub-NeRF for different rays in an end-to-end manner. In contrast to manually assigning training rays to sub-NeRFs, this learnable gating design makes Rad-NeRF generally applicable
diverse scenes lacking prior scene-specifie knowledge. In Section 5.2] we will also show that the gate module can learn assign reasonable gating scores that reflect the object visibility of rays, aligning with our intuition that decoupling training in the ray-dimension is important.
Specifically, we employ a four-layer MLP followed by Sofumax function as the gate module. The gate module takes the starting point and direction (o,d) of a ray as the input, and outputs the gating scores G(r) of multiple sub-NeRI's associated with this ray. Instead of applying any sparsification strategies on the gating score G(r) as in previous work (38), such as top-k gating function [23] we use soft gating scores to enhance the smoothness and consistency of rendered results.
As discussed in Section 3 each ray corresponds to a pixel on the image. Following the volume rendering process,
obtain K rendered colors for each ray, where K is the number of sub- NeRFs. Subsequently, multi-NeRFs' outputs are fused a post-volume-rendrrnn the final rendering results. The fused color ((r) of the ray r can be written as below:
ordering to obtain
K
C(r) = (k())
(3)
k=1
where (r( is the k-th element of gating score G(r) and Cx(() is the rendered color of k-th sub-NeRF for the ray r.
4.2 Depth-based Mutual Learning
By the learnable soft gating design, different sub-NeRFs learn different encodings of the scene. We introduce a mutual learning method to enhance the rendering consistency and robustness of sub-NeRFs, wherein each sub-NeRF not only learns from ground truth but also learns from each other. Due to the lack of the ground truth for per-ray depth, NeRF may fail to learn accurate geometry despite accurately rendering training views, which adversely affects its generalization to novel views. To address this, we perform mutual learning with the rendered depths of sub-NeRFs, which serves as a form of geometric regularization and helps the model find more robust geometric solutions. The per-ray depth estimation D(r) can be written as Equation where t is the i-th sample's ss distance from the starting point on the ray.
D(r)
with
(4)
In practice, we first fuse the rendered depths of sub-NeRFs guided by the gating score G(r). Then use L2 distance to quantify the match of each sub-NeRF's rendered depth Dx(r) and the fused depth Drr). Our depth-based mutual learning loss is defined as below, where is the set of sampled rays:
Lidinl
ΣΣ FERA=1
(e(()) - ((r)1
Compared to directly averaging multiple sub-NeRFs' depth predictions, the gate-guided fused depth D(r) more accurate, as the gating score G(r) can reflect the prediction confidence of each sub-NeRF for the ray
5

---

Table Quantitative results in complex scenes.
TAT
NeRF-360-v2
Free-Dataset
Methods
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR1
SSIM
LPIPS..
NeRF++
20.419
0.663
0.451
27.211
0.728
0.344
24.592
0.648
0.467
MipNeRF360
22.061
0.731
0.357
28.727
0.799
0.255
27.008
0.766
0.295
MipNoRF360showt
20.078
0,617
0.508
25,484
0.631
0.452
24.711
0.648
0.466
DVGO
19.750
0.634
0.498
25.543
0.679
0,380
23,485
0.633
0.479
Instant-NGP
20.722
0.657
0.417
27.309
0.756
0.316
25.951
0.711
0.312
F2-NeRF
26,393
0.746
0.361
26.320
0.779
0.276
Switch-NGP
20.512
0.654
0.432
26.524
0.740
0.331
25.755
0.694
0.341
Block-NGPi 20.783
0.659
0.415
27.436
0.761
0.298
26.015
0.702
0.325
Rad-NeRF
21.708
0.672
0.398
27.871
0.769
0.298
26.449
0.719
0.285
MipNeRF360 roquires nearly one day for training. For a fair comparison, we also report its results with one-hour of training. 1
We adapt Switch-NeRF and Black-NeRF to the Instani-NGP fast training framework.
4.3 The Overall Training Loss
The overall loss function of Rad-NeRF given by:
== Le + Ay + XzLces
(6)
where T.e = ExER (C(r)- (()] (C(r) is the ground truth color value of ray r') the rendering loss. a and X22 are the weights for regularization terms, which are the only typer-parameters to be set. The value of Å chosen from 10-4 and 5 x 10-3 X2 set to x 10-2 on all the datasets. Lee is the balancing regularization on the Coefficient of Variation of the soft gating scores, which prevents the gate module from collapsing onto a specific sub-NeRF. The details of Ley are described and discussed in the Appendix B
5
Experiments
5.1 Datasets and Baselines
Datasets. We use five datasets from different types of scenes evaluate our Rad-NeRF. (1) Object dataset: we take Masked Tanks-And- Temples dataset (MaskTAT) [33 for evaluation, which are photographed objects with masked background; (2) 360-degree inward/outward-facii datasets: we take Tanks-And Aenns--sd-Temples/// (TAT) dataset with unmasked background [I3] and NeRF-360-v2 dataser [ to evaluate on scenes with large dynamic depth range; (3) free shooting-trajecton datasets: we conduct experiments on Free-Datasel (30) and ScanNel dataser (6) which are large outdoor and indoor scenes respectively. Both larger view ranges and more irregular shooting trajectories pose greater challenges for NeRF rendering.
Baselines. We compare our Rad-NeRF with two types of methods: one type uses the grid-based NeRF framework as we do, including PlenOctrees (36), DVGO [25), Instant-NGP [8] and F2-
NeRF 30 The other one is the MLP-based NeRF method, including NeRF NeRF++ [3]]
MipNeRF and MipNeRF330 2 which is inefficient in training and needs almost one day for training in complex scenes. Note that we also implement the NGP-version of Block-NeRF [26].
Switch-NeRF (38) and Mega-NeRF (27) to validate the superiority of Rad-NeRF to other multi-NeRF methods. The implementation details of Mega-NGP are shown the Appendix
5.2 Comparative Studies
Rad-NeRF achieves higher rendering quality than existing single- and multi-NeRF methods. We report the main quantitative results on the complex scenes and the object dataset in Table aand Appendix D respectively. Within no more than one hour of training, Rad-NeRF achieves higher rendering quality compared to other fast training methods and multi-NeRF methods, including Switch-NGP and Blo We can also see that while Rad-NeRF is designed for complex scene

---

Ground Trush
NoRF++
MinNoRF550 Soort
Instank NGP
Rod NeRF
TAT-train
Free-sky
360m2-rrom
Qualitative comparisons on three complex scenes. Rad-NeRF achieves better recovery of details for distant objects and less textured regions such as the wall. (Zoom in for the details, e.g.. sky, banister, roadblock, wall.)
Figure 4:
rendering, it can also improve the rendering performance of objects. We also integrate Rad-NeRF with the recent SOTA single-NeRE framework ZipNeRF [3], named Rad- Rad-ZipNeRRF, in the Appendix Rad-ZipNeRF obtains better rendering performance, validating Rad-NeRF's potential for integration with different frameworks.
Rad-NeRF achieves better recovery of distant details and accurate rendering for less textured regions. The qualitative results are shown in Figure 4 Compared to other methods, Rad-NeRF achieves better rendering quality in both outdoor and indoor scenes. In outdoor scenes, Rad-NeRF produces detailed and realistic rendering results for the sky and other distant objects. indoor scenes, Rad-NeRF generates more accurate details for less textured regions such as the wall. Rad-NeRF takes advantage the gate-guided training decoupling in the ray dimension to boost the model's performance effectively. Results the ScanNet dataset are shown in the Appendix C
The gate module learns reasonably assign gating scores. We visualize how the gate module performs training decoupling in Figure 5 As the two sub-NeRFs exhibit complementary gating (1) the Truck scene, the gate module regions, thereby mitigating
scores, we omit sub-NeRF2's visualization for brevity.
assigns different preferences to sub-NeRF in foreground/backgroum the interference from foreground rays on sub-NeRF1 training with the background region. (2) In the Train scene, sub-NeRF1 exhibits higher preferences for the back side, thereby mitigating the
Truck
Train
Catemillar
High
View-1
View-2
Low
Figure 5: Visualization of the gating scores of sub-NeRF1 on two different views (visualization of sub-NeRF2 is omitted for brevity).

---

204
god valinere fatur
202
PSNR: /d
28.
20.6
284
28
34.66
26.04
25
25.02 2536
20.6
35
2511
2012
Params MB
socne0044
9ccne0226
Figure 6: Scalability study of Rad-NeRF.
Figure 7: Convergence curve ScanNet.
training interference from invisible frontside rays. (3) In the Caterpillar scene, the gating module assigns different preferences foregroundd/acckrrund regions or the different sides of the caterpillar, which are clearly distinguished. The visualization demonstrates that Rad-NeRF learns reasonable ray allocations, matching our intuition. Besides, we observe that in some specific scenes, such as the Truck scene, the gating score visualization indeed shows a significant difference between the edge and the central region, correlating with the aliasing issue. Such observation illustrates that tackling the aliasing issue in some scenarios is another insightful explanation of the Rad-NeRF's effectiveness, which is supplementary to our original motivation targeting scenarios with heavy occlusions.
Scaling up NeRF with the Rad-NeRF framework more effective than scaling the MLP width, increasing the feature grid size, or adding more feature grids. By default, we set the number of sub-NeRFs to 2 in all experiments. As shown in Figure 6 when the number of sub-NeRFs increases, Rad-NeRF consistently obtains average performance gains the ScanNet dataset while only marginally increasing the number of model parameters. Compared with directly increasing the hidden dimension of MLP decoders or the size of the feature grid, Rad-NeRF has better performance- model size scalability. Furthermore, we observe that the model with four sub-NeRFs converges faster than the one with two sub-NeRFs while achieving better rendering quality with the same training iterations, as Figure 7 shows. The ease of training convergence can be attributed to two aspects. On the one hand, the number of learnable parameters and training complexity increases marginally. On the other hand, our gate module (a 4-layer MLP without sinusoidal position encoding) decouples the training in the ray dimension and reduces training interference.
5.3 Comparison with Gaussian Splatting
We additionally compare Rad-NeRF against 3D Gaussian splatting (3DGS) 11 non-neural approach that represents the current state of the art with regard to quality and rendering speed. The comparison is conducted on MaskTAT [13] and ScanNet [6 datasets. MaskTAT is object dataset without point clouds, and ScanNet contains indoor scenes with many less textured regions.
Rad-NeRF performs better than 3DGS in some cases. We report results on Table 9 and show qualitative highlights in Figure X For the MaskTAT dataset, we initialize 3D GS with random points. Our method performs best over 3D GS and Instant-NGP. For the ScanNet dataset, we initialize 3D GS with the point cloud provided by the dataset. However, there are many less textured regions in
3D GS
Instant-NGP
Rad-NeRF
MaskTAT
ScanNet
Barn
Methods
PSNR
PSNR1
3D GS
27.363
26.781 28.074
Instant-NGP
28.752
Rad-NeRF
29.774
28.870
Figure 8: Qualitative comparisons with 3D GS.
Figure 9: Quantitative results.
8

---

Ground Truth
w/o Ledit
wi Limii
Playground
Truck
Figure 10: Depth visualization comparison between w/o Lami and w/ Limt on TAT dataset. Zoom in see the details of sky and ground.
Table Ablation results of gate-guide sate-guided multi-NeRF fusion and depth- based mutual learning.
Method
Metric
M60
Playground
Train
Truck
Avg
PSNRT
19.229
22.863
17.531
23,569
20.798
Uniform fusion
SSIM
0.633
0.694
0.596
0.746
0.667
LPIPS
0.431
0.414
0.451
0.345
0.411
PSNRT 18,912
23.399
17.371
24.665
21.087
w/o depth mutual loss
SSIM
0.621
0.694
0,589
0.758
0.666
LPIPS
0.436
0,402
0.449
0.329
0.404
PSNR 19.051
23.901
19.369
24.509
21.708
Rad-NeRF
SSIM 0.631
0.689
0,612
0.757
0.672
LPIPS
0.429
0.402
0.431
0.333
0.399
indoor scenes that affect the accuracy and density of point clouds. Optical distortion exists in the rendered pictures of 3D GS. In contrast, Rad-NeRO renders more smoothly than all baselines.
Potential combination of Rad-NeRF with 3DGS. NeRF characterized by its neural network- based ray-related predictions, which provide flexibility for cross-scene generalization and enable the application of Rad-NeRF's ray-wise training decoupling approach. contrast, the plain GS framework parametrizes the scene using global, non-ray-related representation, making Rad-NeRF inapplicable. However, Rad-NeRF could potentially be applied to generalizable 3D GS frameworks that integrate neural network-based ray-related predictions 4
5.4 Ablation Studies
In this section, we conduct ablation studies on Rad-NeRF using the TAT dataset 13] takeaways from results summarized below. Some additional ablation studies and analyses presented in the Appendix
The key
Importance of the gate-guided multi-NeRF fusion and depth-based mutual learning. The ablation results the two key components are shown Table Uniform fusion simply averages multi-NeRFs' outputs to get final results without a gate module. In this way, sub-NeRFs focus on all the training rays instead of having their own preferences, which can not effectively improve rendering quality. For the depth-based mutual learning method, reasonable depth prediction, as shown Figure I0 In addition to improving rendering consistency, it also acts geometric regularization to reduce the depth ambiguity and avoid overfitting.
observe that it enables smoother and more
We further provide visualizations of different sub-NeRFs' rendering results in Figure [I which validates that the proposed basth mutual learning scheme will not encourage all sub-NeRFs to converge to the same output. On the one hand, the soft gating module allocates different rays to different sub-NeRFs, making them learn from different views. On the other hand, the depth bbad mutual learning scheme only lets sub-NeRFs learn the depth from each other rather than the overall rendered density or RGB distribution.

---

Sub-NeRFI Rendering
Sub-NeRf Bub-NeRF2 Rendering
Fused Rendering
Playground
WESTER
Thain
Figure 11: Independent and fused rendering results of sub-NeRFs TAT dataset.
Table 3: Ablation results of fusion dimensions.
Table 4: Ablation results of fusion granularity.
Fusion Dimension PSNR1
SSIM
I.PPS.I.
Fusion Gramularity PSNR1
SSIM
IPPPS.
Point-level
20.796
0.661
0.413
Image-level
21.503
0.669
0.408
Ray-level (Ours)
21.708
0.672
0.399
Pixel-level (Ours)
21.708
0.672
0.399
Importance of the ray-level allocation. We evaluate the results of different fusion dimensions in Table Compared to fusing multi-NeRE outputs in the point dimension, our ray-based method performs better, validating the superiority of the visibility-awarr multi-NeRF method.
Importance of pixel-granularity fusion. We compare different fusion granularity in Table 4 In aatagg-grannlariin fusion, all pixels of an image have the same preference for model parameters, which may not be reasonable. An illustrative example is an image capturing both the central object and the background region, such the Truck scene shown in Figure In such case, the rays hitting these two regions should be assigned different model parameters. In contrast, eoxel-granularity fusion provides a more fine-grained understanding of the image and scene.
6 Limitations
As the gating module 4-layer MLP without sinusoidal position encoding) incorporates smoothness prior implicitly, exhibits smooth and close scores to the nearest seen view for unseen views. Consequently, the generalization of the gating module relies on sufficient training data, and thus Rad-NeRF does not perform well in the few-shot setting (see Appendix K for more results). On the contrary, the proposed method suitable for the rendering of complex scenes, which themselves often require sufficient training data.
7 Conclusion
This work proposes a ray-decoupled training framework (Rad-NeRF) for neural rendering- To alleviate the issue of the training interference problem complex scenes, we construct a multi-NeRF framework and decouple the training of NeRFs in the ray dimension. Additionally, we propose depth-basee mutual lcarning method that improves the multi-NeRF rendering consistency and reduces the depth ambiguity, thereby improving generalization to novel views. Extensive experiments across various datasets validate Rad-NeRF's effectiveness and better performance performance-parametee parameter scalability.
We prospect for further exploration to fully exploit the potential of Rad-NeRF. Here, we outline several possible directions:()) As researchers may choose different frameworks based on specific situational requirements. adapting Rad-NeRF to different single-NeRF frameworks including 3D GS (non-neural approach) is valuable next step. (2) The number sub-NeRFs can determined automatically based on scene complexity and training resources.
(3) We hope the newly proposed scaling dimension, which increases the number of sub-NeRFs through ray-wise decoupling, will enable modeling of complex scenes in parameter-efffcient manner.
10

---

Acknowledgments
Lidong Guo, Xuefei Ning, Tianchen Zhao, Jincheng Yu, Yu Wang was supported by the National Key R&D Program of China (2023YFB4502200), the National Natural Science Foundation of China (No. 62325405, 62 104128, U2 U21B2031, 62204164) Tsinghua EE Xilinx Research Fund, Tsinghua- Meituan Joint Institute for Digital Life, and Beijing National Research Center for Information Science and Technology (BNRist).
References
[1] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5855-5864 (2021)
R., Srinivasan, P.P.:
[2] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5470-5479 (2022)
[3] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706 (2023)
[4] Charatan, D., Li, S.I.,, Tagliasacchi, A., Sitzmann, V:: pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19457-19467 (2024)
[5] Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: Tensorf: Tensorial radiance fields. In: European Conference Computer Vision. pp. 333-350. Springer (2022)
[6] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., NieBner, M.: Scannet: Richly- annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference computer vision and pattern recognition. pp. 5828-5838 (2017)
[7] Deng, K., Liu, A., Zhu, J.Y., Ramanan, D.: Depth-supervised nerf: Fewer views and faster training for free. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12882 12891 (2022)
[8] Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: Radiance fields without neural networks. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 5501-5510 (2022)
[9] Goli, L., Reading, C., Sellán, S., Jacobson, A., Tagliasacchi. A.: Bayes' rays: Uncertainty quantification for neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20061-20070 (2024)
[10] Jain, A., Tancik, M., Abbeel, P.: Putting nerf on diet: Semantically consistent few-shot view synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. (2021)
[11] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (2023)
[12] Kim, M., Seo. S., Han, B.: Infonerf: Ray entropy minimization for few-shot neural volume rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12912-12921 (2022)
[13] Knapitsch, A., Park, J., Zhou, Q.Y., Koltun, V: Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG) 36(4), 1-13 (2017)
[14] Liu, S., Zhang, X., Zhang, Z., Zhang, R., Zhu, J.Y., Russell, B.: Editing conditional radiance fields. In: Proceedings of the IEEF/VV international conference on computer vision. pp. 5773-5783 (2021)

---

[15] Baatila,, R., Radwan, N., Sajjadi, M.S., Barron, J.T., Dosovitskiy, A., Duckworth, D.: Nerf in the wild: Neural radiance fields for unconstrained photo collections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7210-7219 (2021)
[16] Mildenhall, B., Srinivasan, P.P., Ortiz-Cayon, R., Kalantari, N.K., Ramamoorthi, R., Ng, R., Kar, A.: Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (ToG) 38(4), 1-14 (2019)
[17] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.; Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM 65(1), 106 (2021)
[18] Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with multireso- lution hash encoding. ACM Transactions Graphies (ToG) 41(4), 1-15 (2022)
[19] Niemeyer, M., Barron, J.T., Mildenhall, B., Sajjadi, M.S., Geiger, A., Radwan, N.: Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5480-5490 (2022)
[20] Oechsle, M., Peng. S., Geiger, A.: Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5589-5599 (2021)
[21] Reiser, C., Peng. S., Liao, Y., Geiger, A.: Kilonerf: Speeding up neural radiance fields with
thousands of tiny mips. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 14335-14345 (2021)
[22] Sabour, S., Vora, S., Duckworth, D., Krasin, I., Fleet, DJJ., Tagliasacchi, A.: Robustnerf: Ignoring distractors with robust losses. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 20626 00266-00636 (2023)
[23] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outra- geously large neural networks: The aparsely-gated mixture-of-expers layer. arXiv preprint LtXiv:1701.06533 (2017)
[24] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le. Q. Hinton, G., Dean, J:: Outra- geously large neural networks: The sparsely-gater mixture-of-eeperr layer. arXiv preprint arXiv:1701.06533 (2017)
[25] Sun, C., Sun, M., Chen, H.T.: Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5459-5469 (2022)
[26] Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P.P., Barron, J.T., Kretzschmar, H.: Block-nerf: Scalable large scene neural view synthesis. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8248 8258 (2022)
[27] Turki, H., Ramanan, D., Satyanarayanan, M.: Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12922-12931 (2022)
[28] Wang, P.. Fan, Z., Chen, T., Wang, Z.: Neural implicit dictionary learning via mixture-off-exper training. In: International Conference on Machine Learning. pp. 22613-22624. PMLR (2022)
[29] Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 106. 10689 (2021)
[30] Wang, P., Liu, Y., Chen, Z., Liu, L., Liu, Z., Komura, T., Thcobalt, C.. Wang, W.: F2-nerf: Fast neural radiance field training with free camera trajectories. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 4150 4159 (2023)
[31] Wei, Y., Liu, S., Rao, Y., Zhao, W., Lu, J., Zhou, J.: Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In: Proceedings of the IEFF/CVF International Conference Computer Vision. pp. 5610-5619 (2021)
12

---

[32] Yang, B., Bao, C.. Zeng. J., Bao, H., Zhang. Y., Cui. Z., Zhang, G.: Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In: European Conference on Computer Vision. pp. 597-614. Springer (2022)
[33] Yang. B., Zhang. Y. Xu, Y. Li. Y., Zhou, H., Bao, H., Zhang, G., Cui, Z.: Learning object-compositional neural radiance field for editable scene rendering. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 13779-137777 13788 (2021)
[34] Yang, J., Pavone, M., Wang, Y.: Freenerf: Improving few-shot neural rendering with free frequency regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8254-8263 (2023)
[35] Yariv, L., Gu, J., Kasten, Y., Lipman, Y:: Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems 34, 4805 4815 (2021)
[36] Yu, A., Li, R., Tancik, M., Li, H., Ng, R., Kanazawa, A.: Plenoctrees for real-time rendering of neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5752-5761 (2021)
[37] Zhang, K., Riegler, G., Snavely, N., Koltun, V.: Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07992 (2020)
[38] Zhenxing, M., Xu, D.: Switch-nerf: Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In: The Eleventh International Conference on Learning Representations (2022)
13

---

Appendix
Table of Contents
A Comparison with Other Multi-NekF Methods
15
B Implementation Details
15
B.1
Implementation Details Rad NeRF
15
B.2 Implementation Details Switch-NGP
B.3
Implementation Details of Block-NGP
16
C
Experiments ScanNet Dataset
17
D Per-Scene Results
18
Additional Ablation Studies
18
F Discussion Mega-NGP
20
G More Scalability Studies
20
H The Training and Inference Efficiency of Rad-NeRF
22
Integration of Rad-NeRF and ZiR-NeRF
22
Additional Visualizations Gating Scores
23
K Limitation under the Few-shot Setting
23
Comparison of Rad-NekF with Uncertainty-bant based Methods
24
14

---

A Comparison with Other Multi-NeRF Methods
The comparison of various multi-NeRE training frameworks is summarized in Table NeRF++ 37 proposes the sphere inversion transformation to map an infinite space to a bounded sphere firstly, and uses two NeRFs to model the 3D points in foreground and background regions, respectively. adopts the manual allocation mode as it manually sets the boundary between foreground and background regions. Block-NeRF [26] and Mega-NeRF [27] are two classical ray- ray-based multi-Nett frameworks, which perform the ray allocation in the grangg-rrrllrtty and sraelllaaullarty. respectively. The former work trains sub-NeRFs in large-scale road scenes with prior knowledge of the image shooting position distribution on the road, and the latter one trains sub-NeRFs in open drone scenes and allocates the rays by partitioning the intersecting positions between the rays and a horizontal plane. However, they are designed for large road scenes and open drone scenes specifically and need manually defined allocation rule, which requires prior scene knowledge and cannot be easily adapted to other types of scenes. Switch-NeRF (38) implements learning-baad
scene partition scheme motivated by Mfxture-of--xpertts (MoE) 24 However, it partitions the scene in the point dimension,
which limits the rendering performance in more complex scenes with occlusions. It is also limited to be only used in open drone scenes. F2-NeRF [30 is another point-based multi-NeRE method, which allocates the 3D points to multiple sub-NeRFs in more elaborate but manual way.
In contrast, our Rad-NeRF performs the allocation and decoupling the training in the ray dimension softly. Acting as ray-based training framework, Rad-NeRF is Visibility-aware" and achieves higher performance in complex scenes. Moreover, compared other multi-NeRF methods, Rad-NeRF boosts rendering quality across differrnt types of scenes without the need for prior scene knowledge.
The "Dimension" column indicates the dimension in which the framework divides the training data into multiple sub-NeRFs; The " "Allocation mode" column indicates whether the framework divides the training data based on the manually designed rule or in a learnable way; The "Target scene" column indicates the scene that the framework is proposed for specifically.
Table S.1: Comparison of multi-NeRF training frameworks. Headers:
Multi-NeRF methods
Dimension
Allocation mode
Target scene
NeRF++
37
point-based ray-based
manual manual manual learnable manual
no constraint large road scene open drone scene open drone scene no constraint
Block-NeRF 26
Mega-NeRF 27
ray-based
Switch-NeRF 38
point-based
F2-NeRF [30
point-based
Rad-NeRF (ours)
ray-based
learnable
no constraint
B Implementation Details
B.1 Implementation Details of Rad-NeRF
Architecture Details. Our Rad-NeRF is built upon Instant-NGP (18) using a third-party PyTorch implementation [1] and costs no more than one hour of training. We follow the original architecture of Instant-NGP with 16 levels of resolution. The hash table length at each resolution is fixed to 219, The density and color MLP comprise one and two hidden layers with 64 channels respectively.
Training Details. For Instance-NGP and our Rad-NeRF, we train the NeRFs for 20k iterations on a single RTX-3090 GPU. We use Adam optimizer with batch size of 8192 rays and learning rate decaying from x 10 2 to x 10 4 For the weights of the regularization terms in Equation 6. X is set to 1 10-4 on NeRF-360-v2 and Free dataset, and is set 5 10-3 on other datasets. We set Xa to x 10-2 on all the datasets. By default, the number of sub-NeRFs is set to 2, and it is sufficient to achieve significant rendering quality improvement.
Some previous work has observed that the gate module tends to converge to an imbalanced state, 38 Such an imbalance
where it always produces large weights for the same few sub-models 23
15

---

Feature Grid
Gate Module
Top Function
Gating score
Point by
Expert
Point
Expert
Feature
Head
Point density & color
Expert
Expert
Figure S.1: The overview of Switch-NGP.
problem exists in Rad-NeRF well. Once the gate module is trapped in local optimum solution, it will always choose specific sub-NekF for rendering and can't effectively decouple the training in the ray dimension.
Following [23 288, we adopt the regularization on the Coefficient of Variation of the soft gating scores, which encourages a balanced allocation of model parameters for training rays. The CV loss function is given by
Var (R)
Lev
(E k=
GA (R)/n))
G (R) Σ Ga(r),
(8)
FER
where G(R) is the set (R)) k-1- Note that work also uses the load-balanced loss to encourage multi-models to receive roughly equal numbers of training examples 23 38 However,
this optimization objective is too strict and unsuitable for our framework.
B.2 Implementation Details of Switch-NGP
Switch-NeRF 38] constructs a point-based multi-NeRt framework based on MLP-based NeRF structure. Given 3D point æ, it first extracts high-level point feature S(x) using linear layer, which will be sent to the gate module to obtain the gating scores. Then, they apply a Top-1 function on the gating scores to determine which NeRF expert should be activated. The output feature of the chosen expert will be multiplied by the gating score corresponding to the expert and obtain the fused point feature. Finally, the fused point feature is sent to the unified MLPs predict the density and color
As illustrated in Figure S.I we build an NGP- of Switch-NeRF, named Switch-NGP Since NGP contains a feature grid in the form of the hash table, we directly use the feature grid to obtain the high-level point feature (xx) of the point Switch-NeRF has validated the importance of unified head, wherein the gating score is multiplied by the high-level features rather than the density color predictions, which makes the gating and prediction more stable in training. We also perform the multi-NeRt fusion the point-feature dimension by inserting extra K feature MLPs before the density MLP. Each expert in Switch-NGP corresponds to tiny feature MIL with two hidden layers and 64 channels.
The training details of Switch-NGP are the same as Rad-NeRF, as described in Section B.1
B.3 Implementation Details of Block-NGP
Block-NeRF 26 applies the multi-NeRF method to the street scene, which allocates model parame- ters in the ray dimension but in the image-level granularity. Specifically, Block-NeRF places one NeRF at each intersection and directly allocates the training images multi-NeRFs according the image shooting positions. We implement an NGP-version Block-NeRF, named Block-NGP, which
16

---

can be applied to various types of scenes without prior knowledge. After getting all the training images, we first use the clustering algorithm (KMeans) cluster the image shooting positions, and the number of clusters is set the same as the number of sub-NeRFs. During the training process, each training image allocated to the corresponding sub-NeRF according the clustering results, and the training of sub-NeRFs is independent.
C Experiments on ScanNet Dataset
We compare Rad-NeRF with other multi-Neke work on ScanNet dataset 6 Compared to other outdoor datasets, ScanNet contains more texture-less regions like the floors and the walls, which poses more challenges for neural rendering. We conduct experiments in four complete scenes in ScanNet, namely scene0046, scene0276, scene0515 and scene0673. The quantitative and qualitative results are shown in Table S.2 and Figure S.2 respecttiv... Our Rad-NeRF outperforms other multi-NeRF methods and renders less blur.
Ground Truth
Instant-NGG
Switch-NGP
Block-NGP
Rad-NcRt
sceneb0146
52cn60673
Figure S.2: Qualitative comparisons on ScanNet dataset. Compared to other multi-NeRF methods, Rad-NeRF renders less blur and achieves better recovery of details.
Table S.2: Quantitative results on ScanNet dataset.
Methods
Metrics
scene0046
scene0276
scene0515
scene0673
Avg
PSNRT 28.504
29,996
28.159
25.278
27.984
NGP
SSIM
0.839
0.835
0.786
0.686
0.786
LPIPS
0.413
0.421
0.448
0.472
0.438
PSNRT 28.135
29,614
27.814
25.140
27.676
Switch-NGP
SSIM
0.834
0,831
0.779
0.684
0,782
LPIPS
0.421
0.431
0.456
0.473
0.445
PSNR 28.728
30.214
28.332
25.444
28.180
Block-NGP
SSIM
0.842
0.840
0.789
0.688
0.790
LPIPS
0,408
0.416
0.443
0.469
0.434
PSNR1
29.440
30.871
29.149
25.759
28,805
Rad-NeRF
SSIM
0.851
0.843
0.800
0.690
0.796
LPIPS
0.396
0.405
0.427
0.469
0.424

---

D Per-Scene Results
We provide the per-scene quantitative results the Mask-TAT dataset, TAT dataset, NeRF-360-v2 dataset and Free dataset in Table [S.3] Table SS.4. Table S.3 and Table respectively. The results are
reported in the metric of PSNR.
Table S.3: Scene breakdown the Mask-TAL dataset.
Methods
Ignatius
Truck
Barn
Caterpillar
Family
Avg
NeRF MipNeRF
25.43
25.36
24.05
23.75
30.29
25.78
29.037
23.19
28.481
28.016
29.009
27.547
PlenOctrees
28.19
26.83
26.8
25.29
32.85
27.99
DVGO
28.16
27.15
27.01
26.00
33.75
28.41
Instant-NGP
28.431
27.562
27.611
26.065
34.092
28.752
Switch-NGP
28.184
27.34
27.472
25.75
33.711
28.491
Block-NGP
28.202
27.621
27.768
26.06
34.081
28.746
Rad-NeRF
29.806
28.163
28.701
27.445
34.756
29.774
Table S.4: Scene breakdown on the TAT dataset.
Methods
M60
Playground
Train
Truck
Avg
NeRF
16.86
21.55
16.64
20.85
18.975
NeRE++
17.964
22.914
18.194
22.603
20,419
MipNeRI-360
20.091
24.27
19.741
24.144
22.062
MipNeRF360.hoo short
18.394
22.682
17.738
21.497
20.078
DVGO
17.292
22.62
17.783
21.306
19.750
Instant-NGP
18.914
22.832
17.707
23.428
20.720
Switch-NGP
18.619
22.661
17.523
23.243
20.512
Block-NGP
18.879
22.555
18.048
23.651
20.783
Rad-NeRF 19.051
23.901
19.369
24.509
21.708
Table S.5: Scene breakdown on the NeRF-360-v2 dataset.
Methods
bicycle
bonsai
counter
garden
kitchen
room
stump
Avg
NeRF
21.818
29.028
26.980
23.640
27.164
30.097
22.934
25.952
NeRF++
21.426
31.670
27.717
24.801
29.468
30.621
24.770
27.210
MipNeRF360
22.861
32.970
29.291
26.014
31.987
32.685
25.278
28.727
MipNeRF360ssort
21.264
28.040
26.366
23.214
26.552
29.636
23.313
25.484
DVGO
21.652
27.919
26.432
23.851
26.282
31.677
20.988
25.543
F2-NeRF
21.311
30.036
25.873
23.694
28.935
29.421
24.251
26.217
Instant-NGH
24.203
31.374
25.665
25.312
30.278
31.534
22.799
27.309
Switch-NGP
23.859
30.012
24.359
25.164
29.865
31.127
21.284
26.524
Block-NGP
24.186
31.684
25.704
25,288
30.382
31.569
23.241
27.436
Rad-NeRF
24.550
32.439
25,230
25.634
31.062
32.863
23.312
27.871
E
Additional Ablation Studies
We add additional ablation studies the TAT dataset to further analyze the mechanism of Rad-NeRF, including structural design, depth-mutual learning, and Ca-baaaceed regularization. The results are shown in Table S.7
Gate-guided depth mutual learning. In Rad-NeRF we use the gate-guided fused depth the target depth to regularize sub-NeRFs' geometry and avoid overfitting. By contrast, when directly use
18

---

Table S.6:
Scene breakdown on the Free dataset
Methods
Hydrant
Lab
Pillar
Road
Sky
Stair
Grass
Avg
NeRF
16.569
17.342
20,944
19.793
15.925
18.731
22.439
18.820
NeRF++
22.948
23.718
26,353
24.916
25.059
27.647
21.504
24.592
MipNeRF360
25.03
26.57
29.22
27.07
26.99
29.79
24.39
27.008
MipNcRF360 short
23.281
24.412
26.789
24.158
25.369
27.139
21.827
24.711
DVGO
22.315
23.123
25.345
23.242
24.736
25.844
19.794
23.485
Instant-NGP
23.29
26.084
28.683
26.302
26.05
28.158
23.088
25.951
F2-NeRF
24.34
25.92
28.76
26.76
26.41
29.19
22.87
26.32
Switch-NGP
23.197
25.901
28.080
26.155
26.034
28.097
22.819
25.755
Block-NGP
23.663
26.682
28.103
25,989
26.283
28.395
22.988
26.015
Rad-NeRF
24.463
25.751
28.871
26.827
27.235
28.562
23.433
26.449
the average of the sub-NeRFs' rendering depths as the target depth, which means all sub-NeRFs have equal regularization strength (Equal DML), the rendering quality will be slightly worse. The results highlight the pivotal role of gate-guided depth mutual learning. Using the gated-guided fused depth the target depth differently penalizes sub-depths based the gating scores and increases the accuracy of the geometry regularization. We also observe that depth mutual learning has no effect in case of uniform fusion due to the low accuracy of the averaged depth.
CV balanced regularization. As introduced in Section B.I we adopt the regularization on the Coefficient of Variation of the soft gating scores to prevent the gate module from collapsing onto a specific sub-NeRF while maintaining sub-NeRF's different specialties. Without CV-balanced regularization, the rendering quality degrades significantly. Besides, we apply the CV regularization only for the first half of the training time and find that the performance is comparable to Rad-NeRF, The results prove that such regularization would not interfere with the learning of the gate module.
Table S.7: Additional ablation results.
Method
Metric
M60
Playground
Train
Truck
Avg
PSNRT
18.929
23. 108
19.012
24.625
21,419
Equal DML
SSIM
0.625
0.686
0.610
0.758
0.670
LPIPS
0.431
0.405
0.432
0.332
0.400
PSNR
18.765
22.839
18.958
24.493
21.264
Independent feature grids
SSIM
0.625
0.697
0.614
0.762
0.675
LPIPS
0.431
0,405
0.417
0,325
0.395
PSNR1
19.229
22.863
17.531
23.569
20.798
Uniform fusion w/o DML
SSIM
0.633
0.694
0.596
0.746
0.667
LPIPS
0.431
0.414
0.451
0.345
0.411
PSNRT
19.005
22.766
17.532
23.513
20.704
Uniform fusion w/ DML
SSIM
0,627
0.695
0.592
0.747
0.665
LPIPS
0.434
0.411
0.453
0.341
0.410
PSNR
18.743
22.795
17.245
23.395
20.545
w/o CV loss
SSIM
0.619
0.683
0.587
0.731
0.655
LPIPS
0.445
0.419
0.465
0.354
0.421
PSNR1
19.114
24.003
19.462
24.518
21.774
Half CV loss
SSIM
0.625
0.689
0.606
0.758
0.670
LPIPS
0.433
0.404
0.430
0.334
0.400
PSNR1
19.051
23,901
19.369
24.509
21.708
Rad-NeRF
SSIM
0.631
0.689
0.612
0.757
0.672
LPIPS
0.429
0.402
0.431
0.333
0.399
19

---

Structural design. In Rad-NeRF, we adopt multi-NeRF structure with a shared feature grid and an ensemble of MLP decoders. We further analyze the reason behind the performance improvement and explore the performance of independent feature grids. As Table (S.7 shows, the model employing shared feature grid (Rad-NeRty outperforms its counterpart with multiple independent feature grids, which highlights the effect of independent MLP decoders rather than feature grids. We attribute this observation and the performance gained by Rad-NeRF to two aspects. (1)Within the hybrid representation, the feature grid is responsible for encoding features of 3D spatial points, while the MLP encoder is designed to encode view information. The crucial design of independent MLP decoders aligns with our visibiliiy-ann aware motivation, thereby enhancing the view-dependent effect. Th) training complexity will also increase as the trainable parameters increase. With the limited amount of training data, increasing the number of feature grids leads to poor convergence. By contrast, as different rays may pass through the same region of 31) space, weight sharing for the feature grid helps to facilitate training. Although the number of learnable parameters hardly increases, Rad-NeRF decouples the training in the ray dimension, helping to increase the model's generalization ability.
F Discussion of Mega-NGP
applies the multi-NeRF method to the drone scenes, allocating model parameters in the ray dimension and the pixel-level granularity. Specifically, it allocates rays by partitioning the intersecting points between rays and scenes. Such method is suitable for drone scenes, where the top-down perspective allows for the approximation of ray-scene intersections by intersecting with a set horizontal plane. However, in unstructured scenes captured by free trajectories, the intersecting points between rays and scenes cannot be determined before the training is completed, limiting the applicability of Mega-NeRF to such scenes.
Mega-NeRF 27
Since there is no straightforward implementation to determine the ray intersections before training. adopt alternative implementation for NGr-version Mega-NeRF, which employs a clustering algorithm to divide rays directly based on their origins and directions. The clustering process is offline and the same as the one in Block-NGP. During the training process, each training pixel is allocated to one corresponding sub-NeRF according to the clustering results. To ensure fair comparison, the model structure of Mega-NGP is the same the one in Rad-NeRF, following the implementation of Block-NGP. We conduct comprehensive evaluation across all datasets and the experimental results shown in Table S.8 Mega-NGP yields similar results Block-NGP, which is less effective than our Rad-NeRF.
Table S.8: Comparison with Mega-NGP and Rad-NeRt
Method
Metric
TAT
360v2
Free Dataset
ScanNet
PSNR
20.843
27.482
25,855
28,100
Mega-NGP
SSIM
0.659
0.761
0.696
0.786
I.PPS.
0.415
0.311
0.332
0.437
PSNRI
21.708
27.87
26.449
28,870
Rad-NeRF
SSIM
0.672
0.769
0.719
0.797
LPIPS
0.399
0.298
0.285
0.424
G More Scalability Studies
We provide the per-scene results of scalability studies on the ScanNet dataset in Table S.9 which are reported in the metric of PSNR.
Furthermore, we observe that the model with four sub-NeRFs while achieving better rendering quality with the same training iterations, as Figure [S.3 shows. The ease of training convergence can be attributed to two aspects. On the one hand, the feature grid shared among multi-NeRF's and thus, the number of learnable parameters increases marginally. On the other hand, as the neural network is better at fitting low-frequency information, our gate module (a 4-layer MLP without sinusoidal position encoding) has implicitly incorporated "smoothness prior", leading to closer rays to be more possibly assigned closer gating scores.
converges faster than the one with two
20

---

Training Loss
Training Accuracy
Baseline
Rad-size2
Rad- size3
Rad-size4
scene0046
Baseline
Rad-size2
Rad-size3 Rad-sizee
Training
Trasking
Beseline
Rad Rad-sise2 Rad-sine3
0,9025
Rad-sizee
Training PSNR
scene0276
-2
Training
0.005
Baseline
27
Rad- Rad-size2
26
Rad-sine:
0.0035
Rad-sized
1590
16400
(3990
19000
19560 20000
2500
10008 12500
15800
17330 20008
Training steps
Training steps
0.903
Bascline
Rad-size2
0,9025
Rad-size3
Rad sized
Scene0515
Training loss
6,22
Training PSNR
Buseline
0001
Rad-size2
24
Rad-size3
Rad-size4
7000
19830
1290
20003
2500
5000
7500
12500
15000
17500
20006
Training steps
Training steps
Baseline
Rad-size2
251
0.35
Rad-sizo3
Scene0673
Training loss
6035
Training PSNR
21
Beseline
Rad-sine2
Rad-sines
Rad-size4
11500
2506
10000 12500
15600
17500 20036
Training steps
Training steps
Figure S.3: Convergence curve on the ScanNet dataset.
21

---

Table S.9: Scene breakdown of scalability studies on the ScanNet dataset.
Method
004600
027600
051500
067304
Avg
Instant-NGP
28.504
29.996
28.159
25.278
27.984
Rad-NeRR-size2
29.440
30.871
29.149
25.759
28.805
Rad-NeRF-size3
29.878
31.242
29.470
25.944
29.134
Rad-NeRF-size4
30.018
31.310
29.679
26.063
29.268
H The Training and Inference Efficiency of Rad-NeRF
We expand the scalability study in the main paper and supplement additional results about training time and inference speed. The comparison results are shown in Figure [S.4] Compared to the Instant- NGP baseline, all methods for scaling up NeRF's capacity require longer training time and exhibit lower inference speed, including scaling the MLP width and different multi-NeRF frameworks. Among these methods, Rad-NeRF achieves the best tradeoff between training/infereen and rendering quality. Since we adopt shared feature grid and multiple independent MLP decoders in the Rad-NeRF framework, a point feature needs to be processed by MLPs in turn, which is the major cause of reduced efficiency. However, multiple independent MLP decoders can be combined into a single MLP through appropriate parameter initialization and freezing, Rad-NeRF can obtain further efficiency improvements and approach the efficiency of scaling the MLP width.
efficiency
NGP with laggs MLP
NGP with luge: MLP
Inat
Indaant NCP
25.8
26.5
Swithh NGP
Switch NGP
28.5-
NeRl
25.5
Beal NeRK
20.4
PSVR
28:
24-
27.8
27.5
1500
2000
2.30
3000
3,00
4000
4500
5000
5500
6000
019
12
13
Triining Time sucond
Inferemce Speed/ EPS
Figure S.4: Scalability study about training/inferrncee efficiency.
I Integration of Rad-NeRF and Zip-NeRF
As a multi-NeRF training framework, Rad-NeRF is essentially orthogonal to the structure and training method of single-NeRF. For the benefit of training efficiency and its wide application, we build and validate Rad-NeRF upon the Instant-NGP. Nevertheless, it can also be integrated with other (a SOTA single-NeRF framework).
single-NeRF frameworks, such as ZipNeRF 3
We implement ZipNeRF version of Red-NeRF, named Rad-ZipNeRF, and evaluate the performance on the 360v2 dataset. Similar to Rad-NeRF, Rad-ZipNeRF adopts a shared feature grid and multiple MLP decoders. The training settings are kept the same as the original paper, including the training iterations and batch size. As shown in Table (S. 10) integrated with Rad-NeRF, ZipNeRF can also obtain performance gains, validating Rad-NeRF's effectiveness and potential for integration with different frameworks.
Considering that different frameworks have different characteristics, researchers may choose different frameworks based on specific situational requirements. Adapting Rad-NeRF to different single-NeRF frameworks remains an interesting point be explored in the future.
We further validate the performance of Rad-NeRF on the Free dataset As the results show, Rad-NeRFT nulti-NeRRF training framework boosts ZipNeRF's performance consistently.
22

---

Table S.10: Comparison with ZipNeRF and Rad Sad-ZippNeRe on the 360v2 dataset.
Methods
bicycle
bonsai
counter
garden
kitchen
stump
Avg
ZipNeRF 21.019
33.052
25.982
24.330
32.843
34.777
25.406
28,201
Rad Rad-ZipNeRF 20.488
33.486
26.372
24.603
33.120
35.795
25.581
28.492
Table S.11: Comparison with ZipNeRF and Rad- -ii-Nei on the free dataset.
Method
Hydrant
Lab
Pillar
Road
Sky
Stair
Grass
Avg
ZipNeRF
25.402
27.827
25.132
28.882
26.993
28.18% 187
18.461
25.841
Rad-ZipNeRE 25.51
28.067
25.348
29.191
27.491
28.339
18.572
26.074
J Additional Visualizations of Gating Scores
In the visualization results of the main paper, we adopt two sub-NeRFs in all scenes of the TAT dataset. With this setting, the two sub-NeRFs exhibit complementary gating scores for the same view and we omitted the visualization of sub-NeRF2 for brevity in the main paper. We also provide the visualization results of the other sub-NeRF in Figure S.5 As shown in Figure S.5 when rendering in an open scene with fewer occlusions, the gating score exhibits different characteristic and smooth transition according the ray directions. This visualization further validates our analysis that as 4-layer MLP without sinusoidal position encoding, the gating module incorporates smoothness prior implicitly. For unseen viewpoints, especially in less-oceluded outdoor scenes, the gating module exhibits smooth and close scores to the nearest seen view. The additional visualization results further prove that our original motivation for tackling heavy occlusion by decoupling sub-NeRF training is valid.
Sul-NeRF1
Sub-NeRF2
Sub-NoRF!
Sub-NeRF2
Higher
View-1
View-
Playground
Figure S.5: Additional visualizations of gating scores on two different views on TAT dataset.
K Limitation under the Few-shot Setting
Previously, we have included the discussion of the limitation under the few-shot setting. This is because rendering under the few-shot setting presents greater challenge to both NeRF's and gating module's generalization ability.
We validate Rad-NeRF's performance in the few-shot setting on the LLFF dataset 16 For 6/9 train- ing views, Rad-NeRF does not exhibit significant benefits or performance improvements compared to Instant-NGP, with all metrics at the same level. This is because insufficient training data affects the training and generalization of the gating module.
However, when rendering with extremely few training data (3 views), Rad-NeRF achieves significantly better rendering quality. We analyze that when training with very few views, the gating module has minimal impact on NeRF's training. Nonetheless, depth- mutual learning between multiple sub- NeRFs could still exhibit an effective geometric regularization effect, thereby improving rendering performance. This analysis is also validated by the visualization results shown in Figure compared the baseline, Rad-NeRF reduces the depth rendering ambiguity and shows better geometry modeling in 3-view setting.
23

---

Table S.12: Rad-NeRF's performance under the few-shot setting
PSNR1
SSIM
LPIPSL
Method
3-view
6-view
9-view
3-view
6-view
9-view
3-view
6-view
9-view
Instant-NGP Rad-NeRF
16,107 16.626
19.594
21.105 20.979
0,419 0.452
0.592
0.663
0.541
0.394
0.353
19.214
0.592
0.661
0.506
0.298
0.344
Inter.N07
Ral.NeRF
Tskeet XCP
Red NcRF
Teatr NGP
Rad Net
Figure S.6: Qualitative comparisons under three few-shot settings on LLFF dataset.
L Comparison of Rad-NeRF with Uncertainty-based Methods
Uinertaiitt--bsee methods consider floaters regions corresponding high uncertainty and remove them by thresholding the scene according to an uncertainty field during rendering. The spatial uncertainty is computed in roughly minute on any existing method. For example, RobustNeRF [2 treated pixels with larger losses as those with high uncertainty, avoiding the misleading effect of outlier points by discarding the training of those pixels. However, it is difficult to distinguish outlier points from the high-frequenee areas that should be learned. Morcover, Instant-NGP [18) regards the spatial points with too low density as regions with high uncertainty and filters these regions when rendering. Although this method works well, still cannot completely eliminate floaters in diffieult scenes and may remove correct regions. As post-hoc uncertainty assessment framework, Bayes Rays [9] acts as post-hoc uncertainty assessment framework, which does not change NeRF's training process, only removing "floater" regions corresponding high uncertainty. However, this solution is not stable and is generally used as an auxiliary solution to improve NeRF's rendering
quality.
Different from uncertainty- based methods, the proposed Rad-NeRF improves rendering quality by tackling the training interference issue. The depth-based mutual learning method also acts geometric regularization to reduce rendering defects. Importantly, Rad-NeRF is essentially orthogonal to these post-training uncertainty removal-based methods and can be integrated with Bayes Rays to obtain further performance improvement.
24

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We state the intuition and corresponding validation experiments in the intro-
duction section T
The contribution of ray-decoupled training framework is demonstrated
clearly.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We create a separate "Limitations" section O in the main paper. We point out the limitation of the proposed Rad-NeRF in the few-shot setting.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided all the needed information (r.g.ntra settings, hyper- meters, and so on) in the Appendix B ensuring the reproducibility of Rad-NeRF.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
26

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide all the code in the supplementary material, with detailed instruc- tions and execution scripts.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
public/guides/ Soaedeess/CodeSSummmsisiooooee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppllligggddpeeeeeeeeeee Cooee///ddeeebmimiii ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the necessary experimental details are stated in the Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Considering the negligible deviation of NeRF rendering results, we do not report enror bars in the quantitative results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We indicate the type of GPU and the time of execution (no more than one hour of training) in the Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: sectps::/neurips.tt httpp//pp....pppp.pptt/tccscsaea
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has no societal impact, since NeRF renders the real scene without any fake generation.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "EZLJQx76eT", "reviewer": "Reviewer_nJqP", "summary": "- The authors decouple the training process of NeRF in the ray dimension and propose a framework where they create an ensemble of sub-NeRFs and train a soft gate module to assign gating scores to these sub-NeRFs based on specific rays.\n- The gating module is a 4-layer MLP followed by a softmax function. The gating module takes in ray starting point and direction and outputs a gating score corresponding to each of the sub-NeRFs\n- The paper proposes a depth-based loss that compares the fused depth with the depth of the individual NeRFs in order to enhance their robustness.\n- Multiple indoor and outdoor scenes from various public datasets were used to show the method's effectiveness.", "questions": "- Uncertainty-based methods, like Bayes Rays, consider floaters as  regions corresponding to high uncertainty and remove them by thresholding the scene according to an uncertainty field during rendering. The spatial uncertainty is computed in roughly a minute on any existing method. What is the advantage of Rad-NeRF over such uncertainty-based methods? \n- Cross-Ray NeRF (CR-NeRF) leverages interactive information across multiple rays to synthesize occlusion-free novel views with the same appearances as the images. They recover the appearance by fusing global statistics, i.e., feature covariance of the rays and the image appearance. Can the method proposed in this paper solve the problems solved by Rad-NeRF?", "limitations": "Yes, the authors have discussed the limitations of the method.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper claims to improve PSNR by 1.5dB with just 0.2% extra parameters.\n- The authors have shown the plug-and-play capability of their method by adding it to existing methods like Zip-NeRF.\n- While methods like Block-NeRF and Mega-NeRF need a manually defined allocation rule and prior scene knowledge, Rad-NeRF does not require prior scene knowledge.", "weaknesses": "- Though the paper claims that their method needs few extra parameters and trains quickly, there only seems to be a slight improvement in quantitative numbers across many scenes.\n- The authors have pointed out that the method does not work for few-shot settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.\n\n**R1-Q1: Performance of Rad-NeRF**\n\nOverall, compared to the Instant-NGP baseline, our scheme shows improvements in the PSNR metric across various datasets: 1.02 on mask-TAT, 0.98 on TAT, 0.57 on 360v2, 0.49 on free dataset and 0.82 on ScanNet dataset. All the improvements are achieved with the default configuration of only two sub-NeRFs (0.02MB extra parameters). Notably, as the number of sub-NeRFs increases, an additional 0.4 PSNR gain is obtained on the ScanNet dataset. We also check the performance of recently published and related work, such as F2-NeRF, which implements a multi-NeRF framework on Instant-NGP. F2-NeRF claims the PSNR metric improvements of 0.15 on 360v2 and 1.91 on the free dataset, which is on par with the Rad-NeRF effect. \n\nAdditionally, on the 360v2 dataset, ZipNeRF obtains 0.74 PSNR gain compared to MipNeRF360. Furthermore, we implement the Rad-ZipNeRF framework (a combined version of Rad-NeRF and ZipNeRF) and achieve an additional 0.3 PSNR improvement. The experimental results prove Rad-NeRF's effectiveness and potential for integration with different single-NeRF frameworks.\n\nFinally, Rad-NeRF goes beyond comparison with existing SOTA single-NeRF methods and extends to the comparison with other multi-NeRF training frameworks based on NGP. In these comprehensive evaluations, Rad-NeRF shows superiority over other multi-NeRF training methods. \n\n**R1-Q2: Limitation under the few-shot setting**\n\nPreviously, we have included the discussion of the limitation under the few-shot setting. This is because that rendering under the few-shot setting presents a greater challenge to both NeRF's and gating module's generalization ability.\n\nWe validate Rad-NeRF's performance in the few-shot setting on the LLFF dataset. For 6/9 training views, Rad-NeRF does not exhibit significant benefits or performance improvements compared to Instant-NGP, with all metrics at the same level. This is because insufficient training data affects the training and generalization of the gating module. \n\n|         |    |  PSNR     |    |      |  SSIM       |  |      |  LPIPS        |  |\n|:-----------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n| Method        | 3-view | 6-view | 9-view | 3-view | 6-view | 9-view | 3-view | 6-view | 9-view |\n| Instant-NGP | 16.107 | 19.594 | 21.105 | 0.419  | 0.592  | 0.663  | 0.541  | 0.394  | 0.353  |\n| Rad-NeRF    | 16.626 | 19.351 | 20.979 | 0.452  | 0.595  | 0.661  | 0.506  | 0.399  | 0.344  |\n\nHowever, when rendering with extremely few training data (3 views), Rad-NeRF achieves significantly better rendering quality. We analyze that when training with very few views, the gating module has minimal impact on NeRF's training. Nonetheless, depth-based mutual learning between multiple sub-NeRFs could still exhibit an effective geometric regularization effect, thereby improving rendering performance. This analysis is also validated by the visualization results in the Global Response. As shown in Figure R.2, compared to the baseline, Rad-NeRF reduces the depth rendering ambiguity and shows better geometry modeling in a 3-view setting.\n\n**R1-Q3: What is the advantage of Rad-NeRF over Bayes Rays?**\n\nAs a post-hoc uncertainty assessment framework, Bayes Rays does not change NeRF's training process, only removing \"floater\" regions corresponding to high uncertainty. However, this solution is not stable and is generally used as an auxiliary solution to improve NeRF's rendering quality. \n\nFor example, RobustNeRF treated pixels with larger losses as those with high uncertainty, avoiding the misleading effect of outlier points by discarding the training of those pixels. However, we validated its effect and found that it was difficult to distinguish outlier points from the high-frequency areas that should be learned. Moreover, Instant-NGP regards the spatial points with too low density as regions with high uncertainty and filters these regions when rendering. Although this method works well, it still cannot completely eliminate floaters in difficult scenes and may remove correct regions. \n\nDifferent from uncertainty-based methods, the proposed Rad-NeRF improves rendering quality by tackling the training interference issue. The depth-based mutual learning method also acts as a geometric regularization to reduce rendering defects. Importantly, Rad-NeRF is essentially orthogonal to these post-training uncertainty removal-based methods and can be integrated with Bayes Rays to obtain further performance improvement. We will incorporate the discussion into the revision.\n\n**R1-Q4: Can the method proposed in CR-NeRF solve the problems solved by Rad-NeRF?**\n\nCR-NeRF works similarly to style transfer/matching NeRF methods, which are primarily proposed to address the challenges of dynamic appearance caused by different capture times and camera settings. Its goal is to control the style/hue of the rendered image from unconstrained image collections. Differently, Rad-NeRF is proposed to tackle the training interference issue in complex scene rendering and improve the rendering performance in normally shot complex scenes. \n\nCR-NeRF proposes to leverage rays' information for modeling appearance, which is an extension and improvement of the appearance modeling scheme in classical NeRF-W. We used to validate the performance of NeRF-W's appearance modeling method on the issue of floaters before and found nearly no improvement. This is because the problem of floaters and other rendering defects in complex scenes is mainly caused by training interference and inaccurate geometry modeling rather than appearance inconsistency. Compared to CR-NeRF, the proposed Rad-NeRF is more suitable to tackle such challenges.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RdRKdvRLZv", "reviewer": "Reviewer_tZHj", "summary": "Traditional NeRF models face challenges in rendering complex scenes due to interference from occluded rays, which leads to inaccurate training data. To address this, the authors propose Rad-NeRF, a framework that decouples the training process in the ray dimension by using multiple sub-NeRFs, each trained with specific rays that contain valid information for the points they observe.\nRad-NeRF employs a soft gate module that assigns gating scores to different sub-NeRFs based on the rays, allowing the model to learn preferences for different rays. Additionally, the framework introduces depth-based mutual learning to enhance consistency and mitigate depth ambiguity among the sub-NeRFs. Essentially, my understanding is that this work builds on Instant NGP by adding multiple sub-MLPs, with the gate module controlling the degree of participation of each MLP during rendering. Experiments demonstrate that Rad-NeRF outperforms existing single and multi-NeRF methods, improving rendering performance with minimal additional parameters.", "questions": "See Weakness.", "limitations": "See Weakness.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-organized, with a clear abstract, introduction, methodology, experiments, and results sections.\n2.\tThe innovative approach of decoupling training in the ray dimension helps mitigate training interference caused by invisible rays. This leads to more accurate and consistent rendering results.\n3.\tThe introduction of a soft gate module that learns the preference of sub-NeRFs for different rays is a novel and effective way to allocate rays dynamically. This approach eliminates the need for manually defined allocation rules, making the method more adaptable and generalizable.\n4.\tRad-NeRF enhances rendering consistency among multiple sub-NeRFs through depth-based mutual learning. This technique also helps reduce depth ambiguity and improves geometric modeling accuracy.\n5.\tRad-NeRF can be integrated with various single-NeRF frameworks, further improving their performance. For instance, integrating with ZipNeRF shows potential for even better rendering outcomes, validating the flexibility and compatibility of Rad-NeRF with different neural rendering approaches.", "weaknesses": "1.\tWhat is the structure of the grid in the network architecture? Is it a multi-resolution grid? Are the features on the grid also learnable?\n2.\tWhat are the training and inference times for this method? It would be beneficial if the authors provided a comparison with other existing methods in the paper.\n3.\tIn the case of multiple networks, if a sub-NeRF receives very low soft gating scores, does it cease to update, or does it still contribute to the computation?\n4.\tAre there any automated methods or heuristics for selecting the optimal number of sub-NeRFs based on scene complexity? What is the relationship between the number of sub-models and the overall performance? Detailed tuning and experimentation are required to ensure optimal performance across different datasets and scenes.\n5.\tCould you provide a visualization of the gating scores for different sub-NeRFs for the same viewpoint? And I am curious about the distribution of gating scores in scenarios without occlusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.\n\n**R2-Q1: Clarification of the network structure**\n\nWithin Rad-NeRF, we adopt a multi-resolution learnable feature grid shared among all sub-NeRFs. Given a 3D point coordinate, we find the surrounding voxels at different resolution levels, index the hash table according to each vertex position, and obtain the final point feature through linear interpolation. During training, loss gradients are backpropagated through the multiple independent MLP decoders, the gating module, and then accumulated in the looked-up learnable feature vectors. We will demonstrate this detail clearly in the revised version.\n\n**R2-Q2: Comparison of training and inference time**\n\nFollowing the reviewer's advice, we expand the scalability study in the main paper by supplementing additional results about training time and inference speed. The comparison results are shown in Figure R.1 (Global Response). Compared to the Instant-NGP baseline, all methods for scaling up NeRF's capacity require longer training time and exhibit lower inference speed, including scaling the MLP width and different multi-NeRF frameworks. Among these methods, Rad-NeRF achieves the best tradeoff between training/inference efficiency and rendering quality. Since we adopt a shared feature grid and multiple independent MLP decoders in the Rad-NeRF framework, a point feature needs to be processed by MLPs in turn, which is the major cause of reduced efficiency. However, as multiple independent MLP decoders can be combined into a single MLP through appropriate parameter initialization and freezing, Rad-NeRF can obtain further efficiency improvements and approach the efficiency of scaling the MLP width.\n\n**R2-Q3:  Does the sub-NeRF cease to update or contribute to the computation if it receives very low soft gating scores?**\n\nIn the Rad-NeRF training framework, even if a sub-NeRF is assigned an extremely low gating score for a particular ray, it still contributes to the rendering of the scene and remains effectively updated.\n\nFirstly, the gating score represents each sub-NeRF's preference for different rays. Although a sub-NeRF is assigned a low score on a certain ray, it still has preferences and contributes to other rays.\n\nSecondly, we implement CV balancing regularization in the training of Rad-NeRF. This regularization prevents the gating module from collapsing onto a specific sub-NeRF while maintaining the different specialties of each sub-NeRF. Consequently, it is rare for a sub-NeRF to contribute extremely little to the total scene during training. \n\nThirdly, depth-based mutual learning is proposed as a geometry regularization to keep sub-NeRFs learning from each other. Even if a sub-NeRF is assigned a low gating score and learns little from the ground truth color value, it still learns from the fused depth value and remains effectively updated.\n\n**R2-Q4: Are there any automated methods or heuristics for selecting the optimal number of sub-NeRFs based on scene complexity?**\n\nWe appreciate the reviewer for the insightful and forward-looking question. In this work, considering the limited computing resources, we mainly adopt the default configuration of two sub-NeRFs and extend it to four sub-NeRFs in the scalability study. Intuitively, more complex and larger scenes require more sub-NeRFs, and the rendering performance of Rad-NeRF will gradually saturate as the number of sub-NeRFs increases. We also conduct additional scalability experiments on MaskTAT, a less complex dataset. As the results show,  the rendering quality saturates when using three sub-NeRFs,  which is similar to the phenomenon observed on the ScanNet dataset, where performance saturation occurs at four sub-NeRFs.\n\n| Method   | Metric   | Baseline   | Size2  | Size3  | Size4  |\n|:--------:|:--------:|:------:|:------:|:------:|:------:|\n| Rad-NeRF | PSNR     | 28.752 | 29.774 | 29.934 | 29.993 |\n|          |     SSIM | 0.914  | 0.920   | 0.925  | 0.924  |\n|          | LPIPS    | 0.140   | 0.130   | 0.125  | 0.124  |\n\nUnder such circumstances, determining the optimal number of sub-NeRFs in the Rad-NeRF training framework is an important problem, which we also point out in the conclusion and outlook of the main paper. A possible approach is to adjust the number of sub-NeRFs dynamically during the training process. By setting an appropriate score threshold, sub-NeRFs with average scores lower than the threshold will be removed. Besides, if most sub-NeRFs do not show a significant preference, additional sub-NeRFs can be added. \n\n**R2-Q5: The visualization of the gating score of different sub-NeRFs for the same viewpoint or in scenarios without occlusions.**\n\nIn the visualization results, we adopt two sub-NeRFs in all scenes of the TAT dataset. With this setting, the two sub-NeRFs exhibit complementary gating scores for the same view and we omitted the visualization of sub-NeRF2 for brevity in the main paper. We supplement the visualization results of the other sub-NeRF in Figure R.4 (Global Response).\n\nWe further provide the gating score visualization in a less-occluded outdoor scene (Playground scene of TAT dataset). As shown in Figure R.4, when rendering in such an open scene with fewer occlusions, the gating score exhibits different characteristic and smooth transition according to the ray directions. This visualization further validates our analysis that as a 4-layer MLP without sinusoidal position encoding, the gating module incorporates smoothness prior implicitly. For unseen viewpoints, especially in less-occluded outdoor scenes, the gating module exhibits smooth and close scores to the nearest seen view.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RdAIOQAAFu", "reviewer": "Reviewer_PUvY", "summary": "This paper proposes an innovative approach to enhance NeRF performance. The key observation is that due to occlusion, some objects may appear in one ray but not in another. While NeRF uses transmittance to mitigate this issue, the paper argues that this may not be sufficient. To address this, the authors propose using multiple MLPs to decode the same feature and jointly learn a gating function to fuse all the information. The results demonstrate that this design successfully improves performance, albeit by a small margin.", "questions": "See above", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed idea is innovative and interesting.\n+ The approach results in a performance improvement.", "weaknesses": "I have several concerns regarding the motivation, methodology, and validation of the proposed approach.\n\n## Motivation: \n\n+ The motivation is not very clear. Specifically, Figure 1 and lines 29-40 argue that poor performance is due to occlusion, leading the authors to propose using multiple MLPs. However, I find Figure 1, especially Figure 1(c), inaccurate. I would expect a CDF instead of a PDF, as NeRF often compensates RGB appearance with poor geometry. Thus, (c) might still have the correct RGB, i.e., its CDF is accurate. \n+ Moreover, Figure 7 does not support the occlusion hypothesis. I suspect the issue might be related to aliasing, where the pixel footprint differs between the center and boundary regions. This can also explain why MipNeRF360 outperforms the proposed method, and the comparison between ZipNeRF and RadZipNeRF in the supplementary material is very close.\n\n## Methodology: \n\nThe approach resembles MOE or a multi-head transformer. However, I have two questions:\n\n+ First, the proposed method applies depth consistency. Isn't this encouraging all MLPs to converge to the same output? This might explain why rad2, rad3, and rad4 in Figure 6 are very similar.\n\n+ Second, regarding the soft gating, the current input is the ray origin and direction. It might be more effective to include the 3D position, as splitting MLPs based on different spatial regions could be more reasonable. Additionally, the analysis of the gating function (Figure 7) lacks depth, and more details would be appreciated.\n\n## Validation: \n\n+ While the rad approach does improve performance, the improvement is relatively marginal. Furthermore, MipNeRF360 outperforms the proposed method, indicating that the proposed approach is not strong enough. \n\n+ In Table 1, why are switch NeRF and block NeRF not better than instant NGP? After ray participation, they should outperform the vanilla version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and thoughtful comments. We address the detailed questions and comments below.\n\n**R3-Q1: Clarification of Figure.1(c) and motivation**\n\nAs the reviewer said, NeRF often compensates for RGB appearance with poor geometry. Figure 1(c) might have the correct RGB with inaccurate geometry. This can be seen in the inaccurate sampling probability presented in Figure 1(c), i.e. the PDF distribution of wi in Equation (1). With Figure 1(c), we aim to illustrate that as the NeRF model is apt to learn a poor geometry, undesirable inter-ray training interference can occur during training. Specifically, when rendering ray-3, it is possible to sample the 3D points near the distant object due to inaccurate sampling. Therefore, the modeling of the distant object will be affected by ray-3 color supervision, resulting in the issue of training interference, which is the main motivation of our work. \n\nWe're not very sure how the reviewer thinks Figure 1(c) should be changed. Could you please provide more details on this comment? We are eager to understand your perspective better and address any concerns you might have.\n\n**R3-Q2: Figure 7 does not support the occlusion hypothesis. I suspect the issue might be related to aliasing, where the pixel footprint differs between the center and boundary regions. This can also explain why MipNeRF360 outperforms the proposed method, and the comparison between ZipNeRF and RadZipNeRF in the supplementary material is very close.**\n\nThank the reviewer for the insightful analysis and careful observation. In the Truck scene, the gating score visualization indeed shows a significant difference between the edge and the central region, correlating with the aliasing issue. We agree that tackling the aliasing issue in some scenarios is an insightful explanation of the Rad-NeRF's effectiveness, which is supplementary to our original motivation targeting scenarios with heavy occlusions. We'll add this discussion to the revision.\n\nThat being said, we find that our original motivation for tackling heavy occlusion by decoupling sub-NeRF training is valid. We further provide more gating score visualization results. As shown in Figure R.3 (Global Response), the gating module assigns different preferences to foreground/background regions or the different sides of the caterpillar. Additionally, Figure R.4 shows that when rendering a less-occluded open scene, the gating score exhibits different characteristics and smooth transitions according to the ray directions.\n\n**R3-Q3: Will the proposed depth regularization encourage all MLPs to converge to the same output?**\n\nThe proposed depth-based mutual learning scheme does not encourage all sub-NeRFs to converge to the same output. We provide visualizations of different sub-NeRFs' rendering results in Figure R.5 (Global Response), which validates our analysis and conclusion.\n\nOn the one hand, the soft gating module allocates different rays to different sub-NeRFs, making them learn from different views. On the other hand, the depth-based mutual learning scheme only lets sub-NeRFs learn the *depth* from each other rather than *the overall rendered density or RGB distribution*. \n\nAs for the training curves in Figure.6, Rad-NeRF with more sub-NeRFs converges faster while achieving better rendering quality with the same training iterations. However, as mentioned in the reply to the reviewer tZHj, the rendering performance of Rad-NeRF will gradually saturate as the number of sub-NeRFs increases. Therefore, rad-4 and rad-3 are relatively similar.\n\n**R3-Q4.1: Regarding the soft gating, the current input is the ray origin and direction. It might be more effective to include the 3D position, as splitting MLPs based on different spatial regions could be more reasonable.**\n\nThanks for this worth-discussing question. Splitting MLPs based on spatial regions (the point-based multi-NeRF framework discussed in our paper) and splitting MLPs based on rays (as far as we know, RadNeRF is the first attempt) are two orthogonal methods targeting **different objectives**. \n\nSpecifically, our ray-based multi-NeRF framework tackles **the training interference from invisible rays**, which is especially of concern in **complex scenes**. Whereas point-based multi-NeRF frameworks that split MLPs based on 3D spatial regions mainly aim to **model large scenes by decomposition**, which cannot effectively address the training interference issue between invisible rays. For instance, with two sub-NeRFs assigned to 3D points around two objects as shown in Figure 1, ray-3 information pertaining to the central object is still used to train the sub-NeRF associated with the distant object due to inaccurate sampling, potentially resulting in training interference. In contrast, constructing a ray-based gating module-based multi-NeRF framework is a more effective scheme for the training interference issue.\n\nWe also conducted a comparison with state-of-the-art point-based multi-NeRF frameworks in Table 1 and conducted the point-based v.s. ray-based ablation study in Table 3. The results show that our ray-based design leads to consistent improvements.\n\n*Due to the character limitation, we'll reply to the remaining questions in the comments.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UwGDWiL50e", "reviewer": "Reviewer_5G52", "summary": "This paper claims that training with those rays with invisible 3D points (occlusions in complex scenes) that do not contain valid information about the point might interfere with the NeRF training. \n\nBased on this intuition, this paper proposes Rad-NeRF to decouple the training process of NeRF in the ray dimension softly, construct an ensemble of sub-NeRFs and train a soft gate module to assign the gating scores to these sub-NeRFs based on specific rays, where the gate module is jointly optimized with the sub-NeRF ensemble to learn the preference of sub-NeRFs for different rays. \n\nThis paper also introduces depth-based mutual learning to enhance the rendering consistency among multiple sub-NeRFs and mitigate the depth ambiguity. \n\nExperiments on five datasets demonstrate that Rad-NeRF can enhance the rendering performance across a wide range of scene types compared with existing Instant-NGP-based methods.", "questions": "The improved 3DGS nowadays is quite good and does not have the ray sampling problem, so I think this work may have limited influence and help a little for our community. I would like to see the rebuttal and other reviews.", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes an Instant-NGP-based ray-decoupled training framework to mitigate the training interference caused by invisible rays by ensemble sub-NeRFs via a jointly optimized gate module.\n\nThis paper also proposes a depth-based mutual learning method to ensure the rendering consistency among multiple sub-NeRFs, which serves as a geometric regularization, alleviating the depth ambiguity and avoiding overfitting.\n\nThis paper conducts extensive experiments. The results show that Rad-NeRF consistently improves rendering quality effectively and obtains better scalability than Instant-NGP-based baselines.", "weaknesses": "My main concern is the novelty. Although this work aims for different tasks, it is quite like Switch-NeRF.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R4-Q1: My main concern is the novelty. Although this work aims for different tasks, it is quite like Switch-NeRF.**\n\nThanks for this question. In our opinion, the most fundamental difference between Switch-NeRF and our Rad-NeRF is that Rad-NeRF is a ray-based multi-NeRF framework (the first ray-based framework as far as we know), while Switch-NeRF is a point-based multi-NeRF framework. And these two types of designs target different objectives. Specifically, Switch-NeRF decomposes 3D points to different sub-NeRFs to **model larger scenes**, and Rad-NeRF decomposes rays to tackle **the training interference from invisible rays**, which is especially of concern in **complex scenes**.\n\nFigure 5 and our newly added Figure R.1 show that scaling the number of sub-NeRFs by decoupling rays to different sub-NeRFs offers better performance-parameter scalability than scaling by decomposing spatial positions or scaling the MLP or spatial grid. We believe that in future applications of scaling NN-based 3D modeling methods for complex scenes, our newly proposed dimension (scaling the number of sub-NeRFs by decoupling rays) is a worth-considering dimension in the compound scaling strategy.\nBesides the major novelty in designing the ray-based multi-NeRF framework, we would also like to mention our introduction of mutual depth supervision between sub-NeRFs. This technique is simple but effective and can be easily adopted in other multi-NeRF frameworks too, providing a new way of unsupervised geometric regularization without the need for manually defined rules.\n\n**R4-Q2: The improved 3DGS nowadays is quite good and does not have the ray sampling problem, so I think this work may have limited influence and help a little for our community.**\n\nThanks for this very worth-discussing topic! We would like to share our thoughts on this topic and welcome any further discussions. Indeed, 3D GS has surpassed NeRF's rendering quality and speed on quite some tasks. Nevertheless, we still think the NeRF framework featured by the parametric neural network representation together with the volume rendering method still has its merits. Maybe one of its most notable strengths lies in its adaptability for generalizable modeling, i.e., in generalizable modeling tasks where we need to leverage the knowledge of existing scenarios to better and more efficiently model some unseen scenarios. In this case, it's very easy to extend the parametric neural network to take in other forms of input other than the position and ray direction. For example, the neural network can take an image, a depth image or even other modalities of information as input to model the current scenario. We think this flexibility is beneficial for pushing towards general 3D modeling or generation.\n\nBesides, there are some scenarios that 3DGS methods cannot handle well, due to the inaccurate and sparsity of point clouds (such as less-textured regions) [1], and the limitation to reason about view-dependent effects (such as reflection and refraction effects)[2]. In these scenarios, NeRF-based methods can offer better visual quality.\n\n[1] PointNeRF++: A multi-scale, point-based Neural Radiance Field\n\n[2] HybridNeRF: Efficient Neural Rendering via Adaptive Volumetric Surfaces", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear All,

We appreciate all the reviewers' time and efforts invested in reviewing our paper. We are encouraged that the reviewers recognize the effectiveness and scalability(nJqP,tZHj,PUvY,5G52), flexibility and compatibility with different neural rendering approaches (nJqP, tZHj), insightful motivation and idea(tZHj,PUvY), effectiveness of depth-based mutual learning (tZHj,5G52), and good presentation (nJqP,tZHj,PUvY,5G52). We are also thankful for all the concerns and suggestions. The concerns and suggestions are helpful, inspiring, and worth further discussion.

We have responded to each of the questions and suggestions carefully. The related supplementary experiments and figures are supplemented to the PDF file in the 'Global Response' and the specific contents are as below:
1. Figure R.1 shows the scalability study of training time and inference speed.
2. Figure R.2 shows the qualitative comparison results under the 3-view setting;
3. Figure R.3 and Figure R.4 supplement some additional visualizations of gating scores;
4. Figure R.5 shows the independent and fused rendering results of sub-NeRFs.

Meta-review: This paper was reviewed by four experts in the field. The initial reviews were mixed, as reviewers are concerned about the validation of the experiment. However, all these concerns were resolved during the rebuttal session, and all reviewers agreed to accept this work. The reviewers appreciate the novelty of the proposed approach, clear presentation, solid experiments, as well as its effectiveness. AC also agrees that this is well-presented and important work and thus would suggest acceptance.

We recommended the authors to carefully read all reviewers’ final feedback, and revise the manuscript as suggested in the final camera-ready version. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)