Rad-NeRF: Ray-decoupled Training of Neural Radiance Field

Authors: lidong  guo
xuefei  ning
yonggan  fu
tianchen  zhao
zhuoliang  kang
jincheng  yu
celine  lin
yu  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 10:46:17

Abstract: Although the neural radiance field (NeRF) exhibits high-fidelity visualization on the rendering task, it still suffers from rendering defects, especially in complex scenes. In this paper, we delve into the reason for the unsatisfactory performance and conjecture that it comes from interference in the training process. Due to occlusions in complex scenes, a 3D point may be invisible to some rays. On such a point, training with those rays that do not contain valid information about the point might interfere with the NeRF training. Based on the above intuition, we decouple the training process of NeRF in the ray dimension softly and propose a Ray-decoupled Training Framework for neural rendering (Rad-NeRF). Specifically, we construct an ensemble of sub-NeRFs and train a soft gate module to assign the gating scores to these sub-NeRFs based on specific rays. The gate module is jointly optimized with the sub-NeRF ensemble to learn the preference of sub-NeRFs for different rays automatically. Furthermore, we introduce depth-based mutual learning to enhance the rendering consistency among multiple sub-NeRFs and mitigate the depth ambiguity. Experiments on five datasets demonstrate that Rad-NeRF can enhance the rendering performance across a wide range of scene types compared with existing single-NeRF and multi-NeRF methods. With only 0.2% extra parameters, Rad-NeRF improves rendering performance by up to 1.5dB. Code is available at https://github.com/thu-nics/Rad-NeRF.

Main paper: 
 introduction: 
 Novel view synthesis is an important task within the domains of computer vision and computer graphics, playing an essential role in a variety of applications, such as autonomous driving, augmented reality, and so on. Recently, Neural Radiance Field (NeRF) [17] has emerged as a promising solution, achieving high-fidelity visualizations on the novel view synthesis task. It implicitly encodes 3D scenes through neural networks and trains the networks using volume rendering. Despite NeRF's excellent scene representation ability, it still suffers from rendering defects when dealing with complex scenes, such as 360-degree unbounded scenes [37, 2] and large scenes with free shooting trajectories [30, 27, 26] . One of the main reasons is the limited model capacity. However, directly increasing the network's size yields marginal performance improvement [18] .Our fundamental intuition is that the training interference from invisible rays affects NeRF's performance. Let us consider a simple case of a 360-degree unbounded scene with a central object (truck) and a distant object (car). As illustrated in Figure 1 (a), a 3D point located on the distant object can be observed from ray-1 and ray-2, but is invisible to ray-3 due to the occlusion presented by the central object. Although NeRF models transmittance in its volume rendering formula, it exhibits low geometric modeling accuracy and inaccurate sampling distribution in complex scenes, especially at the start of training, as Figure 1 (c) shows. So, 3D points on the distant object might be sampled by the ray-3, and the model is trained on these points by the ray-3 color. However, ray-3 does not contain any meaningful information about the distant object, potentially interfering with the NeRF's training. In contrast, considering the different visibility of the object to different rays, our intuition is that rather than using one NeRF, assigning the rays terminating at the distant object to NeRF-1 and the rays terminating at the central object to NeRF-2 could be better, as shown in Figure 1 (b).To verify the above intuition, we manually select two sets of images in the TAT dataset [13] . One set contains 80 images of the train's front side, while the other set includes the former set and 80 backside images. We train two NeRFs using these two sets respectively. As shown in Figure 2 , the model trained on the mixed set performs worse on the front side, which matches our intuition.To mitigate the training interference caused by invisible rays, the intuition solution is to decouple the training of the rays terminating at different regions. To this end, we propose a ray-decoupled training framework for neural rendering (Rad-NeRF). Within the Rad-NeRF framework, an ensemble of sub-NeRFs has different preferences for different rays through a gate module. With the help of the gate module, sub-NeRFs' outputs are fused by post-volume-rendering fusion to yield final rendering results. Notably, the gate module is jointly optimized with NeRF, allowing it to automatically learn the preference of each sub-NeRF for various rays in an end-to-end manner. This learnable gating design makes Rad-NeRF generally applicable to diverse scenes, which stands in contrast to prior multi-NeRF methods [27, 26] that rely on manually defined allocation rules.Additionally, we design a depth-based mutual learning method for the multi-NeRF framework to ensure the rendering consistency among multiple sub-NeRFs. In addition to learning colors, sub-NeRFs teach each other with their rendered depths. Traditional NeRF methods may struggle with generalization to novel views despite accurately rendering training views, as they often fail to capture precise geometry [7, 37] . In contrast, our depth-based mutual learning approach serves as a form of geometric regularization, alleviating the depth ambiguity and avoiding overfitting. To verify the effectiveness of Rad-NeRF, we conduct extensive experiments on various types of datasets. The results show that Rad-NeRF can exhibit anti-aliasing effects and obtain superior geometry modeling, thus consistently improving the rendering quality of novel views. In addition, Rad-NeRF is parameter-efficient and super simple to implement. With only 0.2% extra parameters, Rad-NeRF can increase rendering performance by up to 1.5dB compared to Instant-NGP. By scaling the number of sub-NeRFs through ray-wise decoupling, Rad-NeRF achieves better performance-toparameter scalability than scaling other dimensions, such as the MLP width or the feature grid.2 Related Work 
neural radiance field: 
 Neural Radiance Field (NeRF) [17] has received much attention since it was proposed. It uses MLPs to implicitly represent 3D objects or scenes, achieving realistic rendering results. There have been intensive studies on NeRF's extension, including increasing NeRF's training/inference efficiency [36, 8, 21, 25, 18, 5] , applying NeRF to specific scenes (large/unbounded/poor-textured) [15, 37, 2, 31, 26, 27, 38] , applying NeRF to other tasks (surface reconstruction/scene editing) [35, 20, 29, 14, 33, 32] , increasing NeRF rendering quality in few-shot setting [10, 12, 19, 7] . In this work, we aim to increase NeRF's rendering quality in complex scenes, and propose a multi-NeRF training framework, which can leverage the techniques proposed by these single-NeRF researches.Due to the limited model capacity, the multi-NeRF method is widely adopted to improve the rendering quality, which can be categorized into point-and ray-based multi-NeRF methods.Point-based multi-NeRF method. These methods divide the 3D space in the point-dimension [30, 37, 38] . 3D points in different regions are computed by different sub-NeRFs. For example, NeRF++ [37] proposes the sphere inversion transformation to map an infinite space to a bounded sphere first, and it uses two NeRFs to model the foreground and background regions respectively. Switch-NeRF [38] also partitions the scenes in the point-dimension. These methods do not consider the different visibility of a target region to different views and cause training interference on complex scenes with many occlusions. For example, the front side of an object is not visible when it is observed from the back view or blocked by an occlusion. Training the sub-NeRF with rays that do not contain any valid information about the target region might interfere with the training.Ray-based multi-NeRF methods. These methods allocate training rays to different sub-NeRFs and train sub-NeRFs independently. Block-NeRF [26] and Mega-NeRF [27] perform the ray allocation in the image-granularity and pixel-granularity, respectively. Both of them need a manually defined allocation rule, which requires prior scene knowledge and cannot be easily adapted to other types of scenes. The former work trains sub-NeRFs in large-scale road scenes with prior knowledge of the image shooting position distribution, and the latter one trains sub-NeRFs in open drone scenes and allocates the rays based on the ray intersecting positions with a horizontal plane. However, defining a ray allocation rule for complex scenes lacking prior scene-specific knowledge remains challenging. Another related work is NID [28] , which proposes a mixture-of-experts NeRF for generalizable scene modeling. In this work, different experts serve as the basis to construct the implicit field of different scenes and the gating module takes in the new scene's image as the input (i.e., image-granularity).In this work, we propose a gate-guided multi-NeRF mutual learning framework, performing the allocation and decoupling the training in the ray dimension softly. Compared to other multi-NeRF methods, Rad-NeRF boosts the rendering quality without the need for prior scene knowledge. 
preliminary: 
 NeRF [17] Color Supervision: The fused rendered depth of the ray is used to regularize each sub-NeRF's geometric encoding.! ! = # $% & -% ( & ) " #$% … $ & ! # $ & " # $ & #$! # $ & # # Geometric Regularization: !&'( = # # $+ , ) & -+ -& ) " * )+, #$%the ray and renders the pixel's color Ĉ(r) by discretely summing density σ i and color c i of each point i, which approximates the integral C(r) as follows:EQUATIONEQUATIONwhere t i is the distance between i-th sample's position and the starting point of the ray, δ i = t i+1 -t i is the distance between adjacent samples and T i represents the probability that the ray travels from the start to point i without hitting. The NeRF optimization is based on color supervision.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
rad-nerf: 
 NeRF faces the challenge of limited model capacity when rendering complex scenes [37, 30, 38] . However, directly increasing the number of model parameters yields marginal improvement in the rendering quality [18] , posing an important research question: "how to effectively scale up the capacity of NeRF". While the multi-NeRF methods have been proposed as an effective technique in response to this question, they still face limitations in handling complex scenes (with many occlusions and arbitrary shooting trajectories) due to training interference among invisible rays. In this work, we propose a ray-decoupled training framework (Rad-NeRF), effectively scaling up model's capacity by decoupling training in the ray dimension in a learnable way. Figure 3 gives an overview of Rad-NeRF.Motivated by the intuition and oracle experiment discussed in Section 1, rather than using a single NeRF model, designing a multi-NeRF structure that considers different visibility of the region to different rays and decouple NeRF's training in the ray-dimension could be better. We design a ray-based multi-NeRF model structure and introduce a soft gate module to learn the preference of each sub-NeRF for various rays in a learnable way.Multi-NeRF Structure. As shown in Figure 3 , we employ a shared feature grid among sub-NeRFs and keep MLP decoders independent for the multi-NeRF structure. As different rays may pass through the same region of 3D space, weight sharing for the feature grid helps training, owing to the feature grid's responsibility for encoding features of 3D spatial points. As validated by the Oracle experiment, training with regions A and B facilitates the training of the shared feature grid and improves the rendering quality(PSNR 18.685 vs 18.488). Meanwhile, as the MLP decoder is designed to encode view information, constructing an ensemble of independent MLP decoders helps to decouple the training in the ray dimension, and thus maintains the preference of sub-NeRFs for various rays. Additionally, such structure is a multi-model extension of Instant-NGP [18] , helping to avoid a significant increase in the number of parameters and training complexity. The hybrid representation also maintains high training efficiency.Soft Gate Module. We incorporate a soft gate module to assign gating scores to the sub-NeRFs for each ray. The soft gate module is jointly optimized with NeRF, enabling it to learn the preference of each sub-NeRF for different rays in an end-to-end manner. In contrast to manually assigning training rays to sub-NeRFs, this learnable gating design makes Rad-NeRF generally applicable to diverse scenes lacking prior scene-specific knowledge. In Section 5.2, we will also show that the gate module can learn to assign reasonable gating scores that reflect the object visibility of rays, aligning with our intuition that decoupling training in the ray-dimension is important.Specifically, we employ a four-layer MLP followed by a Softmax function as the gate module. The gate module takes the starting point and direction (o,d) of a ray r as the input, and outputs the gating scores G(r) of multiple sub-NeRFs associated with this ray. Instead of applying any sparsification strategies on the gating score G(r) as in previous work [38] , such as top-k gating function [23] , we use soft gating scores to enhance the smoothness and consistency of rendered results.As discussed in Section 3, each ray corresponds to a pixel on the image. Following the volume rendering process, we can obtain K rendered colors for each ray, where K is the number of sub-NeRFs. Subsequently, multi-NeRFs' outputs are fused in a post-volume-rendering ordering to obtain the final rendering results. The fused color C(r) of the ray r can be written as below:EQUATIONwhere G k (r) is the k-th element of gating score G(r) and Ĉk (r) is the rendered color of k-th sub-NeRF for the ray r.By the learnable soft gating design, different sub-NeRFs learn different encodings of the scene. We introduce a mutual learning method to enhance the rendering consistency and robustness of sub-NeRFs, wherein each sub-NeRF not only learns from ground truth but also learns from each other. Due to the lack of the ground truth for per-ray depth, NeRF may fail to learn accurate geometry despite accurately rendering training views, which adversely affects its generalization to novel views.To address this, we perform mutual learning with the rendered depths of sub-NeRFs, which serves as a form of geometric regularization and helps the model find more robust geometric solutions. The per-ray depth estimation D(r) can be written as Equation 4, where t i is the i-th sample's distance from the starting point on the ray.EQUATIONIn practice, we first fuse the rendered depths of sub-NeRFs guided by the gating score G(r). Then we use L2 distance to quantify the match of each sub-NeRF's rendered depth Dk (r) and the fused depth D(r). Our depth-based mutual learning loss is defined as below, where R is the set of sampled rays:EQUATIONCompared to directly averaging multiple sub-NeRFs' depth predictions, the gate-guided fused depth D(r) is more accurate, as the gating score G(r) can reflect the prediction confidence of each sub-NeRF for the ray r. The overall loss function of Rad-NeRF is given by:EQUATIONwhereL c = r∈R ∥C(r) -C(r)∥ 2 (C(r)is the ground truth color value of ray r) is the rendering loss. λ 1 and λ 2 are the weights for regularization terms, which are the only hyper-parameters to be set. The value of λ 1 is chosen from 1 × 10 -4 and 5 × 10 -3 . λ 2 is set to 1 × 10 -2 on all the datasets. L cv is the balancing regularization on the Coefficient of Variation of the soft gating scores, which prevents the gate module from collapsing onto a specific sub-NeRF. The details of L cv are described and discussed in the Appendix B. 
datasets and baselines: 
 Datasets. We use five datasets from different types of scenes to evaluate our Rad-NeRF. (1) Object dataset: we take Masked Tanks-And-Temples dataset (MaskTAT) [13] for evaluation, which are photographed objects with masked background; (2) 360-degree inward/outward-facing datasets: we take Tanks-And-Temples (TAT) dataset with unmasked background [13] and NeRF-360-v2 dataset [2] to evaluate on scenes with large dynamic depth range; (3) free shooting-trajectory datasets: we conduct experiments on Free-Dataset [30] and ScanNet dataset [6] , which are large outdoor and indoor scenes respectively. Both larger view ranges and more irregular shooting trajectories pose greater challenges for NeRF rendering.Baselines. We compare our Rad-NeRF with two types of methods: one type uses the grid-based NeRF framework as we do, including PlenOctrees [36] , DVGO [25] , Instant-NGP [18] and F2-NeRF [30] . The other one is the MLP-based NeRF method, including NeRF [17] , NeRF++ [37] ,MipNeRF [1] and MipNeRF360 [2] , which is inefficient in training and needs almost one day for training in complex scenes. Note that we also implement the NGP-version of Block-NeRF [26] , Switch-NeRF [38] and Mega-NeRF [27] to validate the superiority of Rad-NeRF to other multi-NeRF methods. The implementation details of Mega-NGP are shown in the Appendix F.Rad-NeRF achieves higher rendering quality than existing single-and multi-NeRF methods.We report the main quantitative results on the complex scenes and the object dataset in Table 1 and Appendix D respectively. Within no more than one hour of training, Rad-NeRF achieves higher rendering quality compared to other fast training methods and multi-NeRF methods, including Switch-NGP and Block-NGP. We can also see that while Rad-NeRF is designed for complex scene The visualization demonstrates that Rad-NeRF learns reasonable ray allocations, matching our intuition. Besides, we observe that in some specific scenes, such as the Truck scene, the gating score visualization indeed shows a significant difference between the edge and the central region, correlating with the aliasing issue. Such observation illustrates that tackling the aliasing issue in some scenarios is another insightful explanation of the Rad-NeRF's effectiveness, which is supplementary to our original motivation targeting scenarios with heavy occlusions.Scaling up NeRF with the Rad-NeRF framework is more effective than scaling the MLP width, increasing the feature grid size, or adding more feature grids. By default, we set the number of sub-NeRFs to 2 in all experiments. As shown in Figure 6 , when the number of sub-NeRFs increases, Rad-NeRF consistently obtains average performance gains on the ScanNet dataset while only marginally increasing the number of model parameters. Compared with directly increasing the hidden dimension of MLP decoders or the size of the feature grid, Rad-NeRF has better performancemodel size scalability. Furthermore, we observe that the model with four sub-NeRFs converges faster than the one with two sub-NeRFs while achieving better rendering quality with the same training iterations, as Figure 7 shows. The ease of training convergence can be attributed to two aspects. On the one hand, the number of learnable parameters and training complexity increases marginally. On the other hand, our gate module (a 4-layer MLP without sinusoidal position encoding) decouples the training in the ray dimension and reduces training interference.We additionally compare Rad-NeRF against 3D Gaussian splatting (3DGS) [11] as a non-neural approach that represents the current state of the art with regard to quality and rendering speed. The comparison is conducted on MaskTAT [13] and ScanNet [6] datasets. MaskTAT is an object dataset without point clouds, and ScanNet contains indoor scenes with many less textured regions.Rad-NeRF performs better than 3DGS in some cases. We report results on Table 9 , and show qualitative highlights in Figure 8 . For the MaskTAT dataset, we initialize 3D GS with random points. Our method performs best over 3D GS and Instant-NGP. For the ScanNet dataset, we initialize 3D GS with the point cloud provided by the dataset. However, there are many less textured regions in indoor scenes that affect the accuracy and density of point clouds. Optical distortion exists in the rendered pictures of 3D GS. In contrast, Rad-NeRF renders more smoothly than all baselines.Potential combination of Rad-NeRF with 3DGS. NeRF is characterized by its neural networkbased ray-related predictions, which provide flexibility for cross-scene generalization and enable the application of Rad-NeRF's ray-wise training decoupling approach. In contrast, the plain 3D GS framework parametrizes the scene using a global, non-ray-related representation, making Rad-NeRF inapplicable. However, Rad-NeRF could potentially be applied to generalizable 3D GS frameworks that integrate neural network-based ray-related predictions [4] .In this section, we conduct ablation studies on Rad-NeRF using the TAT dataset [13] . The key takeaways from our results are summarized below. Some additional ablation studies and analyses are presented in the Appendix E.Importance of the gate-guided multi-NeRF fusion and depth-based mutual learning. The ablation results of the two key components are shown in Table 2 . Uniform fusion simply averages multi-NeRFs' outputs to get final results without a gate module. In this way, sub-NeRFs focus on all the training rays instead of having their own preferences, which can not effectively improve rendering quality. For the depth-based mutual learning method, we observe that it enables a smoother and more reasonable depth prediction, as shown in Figure 10 . In addition to improving rendering consistency, it also acts as a geometric regularization to reduce the depth ambiguity and avoid overfitting.We further provide visualizations of different sub-NeRFs' rendering results in Figure 11 , which validates that the proposed depth-based mutual learning scheme will not encourage all sub-NeRFs to converge to the same output. On the one hand, the soft gating module allocates different rays to different sub-NeRFs, making them learn from different views. On the other hand, the depth-based mutual learning scheme only lets sub-NeRFs learn the depth from each other rather than the overall rendered density or RGB distribution. Importance of the ray-level allocation. We evaluate the results of different fusion dimensions in Table 3 . Compared to fusing multi-NeRFs' outputs in the point dimension, our ray-based method performs better, validating the superiority of the visibility-aware multi-NeRF method.Importance of pixel-granularity fusion. We compare different fusion granularity in Table 4 . In image-granularity fusion, all pixels of an image have the same preference for model parameters, which may not be reasonable. An illustrative example is an image capturing both the central object and the background region, such as the Truck scene shown in Figure 5 . In such a case, the rays hitting these two regions should be assigned different model parameters. In contrast, pixel-granularity fusion provides a more fine-grained understanding of the image and scene. 
limitations: 
 As the gating module (a 4-layer MLP without sinusoidal position encoding) incorporates smoothness prior implicitly, it exhibits smooth and close scores to the nearest seen view for unseen views. Consequently, the generalization of the gating module relies on sufficient training data, and thus Rad-NeRF does not perform well in the few-shot setting (see Appendix K for more results). On the contrary, the proposed method is suitable for the rendering of complex scenes, which themselves often require sufficient training data. 
conclusion: 
 This work proposes a ray-decoupled training framework (Rad-NeRF) for neural rendering. To alleviate the issue of the training interference problem in complex scenes, we construct a multi-NeRF framework and decouple the training of NeRFs in the ray dimension. Additionally, we propose a depth-based mutual learning method that improves the multi-NeRF rendering consistency and reduces the depth ambiguity, thereby improving generalization to novel views. Extensive experiments across various datasets validate Rad-NeRF's effectiveness and better performance-parameter scalability.We prospect for further exploration to fully exploit the potential of Rad-NeRF. Here, we outline several possible directions:(1) As researchers may choose different frameworks based on specific situational requirements, adapting Rad-NeRF to different single-NeRF frameworks including 3D GS (non-neural approach) is a valuable next step. (2) The number of sub-NeRFs can be determined automatically based on scene complexity and training resources. (3) We hope the newly proposed scaling dimension, which increases the number of sub-NeRFs through ray-wise decoupling, will enable modeling of complex scenes in a parameter-efficient manner.  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper has no societal impact, since NeRF renders the real scene without any fake generation.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: This paper does not use existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: The comparison of various multi-NeRF training frameworks is summarized in Table S .1. NeRF++ [37] proposes the sphere inversion transformation to map an infinite space to a bounded sphere firstly, and uses two NeRFs to model the 3D points in foreground and background regions, respectively. It adopts the manual allocation mode as it manually sets the boundary between foreground and background regions. Block-NeRF [26] and Mega-NeRF [27] are two classical ray-based multi-NeRF frameworks, which perform the ray allocation in the image-granularity and pixel-granularity, respectively. The former work trains sub-NeRFs in large-scale road scenes with prior knowledge of the image shooting position distribution on the road, and the latter one trains sub-NeRFs in open drone scenes and allocates the rays by partitioning the intersecting positions between the rays and a horizontal plane. However, they are designed for large road scenes and open drone scenes specifically and need a manually defined allocation rule, which requires prior scene knowledge and cannot be easily adapted to other types of scenes. Switch-NeRF [38] implements a learning-based scene partition scheme motivated by Mixture-of-Experts (MoE) [24] . However, it partitions the scene in the point dimension, which limits the rendering performance in more complex scenes with occlusions. It is also limited to be only used in open drone scenes. F2-NeRF [30] is another point-based multi-NeRF method, which allocates the 3D points to multiple sub-NeRFs in a more elaborate but manual way.In contrast, our Rad-NeRF performs the allocation and decoupling the training in the ray dimension softly. Acting as a ray-based training framework, Rad-NeRF is "visibility-aware" and achieves higher performance in complex scenes. Moreover, compared to other multi-NeRF methods, Rad-NeRF boosts rendering quality across different types of scenes without the need for prior scene knowledge. The "Allocation mode" column indicates whether the framework divides the training data based on the manually designed rule or in a learnable way; The "Target scene" column indicates the scene that the framework is proposed for specifically.NeRF++ [37] point-based manual no constraint Block-NeRF [26] ray-based manual large road scene Mega-NeRF [27] ray-based manual open drone scene Switch-NeRF [38] point-based learnable open drone scene F2-NeRF [30] point Training Details. For Instance-NGP and our Rad-NeRF, we train the NeRFs for 20k iterations on a single RTX-3090 GPU. We use Adam optimizer with a batch size of 8192 rays and a learning rate decaying from 1 × 10 -2 to 3 × 10 -4 . For the weights of the regularization terms in Equation 6, λ 1 is set to 1 × 10 -4 on NeRF-360-v2 and Free dataset, and is set to 5 × 10 -3 on other datasets. We set λ 2 to 1 × 10 -2 on all the datasets. By default, the number of sub-NeRFs is set to 2, and it is sufficient to achieve significant rendering quality improvement.Some previous work has observed that the gate module tends to converge to an imbalanced state, where it always produces large weights for the same few sub-models [23, 28, 38] . Such an imbalance Following [23, 28] , we adopt the regularization on the Coefficient of Variation of the soft gating scores, which encourages a balanced allocation of model parameters for training rays. The CV loss function is given byEQUATIONEQUATIONwhereG(R) is the set G k (R) n k=1. Note that some work also uses the load-balanced loss to encourage multi-models to receive roughly equal numbers of training examples [23, 38] . However, this optimization objective is too strict and unsuitable for our framework.Switch-NeRF [38] constructs a point-based multi-NeRF framework based on MLP-based NeRF structure. Given a 3D point x, it first extracts high-level point feature S(x) using a linear layer, which will be sent to the gate module to obtain the gating scores. Then, they apply a Top-1 function on the gating scores to determine which NeRF expert should be activated. The output feature of the chosen expert will be multiplied by the gating score corresponding to the expert and obtain the fused point feature. Finally, the fused point feature is sent to the unified MLPs to predict the density σ and color c.As illustrated in Figure S.1, we build an NGP-version of Switch-NeRF, named Switch-NGP. Since NGP contains a feature grid in the form of the hash table, we directly use the feature grid to obtain the high-level point feature S(x) of the point x. Switch-NeRF has validated the importance of a unified head, wherein the gating score is multiplied by the high-level features rather than the density or color predictions, which makes the gating and prediction more stable in training. We also perform the multi-NeRF fusion in the point-feature dimension by inserting extra K feature MLPs before the density MLP. Each expert in Switch-NGP corresponds to a tiny feature MLP with two hidden layers and 64 channels.The training details of Switch-NGP are the same as Rad-NeRF, as described in Section B.1.Block-NeRF [26] applies the multi-NeRF method to the street scene, which allocates model parameters in the ray dimension but in the image-level granularity. Specifically, Block-NeRF places one NeRF at each intersection and directly allocates the training images to multi-NeRFs according to the image shooting positions. We implement an NGP-version Block-NeRF, named Block-NGP, which can be applied to various types of scenes without prior knowledge. After getting all the training images, we first use the clustering algorithm (KMeans) to cluster the image shooting positions, and the number of clusters is set the same as the number of sub-NeRFs. During the training process, each training image is allocated to the corresponding sub-NeRF according to the clustering results, and the training of sub-NeRFs is independent.We compare Rad-NeRF with other multi-NeRF work on ScanNet dataset [6] . Compared to other outdoor datasets, ScanNet contains more texture-less regions like the floors and the walls, which poses more challenges for neural rendering. We conduct experiments in four complete scenes in ScanNet, namely scene0046, scene0276, scene0515 and scene0673. The quantitative and qualitative results are shown in Table S . the average of the sub-NeRFs' rendering depths as the target depth, which means all sub-NeRFs have equal regularization strength (Equal DML), the rendering quality will be slightly worse. The results highlight the pivotal role of gate-guided depth mutual learning. Using the gated-guided fused depth as the target depth differently penalizes sub-depths based on the gating scores and increases the accuracy of the geometry regularization. We also observe that depth mutual learning has no effect in the case of uniform fusion due to the low accuracy of the averaged depth.CV balanced regularization. As introduced in Section B.1, we adopt the regularization on the Coefficient of Variation of the soft gating scores to prevent the gate module from collapsing onto a specific sub-NeRF while maintaining sub-NeRF's different specialties. Without CV-balanced regularization, the rendering quality degrades significantly. Besides, we apply the CV regularization only for the first half of the training time and find that the performance is comparable to Rad-NeRF, The results prove that such regularization would not interfere with the learning of the gate module. Structural design. In Rad-NeRF, we adopt a multi-NeRF structure with a shared feature grid and an ensemble of MLP decoders. We further analyze the reason behind the performance improvement and explore the performance of independent feature grids. As Table S .7 shows, the model employing a shared feature grid (Rad-NeRF) outperforms its counterpart with multiple independent feature grids, which highlights the effect of independent MLP decoders rather than feature grids. We attribute this observation and the performance gained by Rad-NeRF to two aspects. (1)Within the hybrid representation, the feature grid is responsible for encoding features of 3D spatial points, while the MLP encoder is designed to encode view information. The crucial design of independent MLP decoders aligns with our visibility-aware motivation, thereby enhancing the view-dependent effect.(EQUATIONThe training complexity will also increase as the trainable parameters increase. With the limited amount of training data, increasing the number of feature grids leads to poor convergence. By contrast, as different rays may pass through the same region of 3D space, weight sharing for the feature grid helps to facilitate training. Although the number of learnable parameters hardly increases, Rad-NeRF decouples the training in the ray dimension, helping to increase the model's generalization ability.Mega-NeRF [27] applies the multi-NeRF method to the drone scenes, allocating model parameters in the ray dimension and the pixel-level granularity. Specifically, it allocates rays by partitioning the intersecting points between rays and scenes. Such a method is suitable for drone scenes, where the top-down perspective allows for the approximation of ray-scene intersections by intersecting with a set horizontal plane. However, in unstructured scenes captured by free trajectories, the intersecting points between rays and scenes cannot be determined before the training is completed, limiting the applicability of Mega-NeRF to such scenes.Since there is no straightforward implementation to determine the ray intersections before training, we adopt an alternative implementation for NGP-version Mega-NeRF, which employs a clustering algorithm to divide rays directly based on their origins and directions. The clustering process is offline and the same as the one in Block-NGP. During the training process, each training pixel is allocated to one corresponding sub-NeRF according to the clustering results. To ensure a fair comparison, the model structure of Mega-NGP is the same as the one in Rad-NeRF, following the implementation of Block-NGP. We conduct a comprehensive evaluation across all datasets and the experimental results are shown in Table S .8. Mega-NGP yields similar results to Block-NGP, which is less effective than our Rad-NeRF. We provide the per-scene results of scalability studies on the ScanNet dataset in Table S .9 which are reported in the metric of PSNR.Furthermore, we observe that the model with four sub-NeRFs converges faster than the one with two sub-NeRFs while achieving better rendering quality with the same training iterations, as Figure S.3 shows. The ease of training convergence can be attributed to two aspects. On the one hand, the feature grid is shared among multi-NeRFs, and thus, the number of learnable parameters increases marginally. On the other hand, as the neural network is better at fitting low-frequency information, our gate module (a 4-layer MLP without sinusoidal position encoding) has implicitly incorporated "smoothness prior", leading to closer rays to be more possibly assigned closer gating scores. S .10, integrated with Rad-NeRF, ZipNeRF can also obtain performance gains, validating Rad-NeRF's effectiveness and potential for integration with different frameworks.Considering that different frameworks have different characteristics, researchers may choose different frameworks based on specific situational requirements. Adapting Rad-NeRF to different single-NeRF frameworks remains an interesting point to be explored in the future.We further validate the performance of Rad-NeRF on the Free dataset [34] . As the results show, Rad-NeRF's multi-NeRF training framework boosts ZipNeRF's performance consistently. Uncertainty-based methods consider floaters as regions corresponding to high uncertainty and remove them by thresholding the scene according to an uncertainty field during rendering. The spatial uncertainty is computed in roughly a minute on any existing method. For example, RobustNeRF [22] treated pixels with larger losses as those with high uncertainty, avoiding the misleading effect of outlier points by discarding the training of those pixels. However, it is difficult to distinguish outlier points from the high-frequency areas that should be learned. Moreover, Instant-NGP [18] regards the spatial points with too low density as regions with high uncertainty and filters these regions when rendering. Although this method works well, it still cannot completely eliminate floaters in difficult scenes and may remove correct regions. As a post-hoc uncertainty assessment framework, Bayes Rays [9] acts as a post-hoc uncertainty assessment framework, which does not change NeRF's training process, only removing "floater" regions corresponding to high uncertainty. However, this solution is not stable and is generally used as an auxiliary solution to improve NeRF's rendering quality.Different from uncertainty-based methods, the proposed Rad-NeRF improves rendering quality by tackling the training interference issue. The depth-based mutual learning method also acts as a geometric regularization to reduce rendering defects. Importantly, Rad-NeRF is essentially orthogonal to these post-training uncertainty removal-based methods and can be integrated with Bayes Rays to obtain further performance improvement.NeurIPS Paper Checklist Justification: We create a separate "Limitations" section 6 in the main paper. We point out the limitation of the proposed Rad-NeRF in the few-shot setting. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://github.com/kwea123/ngp_pl 


Decision: This paper was reviewed by four experts in the field. The initial reviews were mixed, as reviewers are concerned about the validation of the experiment. However, all these concerns were resolved during the rebuttal session, and all reviewers agreed to accept this work. The reviewers appreciate the novelty of the proposed approach, clear presentation, solid experiments, as well as its effectiveness. AC also agrees that this is well-presented and important work and thus would suggest acceptance.

We recommended the authors to carefully read all reviewers’ final feedback, and revise the manuscript as suggested in the final camera-ready version. We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: - The authors decouple the training process of NeRF in the ray dimension and propose a framework where they create an ensemble of sub-NeRFs and train a soft gate module to assign gating scores to these sub-NeRFs based on specific rays.
- The gating module is a 4-layer MLP followed by a softmax function. The gating module takes in ray starting point and direction and outputs a gating score corresponding to each of the sub-NeRFs
- The paper proposes a depth-based loss that compares the fused depth with the depth of the individual NeRFs in order to enhance their robustness.
- Multiple indoor and outdoor scenes from various public datasets were used to show the method's effectiveness.
strength_weakness: strength_weakness: - Though the paper claims that their method needs few extra parameters and trains quickly, there only seems to be a slight improvement in quantitative numbers across many scenes.
- The authors have pointed out that the method does not work for few-shot settings.
questions: questions: - Uncertainty-based methods, like Bayes Rays, consider floaters as  regions corresponding to high uncertainty and remove them by thresholding the scene according to an uncertainty field during rendering. The spatial uncertainty is computed in roughly a minute on any existing method. What is the advantage of Rad-NeRF over such uncertainty-based methods? 
- Cross-Ray NeRF (CR-NeRF) leverages interactive information across multiple rays to synthesize occlusion-free novel views with the same appearances as the images. They recover the appearance by fusing global statistics, i.e., feature covariance of the rays and the image appearance. Can the method proposed in this paper solve the problems solved by Rad-NeRF?
limitations: limitations: Yes, the authors have discussed the limitations of the method.
paper_summary: paper_summary: Traditional NeRF models face challenges in rendering complex scenes due to interference from occluded rays, which leads to inaccurate training data. To address this, the authors propose Rad-NeRF, a framework that decouples the training process in the ray dimension by using multiple sub-NeRFs, each trained with specific rays that contain valid information for the points they observe.
Rad-NeRF employs a soft gate module that assigns gating scores to different sub-NeRFs based on the rays, allowing the model to learn preferences for different rays. Additionally, the framework introduces depth-based mutual learning to enhance consistency and mitigate depth ambiguity among the sub-NeRFs. Essentially, my understanding is that this work builds on Instant NGP by adding multiple sub-MLPs, with the gate module controlling the degree of participation of each MLP during rendering. Experiments demonstrate that Rad-NeRF outperforms existing single and multi-NeRF methods, improving rendering performance with minimal additional parameters.
strength_weakness: strength_weakness: 1.	What is the structure of the grid in the network architecture? Is it a multi-resolution grid? Are the features on the grid also learnable?
2.	What are the training and inference times for this method? It would be beneficial if the authors provided a comparison with other existing methods in the paper.
3.	In the case of multiple networks, if a sub-NeRF receives very low soft gating scores, does it cease to update, or does it still contribute to the computation?
4.	Are there any automated methods or heuristics for selecting the optimal number of sub-NeRFs based on scene complexity? What is the relationship between the number of sub-models and the overall performance? Detailed tuning and experimentation are required to ensure optimal performance across different datasets and scenes.
5.	Could you provide a visualization of the gating scores for different sub-NeRFs for the same viewpoint? And I am curious about the distribution of gating scores in scenarios without occlusions.
questions: questions: See Weakness.
limitations: limitations: See Weakness.
paper_summary: paper_summary: This paper proposes an innovative approach to enhance NeRF performance. The key observation is that due to occlusion, some objects may appear in one ray but not in another. While NeRF uses transmittance to mitigate this issue, the paper argues that this may not be sufficient. To address this, the authors propose using multiple MLPs to decode the same feature and jointly learn a gating function to fuse all the information. The results demonstrate that this design successfully improves performance, albeit by a small margin.
strength_weakness: strength_weakness: I have several concerns regarding the motivation, methodology, and validation of the proposed approach.

## Motivation: 

+ The motivation is not very clear. Specifically, Figure 1 and lines 29-40 argue that poor performance is due to occlusion, leading the authors to propose using multiple MLPs. However, I find Figure 1, especially Figure 1(c), inaccurate. I would expect a CDF instead of a PDF, as NeRF often compensates RGB appearance with poor geometry. Thus, (c) might still have the correct RGB, i.e., its CDF is accurate. 
+ Moreover, Figure 7 does not support the occlusion hypothesis. I suspect the issue might be related to aliasing, where the pixel footprint differs between the center and boundary regions. This can also explain why MipNeRF360 outperforms the proposed method, and the comparison between ZipNeRF and RadZipNeRF in the supplementary material is very close.

## Methodology: 

The approach resembles MOE or a multi-head transformer. However, I have two questions:

+ First, the proposed method applies depth consistency. Isn't this encouraging all MLPs to converge to the same output? This might explain why rad2, rad3, and rad4 in Figure 6 are very similar.

+ Second, regarding the soft gating, the current input is the ray origin and direction. It might be more effective to include the 3D position, as splitting MLPs based on different spatial regions could be more reasonable. Additionally, the analysis of the gating function (Figure 7) lacks depth, and more details would be appreciated.

## Validation: 

+ While the rad approach does improve performance, the improvement is relatively marginal. Furthermore, MipNeRF360 outperforms the proposed method, indicating that the proposed approach is not strong enough. 

+ In Table 1, why are switch NeRF and block NeRF not better than instant NGP? After ray participation, they should outperform the vanilla version.
questions: questions: See above
limitations: limitations: N/A
paper_summary: paper_summary: This paper claims that training with those rays with invisible 3D points (occlusions in complex scenes) that do not contain valid information about the point might interfere with the NeRF training. 

Based on this intuition, this paper proposes Rad-NeRF to decouple the training process of NeRF in the ray dimension softly, construct an ensemble of sub-NeRFs and train a soft gate module to assign the gating scores to these sub-NeRFs based on specific rays, where the gate module is jointly optimized with the sub-NeRF ensemble to learn the preference of sub-NeRFs for different rays. 

This paper also introduces depth-based mutual learning to enhance the rendering consistency among multiple sub-NeRFs and mitigate the depth ambiguity. 

Experiments on five datasets demonstrate that Rad-NeRF can enhance the rendering performance across a wide range of scene types compared with existing Instant-NGP-based methods.
strength_weakness: strength_weakness: My main concern is the novelty. Although this work aims for different tasks, it is quite like Switch-NeRF.
questions: questions: The improved 3DGS nowadays is quite good and does not have the ray sampling problem, so I think this work may have limited influence and help a little for our community. I would like to see the rebuttal and other reviews.
limitations: limitations: The authors adequately addressed the limitations.

Comments: title: Final confirmation from the authors
Thanks for the feedback and further discussion from the authors
Thanks for the feedback and further discussion from the authors
Reply to the comments of Reviewer PUvY
Response to the remaining questions (Q4.2, Q5 and Q6)
comment: Dear Reviewer nJqP,

  As it's near the end of the discussion period. We would like to take this opportunity to check besides the previous question, if there are any remaining concerns. If our rebuttal has addressed all of your concerns, will you reconsider the overall judgment of our paper to be more positive? 

Thanks very much!

 Authors
Thanks for the feedback and we're glad our rebuttal has addressed all the concerns! Regarding the new discussion, we'd like to seize this opportunity to express our perspectives and want to reach a consensus with the reviewer.

---
> The paper lacks theoretical analysis.

We acknowledge that our method doesn't have theoretical analysis, we will add this as a discussion of limitations and future work. Nevertheless, the empirical effectiveness of our method is verified very broadly, including quantitive improvements across multiple datasets, comparison with many baselines (single/multi-NeRF frameworks), plugging into existing state-of-the-art frameworks (ZipNeRF), as well as demonstrating potential in scalability study (the performance-parameter scalability curve is better than other scaling dimensions). Moreover, we also have plenty of oracle experiments, ablation studies, and qualitative results analysis that support the logic behind our method design. We think our current empirical evaluation has established our proposed techniques as a worth-considering technique to try out and a new dimension to scale up the model for complex occluded scenes.

> The proposed architecture is an incremental work by combining the NGP with 'Wang et al. Neural Implicit Dictionary Learning via Mixture-of-Expert Training. ICML 2022'.

Thanks for recommending this work. RadNeRF is distinct from a combination of Wang et al. and NGP in the following aspects: (1) the targeting scenarios and issues, (2) the design logic and the concrete design of the experts and gating module.
1. Wang et al. proposed a multi-expert framework to tackle **the generalizable scene modeling problem**, i.e., how to effectively exploit the knowledge of training scenes to model unseen scenes with few-shot views. In contrast, Rad-NeRF focuses on mitigating **the training interference issue arising from inaccurate sampling when modeling one scene**, this issue is especially relevant when this scene is complex and occluded. 
2. Different experts in Wang et al. serve as the basis to construct the implicit field of **different scenes**. In contrast, different experts in Rad-NeRF serve as the basis for constructing the predictions of **different rays**. Therefore, the gating module in Wang et al. takes in the image of the new scene as the input, whereas the gating module in RadNeRF takes in the ray information as the input. **This routing dimension and granularity are vastly different. And we note that the decoupling at the ray dimension and granularity is the key to solving the training interference between the supervision of multiple rays.**

> Because the design of the hash-table is significantly more important than the subsequent network (as seen in 'Zhu et al. Disorder-invariant Implicit Neural Representation. TPAMI 2024'), it represents a minor innovation by changing one network in the original NGP to two networks (although the paper claims multiple networks, only two are used in experiments).

Regarding the structure, we agree that hash-table-based representation shows impressive performance. Therefore, we choose to build RadNeRF on this state-of-the-art structure to solidly support RadNeRF's practical usage. Indeed, our method is very simple and introduces very few parameters as it only applies the MoE technique onto the MLP part. Nevertheless, instead of considering this a minor innovation, we truly **regard this simplicity and parameter efficiency as an advantage of our work**, as we have emphasized in our introduction, "RadNeRF is parameter-efficient and super simple to implement".

Besides, we would like to note that, the "combination" of multiple ray-partitioned experts and the spatial hash-table grid is not a random combination of techniques. Instead, we carefully choose and ablate this choice. Specifically, the hash-table-based feature grid and the consequent MLP both play crucial roles and have distinct effects. The hash grid needs to capture non-ray-related information, and the MLP needs to capture the ray-related information. We find that letting all experts share the spatial feature grid while having different MLPs to predict the ray-related information achieves the best results. As shown by Table 1, Table S.7, we find that the ray-decopuled training of MLPs with a shared spatial feature grid can achieve 21.708, higher than 21.264 achieved by the ray-decoupled training of standalone NGP (note that both of them are better than the baseline NGP 20.722), with much fewer parameters in the meantime. 

Finally, we have experimented with K=2,3,4 experts (Figure 5 and Figure 6 show results on ScanNet, and we add an experiment on MaskTAT in our original reply).

---
Thanks again for raising your follow-up concerns to us directly. We hope that our further response can help us reach a new consensus.
Thank you for your effort. All of my concerns have been addressed. Although better performance is achieved, the paper lacks theoretical analysis, and the proposed architecture is an incremental work by combining the NGP with 'Wang et al. Neural Implicit Dictionary Learning via Mixture-of-Expert Training. ICML 2022'. Additionally, because the design of the hash-table is significantly more important than the subsequent network (as seen in 'Zhu et al. Disorder-invariant Implicit Neural Representation. TPAMI 2024'), it represents a minor innovation by changing one network in the original NGP to two networks (although the paper claims multiple networks, only two are used in experiments). Consequently, I will change my rating from '6 weak accept' to '4 borderline reject'.
We're very happy that our response has addressed your concerns! Thanks again for all the helpful suggestions and discussions We will add the discussions to the revision.
Thanks for the reply. It addressed my concerns. Based on other reviews and the rebuttal, especially the discussion of novelty and the discussion against 3D GS, which should be added to the camera ready, I raised my rating to Weak Accept.
Thanks for your feedback and we're happy our rebuttal has addressed most of the concerns!
The quantitative results under few-shot settings in R1-Q2 are the average results on all scenes of the LLFF dataset. 

We're open to any further discussions!
Dear reviewer,

Thanks for the feedback and the follow-up discussion! Let us continue the discussion.

---

We find our previous answer is not so clear in conveying our points. We were comparing the NeRF framework that is featured by (1) volume rendering; (2) NN parametrized, and the NN outputs **ray-related** volumetric information,  to the plain 3DGS framework that is featured by (1) rasterization rendering; (2) Globally-parametrized **non-ray-related** information. In one word, we're not suggesting that 3DGS cannot be effectively extended to the generalizable setting, but stating that **the NN-based ray-related prediction, a feature of NeRF, offers flexibility for generalization**. **This feature can surely be integrated with the plain 3DGS framework**, as done in the works mentioned by the reviewer. For example, pixelSplat adopts a neural network to predict **per-ray(pixel)** Gaussian primitive parameters.

Importantly, **the involvement of a neural network in predicting ray-related information allows for the application of the ray-wise training decoupling concept from Rad-NeRF**. This approach can effectively address training interference issues arising from inaccurate spatial sampling in complex and heavily occluded scenes. 

That is to say, although the ray-wise decoupled training method of Rad-NeRF may not be applicable for the plain 3DGS framework with global parametrization, **it could potentially be applied to generalizable 3D GS frameworks that incorporate NN-based ray-related prediction**. This can help mitigate its training interference (Note that regardless of whether rasterization or volume rendering is used, the inaccurate sampling issue will exist, and training interference will arise from the inaccurate sampling if using NN-based ray-related prediction). For instance, a possible way to apply the ray-wise decoupling idea on generalizable-3DGS, taking pixelSplat as the example, is constructing multiple NN-based encoders and fusing the per-ray predictions. Surely, there are still things to design to make this work. Nevertheless, considering that the scope of our paper doesn't include the generalizable setting, we leave it to future work and will add a future work discussion. 

----

Thanks again for this valuable discussion. It has not only helped improve our paper but also deepened our understanding of the topic. We're open to further discussion.
Dear Reviewer tZHj:

Thanks for reviewing this work. Would you mind to check authors' feedback and see if it resolves your concerns or you may have further comments?

Best,
AC
Thanks for the detailed comments. Most of my concerns are addressed. Is the table for sparse view results in **R1-Q2** for one scene of LLFF or all scenes ?
Thanks for the rebuttal. It somehow addressed my concerns. However, I don't agree with the discussion with respect to generalizable modelling, as there are many 3DGS-based works (such as pixelSplat, MVSplat and LaRa) that show impressive generalizability.
Thanks again for all the thoughtful suggestions, which are very helpful in improving our work. We will incorporate them into the revision.
Thanks for the detailed responses!    I am very happy that my concerns are resolved and I would like to raise the score as weak accept.
Thanks for your feedback and we're glad our rebuttal has addressed most of the concerns!

**C1: Motivation of radnerf: I am satisfied that the new figures demonstrate the evidence of addressing occlusion, e.g. rebuttal fig.R.3. I would encourage the authors to add discussion about both occlusion and aliasing in motivation.**

Thanks again for this insightful analysis. We'll add the discussion about aliasing to the revision in the motivation part.  

**C2: As for Fig,1 c in the main paper, I think drawing both pdf and cdf curves, e.g., fig.2 in nues paper(https://arxiv.org/pdf/2106.10689) is better. I think the current version does not look very great.**

Thanks for the suggestion. We will add the CDF curve to the revision.

**C3: Would it increase the overfitting problem if just using ray origins and directions? For example, given the tank example, what happens if you view it from the top? However, I would accept any results about that. I think it is great to have it in paper, even just for failure cases.**

Thanks for this great question. The ray-based gating module in Rad-NeRF does face the overfitting issue, especially under the few-shot setting. We also included the discussion of this limitation in the paper. For Rad-NeRF to achieve its full performance improvement, the soft-gating module needs to see enough input ray origins and directions to generalize well to other views. In the rebuttal to Reviewer nJqP, we validate Rad-NeRF's performance in the few-shot setting. The results do show that Rad-NeRF does not exhibit significant performance improvements compared to Instant-NGP (with all metrics at the same level) under the few-shot setting. Under such circumstances, the gating module has minimal impact on NeRF's training, but our depth-based regularization can still exhibit a positive effect, as shown in Figure R.2 (Global Response).

We further visualize the results of the gating module with limited generalization ability and find that the gating scores are not reasonably allocated, confirming the above limitation. We will supplement all the related results and visualization to the Appendix revision.

**C4: For the question: the analysis of the gating function (Figure 7) lacks depth, and more details would be appreciated. Here "depth" means more analysis. But I am satisfied with the new results in the rebuttal.**

Thank the reviewer for the clarification. We will add the additional gating score visualizations and analysis to the Appendix revision.

**C5: Consideration of performance improvement I am not very sure about this part: "By integrating our method into the ZipNeRF framework, we easily achieve an additional 0.3 PSNR improvement." Is there a new experiment you have did that zipner+radnerf will have further gain? If so, it should be added to the revision. If not, I think it is best to have it to demonstrate radnerf.**

Previously, we list the results of Rad-ZipNeRF on the 360v2 dataset in Appendix H of our paper. Integrated with Rad-NeRF, ZipNeRF can also obtain an additional 0.3 PSNR improvement. During the rebuttal, we further validate the performance of Rad-NeRF on the Free dataset. As the results show, Rad-NeRF's multi-NeRF training framework boosts ZipNeRF's performance consistently. We will add these additional quantitative results in the Appendix revision. 

|    Method   | Metric | Hydrant | Lab    | Pillar | Road   | Sky    | Stair  | Grass  | Avg     |
|-------------|--------|---------|--------|--------|--------|--------|--------|--------|---------|
| ZipNeRF     | PSNR   | 25.402  | 27.827 | 25.132 | 28.882 | 26.993 | 28.187 | 18.461 | 25.841  |
|             | SSIM   | 0.813   | 0.897  | 0.743  | 0.879  | 0.884  | 0.851  | 0.311  | 0.768   |
| Rad-ZipNeRF | PSNR   | 25.510   | 28.067 | 25.348 | 29.191 | 27.491 | 28.339 | 18.572 | **26.074**  |
|             | SSIM   | 0.812   | 0.903  | 0.748  | 0.866  | 0.891  | 0.852  | 0.317  | **0.770**   |
I thank the authors for the detailed responses.  Many of my concerns are addressed, which I appreciate a lot.  Below are some new comments.

1) motivation of radnerf: I am satisfied that the new figures demonstrate the evidence of addressing occlusion, e.g. rebuttal fig.R.3.  I would encourage the authors to add discussion about both occlusion and aliasing in the motivation.

2) As for Fig,1 c in the main paper,  I think drawing both pdf and cdf curves, e.g., fig.2 in nues paper(https://arxiv.org/pdf/2106.10689) is better.  I think the current version does not look very great.

3) for the question: 
> R3-Q4.1: Regarding the soft gating, the current input is the ray origin and direction. It might be more effective to include the 3D position, as splitting MLPs based on different spatial regions could be more reasonable.

I have a follow-up question, would it increase the overfitting problem if just using ray origins and directions?  For example, given the tank example, what happens if you view it from the top?   However,  I would accept any results about that.  I think it is great to have it in the paper, even just for failure cases.  

4) for the question:
> the analysis of the gating function (Figure 7) lacks depth, and more details would be appreciated.
Here "depth" means more analysis.  But I am satisfied with the new results in the rebuttal.

5) as for question:
> R3-Q5: Consideration of performance improvement
I am not very sure about this part:  "By integrating our method into the ZipNeRF framework, we easily achieve an additional 0.3 PSNR improvement."  Is there a new experiment you have did that zipner+radnerf will have further gain?  If so, it should be added in the revision. If not, I think it is best to have it to demonstrate radnerf.

Overall my biggest remaining question is about zipnerf+radnerf.  If the authors show it can help zipnerf, this paper should be accepted.
**R3-Q4.2: The analysis of the gating function (Figure 7) lacks depth, and more details would be appreciated.**

Thanks for the question! We're not sure does the reviewer means to "add the depth information into Figure 7"? If so, considering that we can directly judge the depth information of the scene by observing the rendered RGB image and thus analyze its correlation with the gating score, we did not add the depth visualization results in Figure 7. As shown in the visualization of Global Response, the gating scores are not always directly related to the scene depth, although some score visualizations show consistency with pixel depths in some views. 

If the reviewer is asking for more analyses: We further analyze the additional gating score visualization. As shown in Figure R.3, the gating module can assign different preferences to different sides of the central object. Besides, when the foreground and background of the scene can be clearly distinguished, the preferences of the sub-NeRFs in the fore/background area are also clear (the visualization of view-1 in Figure R.3). However, it is difficult to further summarize the distribution law of the gating scores. Unlike point-based partitioning, the ray-based gating module encodes both the ray origins and directions, incurring higher partition complexity.

**R3-Q5: Consideration of performance improvement**

Thanks for this question. We'd like to share our thoughts and welcome any further discussions.

Firstly, we regard RadNeRF as an easily **integratable** and **workable** strategy to try out in future NeRF applications, as it doesn't require hyperparameter tuning to work well across different types of scenarios, all achieving some improvements (1.02 on mask-TAT, 0.98 on TAT, 0.57 on 360v2, 0.49 on free dataset and 0.82 on ScanNet dataset). For example, in complex scenarios such as the 360v2 dataset, our method can be plugged in and improve the state-of-the-art. Specifically, ZipNeRF obtains 0.74 PSNR gain compared to MipNeRF360. By integrating our method into the ZipNeRF framework, we easily achieve an additional 0.3 PSNR improvement.

Additionally, we compare Rad-NeRF, the first ray-based multi-NeRF framework to our knowledge, with other multi-NeRF training frameworks. The results demonstrate that the ray-based design has distinct advantages over existing point-based designs.

Finally, Figure 5 and our newly added Figure R.1 show that scaling the number of sub-NeRFs by decoupling rays to different sub-NeRFs offers better performance-parameter scalability than scaling by decomposing spatial positions or scaling the MLP or spatial grid. We believe that in future applications of scaling NN-based 3D modeling methods for complex scenes, our newly proposed dimension (scaling the number of sub-NeRFs by decoupling rays) is a worth-considering dimension in the compound scaling strategy.

**R3-Q6: The effect of Switch-NGP and Block-NGP**

Switch-NGP partitions the scenes in the point dimension, which is fundamentally different from our ray-based multi-NeRF framework. In complex scenes with many occlusions and arbitrary shooting trajectories, Switch-NeRF does not consider the different visibility of a target region to different views and can not tackle the training interference effectively, as noted in the reply to Q1. In addition, Switch-NGP does not handle consistency between multiple sub-NeRFs well due to the lack of overlapping regions and proper regularization between sub-NeRFs.

Block-NGP directly allocates the training images to multiple sub-NeRFs according to the image shooting positions. Judging the relationship between images based solely on shooting positions is insufficient for complex scenes, so it will lead to inaccurate partitioning results and unsatisfactory rendering quality.