Title: Rad-NeRF: Ray-decoupled Training of Neural Radiance Field
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural rendering field

Mutual learning

Novel view synthesis

Soft gate module

Complex scenes with occlusions
Abstract: Although the neural radiance field (NeRF) exhibits high-fidelity visualization on the rendering task, it still suffers from rendering defects, especially in complex scenes. In this paper, we delve into the reason for the unsatisfactory performance and conjecture that it comes from interference in the training process. Due to occlusions in complex scenes, a 3D point may be invisible to some rays. On such a point, training with those rays that do not contain valid information about the point might interfere with the NeRF training. Based on the above intuition, we decouple the training process of NeRF in the ray dimension softly and propose a Ray-decoupled Training Framework for neural rendering (Rad-NeRF). Specifically, we construct an ensemble of sub-NeRFs and train a soft gate module to assign the gating scores to these sub-NeRFs based on specific rays. The gate module is jointly optimized with the sub-NeRF ensemble to learn the preference of sub-NeRFs for different rays automatically. Furthermore, we introduce depth-based mutual learning to enhance the rendering consistency among multiple sub-NeRFs and mitigate the depth ambiguity. Experiments on five datasets demonstrate that Rad-NeRF can enhance the rendering performance across a wide range of scene types compared with existing single-NeRF and multi-NeRF methods. With only 0.2% extra parameters, Rad-NeRF improves rendering performance by up to 1.5dB. Code is available at https://github.com/thu-nics/Rad-NeRF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper claims that training with those rays with invisible 3D points (occlusions in complex scenes) that do not contain valid information about the point might interfere with the NeRF training. \n\nBased on this intuition, this paper proposes Rad-NeRF to decouple the training process of NeRF in the ray dimension softly, construct an ensemble of sub-NeRFs and train a soft gate module to assign the gating scores to these sub-NeRFs based on specific rays, where the gate module is jointly optimized with the sub-NeRF ensemble to learn the preference of sub-NeRFs for different rays. \n\nThis paper also introduces depth-based mutual learning to enhance the rendering consistency among multiple sub-NeRFs and mitigate the depth ambiguity. \n\nExperiments on five datasets demonstrate that Rad-NeRF can enhance the rendering performance across a wide range of scene types compared with existing Instant-NGP-based methods.\n\nSTRENGTHS:\nThis paper proposes an Instant-NGP-based ray-decoupled training framework to mitigate the training interference caused by invisible rays by ensemble sub-NeRFs via a jointly optimized gate module.\n\nThis paper also proposes a depth-based mutual learning method to ensure the rendering consistency among multiple sub-NeRFs, which serves as a geometric regularization, alleviating the depth ambiguity and avoiding overfitting.\n\nThis paper conducts extensive experiments. The results show that Rad-NeRF consistently improves rendering quality effectively and obtains better scalability than Instant-NGP-based baselines.\n\nWEAKNESSES:\nMy main concern is the novelty. Although this work aims for different tasks, it is quite like Switch-NeRF.\n\nQUESTIONS:\nThe improved 3DGS nowadays is quite good and does not have the ray sampling problem, so I think this work may have limited influence and help a little for our community. I would like to see the rebuttal and other reviews.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an innovative approach to enhance NeRF performance. The key observation is that due to occlusion, some objects may appear in one ray but not in another. While NeRF uses transmittance to mitigate this issue, the paper argues that this may not be sufficient. To address this, the authors propose using multiple MLPs to decode the same feature and jointly learn a gating function to fuse all the information. The results demonstrate that this design successfully improves performance, albeit by a small margin.\n\nSTRENGTHS:\n+ The proposed idea is innovative and interesting.\n+ The approach results in a performance improvement.\n\nWEAKNESSES:\nI have several concerns regarding the motivation, methodology, and validation of the proposed approach.\n\n## Motivation: \n\n+ The motivation is not very clear. Specifically, Figure 1 and lines 29-40 argue that poor performance is due to occlusion, leading the authors to propose using multiple MLPs. However, I find Figure 1, especially Figure 1(c), inaccurate. I would expect a CDF instead of a PDF, as NeRF often compensates RGB appearance with poor geometry. Thus, (c) might still have the correct RGB, i.e., its CDF is accurate. \n+ Moreover, Figure 7 does not support the occlusion hypothesis. I suspect the issue might be related to aliasing, where the pixel footprint differs between the center and boundary regions. This can also explain why MipNeRF360 outperforms the proposed method, and the comparison between ZipNeRF and RadZipNeRF in the supplementary material is very close.\n\n## Methodology: \n\nThe approach resembles MOE or a multi-head transformer. However, I have two questions:\n\n+ First, the proposed method applies depth consistency. Isn't this encouraging all MLPs to converge to the same output? This might explain why rad2, rad3, and rad4 in Figure 6 are very similar.\n\n+ Second, regarding the soft gating, the current input is the ray origin and direction. It might be more effective to include the 3D position, as splitting MLPs based on different spatial regions could be more reasonable. Additionally, the analysis of the gating function (Figure 7) lacks depth, and more details would be appreciated.\n\n## Validation: \n\n+ While the rad approach does improve performance, the improvement is relatively marginal. Furthermore, MipNeRF360 outperforms the proposed method, indicating that the proposed approach is not strong enough. \n\n+ In Table 1, why are switch NeRF and block NeRF not better than instant NGP? After ray participation, they should outperform the vanilla version.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTraditional NeRF models face challenges in rendering complex scenes due to interference from occluded rays, which leads to inaccurate training data. To address this, the authors propose Rad-NeRF, a framework that decouples the training process in the ray dimension by using multiple sub-NeRFs, each trained with specific rays that contain valid information for the points they observe.\nRad-NeRF employs a soft gate module that assigns gating scores to different sub-NeRFs based on the rays, allowing the model to learn preferences for different rays. Additionally, the framework introduces depth-based mutual learning to enhance consistency and mitigate depth ambiguity among the sub-NeRFs. Essentially, my understanding is that this work builds on Instant NGP by adding multiple sub-MLPs, with the gate module controlling the degree of participation of each MLP during rendering. Experiments demonstrate that Rad-NeRF outperforms existing single and multi-NeRF methods, improving rendering performance with minimal additional parameters.\n\nSTRENGTHS:\n1.\tThe paper is well-organized, with a clear abstract, introduction, methodology, experiments, and results sections.\n2.\tThe innovative approach of decoupling training in the ray dimension helps mitigate training interference caused by invisible rays. This leads to more accurate and consistent rendering results.\n3.\tThe introduction of a soft gate module that learns the preference of sub-NeRFs for different rays is a novel and effective way to allocate rays dynamically. This approach eliminates the need for manually defined allocation rules, making the method more adaptable and generalizable.\n4.\tRad-NeRF enhances rendering consistency among multiple sub-NeRFs through depth-based mutual learning. This technique also helps reduce depth ambiguity and improves geometric modeling accuracy.\n5.\tRad-NeRF can be integrated with various single-NeRF frameworks, further improving their performance. For instance, integrating with ZipNeRF shows potential for even better rendering outcomes, validating the flexibility and compatibility of Rad-NeRF with different neural rendering approaches.\n\nWEAKNESSES:\n1.\tWhat is the structure of the grid in the network architecture? Is it a multi-resolution grid? Are the features on the grid also learnable?\n2.\tWhat are the training and inference times for this method? It would be beneficial if the authors provided a comparison with other existing methods in the paper.\n3.\tIn the case of multiple networks, if a sub-NeRF receives very low soft gating scores, does it cease to update, or does it still contribute to the computation?\n4.\tAre there any automated methods or heuristics for selecting the optimal number of sub-NeRFs based on scene complexity? What is the relationship between the number of sub-models and the overall performance? Detailed tuning and experimentation are required to ensure optimal performance across different datasets and scenes.\n5.\tCould you provide a visualization of the gating scores for different sub-NeRFs for the same viewpoint? And I am curious about the distribution of gating scores in scenarios without occlusions.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The authors decouple the training process of NeRF in the ray dimension and propose a framework where they create an ensemble of sub-NeRFs and train a soft gate module to assign gating scores to these sub-NeRFs based on specific rays.\n- The gating module is a 4-layer MLP followed by a softmax function. The gating module takes in ray starting point and direction and outputs a gating score corresponding to each of the sub-NeRFs\n- The paper proposes a depth-based loss that compares the fused depth with the depth of the individual NeRFs in order to enhance their robustness.\n- Multiple indoor and outdoor scenes from various public datasets were used to show the method's effectiveness.\n\nSTRENGTHS:\n- The paper claims to improve PSNR by 1.5dB with just 0.2% extra parameters.\n- The authors have shown the plug-and-play capability of their method by adding it to existing methods like Zip-NeRF.\n- While methods like Block-NeRF and Mega-NeRF need a manually defined allocation rule and prior scene knowledge, Rad-NeRF does not require prior scene knowledge.\n\nWEAKNESSES:\n- Though the paper claims that their method needs few extra parameters and trains quickly, there only seems to be a slight improvement in quantitative numbers across many scenes.\n- The authors have pointed out that the method does not work for few-shot settings.\n\nQUESTIONS:\n- Uncertainty-based methods, like Bayes Rays, consider floaters as  regions corresponding to high uncertainty and remove them by thresholding the scene according to an uncertainty field during rendering. The spatial uncertainty is computed in roughly a minute on any existing method. What is the advantage of Rad-NeRF over such uncertainty-based methods? \n- Cross-Ray NeRF (CR-NeRF) leverages interactive information across multiple rays to synthesize occlusion-free novel views with the same appearances as the images. They recover the appearance by fusing global statistics, i.e., feature covariance of the rays and the image appearance. Can the method proposed in this paper solve the problems solved by Rad-NeRF?\n\nLIMITATIONS:\nYes, the authors have discussed the limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}