UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections

Authors: fangjinhua  wang
marie-julie  rakotosaona
michael  niemeyer
richard szeliski google
marc  pollefeys
federico  tombari
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:07:27

Abstract: Neural 3D scene representations have shown great potential for 3D reconstruction from 2D images. However, reconstructing real-world captures of complex scenes still remains a challenge. Existing generic 3D reconstruction methods often struggle to represent fine geometric details and do not adequately model reflective surfaces of large-scale scenes. Techniques that explicitly focus on reflective surfaces can model complex and detailed reflections by exploiting better reflection parameterizations. However, we observe that these methods are often not robust in real scenarios where non-reflective as well as reflective components are present. In this work, we propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. We investigate both camera view as well as reflected view-based color parameterization techniques and find that explicitly blending these representations in 3D space enables reconstruction of surfaces that are more geometrically accurate, especially for reflective surfaces. We further combine this representation with a multi-resolution grid backbone that is trained in a coarse-to-fine manner, enabling faster reconstructions than prior methods. Extensive experiments on object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF 360 and Ref-NeRF real demonstrate that our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces, leading to the best overall performance. Project page: https://fangjinhuawang.github.io/UniSDF.

Main paper: 
 introduction: 
 Given multiple images of a scene, accurately reconstructing a 3D scene is an open problem in 3D computer vision. 3D meshes from reconstruction methods can be used in many downstream applications, e.g., scene understanding, robotics, and creating 3D experiences for augmented/virtual reality [36, 51] . Typical aspects of real-world scenes such as uniformly colored areas or non-Lambertian surfaces remain challenging.As a traditional line of research, multi-view stereo methods [39, 49, 16, 46] usually estimate depth maps with photometric consistency and then reconstruct the surface as a post-processing step, e.g., point cloud fusion with screened Poisson surface reconstruction [19] or TSDF fusion [10] . However, they cannot reconstruct reflective surfaces since their appearances are not multi-view consistent.Figure 1 : Comparison of surface normals (top) and RGB renderings (bottom) on "garden spheres" [44] . While the state-of-the-art methods Ref-NeRF [44] , ENVIDR [22] , and Neuralangelo [21] struggle to reconstruct reflective elements or fine geometric details, our method accurately models both, leading to high-quality mesh reconstructions of all parts of the scene. Best viewed when zoomed in.Neural radiance fields (NeRF). As a seminal method in view synthesis, NeRF [28] represents a scene as a continuous volumetric field with an MLP, with position and camera view direction as inputs, and renders an image using volumetric ray-tracing. Since NeRF is slow to train, some methods [30, 42, 8] use voxel-grid-like data structures to accelerate training. Many follow-up works apply NeRFs to different tasks, e.g., sparse-view synthesis [54, 31, 43] , real-time rendering [9, 34, 18, 53] , 3D generation [33, 23, 7] and pose estimation [24, 41, 59] . For the 3D reconstruction task, there are many methods [32, 50, 47, 55, 13, 21, 37, 26, 35] integrating NeRF with signed distance functions, a common implicit function for geometry. Specifically, they transform SDFs back to volume density for volume rendering. However, we observe that they are unable to reconstruct shiny / reflective surfaces since NeRF's camera view direction parameterization for the color prediction does not accurately model reflective parts of the scene.NeRFs for reflections. To render reflective appearance, [40, 5, 57, 58] extend NeRF and decompose a scene into physical components with strong simplifying assumptions, e.g., known lighting [40] or no self-occlusion [5, 57] . Recently, Ref-NeRF [44] reparameterizes the appearance prediction with separate diffuse and reflective components by using the reflected view direction, which improves the rendering of specular surfaces. As a result, recent works [51, 22, 25, 14, 27] adopt this representation to reconstruct glossy surfaces. While leading to strong view-synthesis for reflections, we find that reflected view radiance field approaches often lead to overly smooth reconstructions with missing details and that their optimization is not stable on real-world scenes. In contrast to existing methods with a single radiance field, we propose to seamlessly combine reflected view and camera view radiance fields into a novel unified radiance field with learnable weight, which is robust for reconstruction in challenging scenes with reflective surfaces. The recent preprint Factored-NeuS [12] also uses camera view and reflected view radiance fields. It separately supervises the rendered colors of two radiance fields with ground-truth color, instead of learning a weight field to combine them like ours. We find that our approach to combine two radiance fields with learnable weight is simpler to train and leads to better reconstruction. Other recent methods [17, 56, 52] use weight to compose the colors from camera view radiance field(s) to render reflections, similarly to us. [17, 56] can only handle planar reflections, e.g., mirrors, and require ground-truth masks of reflective objects to supervise the weight. In contrast, our method can handle non-planar reflective objects, e.g., spheres. As we learn the weight to compose camera view and reflected view radiance fields without supervision, our method does not require additional input and can be trained from only RGB images. MS-NeRF [52] uses multiple volume density fields and how to reconstruct the underlying surface is undefined. In contrast, we use a single SDF field to represent geometry and can hence directly extract the iso-surface from it.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Please refer to L9-19 (abstract) and L46-57 (introduction).Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations in Sec. H of supplementary.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we first review the basic elements of NeRF [28] . We then describe the architecture and training strategy of our method.In NeRF [28] , a 3D scene is represented by mapping a position x and ray direction d to a volumetric density σ and color c using MLP. For a pixel in the target viewpoint and its corresponding ray r = o + td, distance values t i are sampled along the ray. The density σ i is predicted by a spatial MLP that receives the position x as input, while the directional MLP that predicts the color c i uses the bottleneck vector b(x) from the density MLP and the view direction d as input. The final color C is rendered as:EQUATIONwhereα i = 1 -exp(-σ i δ i ) is opacity, δ i = t i -t i-1is the distance between adjacent samples, andT i = i-1 j=1 (1 -α j )is the accumulated transmittance. The model is trained by minimizing the loss between the predicted and ground truth color:EQUATIONNote that Mildenhall et al. [28] use a single-layer directional MLP and thus often describe the combination of NeRF's spatial and view dependence MLPs as a single MLP. Given a set of known images of a scene that potentially contains reflective surfaces, our goal is to optimize a neural implicit field and reconstruct the scene with high fidelity and geometric accuracy.We propose UniSDF, a method that enables us to seamlessly combine camera view radiance fields and reflected view radiance fields to reconstruct both (a) non-reflective surfaces, diffuse reflective surfaces and complex surfaces with both reflective and non-reflective areas as well as (b) highly specular surfaces with a well defined and detailed reflected environment. Our pipeline is shown in Fig. 2 . We generate two radiance fields that are parameterized by camera view directions or reflected view directions and combine them at the pixel level using a learned rendered weight.Volume rendering the SDF. We represent the scene geometry using a signed distance field (SDF), which defines the surface S as the zero level set of SDF d:S = {x : d(x) = 0}.(3) To better reconstruct large-scale scenes, we follow Mip-NeRF 360 [3] and transform x into a contracted space with the following contraction:EQUATIONFor volume rendering, we compute the volume density σ(x) from the signed distance d(x) as:σ(x) = αΨ β (d(x)), where Ψ β is the cumulative distribution function of a zero-mean Laplace distribution with learnable scale parameter β > 0. The surface normal at x can be computed as the gradient of the signed distance field: n = ∇d(x)/||∇d(x)||.Hash Encoding with iNGP. To accelerate training and improve reconstruction of high-frequency details, we use iNGP [30] to map each position x to a higher-dimensional feature space. Specifically, the features {γ l (x)} from the pyramid levels of iNGP are extracted with trilinear interpolation and then concatenated to form one single feature vector γ(x), which is passed to the SDF MLP.Camera View & Reflected View Radiance Fields. In contrast to most existing methods [28, 2, 44] that use a single radiance field, we propose to combine a camera view radiance field and a reflected view radiance field to better represent reflective and non reflective surfaces.We follow NeRF [28] for representing our camera view radiance field c cam , which is computed from features defined at each position and the camera view direction:EQUATIONFigure 3 : Visualization of the color of reflected view radiance field, color of camera view radiance field, learned weight W, composed color and surface normal on "sedan" and "garden spheres" scenes [44] . Our method assigns high weight (red color) for reflective surfaces, e.g., window and hood of sedan, spheres, without any supervision.where b is the bottleneck feature vector from SDF MLP, n is the normal at x and d is the camera view direction. Similarly to recent works [50, 47] , we notice that using surface normals as input leads to better quality.We represent the reflected radiance field c ref with an MLP f ref as:EQUATIONwhere ω r is the reflected view direction around the normal n. In Ref-NeRF [44] , it is shown that for BRDFs under a limited set of conditions, view-dependent radiance is a function of ω r only. Unlike Ref-NeRF, which uses separate diffuse and specular components, we only use the specular component, leading to a simpler architecture. Additionally, we observe that using separate diffuse and specular components can lead to optimization instabilities resulting in geometry artifacts (see supp. mat. for details).The main difference between two radiance fields is the view directional input of the MLP. As shown in Fig. 3 , our method mainly uses the reflected view radiance field to represent highly specular reflections such as the tree reflections in the garden spheres or the environment reflection on the sedan car. The camera view radiance field is used to represent more diffuse reflections.Learned composition. We compose two radiance fields using a learnable weight field in 3D. Specifically, we use an MLP f w to learn the weight values w:EQUATIONWe compose the signals at the pixel level. We first volume render W, C ref , C cam following Eq. 1.We then compose the colors for each pixel as follows:EQUATIONIn Fig. 3 , the weight W detects reflections well and assigns high weight to reflected view radiance field in reflective regions. The surface normals show that our model accurately reconstructs both reflective and non-reflective surface geometry.Motivation of composing radiance fields. Disambiguating the influence of geometry, color and reflection is an ill-posed problem in 3D reconstruction from images. NeRF-based methods [47, 50, 21] with camera view radiance field show their robustness in real-world scenes [1] , while having difficulty with reflections [44, 14] . Ref-NeRF based methods [25, 14, 51] with reflected view radiance field usually perform well under restricted conditions, e.g., highly specular objects [44] , while we experimentally find their performance degrades in real-world scenes, e.g., NeRO [25] and Ref-NeuS [14] on DTU [1] (Tab. 1), BakedSDF [51] on Mip-NeRF 360 dataset [3] (Fig. 4 ). Therefore, to extend theoretically justified Ref-NeRF representation with robust scene representations, we propose to exploit the advantages of two radiance fields by combining them with learnable weight. Moreover, since each type of radiance field is specialized for different levels of reflection strength and complexity, we observe that the reconstructed geometries while using the two types of radiance are often complementary (Fig. 5 ). In our method, we explicitly intertwine the radiance fields in 3D to continuously determine and use the most adapted parametrization for each surface area.Coarse-to-fine training. We observe that directly optimizing all the features in our multi-resolution hash grid leads to overfitting of training images, in particular to specular appearance details, which in turn results in incorrect geometry as we show in Fig. 7 (a). We observe that this model tends to fake specular effects by embedding emitters inside the surface exploiting the numerous learnable features in the hash grid. Therefore, we propose to instead optimize the hash grid features in a coarse-to-fine fashion, similarly to [21, 37] , to avoid overfitting and promote smoother and more realistic surfaces. Specifically, we start with L init coarse pyramid levels in the beginning of training, and introduce a new level with higher resolution every T 0 training fraction (see implementation details in Sec. 4.1).Regularization. Following prior works [50, 47] , we use an eikonal loss [15] to encourage d(x) to approximate a valid SDF:EQUATIONTo promote normal smoothness, we constrain the computed surface normal n to be close to a predicted normal vector n ′ . n ′ is predicted by the SDF MLP and normalized. We use the normal smoothness loss L p [44] as:EQUATION)We also use the orientation loss L o from Ref-NeRF [44] to penalize normals that are "back-facing", using:EQUATIONFull loss function. The full loss function L includes the color loss L color of composed color C and the regularizations, which is written as follows:EQUATION4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [No] Justification: The datasets used in this paper are all publicly available. We are unable to immediately release the code, but plan it for the future.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental settings: 
 Datasets. We extensively evaluate our method on four different types of datasets. The DTU dataset [1] is an indoor object-centric dataset with ground truth point clouds. Following prior works [50, 47] , we use the same 15 scenes for evaluation. The Shiny Blender dataset [44] contains six different shiny objects that are rendered in Blender under conditions similar to the NeRF dataset. The Mip-NeRF 360 dataset is proposed in [3] and contains complex unbounded indoor and outdoor scenes captured from many viewing angles. We further evaluate on the three large-scale scenes with reflections that are introduced in Ref-NeRF [44] , which consists of the scenes "sedan", "garden spheres" and "toycar". For simplicity, we name these 3 scenes the "Ref-NeRF real dataset".Implementation details. Based on the Mip-NeRF 360 codebase [29] , we implement our method in Jax [6] with the re-implementation of VolSDF [50] and iNGP [30] . In our iNGP hierarchy of grids and hashes, we use 15 levels from 32 to 4096, where each level has 4 channels. For coarse to fine training, we set L init = 4 and T 0 = 2%. Similar to mip-NeRF 360 [3] , we use two rounds of proposal sampling and then a final NeRF sampling round. Following Zip-NeRF, we penalize the sum of the mean of squared grid/hash values at each pyramid level with a loss multiplier as 0.1. Our models are all trained on 8 NVIDIA Tesla V100-SXM2-16GB GPUs with a batch size of 2 14 . We train 25k steps on DTU / Shiny Blender and 100k steps on Mip-NeRF 360 / Ref-NeRF real datasets, which takes 0.75h and 3.50h respectively. See the supplement for more details. Baselines. We compare our method to state-of-the-art volumetric implicit methods in surface reconstruction [47, 11, 13, 21, 51, 25, 14, 22, 12] and view synthesis [3, 44, 4] . Neuralangelo [21] and Zip-NeRF [4] are hash grid-based state-of-the-art methods for reconstruction and view synthesis, respectively. Tailored for handling reflections, [25, 14, 22, 12] are top performing methods for reconstructing and rendering objects with reflective surfaces. BakedSDF [51] is a top performing method for reconstructing high quality mesh of unbounded scenes with reflective surfaces.PSNR ↑ SSIM ↑ LPIPS ↓ MAE • ↓ Acc* ↓ PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ Mip-NeRFTo further evaluate the effectiveness of our method, we propose two custom baselines, named "CamV" and "RefV". Using the same backbone as our method, "CamV" uses only the camera view radiance field, while "RefV" uses only the reflected view radiance field following Ref-NeRF [44] . Note that for both baselines, we also use our coarse-to-fine training strategy to improve performance.DTU. We evaluate the reconstruction quality on DTU dataset [1] . Following prior works, we extract the mesh at 512 resolution. For Neuralangelo [21] , we report the reproduced results from the official implementation. As shown in Tab. 1, Geo-NeuS [13] performs best on DTU and our method outperforms the remaining methods. Geo-NeuS heavily relies on supervision from accurate SfM point cloud and photometric consistency constraint to achieve top performance, while our method uses rendering loss only like Neuralangelo [21] . For reflective regions, the SfM reconstruction is inaccurate for supervision [14] and the multi-view photometric consistency is not guaranteed. We show that Geo-NeuS performs worse on Shiny Blender [44] in Tab. 2. Besides, we observe that NERO [25] and Ref-NeuS [14] perform worse than their baseline NeuS [47] . Though these methods perform well on objects with strong reflections, e.g., Shiny Blender [44] , they are not robust in general real-world scenes without strong reflections.Shiny Blender. We summarize the rendering metrics, mean angular error (MAE) of normals and accuracy (Acc) of mesh in Tab. 2. Our method performs best in PSNR and on par with ENVIDR [22] in SSIM, LPIPS and MAE. Note that ENVIDR additionally uses an environment MLP, which we do not require, to improve rendering and reconstruction. Besides, we find ENVIDR unrobust in real-world scenes with geometry artifacts on both reflective and non-reflective surfaces, shown in Fig. 1 . For mesh quality, our method performs on par with Ref-NeuS [14] in Acc, while performing much better than it on DTU [1] (Tab. 1). This demonstrates the roubustness of our method in various types of scenes. Moreover, our method explicitly outperforms Geo-NeuS [13] , Neuralangelo [21] and Zip-NeRF [4] in all metrics.Figure 4 : Qualitative comparison with BakedSDF [51] on "bicycle" and "officebonsai" scenes of Mip-NeRF 360 dataset [3] . BakedSDF produces hole structures in many regions (highlighted with dotted orange boxes) and less details of fine structures (highlighted with red boxes), while our method reconstructs more complete surfaces and better details. Best viewed when zoomed in. Mip-NeRF 360 dataset. As shown in Tab. 2, our method performs on par with Zip-NeRF [4] in rendering. Note that Zip-NeRF focuses on view synthesis, while we focus on surface reconstruction. Compared with BakedSDF [51] and Neuralangelo [21] , our performance is much better in all metrics. As shown in Fig. 4 , our method reconstructs more complete surfaces and better details, while BakedSDF shows hole artifacts and struggles to reconstruct fine geometric details.Ref-NeRF real dataset. As shown in Tab. 2, evaluation on this dataset is challenging for all methods.Our method outperforms Ref-NeRF [44] and Neuralangelo [21] in SSIM and LPIPS, Zip-NeRF [4] in PSNR and SSIM, and BakedSDF [51] in LPIPS. For surface reconstruction, Neuralangelo [21] cannot reconstruct reflective spheres well as shown in Fig. 1 , while our method accurately reconstructs the smooth surface of the reflective spheres and the fine details on the statue.The four datasets that we evaluate on include various scene types, ranging from object-level to unbounded scenes, with and without reflections. While some methods perform best on specific datasets, e.g., Geo-NeuS [13] on DTU and Zip-NeRF [4] on Mip-NeRF 360 dataset [3] , we show their performance degrades on other types of datasets, e.g., Shiny Blender [44] .In contrast, our method shows competitive performance on all datasets and performs best overall (see averaged metrics in Tab. 2). This demonstrates the robustness of our method to various scene types.Custom baselines comparison. We compare our method with our two custom baselines, CamV and RefV, on the DTU [1] , Mip-NeRF 360 [3] , and Ref-NeRF real [44] datasets. As shown in Tab. 3, our method outperforms the two baselines in all metrics on all three datasets. Besides, CamV mostly outperforms RefV, while RefV fails on one scene in DTU. This shows that the camera view radiance field is usually more robust than the reflected view radiance field, although this method does not reconstruct the geometry of reflective regions well.Fig. 5 shows a qualitative comparison, where RefV reconstructs smooth surface for the reflective back window but has artifacts on the side for "sedan", while CamV fails to reconstruct accurate surfaces because of the reflections. On the "toycar" scene, RefV fails to reconstruct the correct geometry, while CamV reconstructs shiny surfaces better while showing artifacts on the hood. For RefV, we sometimes observe optimization issues with separate diffuse and specular components, where the specular component may be blank throughout training and the diffuse component (w./o. directional input) wrongly represents the view-dependent appearance with incorrect geometry (see supp. mat. for details). By coupling two difference radiance fields continuously in 3D, our method represents the appearance and geometry better than the baselines that only use a single radiance field, leading to higher-quality reconstructed surfaces. Mesh visualization. In Fig. 6 , we visualize our meshes on Shiny Blender [44] and Mip-NeRF 360 dataset [3] . Our method can accurately reconstruct the surfaces of reflective objects as well as large-scale scenes with fine geometric details.Coarse-to-fine training. As shown in Fig. 7 (a), the reconstructions of "sedan" [44] contain artifacts on the specular window and hood without training in a coarse-to-fine manner. With all feature pyramid grids activated in the beginning, the hash grid backbone can easily overfit to the specular effects with wrong geometry.Unification of radiance fields. In Tab. 3, we show that composing two radiance fields can consistently improve performance. In this ablation, we justify the effectiveness of unification with learnable weights. Since the main difference of two radiance fields is the view directional input, we design a baseline with a single radiance field that takes both camera view d and reflected view ω r as inputs.Structurally, this baseline has the same input information as our radiance fields and weight field.As shown in Fig. 7 (b), our method performs better in reconstruction, while the baseline cannot reconstruct reflective surfaces well on both scenes. 
conclusion: 
 In this paper, we have presented UniSDF, a novel algorithm that learns to seamlessly combine radiance fields for robust and accurate reconstruction of complex scenes with reflections. We find that camera view radiance fields, e.g., NeRF, are robust to complex real settings but cannot reconstruct reflective surfaces well, while reflected view radiance fields, e.g., Ref-NeRF, can effectively reconstruct highly specular surfaces but struggle in real-world settings and to represent other types of surfaces. By adaptively combining camera view and reflected view radiance fields with a learnable weight field, our method significantly outperforms the baselines with either single radiance field. Together with a hash grid backbone to accelerate training and improve reconstruction details, our method performs superior to or on par with state-of-the-art methods, tailored for handling reflections or not, in reconstruction and rendering on different types of scenes, ranging from object-level to unbounded scenes, with and without reflections. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We specify the experimental details in Sec. 4.1 and Sec. B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We extensively evaluate our method on 4 datasets, which contain 33 scenes in total, and our method needs to be trained for each scene individually with 8 GPUs. Due to the limited computing resources and time, reporting error bars would be too computationally expensive for us.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discuss the potential societal impacts in Sec. G of supplementary. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original owners of codes and datasets that we use in Sec. 4.1. We mention their licenses in Sec. I of supplementary.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Our paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the supplementary, we first discuss more experimental details, including dataset and implementation details. Second, we discuss BakedSDF [51] in more detail. We show that BakedSDF is often not stable and we discuss how we finetune this method to improve performance on several scenes. Third, we summarize the detailed evaluation results on DTU [1] , Shiny Blender [44] , Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets. We also qualitatively compare with the state-of-the-artmethods [13, 51, 21, 14, 4] . Fourth, we include more ablation study. Fifth, we discuss the comparison with two custom baselines, RefV and CamV, in more detail. Finally, we discuss the potential societal impacts and limitations of our method.We use the same data splits as prior works for fair comparison. On DTU [1] , following [47, 13] , we use all the images for surface reconstruction. On Shiny Blender [44] , Mip-NeRF 360 [3] , and Ref-NeRF real [44] datasets, we follow the official protocol and use the official training / testing split for training and testing.Network Architecture. In addition to the iNGP [30] structure that we have introduced in the main paper, we further discuss the details of the MLP architectures. Specifically, the SDF MLP f has 2 layers with 256 hidden units and outputs the bottleneck feature vector b with size 256. The two radiance MLP f cam , f ref have 4 layers with 256 hidden units. Besides, the weight MLP f w has a single layer with 256 hidden units.Recall that following Mip-NeRF 360 [3] , we use two rounds of proposal sampling and then a final NeRF sampling round. The proposal sampling is used to bound the scene geometry and recursively generate more detailed sample intervals, while the final NeRF sampling is used to render the final set of intervals into an image. We set the number of samples for these 3 sampling rounds as 64, 32, 32 for the object-level DTU [1] and Shiny Blender [44] , and 64, 64, 32 for unbounded Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets.In Sec. 3 of the main paper, we mainly introduce model details of the final NeRF sampling round, where the color is rendered, for simplicity. Thus, we introduce the details for proposal sampling rounds here. Specifically, the proposal sampling rounds only have a SDF MLP, i.e.no radiance MLP and weight MLP, since color is not rendered in these rounds. Moreover, the two proposal sampling rounds share a SDF MLP, which is different from the SDF MLP in the NeRF sampling round. Contrary to Zip-NeRF [4] that uses a distinct iNGP for each sampling round, we use a single iNGP that is shared by all sampling rounds. We find that this produces similar performance as using multiple iNGPs but explicitly simplifies the model.Loss Function. In the loss function (Eq. 12 in the main paper, which is for the final NeRF sampling round), we set λ 1 = 10 -4 and λ 3 = 10 -3 . Moreover, we set λ 2 = 10 -4 for Shiny Blender [44] , and λ 2 = 10 -3 for DTU [1] , Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets. For proposal sampling rounds, we replace L color with L prop , the proposal loss described in Mip-NeRF 360 [3] .Training Details. For training, we use the Adam [20] optimizer with β 1 = 0.9, β 2 = 0.999, ϵ = 10 -6 . We warm up the learning rate in the first 2% iterations and then decay the it logarithmically from 5 × 10 -3 to 5 × 10 -4 .In our experiments, we find that the optimization of BakedSDF [51] is sensitive and often fails completely on Shiny Blender [44] and Ref-NeRF real dataset [44] , as shown in Fig. 8 .Figure 8 : Final image rendering and normal of original BakedSDF [51] on "garden spheres" scene [44] . The training is not stable leading to degraded results (see text).BakedSDF only uses eikonal loss, L eik , for regularization, where the corresponding loss weight is set to 0.1 by default. We experimentally find that decreasing the eikonal loss weight can stabilize the training and thus carefully tune it for each scene. For Shiny Blender [44] , we set the eikonal loss weight as 10 -2 for "toaster", "helmet" and "coffee", and 10 -1 for "ball". Unfortunately, we could not find the best eikonal loss weight for "car" and "teapot" scenes since the training does not produce reasonable geometry. For completeness, we report the rendering metrics on these two scenes with eikonal loss weight as 10 -2 . For Ref-NeRF real dataset [44] , we set the eikonal loss weight as 10 -2 for "sedan" and "toycar", and 10 -5 for "garden spheres".Tab. 4, Tab. 5, Tab. 6 and Tab. 7 contain the detailed metrics for each individual scene on DTU [1] , Shiny Blender [44] , Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets respectively.We qualitatively compare with state-of-the-art methods [21, 51, 4] on Shiny Blender, Mip-NeRF 360 and Ref-NeRF real datasets in Fig. 9 . Our method is robust and demonstrates competitive performance on various scene types, ranging from object-level to unbounded scenes, with and without reflections.Shiny Blender. In addition to the MAE error for evaluating the surface normal accuracy, we also evaluate the mesh quality. Though Chamfer Distance includes both accuracy and completeness, Ref-NeuS [14] points out that the ground-truth meshes of Shiny Blender [44] are double-layered, which results in many redundant points in the ground truth. Therefore, we follow Ref-NeuS [14] and only evaluate the accuracy (Acc) of reconstructed meshes. Besides, since ground-truth meshes of "ball" and "teapot" are unavailable, we only evaluate on the remaining four scenes following Ref-NeuS.As shown in Fig. 10 , since the objects in Shiny Blender are highly reflective, our method automatically assigns high weights for the reflected view radiance field in most regions. Though our learned weight is not supervised, it can successfully detect the reflective surfaces. Besides, our reflected view radiance field represents the highly specular reflections of the surrounding environment, which is similar to the observation in Fig. 3 of the main paper.We further qualitatively compare the surface normals with Geo-NeuS [13] , Neuralangelo [21] and Ref-NeuS [14] in Fig. 11 . For "teapot" scene, Geo-NeuS fails to reconstruct the smooth surface of the teapot. Neuralangelo reconstructs incorrect surface elements under the object. Ref-NeuS reconstructs overly smooth surfaces without fine details. For "ball" scene, Geo-NeuS produces slight artifacts on the surface, while Neuralangelo reconstructs lots of incorrect surface elements inside the ball. In contrast, our method reconstructs the surfaces more accurately on both scenes.In the main paper, we have compared our method with two custom baselines, CamV and RefV, both quantitatively (Tab. 3) and qualitatively (Fig. 5 ). In Fig. 12 , we further visualize the qualitative results on scan 37 of DTU [1] . On the one hand, RefV reconstructs holes on the objects with or without reflections, which is similar to the artifacts that BakedSDF [51] shows in Fig. 4 of the main paper. On the other hand, though the surface is a little noisy, CamV reconstructs shiny objects relatively well. Note that in Fig. 5 of the main paper, CamV also reconstructs the shiny surfaces of "toycar" well, despite having some small artifacts. Since scan 37 of DTU and "toycar" of Ref-NeRF real dataset mainly contain reflective surfaces that are less specular, we can infer that camera view radiance field can handle less specular reflections to some extent.As shown in Fig. 13 , we sometimes observe that RefV has optimization issues with separate diffuse and specular components. Specifically, the specular component may be empty throughout training, Figure 10 : Visualization of the color of reflected view radiance field, color of camera view radiance field, learned weight W (red color represents large weight), composed color and surface normal on "helmet" and "car" scenes [44] . Best viewed when zoomed in.Figure 11 : Qualitative comparison of surface normal on "teapot" and "ball" scenes [44] . Best viewed when zoomed in.while the diffuse component represents both view-dependent and non view-dependent appearance.Since diffuse component depends only on the 3D position x, the view-dependence is represented with incorrect geometry, as shown in the results of "toycar" in Fig. 13 . We believe that this issue may be related to the high frequency signals that iNGP [30] can encode. In RefV, the diffuse component is parameterized by the feature vector γ from iNGP, which is capable of representing very high frequency signal. Therefore, the diffuse component may take advantage of the high capacity of the iNGP representation to model the view-dependent appearance with geometry only, leading to an incorrect reconstruction. This is especially true for small-scale scenes with relatively simple view-dependent appearance, e.g., "toycar".Normal. Recall that we use normal n, the gradient of the signed distance field, for computing reflected view direction ω r and loss function Eq. 11. In contrast, Ref-NeRF [44] uses predicted normal n ′ instead. In this ablation study, we follow Ref-NeRF [44] and use the predicted normal n ′ for computing reflected view direction ω r and loss function Eq. 11. As shown in Tab. 8, our method performs better than the baseline in rendering. We visualize the surface normal in Fig. 14 . Our method successfully reconstructs the reflective surfaces, while the baseline reconstructs artifacts on them. [44] 0.955 0.995 0.958 0.998 0.922 0.974 ENVIDR [22] 0.972 0.997 0.993 0.999 0.955 0.984 NERO [25] 0.949 0.974 0.971 0.995 0.929 0.956 Ref-NeuS [14] 0.919 0.989 0.971 0.981 0.903 0.941 BakedSDF [51] 0.807 0.979 0.990 0.967 0.939 0.978 Ours 0.954 0.993 0.990 0.998 0.945 0.973 LPIPS ↓ Mip-NeRF 360 [3] 0.062 0.189 0.127 0.094 0.144 0.118 Zip-NeRF [4] 0.060 0.231 0.115 0.010 0.127 0.128 Geo-NeuS [13] 0.080 0.082 0.082 0.024 0.134 0.110 Neuralangelo [21] 0.066 0.186 0.085 0.017 0.123 0.091 Ref-NeRF [44] 0.041 0.059 0.075 0.004 0.095 0.078 ENVIDR [22] 0.031 0.020 0.022 0.003 0.097 0.044 NERO [25] 0.074 0.094 0.050 0.012 0.089 0.110 Ref-NeuS [14] 0.076 0.058 0.046 0.025 0.114 0.119 BakedSDF [51] 0 Positive impact. Our method can accurately reconstruct complex scenes with reflections, which is a challenging task for existing methods. The accurate reconstruction can be used for many downstream applications, e.g., robotics and creating 3D experiences for augmented/virtual reality.Negative impact. Similar to most existing volumetric implicit methods [28, 3, 4, 21, 44] , our method needs to be individually trained for each scene. CamV reconstructs more noisy surface on the red handle with reflections (highlighted with red box and zoomed in), while RefV generates holes on the shiny objects and even the brick without any reflections. Best viewed when zoomed in.NeRF 360 [3] and Ref-NeRF real [44] datasets), we use 8 NVIDIA Tesla V100-SXM2-16GB GPUs simultaneously, which consumes much energy.Figure 13 : Visualization of diffuse color component, specular color component and surface normal for RefV on "sedan" and "toycar" scene [44] . RefV successfully decomposes two color components for "sedan", while it fails on "toycar" with blank specular component. Best viewed when zoomed in. Figure 14 : Ablation study of normals on "sedan" and "garden spheres" scene [44] . Best viewed when zoomed in.In this work, we mainly focus on robust surface reconstruction of scenes with reflections. Our method cannot be used for editing tasks such as relighting. Besides, similar to most NeRF methods [3, 21, 4] , our method requires the estimated poses from SfM, e.g., COLMAP [38] , as input for real-world scenes. However, SfM needs to perform feature matching, which is based on multi-view photometric consistency, for pose estimation. As we discuss in the main paper, multi-view photometric consistency is not guaranteed with reflective surfaces and thus SfM may produce inaccurate poses, which may lead to inaccurate reconstruction and rendering. In addition, our method cannot accurately reconstruct the reflective surfaces with sparse input views. It is challenging to determine reflections in this case because of the ambiguity. Note that both Shiny Blender [44] and Ref-NeRF real dataset [44] carefully provide dense 360 degree views around the reflective objects so that it is easier to detect the view-dependent reflective appearance.code:• multinerf [29] : Apache License 2.0. datasets:• DTU [1]: we do not find the license.• Shiny Blender [44] : coffee (CC-0 license), toaster (CC-BY license), car (CC-0 license), helmet (CC-0 license). • Mip-NeRF 360 dataset [3] : CC-BY license.• Ref-NeRF real dataset [44] : it is captured by the authors of Ref-NeRF [44] , we do not find the license.NeurIPS Paper Checklist 


Decision: The paper describes a specific self-contained improvement to NeRF: instead of using the camera direction only as in classic NeRF and instesd of using the reflected direction in combination with some "proto-shading" information as in RefNeRF, the proposal is to simply use both and learn how to blend between them. The idea is not extremely tight to NeRF, it could also be applied to other view-dependent appearance.

All reviewers understood and liked the idea, result and exposition.

The downside is, that the decomposition does not in the end provide any new insight or result in a semantic or physical separation. 

The rebuttal admitted this limitation, providing some discussion and was generally well received to also clarify some details.

The AC read and and understood the paper in detail, certifies the reviews and the dsicussion were conducted orderly and hence can only recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a new strategy for modelling view-dependent effects in Neural Radiance Field-based scene models. Existing approaches have used networks conditioned on camera view directions, as well as on reflected view directions using surface normals, but this work proposes and validates the idea that both model types should be used simultaneously. This is achieved by employing a learned weight model that mixes the two color predictions, thereby allowing regions where the color is not well explained by the reflection-based model to be handled by the view direction-based model. This is shown through ablations to be a better strategy than either strategy individually, and is also compared to a number of baseline methods which employ reflection-based modelling of view-dependent effects.
strength_weakness: strength_weakness: The only notable weakness is that the proposed strategy is a fairly minor deviation from previously proposed methods. However, I think this is largely mitigated by the non-obvious nature of the change and the in-depth experimental evaluation.
questions: questions: Given the increasing prevalence of methods like 3DGS which do not use neural networks for modelling view-dependence but rather simpler approaches like spherical harmonics, do you see a way of applying the ideas of this paper to such methods?
limitations: limitations: The only limitation I see is the reduced interpretability/editability of the model compared to other approaches, but this is addressed by the authors. I see no issues with societal impact beyond generic concerns that are relevant to all scene modelling methods.
paper_summary: paper_summary: The paper proposes a method to reconstruct scenes containing both reflective surfaces and non-reflective surfaces with high fidelity. Specifically, it trains a camera view radiance field and a reflected view radiance field separately, combining them by a learnable weight. The method is evaluated on four datasets, which covers different situations including objects and 360 scenes, shiny objects, real-world captures. And the method outperforms or is on par with SOTA on all these datasets, demonstrating its effectiveness.
strength_weakness: strength_weakness: 1. The method involves two radiance fields and one weight field, and it has to perform volume rendering for colors from both radiance fields and the weights, which is very time-consuming even with implementation of iNGP.
2. I wonder if you have a possible explanation for why some reflected view radiance field methods is not robust to real-world scene.
3. I noticed the comparison with Factored-NeuS is only performed on DTU. Since Factored-NeuS is designed to also handle glossy objects, I wonder if how it look like when compared to this paper on Shiny Blender and Ref-NeRF real datasets.
4. As the authors mention in the section of limitations, this method requires posed images and SfM can fail on reflective surfaces. While this is a common issue, I still wonder how robust this method is to inaccurate camera pose. Even a failure case is fine.
questions: questions: Please see weaknesses.
limitations: limitations: The authors listed a quite complete list of limitations. I think some are common issues for many works like requiring camera pose and dense views. With that being said, in the future the authors can consider extending this work to sparse views and how to eliminate the requirement for several fields and volume rendering, making the algorithm more efficient.
paper_summary: paper_summary: The paper tackles the problem of 3D reconstruction in the presence of highly reflective objects. To address the problem, they propose to learn an SDF-based neural representation. Different from prior work, they use two radiance branches in their representation, one conditioned on the camera viewing direction rotated about the normal and one conditioned on the regular viewing direction. The outputs of the two networks are combined using a learned weight to arrive at the final rendered radiance. 

The authors argue that while the reflected direction is better at capturing the details in reflective surfaces, the viewing direction is more robust to real data. 

Through extensive experiments, they show their method is competitive with prior art in both the tasks of NVS and mesh recovery (as measured by Chamfer Distance).
strength_weakness: strength_weakness: Significance: I think the method is generally quite a straightforward extension of existing work and the paper's contributions are fairly limited. As I see it both the radiance branches have previously existed and the Ref-NeRF parametrization could be considered more “physically based”, the authors forgo a more mathematically solid setup in favour of an increase in performance. 

I think the paper would benefit a bit from providing more insight into the reasons for some of the phenomena i.e. why do the weights separate like that for more specular/diffuse regions even though both are conditioned on some sort of parametrization of the viewing direction etc. 

I am sympathetic however to the authors that some of these phenomena just occur and might be hard to reason about, but some of the simulated experiments could provide more insight into how the optimization behaves.
questions: questions: 1) Is there a stopgrad on any components of the normal smoothness loss (predicted vs computed normal in Eq 10)? 

2) I see there is an ablation on this, but why do the authors think the weight to mix the outputs of the two radiance branches is necessary, why doesn’t this factor just get automatically learned by the two networks? 

3) The authors seem to have ran ENVIDR on the Ref-NeRF real dataset (as evidenced by Fig 1?), but do not include the results in Table 2? Why? 

4) Have the authors tried passing in $n^\prime$ instead of $n$ to the radiance network or also calculating $w_r$ about it? If I understand correctly $n^\prime$ is supposed to be “smoother”, perhaps this would make results more robust for just RefV?
limitations: limitations: Limitations have been adequately discussed in the supplement.
paper_summary: paper_summary: This paper proposes UniSDF, an improved NeuS architecture capable of reconstructing photorealistic reflective surfaces and robustly working in real-world scenes. The method models reflective color and base color as separate MLPs, using learned weights to blend them and obtain the final colors. Qualitative and quantitative results on several datasets demonstrate that the method surpasses other baselines and results in fewer geometric artifacts.
strength_weakness: strength_weakness: The biggest weakness is the lack of physical interpretation. In Eq. 8, the final color is a linear combination of the camera view and reflected view radiance fields, resulting in unclear physical meaning for each color component. Although the method focuses on novel view synthesis and geometry reconstruction regardless of the underlying physical components, this lack of physical meaning leads to ambiguity about the assumptions the authors have for the reconstructed scene.

For example, the model doesn't account for how reflections change with different surface roughness, raising the question of whether it will perform well on surfaces with spatially varying roughness. Additionally, the model does not explicitly trace the reflective ray but instead models the color of the reflective ray through an MLP. This raises concerns about its ability to handle near-field lighting conditions effectively.

Furthermore, I am curious about the differences between Ref-NeRF's equation c = diffuse + specular × tint and Eq. 8 c=(1−w) c_cam+w c_ref. Will these two modeling approaches yield similar performance?
questions: questions: 1. Title: It's unclear from the abstract and introduction where the "unifying" concept comes from. "Unifying scene representation" suggests combining several scene representations, but theoretically, it is still a NeuS-based representation.
	2. Missing Citations: 
		a. MS-NeRF: Multi-space Neural Radiance Field: This paper also presents a similar idea of linearly combining different color fields. 
		b. Several Concurrent Works Regarding Reflective NeRF: 
			i. NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections
			ii. SpecNeRF: Gaussian Directional Encoding for Specular Reflections 
		c. Reflection Handling in Gaussian Splatting: 
			i. MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections 
			ii. Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting
	3. Illustration of Existing Works: In line 131, I wouldn't call Ref-NeRF[43] as using a single radiance field, as it uses two MLPs to model diffuse color and specular color separately (though the specular MLP is conditioned on the feature of the diffuse MLP)
	4. Hyperparameters in Final Loss: The final loss contains several hyperparameters for controlling the strength of each term. In my experience, these weights play an important role. It would be beneficial to conduct small ablations or illustrate how these weights were decided.
	5. Input Redundancy: The f_ref and f_cam functions take in both the spatial coordinate x and the bottleneck feature b from the SDF MLP. Given that b is already a spatial feature vector, what's the point of adding x as one of the inputs? This seems a bit redundant.
	6. Normals and Orientation Loss: There are two normals being produced: n is the normal from the SDF field, and n′ is the MLP-predicted normal. For Eq. 11, Ref-NeRF uses n′ for orientation loss. It's unclear if this is a typo or if n is actually used for the orientation loss.
	7. Volumetric Rendering SDF: In line 124, it would be better to cite which volumetric rendering SDF system is used. I assume it's NeuS.
limitations: limitations: There are no potential negative society impact.

Comments: comment: I appreciate the author's efforts on answering my questions especially providing additional results in such a short time. My concerns / questions are solved. I'll keep my positive rating.
Thank you for the comments. We will add an ablation study about the normal that we use, i.e. SDF normal $n$ or predicted normal $n’$, for the reflected view direction and loss formulations in the paper.
Thank you very much for the response to my questions. I especially appreciate the response in the global rebuttal. 

I'm not fully satisfied with the answer to the question about specular/diffuse separation--I feel like the authors are just reiterating what's happening, as opposed to providing any reasoning. 

I also think since reviewer 5Tqb and I both had questions about this, it would be useful to run a quick (even on a smaller subset of scenes) the effect of using different normals in the reflected view direction and/or loss formulations for the camera ready version. 

Nonetheless, I think the experimental results, alongside some of the arguments laid out in the global rebuttal, such as the comment of Ref-NeRF not being a complete physical model (not explicitly handling interreflections, etc), make a stronger case for the paper. I am happy to increase my score.