UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections
Authors: Fangjinhua Wang
Marie-Julie Rakotosaona
Michael Niemeyer
Richard Szeliski
Marc Pollefeys
Federico Tombari
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D reconstruction
novel view synthesis
reflection
Primary area: machine_vision
TL;DR: We unify camera view and reflected view radiance field parameterizations and combine it with a multi-resolution grid backbone to achieve high-quality reconstruction of complex scenes with reflections.

Abstract: Neural 3D scene representations have shown great potential for 3D reconstruction from 2D images. However, reconstructing real-world captures of complex scenes still remains a challenge. Existing generic 3D reconstruction methods often struggle to represent fine geometric details and do not adequately model reflective surfaces of large-scale scenes. Techniques that explicitly focus on reflective surfaces can model complex and detailed reflections by exploiting better reflection parameterizations. However, we observe that these methods are often not robust in real scenarios where non-reflective as well as reflective components are present. In this work, we propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. We investigate both camera view as well as reflected view-based color parameterization techniques and find that explicitly blending these representations in 3D space enables reconstruction of surfaces that are more geometrically accurate, especially for reflective surfaces. We further combine this representation with a multi-resolution grid backbone that is trained in a coarse-to-fine manner, enabling faster reconstructions than prior methods. Extensive experiments on object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF 360 and Ref-NeRF real demonstrate that our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces, leading to the best overall performance. Project page: https://fangjinhuawang.github.io/UniSDF.

UniSDE: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections
Fangjinhua Wang" ETH Zurich
Marie-Julie Rakotosaona
Michael Niemeyer Google
Richard Szeliski Google
Google
Marc Pollefeys
Federico Tombari
ETH Zurich
Google
Abstract
Neural 3D scene representations have shown great potential for 3D reconstruc- tion from 2D images. However, reconstructing real-world captures of complex scenes still remains a challenge. Existing generic 3D reconstruction methods often struggle to represent fine geometric details and do not adequately model reflective surfaces of large-scale scenes, Techniques that explicitly focus on reflective sur- faces can model complex and detailed reflections by exploiting better reflection parameterizations. However, we observe that these methods are often not robust real scenarios where non-reflective well reflective components this work, we propose UniSDE, a general purpose 3D reconstruction method that reconstruct large complex scenes with reflections. We investigate both camera view as well as reflected view-based color parameterization techniques and find that explicitly blending these representations in 3D space enables reconstruction of surfaces that are more geometrically accurate, especially for reflective surfaces. We further combine this representation with a multi-resolution grid backbone that is trained in coarse-to torsee-to-finn fine manner, enabling faster reconstructions than prior methods. Extensive experiments on object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF 360 and Ref-NeRF real demonstrate that our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces, leading to the best overall performance. Project page: attpss//ffane inggawanngggggg gi github. io/UniSDF.
present. In
Introduction
Given multiple images of scene, accurately reconstructing a 3D scene is an open problem in 3D computer vision. 3D meshes from reconstruction methods can be used in many downstream applications, e.g., scene understanding, robotics, and creating 3D experiences for augmented/virtur reality [36, 51]. Typical aspects of real-world scenes such as uniformly colored areas or non- Lambertian surfaces remain challenging.
As traditional line of research, multi-view stereo methods [39, 49, 16, 46] usually estimate depth maps with photometric consistency and then reconstruct the surface as a post-processing step, e-g., point cloud fusion with screened Poisson surface reconstruction [19] or TSDF fusion [10]. However, they cannot reconstruct reflective surfaces since their appearances are not multi-view consistent.
"This work conducted during internship Google.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Ref-NeRF
ENVIDR
Neuralangelo
Ours
Our mesh
Figure I: Comparison of surface normals (top) and RGB renderings (bottom) "garden spheres" [44]. While the state-of-the-art methods Ref-NeRF [44], ENVIDR [22]. and Neuralangelo [21] struggle to reconstruct reflective elements fine geometric details, method accurately models both, leading to high-quality mesh reconstructions of all parts of the scene. Best viewed when zoomed in.
Recently, Neural Radiance Fields (NeRF) [28] render compelling photo-realistic images by pa- rameterizing a scene as a continuous function of radiance and volume density using a multi-layer perceptron (MLP). More recent works [30, 8, 42, 4] replace augment MLPs with grid based data structures to accelerate training. For example, Instant-NGP (iNGP) [30] uses pyramid of grids and hashes encode features and tiny MLP process them. Motivated by NeRF, neural implicit reconstruction methods [50, 47] combine signed distance functions (SDF) with volume rendering, and produce smooth and complete surfaces. For acceleration, recent works [21, 37] rely on hash grid representations and reconstruct surfaces with finer details. However, these NeRF-based methods cannot accurately reconstruct reflective surfaces [44, 14].
To better represent the reflective appearance, Ref-NeRF [44] parameterizes the appearance using reflected view direction that exploits the surface normals, while NeRF uses the camera view direction. Recently, some works [51, 22, 25, 14] adopt this reflected view parameterization and successfully reconstruct reflective surfaces. We observe that while reflected view radiance fields can effectively reconstruct highly specular reflections, they struggle to represent more diffuse or ambiguous reflection types and fine details that can found real scenes. In contrast, we find that direct camera view radiance fields are more robust to difficult surfaces in real settings, although the reconstructions still present artifacts for reflective scenes. In this paper, we seamlessly bring together reflected view and camera view radiance fields into novel unified radiance field for representing 3D real scenes accurately in the presence of reflections. Our method robust for reconstructing both real challenging scenes and highly reflective surfaces.
The proposed method, named UniSDF, performs superior on par with respective state-of-the methods which are tailored for a specific scene type. UniSDF can be applied to any type of dataset, ranging from DTU [1], Shiny Blender [44], Mip-NeRF 360 dataset [3], to Ref-NeRF real dataset [44], leading to the overall best performance. It demonstrates the capability to accurately reconstruct complex scenes with large scale, fine details and reflective surfaces as we see Fig. 1.
In summary, we propose a novel algorithm that learss to scamlessly combine two radiance fields with learnable weight field while exploiting the advantages of each representation. Our method produces high quality surfaces in both reflective and non-reflective regions.
2 Related Works
Multi-view stereo (MVS). Many traditional [39, 48] and learning based [49, 16, 46, 45] MVS methods first estimate multi-view depth maps and then reconstruct the surface by fusing depth maps in post-processing step. As the core step, depth estimation is mainly based on the photometric consistency assumption across multiple views. However, this assumption fails for glossy surfaces with reflections, and thus MVS methods cannot reconstruct them accurately.

---

Neural radiance fields (NeRF). As a seminal method in view synthesis, NeRF [28] represents a scene as a continuous volumetric field with an MLP, with position and camera view direction as inputs, and renders an image using volumetric ray-tracing. Since NeRF is slow to train, some methods [30, 42, 8] use voxel-grid-like data structures to accelerate training. Many follow-up works apply NeRFs to different tasks, e.g., sparse-view synthesis [54, 31, 43], real-time rendering [9, 34, 18, 53], 3D generation [33, 23, 7] and pose estimation [24, 41, 59]. For the 3D reconstruction task, there are many methods [32, 50, 47, 55, 13, 21, 37, 26, 35] integrating NeRF with signed distance functions, a common implicit function for geometry. Specifically, they transform SDFs back to volume density for volume rendering. However,
observe that they unable to reconstruct shiny / reflective surfaces since NeRF's camera view direction parameterization for the color prediction does not accurately model reflective parts of the scene.
NeRFss for reflections. To render reflective appearance, [40, 5, 57, extend NeRF and decompose scene into physical components with strong simplifying assumptions, e-g. known lighting [40] or no self-occlusion [5, 57]. Recently, Ref-NeRF [44] reparameterizes the appearance prediction with separate diffuse and reflective components by using the reflected view direction, which improves the rendering of specular surfaces. As a result, recent works [51, 22, 25, 14, 27] adopt this representation to reconstruct glossy surfaces. While leading to strong view-synthesis for reflections, we find that reflected view radiance field approaches often lead to overly smooth reconstructions with missing details and that their optimization is not stable real-world scenes. In contrast to existing methods with a single radiance field, we propose to seamlessly combine reflected view and camera view radiance fields into novel unified radiance field with learnable weight, which robust for reconstruction in challenging scenes with reflective surfaces. The recent preprint Factored-NeuS [12] also uses camera view and reflected view radiance fields. It separately supervises the rendered colors of two radiance fields with ground-truth color, instead of learning weight field to combine them like ours. We find that our approach to combine two radiance fields with learnable weight is simpler to train and leads to better reconstruction. Other recent methods [17, 56, 52] use weight to compose the colors from camera view radiance field(s) render reflections, similarly us. [17, 56] can only handle planar reflections, e.g., mirrors, and require ground-truth masks of reflective objects to supervise the weight. In contrast, our method can handle non-planar reflective objects, e-g., spheres. As we learn the weight to compose camera view and reflected view radiance fields without supervision, our method does not require additional input and can be trained from only RGB images. MS-NeRF [52] uses multiple volume density fields and how to reconstruct the underlying surface is undefined. In contrast, we use a single SDF field to represent geometry and can hence directly extract the iso-surface from it.
3 Method
In this section, we first review the basic elements of NeRF [28]. We then describe the architecture and training strategy of our method.
3.1 NeRF Preliminaries
In NeRF [28], a 3D scene is represented by mapping a position x and ray direction a volumetric density a and color using MLP. For pixel in the target viewpoint and its corresponding ray r o + td, distance values t; are sampled along the ray. The density a; is predicted by a spatial MLP that receives the position x as input, while the directional MLP that predicts the color uses the bottleneck vector bxx) from the density MLP and the view direction as input. The final color C is rendered as:
C = Σ W.CC, W. = Ti
(I)
where d; = 1 exp(-x,ppp is opacity, 8, = 4 - 4-11 is the distance between adjacent samples, and T: = |=1 az) is the accumulated transmittance. The model is trained by minimizing the loss between the predicted and ground truth color:
Leoker
Note that Mildenhall et al. [28] use single-layer directional MIP and thus often describe the combination of NcRF's spatial and view dependence MLPs as a single MLP.
3

---

Craf
Radiance MLP Fref
Reflection
iNGP
SDF
Radiance
Gram
Eq.
MLP
d
feam
f
n'
b
W
Weight fw
Figure 2: Pipeline of UniSDF. We combine the camera view radiance field and reflected view radiance field in 3D. Given position x, we extract iNGP features y and input them to an MLP f that estimates signed distance value d used to compute the NeRF density. We parametrize the camera view and reflected view radiance fields with two different MIPs Seam and fref respectively. Finally, we learn continuous weight field that is used to compute the final color weighted composite W of the radiance fields colors Ceam and Cref after volume rendering, Eq. 8.
3.2
UniSDF
Given a of known images of scene that potentially contains reflective surfaces, our goal is to optimize neural implicit field and reconstruct the scene with high fidelity and geometric accuracy. We propose UniSDF, method that enables to seamlessly combine camera view radiance fields and reflected view radiance fields to reconstruct both (a) non-reflective surfaces, diffuse reflective surfaces and complex surfaces with both reflective and non-reflective areas well as (b) highly specular surfaces with a well defined and detailed reflected environment. Our pipeline is shown in Fig. We generate two radiance fields that are parameterized by camera view directions or reflected view directions and combine them at the pixel level using a learned rendered weight.
Volume rendering the SDE We represent the scene geometry using a signed distance field (SDF), which defines the surface S as the zero level set of SDF d:
S == fx: = 0}. To better reconstruct large-scale scenes, follow Mip-NeRF 360 [3] and transform into contracted space with the following contraction:
(3)
|x|| S ||x||
contract(x)) = (2-111)
[
|
For volume rendering, we compute the volume density (xx from the signed distance d(x) as: (x) = aM2 (d(x)), where V is the cumulative distribution function of a zero-mean Laplace distribution with learnable scale parameter 3 0. The surface normal at can be computed as the gradient of the signed distance field: n = Vd(())/l
Hash Encoding with iNGP. To accelerate training and improve reconstruction of high-frequency details, we use iNGP [30] to map each position x to a higher-dimensionall feature space. Specifically, the features (x)} from the pyramid levels of iNGP are extracted with trilinear interpolation and then concatenated form one single feature vector (x),, which is passed the SDF MLP.
Camera View & Reflected View Radiance Fields. In contrast to most existing methods [28, 2, 44] that use a single radiance field, we propose to combine camera view radiance field and reflected view radiance field to better represent reflective and non reflective surfaces.
We follow NeRF [28] for representing our camera view radiance field Coam, which is computed from features defined at each position and the camera view direction:
Coom
Scom (x, d, n. b);
(5)

---

Color reflected view radiance field
Color camera view radiance field
Leamedd weight
Composed color
Normal
Figure 3: Visualization of the color of reflected view radiance field, color of camera view radiance field, learned weight W, composed color and surface normal on "sedan" and "garden spheres" scenes [44]. Our method assigns high weight (red color) for reflective surfaces, e.g., window and hood of sedan, spheres, without any supervision.
where b is the bottleneck feature vector from SDF MLP, is the normal at and d is the camera view direction. Similarly to recent works [50, 47], we notice that using surface normals as input leads to better quality.
We represent the reflected radiance field Cref with an MLP fref as:
Cref = fref Wy, n, b),
(6)
where Wp is the reflected view direction around the normal In Ref-NeRF [44], it is shown that for BRDFs under a limited set of conditions, view-dependent radiance is a function of w, only. Unlike Ref-NeRF, which separate diffuse and specular components, component, leading to a simpler architecture. Additionally, we observe that using separate diffuse and specular components can lead optimization instabilities resulting geometry artifacts (see supp. mat. for details).
only the specular
The main difference between two radiance fields is the view directional input of the MLP. As shown in Fig. 3, our method mainly uses the reflected view radiance field to represent highly specular reflections such the tree reflections in the garden spheres or the environment reflection the sedan car. The camera view radiance field is used to represent more diffuse reflections.
Learned composition. We compose two radiance fields using learnable weight field in 3D. Specifically, we use an MLP fo to learn the weight values w:
w = sigmoid (fw n, b)).
We compose the signals at the pixel level. first volume render W, Crefi Coam following Eq. We then compose the colors for each pixel as follows:
C W Cref + (1- W) Cean.
(8)
In Fig. 3, the weight W detects reflections well and assigns high weight to reflected view radiance field in reflective regions. The surface normals show that our model accurately reconstructs both reflective and non-reflective surface geometry.
Motivation of composing radiance fields. Disambiguating the influence of geometry, color and reflection is an ill-posed problem in 3D reconstruction from images. NeRF-based methods [47, 50,
21] with camera view radiance field show their robustness real-world scenes [1], while having difficulty with reflections [44, 14]. Ref-NeRF based methods [25, 14, 51] with reflected view radiance field usually perform well under restricted conditions, e-g., highly specular objects [44], while we experimentally find their performance degrades in real-world scenes, e.g., NeRO [25] and Ref-NeuS [14] on DTU [1] (Tab. 1), BakedSDF [51] on Mip-NeRF 360 dataset [3] (Fig. 4). Therefore, extend theoretically justified Ref-NeRF representation with robust scene representations,

---

we propose to exploit the advantages of two radiance fields by combining them with learnable weight. Moreover, since each type of radiance field is specialized for different levels of reflection strength and complexity, we observe that the reconstructed geometries while using the two types of radiance are often complementary (Fig. 5). In our method, we explicitly intertwine the radiance fields in 3D to continuously determine and use the most adapted parametrization for each surface area.
3.3 Training and Regularization
Coarse-to-fing training. We observe that directly optimizing all the features in our multi-resolution hash grid leads overfitting of training images, in particular specular appearance details, which in turn results in incorrect geometry as we show in Fig. (a). We observe that this model tends to fake specular effects by embedding emitters inside the surface exploiting the numerous learnable features in the hash grid. Therefore, we propose to instead optimize the hash grid features a coarse-to-fine fine fashion, similarly to [21, 37], to avoid overfitting and promote smoother and more realistic surfaces. Specifically, we start with Linit coarse pyramid levels in the beginning of training, and introduce a new level with higher resolution every To training fraction (see implementation details in Sec. 4.1).
Regularization. Following prior works [50, 47], we use an eikonal loss [15] to encourage d(x) to approximate a valid SDF:
Len Ex [() Vd(x))l 1)
To promote normal smoothness, we constrain the computed surface normal n to be close a predicted normal vector n'. n' is predicted by the SDF MLP and normalized. We use the normal smoothness loss Cp [44] as:
C,
Σ
wi n n'|1?
(10)
We also use the orientation loss Co from Ref-NeRF [44] to penalize normals that are "back-facing", using:
Lo
Σ w; max(0, n dd).
(11)
Full loss function.
The full loss function C includes the color loss Coolor of composed color C and the regularizations, which is written as follows:
L = Leolor + X Leik + dohap + d36..
(12)
4 Experiments
4.1 Experimental Settings
Datasets. We extensively evaluate our method on four different types of datasets. The DTU dataset [1] is an indoor object-centric dataset with ground truth point clouds. Following prior works [50, 47], we use the same 15 scenes for evaluation. The Shiny Blender dataset [44] contains six different shiny objects that are rendered in Blender under conditions similar to the NeRF dataset. The Mip-NeRF 360 dataset is proposed in [3] and contains complex unbounded indoor and outdoor scenes captured from many viewing angles. We further evaluate the three large-scale scenes with reflections that are introduced Ref-NeRF [44], which consists of the scenes "sedan", "garden spheres" and "toycar". For simplicity, we name these scenes the "Ref-NeRF real dataset".
Implementation details. Based on the Mip-NeRF 360 codebase [29], we implement our method in Jax [6] with the re-implementation of VoISDF [50] and iNGP [30]. In our iNGP hierarchy of grids and hashes, we use 15 levels from 32 to 4096, where each level has 4 channels. For coarse to fine training, we set Lmni = 4 and To 2%. Similar to mip-NeRF 360 [3], we use two rounds of proposal sampling and then a final NeRF sampling round. Following Zip-NeRF, we penalize the sum of the mean of squared grid/hash values at each pyramid level with a loss multiplier as O. Our models are all trained on 8 NVIDIA Tesla V100-SXM2-16GI GPUs with a batch size of 2 We train 25k steps on DTU/ Shiny Blender and 100k steps on Mip-NeRF 360 Ref-NeRF real datasets, which takes 0.75h and 3.50h respectively. See the supplement for more details.

---

Table 1: Quantitative results of Chamfer Distance (C.D.) on DTU [1]. Red, orange and yellow indicate the first, second and third best methods. t: Factored-NeuS [12] does not provide result for scan 69, Its result is the average error of the other 14 scenes.
Methods
NeuS [47]
WerpalWarp [11]
Geo-NeuS [13]
Neuralangelo [21]
C.D. (mm) 4
0.87
0.68
0.51
1.07
Methods
NERO [25]
Ref- [14]
Factored-NeuSt [12]
Ours
C.D. (mm)
1.04
1.93
0.77
0.64
Table 2: Quantitative results on Shiny Blender [44], Mip-NeRF 360 dataset [3] and Ref-NeRF real dataset [44]. 'Mean' represents the average rendering metrics on all datasets. Red, orange, and yellow indicate the first, second, and third best methods for each metric. *: We follow Ref-NeuS [14] and evaluate accuracy of mesh on four scenes (car, helmet, toaster, coffee). See supp. mat. for details.
Methods
Shiny
Mip-NeRR dataset
Ref-NeRE real dataset
Mean
PSNR
SSIM
LPIPS
MAR L Apc
PSNR 1 SSIM
LPIPS
PSNR
SSIM 1 LPIPS 1
PSNR
SSIM
LPIPS
Mip-NaRt NERF [30
25.19
0.939
0.122
27.69
0.791
0.237
24.27
0.650
0.276
2582
0.791
0.212
Zip-NoRF
29.2
0.942
28.53
0.828
27.15
0.183
Geo-Neu8 [13]
28.78
0.945
0.085
10:52
Neuralangelo (21)
30.68
0.949
0.095
14.16
1.81
25.08
0.699
0.332
23.70
0.608
0.330
26.99
0.52
0.252
RGF-N
35.96
0.967
0.058
13.38
24.06
0.389
0333
ENVIDR
35.85
0.983
0.066
NeRO [25]
29.84
0.962
0.072
Ref-Nou8 [14]
27.40
0.951
0.073
5:34
0.85
Factored-Neut (12)
30.80
01.554
0.076
5.31
1.90
BakedSDP
25.60
0.943
0,090
26.42
0.738
0.314
24.43
0.636
0.325
25.48
0,772
0.243
Ours
36.82
0.976
0.043
4.76
1.06
27.67
0.808
0.213
23.70
29.40
03007
0.174
Baselines. We compare our method to state-of-the-ar volumetric implicit methods in surface reconstruction [47, 11, 13, 21, 51, 25, 14, 22, 12] and view synthesis [3, 44, 4]. Neuralangelo [21] and Zip-NeRF [4] are hash grid-based state-of-the-ar methods for reconstruction and view synthesis, respectively. Tailored for handling reflections, [25, 14, 22, 12] are top performing methods for reconstructing and rendering objects with reflective surfaces. BakedSDF [51] is a top performing method for reconstructing high quality mesh of unbounded scenes with reflective surfaces.
To further evaluate the effectiveness of our method, we propose two custom baselines, named "C and "RetV", Using the same backbone as our method, "CamV" uses only the camera view radiance field, while "RefV" uses only the reflected view radiance field following Ref-NeRF [44]. Note that for both baselines, we also use our coarse-to- to-fine training strategy to improve performance.
4.2 Evaluation Results
DTU. We evaluate the reconstruction quality on DTU dataset [1]. Following prior works, we extract the mesh at 512 resolution. For Neuralangelo [21], we report the reproduced results from the official implementation. As shown in Tab. 1, Geo-NeuS [13] performs best on DTU and our method outperforms the remaining methods. Geo-NeuS heavily relies on supervision from accurate SiM point cloud and photometric consistency constraint to achieve top performance, while our method uses rendering loss only like Neuralangelo [21]. For reflective regions, the SIM reconstruction is inaccurate for supervision [14] and the multi-view photometric consistency is not guaranteed. We show that Geo-NeuS performs worse on Shiny Blender [44] in Tab. Besides, we observe that NERO [25] and Ref-Neut [14] perform worse than their baseline NeuS [47]. Though these methods perform well on objects with strong reflections, e.g., Shiny Blender [44], they are not robust in general real-world scenes without strong reflections.
Shiny Blender. We summarize the rendering metrics, mean angular error (MAE) of normals and accuracy (Acc) of mesh in Tab. 2. Our method performs best in PSNR and on par with ENVIDR [22] in SSIM, LPIPS and MAE. Note that ENVIDR additionally uses an environment MLP, which we do not require, to improve rendering and reconstruction. Besides, we find ENVIDR unrobust in real-world scenes with geometry artifacts on both reflective and non-reflective surfaces, shown in Fig. 1. For mesh quality, our method performs on par with Ref-NeuS [14] in Acc, while performing much better than it on DTU [1] (Tab. 1). This demonstrates the roubustness of our method in various types of scenes. Moreover, our method explicitly outperforms Geo-NeuS [13], Neuralangelo [21] and ZiR-NeRF [4] in all metrics.
7

---

bicycle (BakedSDF)
bicycle (Ours)
officcbensai (BakedSDF)
officebensai (Ouns)
Figure 4: Qualitative comparison with BakedSDF [51] on "bicycle" and "officebonsai" scenes of Mip-NeRF 360 dataset [3]. BakedSDF produces hole structures in many regions (highlighted with dotted orange boxes) and less details of fine structures (highlighted with red boxes), while our method reconstructs more complete surfaces and better details. Best viewed when zoomed in.
Table 3: Quantitative comparison with two custom baselines. Best results are in bold. *; RefV fails on scan 110 of DTU [1], the reported chamfer distance (C.D.) is the average of other 14 scenes.
DTU
Mip-NeRF 360 datasct
Ref-NeRF real dataset
Methods
C.D. (mm) +
PSNR
SSIM 1
LPIPS +
PSNR
SSIM +
LPIPS +
CamV
0.85
27.26
0.800
0.225
23.30
0.622
0.283
RefV
0.89*
26.74
0.794
0.223
23.02
0.615
0.301
Ours
0.64
27.67
0.808
0.213
23.70
0.636
0.265
Mip-NeRF 360 dataset. As shown in Tab. 2, our method performs on par with Zip-NeRF [4] in rendering. Note that Zip-NeRF focuses on view synthesis, while we focus surface reconstruction. Compared with BakedSDF [51] and Neuralangelo [21], our performance is much better in metrics. As shown in Fig. 4, our method reconstructs more complete surfaces and better details, while BakedSDF shows hole artifacts and struggles to reconstruct fine geometric details.
Ref-NeRF real dataset. As shown in Tab. 2, evaluation on this dataset challenging for all methods. Our method outperforms Ref-NeRF [44] and Neuralangelo [21] in SSIM and LPIPS, Zip-NeRF [4] in PSNR and SSIM, and BakedSDF [51] in LPIPS. For surface reconstruction, Neuralangelo [21] cannot reconstruct reflective spheres well shown in Fig. 1, while method accurately reconstructs the smooth surface of the reflective spheres and the fine details on the statue.
Summary of evaluation. The four datasets that we evaluate on include various scene types, ranging from object-level unbounded scenes, with and without reflections. While some methods perform best on specific datasets, e.g., Geo-NeuS [13] on DTU and Zip-NeRF [4] on Mip-NeRF 360 dataset [3], show their performance degrades on other types of datasets, e.g., Shiny Blender [44].
In contrast, our method shows competitive performance on all datasets and performs best overall (see
averaged metrics in Tab. 2). This demonstrates the robustness of our method to various scene types.
Custom baselines comparison. We compare our method with our two custom baselines, CamV and RefV, on the DTU [1], Mip-NeRF 360 [3], and Ref-NeRF real [44] datasets. As shown in Tab. 3, our method outperforms the two baselines in all metrics on all three datasets. Besides, CamV mostly outperforms Ref',, while RefV fails on one scene in DTU. This shows that the camera view radiance field usually more robust than the reflected view radiance field, although this method does not reconstruct the geometry of reflective regions well.
Fig. 5 shows qualitative comparison, where RefV reconstructs smooth surface for the reflective back window but has artifacts on the side for "sedan", while CamV fails to reconstruct accurate surfaces because of the reflections. On the "toycar" scene, ReIV fails to reconstruct the correct geometry, while Cam V reconstruets shiny surfaces better while showing artifacts on the hood. For Rei', we sometimes observe optimization issues with separate diffuse and specular components, where the specular component may be blank throughout training and the diffuse component (w./o. directional input) wrongly represents the view-dependent appearance with incorrect geometry (see supp. mat. for details). By coupling two difference radiance fields continuously in 3D, our method represents the appearance and geometry better than the baselines that only use single radiance field, leading to higher-quality reconstructed surfaces.
8

---

RefV
CamV
Ours
Figure 5: Qualitative comparison of surface normals with two baselines, RefV and CamV "sedan" and "toycar" scenes [44]. Best viewed when zoomed in.
helmet
coffee
teapot
ball
bicycle
kitchenlego
Figure 6: Visualization of our meshes. Best viewed when zoomed in.
w.oo, coarse-to- fine
Ours
learned composition
Ours
(a)
(b)
Figure 7: Ablation study of our method. Best viewed when zoomed in.
Mesh visualization. In Fig. 6, we visualize meshes Shiny Blender [44] and Mip-NeRF 360 dataset [3]. Our method can accurately reconstruct the surfaces of reflective objects as well as large-scale scenes with fine geometric details.
4.3 Ablation Study
Coarse-to-fine training. As shown Fig. 7 (a), the reconstructions of "sedan" [44] contain artifacts the specular window and hood without training in coarse-to-fine manner. With all feature pyramid grids activated in the beginning. the hash grid backbone can easily overfit to the specular effects with wrong geometry.
Unification of radiance fields. In Tab. 3, we show that composing two radiance fields can consis- tently improve performance. In this ablation, justify the effectiveness of unification with learnable weights. Since the main difference of two radiance fields is the view directional input, we design baseline with single radiance field that takes both camera view d and reflected view w, as inputs. Structurally, this baseline has the same input information as our radiance fields and weight field. As shown in Fig. 7 (b), method performs better in reconstruction, while the baseline cannot reconstruct reflective surfaces well on both scenes.
5 Conclusion
In this paper, we have presented UniSDR a novel algorithm that learns to seamlessly combine radiance fields for robust and accurate reconstruction of complex scenes with reflections. We find that camera

---

view radiance fields, e.g., NeRF, are robust to complex real settings but cannot reconstruct reflective surfaces well, while reflected view radiance fields, e-g., Ref-NeRF, can effectively reconstruct highly specular surfaces but struggle in real-world settings and to represent other types of surfaces. By adaptively combining camera view and reflected view radiance fields with learnable weight field, our method significantly outperforms the baselines with either single radiance field. Together with a hash grid backbone to accelerate training and improve reconstruction details, our method performs superior to on par with state-of-the-ar methods, tailored for handling reflections not, in reconstruction and rendering on different types scenes, ranging from object-level to unbounded scenes, with and without reflections.
We would like to thank Dor Verbin, Peter Hedman, Ben Mildenhall and Pratul P. Srinivasan for feedback and comments.
Acknowledgement.
References
[1] Henrik Aanæs, Rasmus Ramsbol Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scale data for multiple-view stereopsis. IJV,, 2016. 2, 6, 7, 8, 14, 15, 16, 19, 21
[2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman. Ricardo Bautin-Bruaall, and Pratul P Srinivasan. Mip-NeRF: multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021. 4
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- NeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022. 2, 4, 5, 6, 7, 8, 9, 14, 15, 16, 18, 19, 20, 21
[4] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Zip- NeRF: Anti-aliased grid-based neural radiance fields. In ICCV, 2023. 2, 7, 8, 14, 15, 16, 18, 19, 20
[5] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In ICCV, 2021. 3
[6] James Bradbury, Roy Frostig, Peter Hawkins, Matthew Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. composable transformations of Python+NumPy programs, 2018. 6
JAX:
[7] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3d generative adversarial networks. In CVPR, 2022. 3
[8] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensorRF: Tensorial radiance fields. In ECCV. Springer, 2022. 3,3
[9] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. MobileNeRF: Exploiting the polygon rasterization pipeline for efficient neural field rendering mobile architectures. In CVPR, 2023. 3
[10] Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303-312, 1996. 1
[11] François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. In CVPR, 2022. 7, 16
[12] Yue Fan, Ivan Skorokhodov, Oleg Voynov, Savva Ignatyev, Evgeny Burnaev, Peter Wonka, and Yiqun Wang. Factored-neus: Reconstructing surfaces, illumination, and materials of possibly glossy objects. arXiv preprint arXiv:2305. 2023. 3, 7, 16
[13] Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-Neus: Geometry- consistent neural implicit surfaces learning for multi-view reconstruction. NeurlPS, 35, 2022. 3, 7, 8. 14, 15, 16, 18
10

---

[14] Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ying-Cong Chen. Ref-neus: Ambiguity- reduced neural implicit surface learning for multi-view reconstruction with reflection. arXiv preprini arXiv:2303. 10840, 2023. 2, 3,3,5, 5, 7, 14, 15, 16, 18
[15] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002. 10099, 2020. 6
[16] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution view stereo and stereo matching. In CVPR, 2020. 1,2
[17] Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Hai Zhang. Nerfren: Neural radiance fields with reflections. CVPR, 2022.
[18] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV, 2021. 3
[19] Michael Kazhdan and Hugues Hoppe. Screened Poisson surface reconstruction. ACM Transac- tions on Graphics (ToG), 3233):1-13, 2013.
[20] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
14
[21] Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming- Yu Liu, and Chen-Hsuae Lin. Neuralangelo: High-fidelity neural surface reconstruction. In CVPR, 2023. 2, 3, 5, 6, 7, 8, 14, 15, 16, 18, 19, 20
[22] Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijayku- mar. ENVIDR: Implicit differentiable renderer with neural environment lighting. arXiv preprint arXiv:2303.13022 2023. 2, 7, 18
[23] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung-Y Yi Lin. Magic3D: High-resolution text-to-3d content creation. In CVPR, 2023. 3
[24] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: Bundle-adjusting neural radiance fields. In ICCV, 2021. 3
[25] Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long. Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. arXiv preprint arXiv:2305. /7398, 2023. 2, 3, 5, 7, 16, 18
[26] Xiaoxiao Long, Cheng Lin, Peng Wang. Taku Komura, and Wenping Wang. SparseNeuS: Fast generalizable neural surface reconstruction from sparse views. In ECCV, 2022. 3
[27] Li Ma, Vasu Agrawal, Haithem Turki, Changil Kim, Chen Gao, Pedro Sander, Michael Zollhöfer, and Christian Richardt. Specnerf: Gaussian directional encoding for specular reflections. In CVPR, 2024. 3
[28] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021. 2, 3, 4, 18
[29] Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman, Ricardo Martin-Brualle and Jonathan Barron. MultiNeRF: A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF, 2022. 6, 20
[30] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. SIGGRAPH, 2022. 2, 3, 4, 6, 14, 17
[31] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. CVPR, 2022.
[32] Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, 2021. 3
11

---

[33] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-36 using 2d diffusion. Xrviv preprint arXiv:2209. 14988, 2022.
[34] Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. MERF: Memory -eemory-effcientt radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics (TOG), 42(4):1-12, 2023.
[35] Yufan Ren, Fangjinhua Wang, Tong Zhang, Mare Pollefeys, and Sabine Süsstrunk.. Volrecon: Volume rendering of signed ray distance functions for generalizable multi-view reconstruction. arXiv preprini wwrxx:2222.080060 2022. 3
[36] Gernot Riegler and Vladlen Koltun. Stable view synthesis. In CVPR, 2021.
[37] Radu Alexandru Rosu and Sven Behnke. PermutoSDF: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices. In CVPR, 2023. 2, 3,
[38] Johannes L Schonberger and Sun-Michael Frahm. Structure-from-moots revisited. In CVPR,
2016. 20
[39] Johannes L Schönberger, Enliang Zheng, Jan- Michael Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016. 1,
[40] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
[41] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison. iMAP: Implicit mapping and positioning in real-time. In ICCV, 2021. 3
[42] Cheng Sun, Min Sun, and Hwann-Tzow Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022. 2,
[43] Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. SPARE: Neural radiance fields from sparse and noisy poses. In CVPR, 2023. 3
[44] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. In CVPR, 2022. 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16. 17, 18, 19, 20, 21
[45] Fangjinhua Wang. Silvano Galliani, Christoph Vogel, and Mare Pollefeys. IterMVS Iterative probability estimation for efficient multi-view stereo. In CVPR, 2022. 2
[46] Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Mare Pollefeys. PatchmatchNet: Learned multi-view patchmatch stereo. In CVPR, 2021. 1,2
[47] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021. 2, 3, 5, 6, 7, 14, 16
[48] Qingshan Xu and Wenbing Tao. Multi-scale geometric consistency guided multi-view stereo.
CVPR, 2019. 2
[49] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. MVSNet: Depth inference for unstructured multi-view stereo. In ECCV, 2018. 1,2
[50] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. NewrIPS, 34, 2021. 2,
[51] Lior Yariy, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T' Barron, and Ben Mildenhall. BakedSDF: Meshing neural sdfs for real-time view synthesis. arXiv preprint 1rX.,.2302.14859 2023. 1, 2, 3, 5, 7, 8, 14, 15, 16, 18, 19
[52] Ze-Xin Yin, Jiaxiong Qiu, Ming-Ming Cheng, and Bo Ren. Multi-space neural radiance fields. In CVPR, 2023.
12

---

[53] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa, PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021. 3
[54] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixeINeRF: Neural radiance fields from one few images. In CVPR, 2021. 3
[55] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction. NeurIPS, 35, 2022.
[56] Junyi Zeng, Chong Bao, Rui Chen, Zilong Dong, Guofeng Zhang, Hujun Bao, and Zhaopeng Cui. Mirror-nerf: Learning neural radiance fields for mirrors with whitted-style ray tracing. In Proceedings of the 31st ACM International Conference on Multimedia, 2023.
[57] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021.
[58] Xiuming Zhang, Pratul Srinivasan, Boyang Deng, Paul Debevec, William Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under unknown illumination. ACM Transactions on Graphics (ToG), 40(6), 2021.
[59] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. NICE-SLAM: Neural implicit scalable encoding for SLAM. In CVPR, 2022. 3
13

---

A Appendix supplemental material
In the supplementary, we first discuss more experimental details, including dataset and implementation details. Second, we discuss BakedSDF [51] more detail. We show that BakedSDF often not stable and we discuss how we finetune this method to improve performance on several scenes. Third, we summarize the detailed evaluation results on DTU [1], Shiny Blender [44], Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets. We also qualitatively compare with the state-of-the-arr methods [13, 51, 21, 14, 4]. Fourth, we include more ablation study. Fifth, we discuss the comparison with two custom baselines, RefV and CamV, in more detail. Finally, we discuss the potential societal impacts and limitations of our method.
B Experimental Settings
B.1 Dataset Details
We use the same data splits as prior works for fair comparison. On DTU [1], following [47, 13]. we use all the images for surface reconstruction. On Shiny Blender [44], Mip-NeRF 360 [3], and Ref-NeRF real [44] datasets, follow the official protocol and use the official training testing split for training and testing.
B.2 Implementation Details
Network Architecture. In addition to the iNGP [30] structure that we have introduced in the main paper, we further discuss the details of the MLP architectures. Specifically, the SDF MLP f has 2 layers with 256 hidden units and outputs the bottleneck feature vector with size 256. The two radiance MI.P feam, fref have 4 layers with 256 hidden units. Besides, the weight MLP fa has single layer with 256 hidden units.
Recall that following Mip-NeRF 360 [3], we use two rounds of proposal sampling and then a final NeRF sampling round. The proposal sampling is used bound the scene geometry and recursively generate more detailed sample intervals, while the final NeRF sampling is used to render the final of intervals into image. We set the number of samples for these sampling rounds 64, 32, 32 for the object-level DTU [1] and Shiny Blender [44], and 64, 64, 32 for unbounded Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets.
In Sec. 3 of the main paper, we mainly introduce model details of the final NeRF sampling round, where the color is rendered, for simplicity. Thus, introduce the details for proposal sampling rounds here. Specifically, the proposal sampling rounds only have a SDF MLP, i.e.no radiance MLP and weight MLP, since color is not rendered in these rounds. Moreover, the two proposal sampling rounds share a SDF MLP, which is different from the SDF MLP in the NeRF sampling round. Contrary to Zip-NeRF [4] that uses distinct iNGP for each sampling round, we use single iNGP that is shared by all sampling rounds. We find that this produces similar performance as using multiple iNGPs but explicitly simplifies the model.
Loss Function. In the loss function (Eq. 12 in the main paper, which for the final NeRF sampling set λ 10 for Shiny Blender [44], and
round), set λ1 10 and λ3 10 3 Moreover, X2 = 10-3 for DTU Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets. For proposal sampling rounds, we replace Leotor with Lpop. the proposal loss described in Mip-NeRF 360 [3].
Training Details. For training, we use the Adam [20] optimizer with 31 0.9, 82 = 0.999, 10-6, We warm up the learning rate in the first 2% iterations and then decay the it logarithmically from x 10 5 10-
C BakedSDF
In our experiments, we find that the optimization of BakedSDF [51] sensitive and often fails completely on Shiny Blender [44] and Ref-NeRF real dataset [44], shown in Fig, 8.
14

---

Image
Normal
Figure 8: Final image rendering and normal of original BakedSDF [51] on "garden spheres"
Scene [44]. The training is not stable leading to degraded results (see text).
BakedSDF only uses eikonal loss, Leik, for regularization, where the corresponding loss weight is set to 1 by default. We experimentally find that decreasing the eikonal loss weight can stabilize the training and thus carefully tune for each scene. For Shiny Blender [44], we set the eikonal loss weight as 10 2 for "toaster", "helmet" and "coffee", and 10 for "ball". Unfortunately, we could find the best eikonal loss weight for "car" and "teapot" scenes since the training does not produce reasonable geometry. For completeness, we report the rendering metrics on these two scenes with eikonal loss weight as 10-2 2 For Ref-NeRF real dataset [44], we set the eikonal loss weight as 10-2 for "sedan" and "toycar", and 10-5 for "garden spheres".
D Detailed Evaluation Results
Tab. 4, Tab. 5, Tab. and Tab. 7 contain the detailed metrics for each individual scene on DTU [1], Shiny Blender [44], Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets respectively.
We qualitatively compare with state-of-the-ars methods [21, 51, 4] on Shiny Blender, Mip-NeRF 360 and Ref-NeRF real datasets in Fig. 9, Our method is robust and demonstrates competitive performance on various scene types, ranging from object-level to unbounded scenes, with and without reflections.
Shiny Blender. In addition to the MAE error for evaluating the surface normal accuracy, also evaluate the mesh quality. Though Chamfer Distance includes both accuracy and completeness, Ref-NeuS [14] points out that the ground-truth meshes of Shiny Blender [44] double-layered, which results in many redundant points in the ground truth. Therefore, we follow Ref-NeuS [14] and only evaluate the accuracy (Acc) of reconstructed meshes. Besides, since ground-trie truth meshes of "ball" and "teapot" are unavailable, we only evaluate on the remaining four scenes following Ref-NeuS.
As shown Fig. 10, since the objects in Shiny Blender are highly reflective, our method automatically assigns high weights for the reflected view radiance field in most regions. Though our learned weight is not supervised, it can successfully detect the reflective surfaces. Besides, our reflected view radiance field represents the highly specular reflections of the surrounding environment, which is similar to the observation in Fig. 3 of the main paper.
We further qualitatively compare the surface normals with Geo-Ner Geo-NeuS [13], Neuralangelo [21] and Ref-NeuS [14] in Fig. 11. For "teapot" scene, Geo-NeuS fails to reconstruct the smooth surface the teapot. Neuralangelo reconstructs incorrect surface elements under the object. Ref-NeuS reconstructs overly smooth surfaces without fine details. For "ball" scene, Geo-NeuS produces slight artifacts on the surface, while Neuralangelo reconstructs lots of incorrect surface elements inside the ball. In contrast, our method reconstructs the surfaces more accurately on both scenes.
E Custom Baselines Comparison
In the main paper, we have compared our method with two custom baselines, Cam and RefV, both quantitatively (Tab. 3) and qualitatively (Fig. 5).
15

---

15.250B
19:710E
25.91dB
29.1 29.138B
Ground Truth
Zip-NeRF
Neurakagello
BakedSDF
Ours
Figure 9: Qualitative comparison with state-of-the-art methods [21, 51, 4] on Shiny Blender [44], Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets. PSNR values for each image patch are inset. Best viewed when zoomed in.
Table 4: Quantitative Chamfer distance (4) of individual scenes DTU dataset [1]. Red, orange and yellow indicate the first, second and third best performing methods for each scene. f: Factored- NeuS [12] provides valid results for 14 scenes, except for scan 69.
Methods
24
37
40
55
63
65
69
83
97
105
106
110
114
118
122
NeuS [47]
1.37
1.21
0.73
0.40
1.20
0.70
0.72
1.01
1.16
0.82
0.66
1.69
0.39
0.49
0.51
NeuralWarp [11]
0.49
0.71
0.38
0.38
0.79
0.81
0.82
1.20
1.06
0.68
0.66
0.74
0.41
0.63
0.51
Geo-NeuS [13]
0.38
0.53
0.34
0.36
0.80
0.45
0.41
1.03
0.84
0.55
0.46
0.47
0.29
0.36
0.35
Neuralangelo [21]
0.49
1.05
0.95
0.38
1.22
1.10
2.16
1.68
1.78
0.93
0.44
1.46
0.41
1.13 0.97
NERO [25]
1.10
1.13
1.26
0.46
1.32
1.93
0.87
1.61
1.47
1.10
0.70
1.14
0.39
0.52
0.57
Ref-NeuS [14]
1.17
0.43
4.41
3.19
1.45
3.46
1.20
0.74
1.94
0.49
0.66
Factored--Nen -NuStt [12]
0.82
1.05
0.85
0.40
0.99
0.59
1.44
1.15
0.81
0.58
0.89
0.36
0.44
0.46
Ours
0.54
0.84
0.66
0.51
0.76
0.64
0.71
0.70
0.86
0.57
0.69
0.65
0.45
0.56
0.50
In Fig. 12, we further visualize the qualitative results on scan 37 of DTU [1]. On the one hand, RefV reconstructs holes the objects with without reflections, which is similar to the artifacts that BakedSDF [51] shows in Fig. 4 of the main paper. On the other hand, though the surface is a little noisy, CamV reconstructs shiny objects relatively well. Note that in Fig. of the main paper, CamV V also reconstructs the shiny surfaces of "toycar" well, despite having some small artifacts. Since scan 37 of DTU and "toycar" of Ref-NeRF real dataset mainly contain reflective surfaces that less specular, we can infer that camera view radiance field can handle less specular reflections to some extent.
shown in Fig. 13, we sometimes observe that Ref' has optimization issues with separate diffuse and specular components. Specifically, the specular component may be empty throughout training,
16

---

Color reflected radiance field
Color of camera view radiance field
Learred weight
Composed solee
Normal
Figure 10: Visualization of the color of reflected view radiance field, color of camera view radiance field, learned weight W (red color represents large weight), composed color and surface normal on "helmet" and "car" scenes [44]. Best viewed when zoomed in.
Ground truth
Geo-NeuS
Neuralangelo
Ref-NeuS
Ours
Figure 11: Qualitative comparison of surface normal on "teapot" and "ball" scenes [44]. Best viewed when zoomed in.
while the diffuse component represents both view-dependent and non view-dependent appearance. Since diffuse component depends only on the 3D position x, the view-dependence is represented with incorrect geometry, shown in the results of "toycar" Fig. 13. We believe that this issue may be related to the high frequency signals that iNGP [30] can encode. In RefV, the diffuse component parameterized by the feature vector 7 from iNGP, which capable of representing very high frequency signal. Therefore, the diffuse component may take advantage of the high capacity of the iNGP representation model the view-dependent appearance with geometry only, leading to an incorrect reconstruction. This is especially true for small-scale scenes with relatively simple view-dependent appearance, e.g., "toycar".
F Ablation Study
Normal. Recall that we use normal the gradient of the signed distance field, for computing
reflected view direction w, and loss function Eq. 11. In contrast, Ref-NeRF [44] uses predicted normal n' instead. In this ablation study, we follow Ref-NeRF [44] and use the predicted normal n' for computing reflected view direction wi and loss function Eq. 11. As shown in Tab. 8, our method performs better than the baseline in rendering. We visualize the surface normal in Fig. 14. Our method successfully reconstructs the reflective surfaces, while the baseline reconstructs artifacts on them.
17

---

Table 5: Quantitative results of individual scenes on Shiny Blender [44]. BakedSDF [S1] fails on "car" and "teapot" scenes without producing reasonable geometry. Thus we do not report its MAE metric on these scenes. *: We follow Ref-NeuS [14] and evaluate accuracy of mesh on four scenes (car, helmet, toaster, coffee). Red, orange, and yellow indicate the first, second, and third best performing algorithms for each scene.
Methods
car
ball
helmet teapot
toaster
coffee
PSNR 1
Mip-NeRF [3]
26.48
27.69
27.41
17.84
22.47
31.07
Zip-NeRF [4]
27.46
21.70
26.87
45.45
23.23
30.76
Geo-NeuS [13]
26.92
35.50
25.64
32.62
23.49
28.49
Neuralangelo [21]
27.58
27.87
28.68
44.38
23.42
32.16
Ref-NeRF [44]
30.82
47.46
29.68
47.90
25.70
34.21
ENVIDR [22]
29.88
41.03
36.98
46.14
26.63
34.45
NERO [25]
25.53
30.26
29.20
38.70
26.46
28.89
Ref-NeuS [14]
24.30
34.57
28.07
28.73
22.68
26.09
BakedSDF [S1]
10.03
31.35
35.50
17.84
23.84
35.06
Ours
29.86
44.10
38.84
48.76
26.18
33.17
SSIM f
Mip-NeRF 360 [3]
0.922
0.937
0.939
0.967
0.900
0.966
Zip-NeRF [4]
0.932
0.906
0.946
0.996
0.910
0.966
Geo-NeuS [13]
0.922
0.982
0.946
0.985
0.884
0,951
Neuralangelo [21]
0.935
0.931
0.953
0.995
0.910
0.970
Ref-NeRF [44]
0.955
0.995
0.958
0.998
0.922
0.974
ENVIDR [22]
0.972 0.997
0.993
0.999
0.955
0.984
NERO [25]
0.949 0,974
0.971
0.995
0.929
0.956
Ref-NeuS [14]
0.919 0.989
0.971
0.981
0,903
0.941
BakedSDF [51]
0,807 0.979
0.990
0.967
0.939
0.978
Ours
0.954
0.993
0.990
0.998
0.945
0.973
LPIPS
Mip-NeRF 360 [3]
0.062
0.189
0.127
0.094
0.144
0.118
Zip-NeRF [4]
0.060
0.231
0.115
0.010
0.127
0.128
Geo-NeuS [13]
0.080
0.082
0.082
0.024
0.134
0.110
Neuralangelo [21]
0.066 0.186
0.085
0.017
0.123
0.091
Ref-NeRF [44]
0.041 0.059
0.075
0.004
0.095
0.078
ENVIDR [22]
0.031 0.020
0.022
0.003
0.097
0.044
NERO [25]
0.074 0.094
0.050
0.012
0.089
0.110
Ref-NeuS [14]
0.076 0.058
0.046
0.025
0.114
0.119
BakedSDF [S1]
0.197 0.094
0.019
0.079
0.079
0.072
Ours
0.047
0.039
0.021
0.004
0.072
0.078
MAE
Geo-NeuS [13]
12.15
4.12
19.77
16.23
9.79
Neuralangelo [21]
12.34
35.63
9.23
4.98
14.14
8.61
Ref-NeRF [44]
14.93
1.55
29.48
9.23
42.87
12.24
BakedSDF [51]
0.44
1.74
12.24
3.31
ENVIDR [22]
7.10
0.74
1.66
2.47
6.45
9.23
Ref-NeuS [14]
7.68
0.50
1.94
10.25
5.95
5.73
Ours
6.88
0.45
1.72
2.80
8.71
8.00
Acc*
Geo-NeuS [13]
0.72
0.74
4.14
0.90
Neuralangelo [21]
1.89
1.71
2.99
0.63
Ref-NeuS [14]
0.50
0.53
0.54
1.83
Ours
0.58
0.46
2.02
1.17
G
Potential Societal Impacts
Positive impact. Our method can accurately reconstruct complex scenes with reflections, which is challenging task for existing methods. The accurate reconstruction can be used for many downstream applications, e.g., robotics and creating 3D experiences for augmented/virtual reality.
Negative impact. Similar to most existing volumetric implicit methods [28, 3, 4, 21, 44], our method needs to be individually trained for each scene. Though the training is fast (3.50h on Mip-
18

---

Table 6: Quantitative results of individual scenes Mip-NeRF 360 dataset [3]. Red, orange, and yellow indicate the first, second, and third best performing algorithms for each scene.
PSNR
Outdoor
Indoor
bicycle
flowers
garden
stump
treehill
room
counter
kitchen
bonsai
Mip-NeRF 360 [3]
24.40
21.64
26.94
26.36
22.81
31.40
29.44
32.02
33.11
Zip-NeRF [4]
25.80
22.40
28.20
27.55
23.89
32,65
29.38
32.50
34,46
Neuralangelo [21]
23.78
21.03
23.76
21.38
22.83
28.76
25.39
30.40
28.39
BakedSDF [S1]
23.05
20.55
26.44
24.39
22.55
30.68
27.99
30.91
31,26
Ours
24.67
21.83
27.46
26.39
23.51
31.25
29.26
31.73
32.86
SSIM
Outdoor
Indoor
bicycle
flowers
garden
stump
treehill
counter
kitchen
bonsai
Mip-NeRF 360 [3|
0.699
0.583
0.816
0.746
0.632
0.913
0.895
0.920
0.939
Zip-NeRF [4]
0.769
0.642
0.860
0.800
0.681
0.925
0,902
0.928
0.949
Neuralangelo [21]
0.605
0.508
0.635
0.502
0.623
0.869
0.796
0.891
0.857
BakedSDF [51]
0.588
0.504
0.793
0.662
0,543
0.892
0.845
0,903
0.911
Ours
0.737
0.606
0.844
0.759
0.670
0.914
0.888
0.919
0.939
LPIPS
Outdoor
Indoor
bicycle
flowers
garden
stump
treehill
counter
kitchen
bonsai
Mip-NeRI- 360
0.289
0.345
0.164
0.254
0.338
0.211
0.203
0.126
0.177
Zip-NeRF [4]
0.208
0.273
0.118
0.193
0.242
0.196
0.185
0.116
0.173
Neuralangelo [21]
0.390
0.419
0.326
0.440
0.360
0.269
0.310
0.172
0.312
BakedSDF [51]
0.400
0.437
0.204
0.343
0.270
0.293
0.244
Ours
0.243
0.320
0.136
0.242
0.265
0.206
0.206
0.124
0.184
Table 7: Quantitative results of individual scenes on the Ref-NeRF real dataset [44]. Red, orange,
and yellow indicate the first, second, and third best methods for each metric.
Sedan
Toycar
Garden Spheres
Methods
PSNR
SSIM f
LPIPS
PSNR f
SSIM
LPIPS |
PSNR
SSIM
LPIPS
Mip-NeRF 360 [3]
25.56
0.708
24.32
0.654
0.256
22.94
0.587
0.268
Zip-NeRF
25.85
0.733
0.260
23.41
0.626
0.243
21.77
0.545
0.238
Neuralangelo [21]
24.82
0.656
0.384
24.28
0.638
0.293
22.03
0.529
0.313
Ref-NeRF [44]
25.20
0.639
0.406
24.40
0.627
0.292
22.57
0.502
0.366
BakedSDF [51]
25.70
0.700
0.332
24,51
0.655
0.280
23.08
0.553
0.363
Ours
24.68
0.700
0.309
24.15
0.639
0.245
22.27
0.567
0.243
Reference Image
CamVV (CD: 0.88)
RefV (CD: 1,12)
Ours (CD: 0.84)
Comparison with two baselines, CamV and RefV, on scan 37 of DTU [1] (CD Chamfer distance error). CamV reconstructs more noisy surface the red handle with reflections (highlighted with red box and zoomed in), while RefV generates holes on the shiny objects and even the brick without any reflections. Best viewed when zoomed in.
Figure 12:
NeRF 360 [3] and Ref-NeRF real [44] datasets), we use NVIDIA Tesla V100-SXM2-1CH 16GB GPUs simultancously, which consumes much energy.
19

---

Diffuse
Specular
Normal
Figure 13: Visualization of diffuse color component, specular color component and surface normal for RefV "sedan" and "toycar" scene [44]. RefV successfully decomposes color components for "sedan", while fails on "toycar" with blank specular component. Best viewed when zoomed in.
Table 8: Ablation study of normals on the Ref-NeRF real dataset [44].
Methods
PSNR
SSIM f
LPIPS |
w. predicted normal
23.56
0.630
0.268
Ours
23.70
0.636
0.265
w./ predicted normal
Ours
w./ predicted normal
Ours
Figure 14: Ablation study of normals on "sedan" and "garden spheres" scene [44]. Best viewed when zoomed in.
H
Limitations
In this work, we mainly focus on robust surface reconstruction of scenes with reflections. Our method cannot be used for editing tasks such relighting. Besides, similar most NeRF methods [3, 21, 4], our method requires the estimated poses from SfM, egg., COLMAP [38], as input for real-world scenes. However, SIM needs to perform feature matching, which based on multi-view photometric consistency, for pose estimation. As we discuss in the main paper, multi-view photometric consistency guaranteed with reflective surfaces and thus SfM may produce inaccurate poses, which may lead to inaccurate reconstruction and rendering. In addition, our method cannot accurately reconstruct the reflective surfaces with sparse input views. is challenging determine reflections in this case because of the ambiguity. Note that both Shiny Blender [44] and Ref-NeRF real dataset [44] carefully provide dense 360 degree views around the reflective objects that it is easier to detect the view-dependent reflective appearance.
I Licenses for Existing Assets
code:
multinerf [29]: Apache License 2.0.
datasets:
20

---

DTU [1]: we do not find the license.
Shiny Blender [44]: coffee (CC-0 license), toaster (CC-BY license), car (CC-0 license).
helmet (CC-0 license).
Mip-NeRF 360 dataset [3]: CC-BY license.
Ref-NeRF real dataset [44]: it is captured by the authors of Ref-NeRF [44], we do not find the license.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to L9-19 (abstract) and L46-57 (introduction).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Sec. of supplementary.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss the details of experiments in Sec. 4. and Sec. B, including network architecture, training details and dataset details.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[No]
Justification: The datasets used in this paper are all publicly available. We are unable to immediately release the code, but plan it for the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.c cc/
http://uuiees//ooeeuuiisssssssse ooloocyooolyyooooooolloooommllcl for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips.cc..occc wwtpppccccgeeeeeeecceccccceeeeee muullcc/uuieee//// Seuudee/Cooeeuueeeeeeeeeeeeeeeee wwt.SSummisiioPooii. asiionylicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify the experimental details in Sec. 4.1 and Sec. B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We extensively evaluate our method on 4 datasets, which contain 33 scenes in total, and our method needs to trained for each scene individually with GPUs. Due to the limited computing resources and time, reporting error bars would be too computationally expensive for us.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As discussed in Sec. 4.1, our models are all trained on 8 NVIDIA Tesla V100-SXM2-16GGB GPUs. We train 25k steps DTU/ Shiny Blender and 100k steps Mip-NeRF 360 Ref-NeRF real datasets, which takes 0.75h and 3.50h respectively.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httttp:///neuippe
inns?
Answer: [Yes]
Justification: We carefully checked the Code of Ethics and can confirm that our research conforms with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impacts in Sec, of supplementary.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original owners of codes and datasets that we use in Sec. 4.1. We mention their licenses in Sec. I of supplementary.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "EhwZwOtiXS", "reviewer": "Reviewer_aZTE", "summary": "This paper proposes a new strategy for modelling view-dependent effects in Neural Radiance Field-based scene models. Existing approaches have used networks conditioned on camera view directions, as well as on reflected view directions using surface normals, but this work proposes and validates the idea that both model types should be used simultaneously. This is achieved by employing a learned weight model that mixes the two color predictions, thereby allowing regions where the color is not well explained by the reflection-based model to be handled by the view direction-based model. This is shown through ablations to be a better strategy than either strategy individually, and is also compared to a number of baseline methods which employ reflection-based modelling of view-dependent effects.", "questions": "Given the increasing prevalence of methods like 3DGS which do not use neural networks for modelling view-dependence but rather simpler approaches like spherical harmonics, do you see a way of applying the ideas of this paper to such methods?", "limitations": "The only limitation I see is the reduced interpretability/editability of the model compared to other approaches, but this is addressed by the authors. I see no issues with societal impact beyond generic concerns that are relevant to all scene modelling methods.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper presents a simple, yet effective idea that would be easy to incorporate into other works which use reflection-based models. I think a lot of the contribution is in showing such a simple but surprising result which is not obvious to try.\n\nThe quality and clarity of the writeup is high, and the evaluation seems quite thorough in comparing to other relevant works and on relevant datasets. I think the value of the proposed strategy is shown quite clearly by the experiments.", "weaknesses": "The only notable weakness is that the proposed strategy is a fairly minor deviation from previously proposed methods. However, I think this is largely mitigated by the non-obvious nature of the change and the in-depth experimental evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Applying the idea in 3D Gaussian Splatting**\n\nThank you for your interesting suggestion. Recently, 3DGS techniques are advancing rapidly and there are some concurrent works trying to reconstruct the surface [1*] or render reflections [2*]. GaussianShader [2*] introduces shading attributes for each 3D Gaussian, such as diffuse, tint, roughness and normal, to model reflections. Its rendering is similar to Ref-NeRF. Therefore, we think it is possible to apply our idea to 3DGS. We can combine the original 3DGS and GaussianShader representation with an optimizable “weight” attribute, similar to the weight field in our method, that is assigned to each 3D Gaussian point. We consider this as a potential future work. \n\n>**Reduced editability**\n\nWe discuss this limitation in the paper (Sec. G). Though our method does not support editability, with the high-quality mesh extracted from our method, it is possible to perform editing tasks such as relighting. For example, recent single image relighting methods [3*] use diffusion models for relighting and adopt “radiance cues” -- renderings of the object’s geometry with various roughness levels under the target environment illumination -- as conditioning. The mesh extracted from our method can be used to render such radiance cues.\n\n[1*] Huang et al. 2D Gaussian Splatting for Geometrically Accurate Radiance Fields.  SIGGRAPH, 2024\n\n[2*] Jiang et al. GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces. CVPR 2024\n\n[3*] Zeng et al. DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation. SIGGRAPH, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m3snpHxHsv", "reviewer": "Reviewer_dks9", "summary": "The paper proposes a method to reconstruct scenes containing both reflective surfaces and non-reflective surfaces with high fidelity. Specifically, it trains a camera view radiance field and a reflected view radiance field separately, combining them by a learnable weight. The method is evaluated on four datasets, which covers different situations including objects and 360 scenes, shiny objects, real-world captures. And the method outperforms or is on par with SOTA on all these datasets, demonstrating its effectiveness.", "questions": "Please see weaknesses.", "limitations": "The authors listed a quite complete list of limitations. I think some are common issues for many works like requiring camera pose and dense views. With that being said, in the future the authors can consider extending this work to sparse views and how to eliminate the requirement for several fields and volume rendering, making the algorithm more efficient.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments. The method was compared to several SOTA methods and outperforms or is on par with them, showing its ability to reconstruct complicated scenes with reflective surfaces on four datasets. \n2. The paper is well written. And the figures clearly serve for their purposes.\n3. The author did a great job in analyzing the limitations and social impact, which is highly encouraged.", "weaknesses": "1. The method involves two radiance fields and one weight field, and it has to perform volume rendering for colors from both radiance fields and the weights, which is very time-consuming even with implementation of iNGP.\n2. I wonder if you have a possible explanation for why some reflected view radiance field methods is not robust to real-world scene.\n3. I noticed the comparison with Factored-NeuS is only performed on DTU. Since Factored-NeuS is designed to also handle glossy objects, I wonder if how it look like when compared to this paper on Shiny Blender and Ref-NeRF real datasets.\n4. As the authors mention in the section of limitations, this method requires posed images and SfM can fail on reflective surfaces. While this is a common issue, I still wonder how robust this method is to inaccurate camera pose. Even a failure case is fine.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1: Time consuming**\n\nOur method has three fields and is based on volume rendering. Thus our rendering efficiency is not high. Recently, 3D Gaussian Splatting has become popular in view synthesis and there are some concurrent works focusing on reconstructing the surface [1*] or rendering reflections [2*]. Therefore, we think extending our pipeline with 3D Gaussian Splatting is a potential future direction to improve efficiency. Thank you for your suggestion to explore eliminating the multiple fields, this is an interesting direction for future works. \n\n>**W2: Explanation for why some reflected view radiance field methods are not robust to real-world scene**\n\nDisambiguating the influence of geometry, color and reflection is an ill-posed problem in image-based 3D reconstruction. Methods such as Ref-NeRF assume that there are no inter-reflections or non-distant illumination, which is not often the case in real world data. Please see the global rebuttal for more details. \n\nFor the reflected view radiance field, the view-dependent appearance mainly depends on $\\mathbf{\\omega}_r$, the reflected view direction around the normal $\\mathbf{n}$. However, the geometry and surface normal $\\mathbf{n}$ are unknown in the beginning and need to be optimized during training. Therefore, the directional input $\\mathbf{\\omega}_r$ of view-dependent appearance keeps changing during training. We conjecture that this makes the optimization process more complex and ill-posed.\n\n>**W3: Results of Factored-NeuS**\n\nThe metrics of Factored-NeuS on DTU are the official results from their paper (the public ICLR 2024 submission). We just found that Factored-NeuS was open-sourced recently. Thus we evaluate Factored-NeuS on ShinyBlender dataset and summarize the results as follows. Our method performs better than Factored-NeuS in both rendering and surface accuracy. As shown in Fig.2 of the rebuttal PDF, Factored-NeuS reconstructs explicit artifacts on the “helmet” scene, while our method reconstructs the surface more accurately.\n\nMethods  |  PSNR $\\uparrow$  |  SSIM$\\uparrow$  |  LPIPS$\\downarrow$  | MAE $\\downarrow$  |  Acc $\\downarrow$\n--|--|--|--|--|--\nFactored-NeuS  | 30.89 |  0.954 |  0.076 |  5.31 | 1.90\nOurs | **36.82**  |  **0.976**  |  **0.043** |  **4.76**  | **1.06**\n\n\n>**W4: Results with inaccurate camera pose**\n\nThank you for the suggestion. In this rebuttal, we evaluate some scenes of the Ref-NeRF real dataset on camera poses with uniform noise.  Specifically, we first add random translation noise (with a range as [-0.01,0.01]) along each axis for each camera pose. Note that before adding noise, the scene is already normalized to a unit cube following Mip-NeRF360. Second, we add random rotation noise to the camera poses. For each camera pose, we randomly sample a 3d unit vector and rotate the camera pose around it  (with a range as [-1,1] degrees). As shown in Fig.3 of the rebuttal PDF, both the rendering and surface become worse. Integrating existing methods for pose optimization during training could be an interesting future direction. \n\n[1*] Huang et al. 2D Gaussian Splatting for Geometrically Accurate Radiance Fields.  SIGGRAPH, 2024\n\n[2*] Jiang et al. GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces. CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k5Fkpv7LK0", "reviewer": "Reviewer_2MRi", "summary": "The paper tackles the problem of 3D reconstruction in the presence of highly reflective objects. To address the problem, they propose to learn an SDF-based neural representation. Different from prior work, they use two radiance branches in their representation, one conditioned on the camera viewing direction rotated about the normal and one conditioned on the regular viewing direction. The outputs of the two networks are combined using a learned weight to arrive at the final rendered radiance. \n\nThe authors argue that while the reflected direction is better at capturing the details in reflective surfaces, the viewing direction is more robust to real data. \n\nThrough extensive experiments, they show their method is competitive with prior art in both the tasks of NVS and mesh recovery (as measured by Chamfer Distance).", "questions": "1) Is there a stopgrad on any components of the normal smoothness loss (predicted vs computed normal in Eq 10)? \n\n2) I see there is an ablation on this, but why do the authors think the weight to mix the outputs of the two radiance branches is necessary, why doesn’t this factor just get automatically learned by the two networks? \n\n3) The authors seem to have ran ENVIDR on the Ref-NeRF real dataset (as evidenced by Fig 1?), but do not include the results in Table 2? Why? \n\n4) Have the authors tried passing in $n^\\prime$ instead of $n$ to the radiance network or also calculating $w_r$ about it? If I understand correctly $n^\\prime$ is supposed to be “smoother”, perhaps this would make results more robust for just RefV?", "limitations": "Limitations have been adequately discussed in the supplement.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "I like the problem being attempted in the paper, I also like the insight that the viewing direction conditioning is more robust, while the reflected direction produces better geometry for reflective surfaces in controlled environments. The results are also impressive. \n\nQuality: The ablation studies are well/extensively done. \n\nClarity: The paper is very well written and easy to follow. The figures are all well understandable and the work should be easily reproducible from just the description.", "weaknesses": "Significance: I think the method is generally quite a straightforward extension of existing work and the paper's contributions are fairly limited. As I see it both the radiance branches have previously existed and the Ref-NeRF parametrization could be considered more “physically based”, the authors forgo a more mathematically solid setup in favour of an increase in performance. \n\nI think the paper would benefit a bit from providing more insight into the reasons for some of the phenomena i.e. why do the weights separate like that for more specular/diffuse regions even though both are conditioned on some sort of parametrization of the viewing direction etc. \n\nI am sympathetic however to the authors that some of these phenomena just occur and might be hard to reason about, but some of the simulated experiments could provide more insight into how the optimization behaves.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Why learned weights separate specular/diffuse regions without explicit supervision**\n\nIn Ref-NeRF, the ablation study shows that when rendering reflective surfaces, using the reflected view direction as the MLP’s input is explicitly better than using the camera view direction of NeRF. In our method, the main difference between two radiance fields is the view directional input. The reflected view radiance field uses reflected view direction as Ref-NeRF, while the camera view radiance field uses camera view direction. Similar to the findings in Ref-NeRF, our reflected view radiance field can model the appearance of reflective regions better than the camera view radiance field. Thus the weight $\\mathbf{W}$, which corresponds to the rendered color $\\mathbf{C}\\_{ref}$ of reflected view radiance field (Eq.8), for these reflective regions increases during optimization to reduce the composed color loss $\\mathcal{L}\\_{\\text{color}}$. Conversely, when the color computed from the camera view radiance field is better, the weight $\\mathbf{W}$ decreases.\n\n>**Q1: Stopgrad on any components of the normal smoothness loss**\n\nAs in Ref-NeRF, we do not use stopgrad on any components.\n\n>**Q2: Weight field to mix the outputs of the two radiance branches**\n\nThank you for the suggestion. For this rebuttal, we evaluate our method while removing the weight field. The final color is composed as $\\mathbf{C} = \\mathbf{C}\\_{ref} + \\mathbf{C}\\_{cam}$. We evaluate on the Ref-NeRF real dataset and summarize the results as follows. We observe that our method performs better than this ablation. \n\nAs shown in Fig.3, the learned weight field has the advantage of leading to better interpretability as it can be used to detect highly reflective regions from different viewpoints with volume rendering. This is a challenging task because reflection is an attribute of surfaces that objects or parts of objects have. Therefore, it is difficult to detect such regions using existing semantic models. For example, we tried state-of-the-art open vocabulary semantic segmentation methods with “reflective objects” or “reflective surfaces” as prompt, and they were unable to successfully segment the reflective objects in the image. \n\nMethods | PSNR $\\uparrow$ | SSIM$\\uparrow$ | LPIPS$\\downarrow$  \n--|--|--|--\nw./o. weight field | 23.39 | 0.625 | 0.279\nours | **23.70** | **0.636** | **0.265**\n\n\n>**Q3: Results of ENVIDR**\n\nThe metrics and rendered images of ENVIDR are the official results provided by the authors of ENVIDR. On the Ref-NeRF real dataset, ENVIDR is evaluated on “gardenspheres” only. In the limitation section of the ENVIDR paper, it is said that ENVIDR is unable to handle unbounded scenes, while the “sedan” scene of Ref-NeRF real dataset is unbounded.\n\n>**Q4: Use $n’$ instead of $n$**\n\nAs shown in [46,49], the normals, estimated as the gradients, of the SDF field are well-defined and much smoother than those computed from the volume density field of NeRF. Compared with the noisy normal of the volume density field of Ref-NeRF, we use a SDF field as our geometry representation and thus the normal is already smoother and more accurate. So we only use $n’$ for regularization in Eq.10. Note that BakedSDF [50] and Ref-NeuS [14] also use the SDF normal $n$ for their reflected view radiance fields.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H5UvnyGcgg", "reviewer": "Reviewer_5Tqb", "summary": "This paper proposes UniSDF, an improved NeuS architecture capable of reconstructing photorealistic reflective surfaces and robustly working in real-world scenes. The method models reflective color and base color as separate MLPs, using learned weights to blend them and obtain the final colors. Qualitative and quantitative results on several datasets demonstrate that the method surpasses other baselines and results in fewer geometric artifacts.", "questions": "1. Title: It's unclear from the abstract and introduction where the \"unifying\" concept comes from. \"Unifying scene representation\" suggests combining several scene representations, but theoretically, it is still a NeuS-based representation.\n\t2. Missing Citations: \n\t\ta. MS-NeRF: Multi-space Neural Radiance Field: This paper also presents a similar idea of linearly combining different color fields. \n\t\tb. Several Concurrent Works Regarding Reflective NeRF: \n\t\t\ti. NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections\n\t\t\tii. SpecNeRF: Gaussian Directional Encoding for Specular Reflections \n\t\tc. Reflection Handling in Gaussian Splatting: \n\t\t\ti. MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections \n\t\t\tii. Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting\n\t3. Illustration of Existing Works: In line 131, I wouldn't call Ref-NeRF[43] as using a single radiance field, as it uses two MLPs to model diffuse color and specular color separately (though the specular MLP is conditioned on the feature of the diffuse MLP)\n\t4. Hyperparameters in Final Loss: The final loss contains several hyperparameters for controlling the strength of each term. In my experience, these weights play an important role. It would be beneficial to conduct small ablations or illustrate how these weights were decided.\n\t5. Input Redundancy: The f_ref and f_cam functions take in both the spatial coordinate x and the bottleneck feature b from the SDF MLP. Given that b is already a spatial feature vector, what's the point of adding x as one of the inputs? This seems a bit redundant.\n\t6. Normals and Orientation Loss: There are two normals being produced: n is the normal from the SDF field, and n′ is the MLP-predicted normal. For Eq. 11, Ref-NeRF uses n′ for orientation loss. It's unclear if this is a typo or if n is actually used for the orientation loss.\n\t7. Volumetric Rendering SDF: In line 124, it would be better to cite which volumetric rendering SDF system is used. I assume it's NeuS.", "limitations": "There are no potential negative society impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The method proposes a new network architecture to better represent the reflective radiance field, although most of the concepts are derived from previous works. There are some original contributions, such as the blending between two fields inspired by NeRFRen, which is extended to non-planar surfaces, and the coarse-to-fine training from Neuralangelo, used here to reduce the ambiguity of reflections. Therefore, while the method is novel, its significance is not substantial.\n\nAnother strength is the robust and high-quality reconstruction demonstrated in the paper and supplementary material, which surpasses other baselines. This is valuable since this problem is crucial in real-world 3D reconstruction scenarios.", "weaknesses": "The biggest weakness is the lack of physical interpretation. In Eq. 8, the final color is a linear combination of the camera view and reflected view radiance fields, resulting in unclear physical meaning for each color component. Although the method focuses on novel view synthesis and geometry reconstruction regardless of the underlying physical components, this lack of physical meaning leads to ambiguity about the assumptions the authors have for the reconstructed scene.\n\nFor example, the model doesn't account for how reflections change with different surface roughness, raising the question of whether it will perform well on surfaces with spatially varying roughness. Additionally, the model does not explicitly trace the reflective ray but instead models the color of the reflective ray through an MLP. This raises concerns about its ability to handle near-field lighting conditions effectively.\n\nFurthermore, I am curious about the differences between Ref-NeRF's equation c = diffuse + specular × tint and Eq. 8 c=(1−w) c_cam+w c_ref. Will these two modeling approaches yield similar performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Explicitly tracing the reflective ray**\n\nIn this work, similar to many recent methods that are tailored to handle reflections [12,14,22,50], we follow Ref-NeRF and parameterize part of view-dependent appearance as a function of the reflected view direction $\\mathbf{\\omega}\\_r$. Explicitly tracing the reflective ray is an interesting direction for us and it is shown effective in the concurrent work NeRF-Casting.\n\n>**Ref-NeRF's model v.s. our model**\n\nOur custom baseline “RefV” uses the same physics based model as Ref-NeRF (L215), where we separate the color into diffuse and specular components. As shown in Table.3, our method outperforms RefV in both surface reconstruction and rendering on 3 datasets. Fig.5 shows that RefV reconstructs artifacts on the surfaces, while our method performs better. We also observe that when using the hash grid backbone, RefV may have optimization issues with separate diffuse and specular components as visualized in Fig.13. We discuss this in L533-543.\n\n>**Q1: Unifying neural representations**\n\nWhat we mean is that we combine the two neural representations for the radiance field, i.e., camera view and reflected view radiance field. We will rephrase the term “scene representation” in the abstract.\n\n>**Q2: Missing citations**\n\nThank you for pointing this out. We will add them in our paper. MS-NeRF focuses on view synthesis and combines $K \\geq 2$ camera view radiance fields, where each camera view radiance field has a volume density field. It is unclear how to extract the surface since there are $K$ different volume density fields and iso-surfaces can be extracted from each of them. In contrast, we use a single SDF field to represent geometry and directly extract iso-surface from it. \n\n>**Q3: Ref-NeRF as a single radiance field**\n\nThank you for pointing this out. We will rephrase.\n\n>**Q4: Ablation study on loss weights**\n\nWe perform an ablation study on the eikonal loss weight $\\lambda\\_1$ on Ref-NeRF real dataset and summarize the results as follows. Setting $\\lambda\\_1=10^{-4}$ produces the best performance. If $\\lambda\\_1$ is too large (e.g., $\\lambda\\_1=10^{-2}$), we observe that the training becomes unstable and may fail. This is similar to our findings on BakedSDF (Sec. C). \n\nAs shown in [46,49], the normals, estimated as the gradients, of the SDF field are well-defined and more accurate than those computed from the volume density field. Recent BakedSDF [50] and Ref-NeuS [14] combined SDF field with reflected view radiance field to reconstruct reflective surfaces. They found that using eikonal loss $\\mathcal{L}\\_{\\text{eik}}$ for regularization is enough to get accurate normals and thus it is not necessary to use the normal smoothness loss $\\mathcal{L}\\_p$ or the orientational loss $\\mathcal{L}\\_o$ like Ref-NeRF. In our method, we still use $\\mathcal{L}\\_p$ and $\\mathcal{L}\\_o$ for additional regularization. However, since we use eikonal loss already, we simply set  $\\lambda\\_2, \\lambda\\_3$ to relatively small values and do not tune them a lot. We only slightly increase the weight $\\lambda\\_2$ for normal smoothness loss $\\mathcal{L}\\_p$ on real data because of the potential noise (L481), similar to Ref-NeRF. \n\nMethods | PSNR $\\uparrow$ | SSIM$\\uparrow$ | LPIPS$\\downarrow$ \n--|--|--|--\n$\\lambda\\_1=10^{-2}$ | 20.99 | 0.509 | 0.413\n$\\lambda\\_1=10^{-3}$ | 23.24 | 0.611 | 0.295\n$\\lambda\\_1=10^{-4}$ (Ours) | **23.70** | **0.636** | **0.265**\n$\\lambda\\_1=10^{-5}$ | 23.62 | 0.633 | 0.268\n\n\n>**Q5: Input redundancy**\n\nIt is correct that $\\mathbf{b}$ is dependent on $\\mathbf{x}$. The reason that we use both $\\mathbf{b}$ and $\\mathbf{x}$ is to align with the state-of-the-art implicit reconstruction pipelines such as VolSDF [49] and NeuS [46], which use both $\\mathbf{b}$ and $\\mathbf{x}$ to compute radiance.\n\n>**Q6: Normals and orientation loss**\n\nSince the normal estimated from the volume density gradient are often extremely noisy, Ref-NeRF uses the predicted normal $n’$ throughout its pipeline, e.g., computing reflected view direction $\\mathbf{\\omega}\\_r$, including for orientation loss. As shown in [46,49], the normals, estimated as the gradients, of the SDF field are well-defined and much smoother than those computed from the volume density field. Compared with the noisy normal of the volume density field of Ref-NeRF, we use a SDF field as our geometry representation and thus the normal is already smoother and more accurate. So we use $n$ in our pipeline (also for orientation loss) and only use the predicted normal $n’$ for regularization in Eq.10. Note that BakedSDF [50] and Ref-NeuS [14] also use the SDF normal $n$ for their reflected view radiance fields.\n\n>**Q7: Volumetric rendering the SDF**\n\nSorry, we missed the reference here. We use VolSDF [49] as our representation (L199).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback. In this global rebuttal, we address the common questions raised by the reviewers as follows:

> **Physical interpretation**

In this work, we mainly focus on robust surface reconstruction of real-world complex scenes with both reflective and non reflective surfaces. Current NeRF methods designed to handle reflections are mainly evaluated on synthetic datasets or scenes captured in well-controlled environments. It is challenging but important to extend them to more general scenes.

**Intuition on the need of a dual representation.**
To render reflective surfaces, Ref-NeRF [43] separates the color into physical components such as diffuse and specular components. During our extensive experiments, we find that this representation is not robust in complex real-world scenarios. As mentioned as a limitation in the Ref-NeRF paper, the representation does not “explicitly model inter-reflections or non-distant illumination”. Therefore, complex reflections that we can find in real world scenes will not be handled by similar methods.  As discussed in L157-169, we experimentally observe advantages and limitations using existing camera view and reflected view radiance fields separately.  Thus we decide to exploit the advantages of two radiance fields by combining them with a learnable weight. 

**Ambiguous interpretation of reflections as geometry.** 
Our custom baseline “RefV” is a physically-based model, where we follow Ref-NeRF and separate the color into diffuse and specular components. We observe that accurately separating diffuse and specular components using RefV only is a difficult task, as shown in Fig.13. In particular, view dependent effects can be represented as either geometry artifacts (holes and bumps) or with specularity. We discuss this problem in L533-543.

**Ambiguous interpretation of reflections as diffuse colors.**
Moreover, diffuse reflected components can be represented as both diffuse color and specular reflections due to the low frequency.  We observe that separating reflections and color is more challenging when the distant reflected world is less detailed. For example, the gardensphere scene is easier to reconstruct with RefV only as the background (trees, buildings) is clearly reflected. On the other hand, the sedan scene is more challenging as the reflections of the background on the car are more blurry. This observation coincides with the limitations listed in the Ref-NeRF paper.  

**Ambiguous interpretation of diffuse colors as reflections.**
We find that even if the color decomposition like Ref-NeRF does not fail, the color components may not be accurate enough. For example, BakedSDF [50] follows Ref-NeRF and uses diffuse and specular color components. However, we find that BakedSDF does not accurately separate these two components. As shown in Fig.1 of the rebuttal PDF, the specular component wrongly represents the diffuse colors of the objects that have almost no reflections, such as leaves and grass.  

Due to the many limitations of physics based models for real world scenes, we propose to extend such methods by unifying a physics based radiance field with an additional camera view radiance field. Therefore leading to more robust reconstructions, as highlighted by reviewers.

Meta-review: The paper describes a specific self-contained improvement to NeRF: instead of using the camera direction only as in classic NeRF and instesd of using the reflected direction in combination with some "proto-shading" information as in RefNeRF, the proposal is to simply use both and learn how to blend between them. The idea is not extremely tight to NeRF, it could also be applied to other view-dependent appearance.

All reviewers understood and liked the idea, result and exposition.

The downside is, that the decomposition does not in the end provide any new insight or result in a semantic or physical separation. 

The rebuttal admitted this limitation, providing some discussion and was generally well received to also clarify some details.

The AC read and and understood the paper in detail, certifies the reviews and the dsicussion were conducted orderly and hence can only recommend acceptance.
Decision: Accept (poster)