Title: UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D reconstruction

novel view synthesis

reflection
Abstract: Neural 3D scene representations have shown great potential for 3D reconstruction from 2D images. However, reconstructing real-world captures of complex scenes still remains a challenge. Existing generic 3D reconstruction methods often struggle to represent fine geometric details and do not adequately model reflective surfaces of large-scale scenes. Techniques that explicitly focus on reflective surfaces can model complex and detailed reflections by exploiting better reflection parameterizations. However, we observe that these methods are often not robust in real scenarios where non-reflective as well as reflective components are present. In this work, we propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. We investigate both camera view as well as reflected view-based color parameterization techniques and find that explicitly blending these representations in 3D space enables reconstruction of surfaces that are more geometrically accurate, especially for reflective surfaces. We further combine this representation with a multi-resolution grid backbone that is trained in a coarse-to-fine manner, enabling faster reconstructions than prior methods. Extensive experiments on object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF 360 and Ref-NeRF real demonstrate that our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces, leading to the best overall performance. Project page: https://fangjinhuawang.github.io/UniSDF.

UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections

Abstract

Neural 3D scene representations have shown great potential for 3D reconstruction from 2D images. However, reconstructing real-world captures of complex scenes still remains a challenge. Existing generic 3D reconstruction methods often struggle to represent fine geometric details and do not adequately model reflective surfaces of large-scale scenes. Techniques that explicitly focus on reflective surfaces can model complex and detailed reflections by exploiting better reflection parameterizations. However, we observe that these methods are often not robust in real scenarios where non-reflective as well as reflective components are present. In this work, we propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. We investigate both camera view as well as reflected view-based color parameterization techniques and find that explicitly blending these representations in 3D space enables reconstruction of surfaces that are more geometrically accurate, especially for reflective surfaces. We further combine this representation with a multi-resolution grid backbone that is trained in a coarse-to-fine manner, enabling faster reconstructions than prior methods. Extensive experiments on object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF 360 and Ref-NeRF real demonstrate that our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces, leading to the best overall performance. Project page: https://fangjinhuawang.github.io/UniSDF .

1 Introduction

Given multiple images of a scene, accurately reconstructing a 3D scene is an open problem in 3D computer vision. 3D meshes from reconstruction methods can be used in many downstream applications, e . g ., scene understanding, robotics, and creating 3D experiences for augmented/virtual reality [36, 51]. Typical aspects of real-world scenes such as uniformly colored areas or nonLambertian surfaces remain challenging.

As a traditional line of research, multi-view stereo methods [39, 49, 16, 46] usually estimate depth maps with photometric consistency and then reconstruct the surface as a post-processing step, e . g ., point cloud fusion with screened Poisson surface reconstruction [19] or TSDF fusion [10]. However, they cannot reconstruct reflective surfaces since their appearances are not multi-view consistent.

Ref-NeRF

ENVIDR

Ours

Our mesh

Recently, Neural Radiance Fields (NeRF) [28] render compelling photo-realistic images by parameterizing a scene as a continuous function of radiance and volume density using a multi-layer perceptron (MLP). More recent works [30, 8, 42, 4] replace or augment MLPs with grid based data structures to accelerate training. For example, Instant-NGP (iNGP) [30] uses a pyramid of grids and hashes to encode features and a tiny MLP to process them. Motivated by NeRF, neural implicit reconstruction methods [50, 47] combine signed distance functions (SDF) with volume rendering, and produce smooth and complete surfaces. For acceleration, recent works [21, 37] rely on hash grid representations and reconstruct surfaces with finer details. However, these NeRF-based methods cannot accurately reconstruct reflective surfaces [44, 14].

To better represent the reflective appearance, Ref-NeRF [44] parameterizes the appearance using reflected view direction that exploits the surface normals, while NeRF uses the camera view direction. Recently, some works [51, 22, 25, 14] adopt this reflected view parameterization and successfully reconstruct reflective surfaces. We observe that while reflected view radiance fields can effectively reconstruct highly specular reflections, they struggle to represent more diffuse or ambiguous reflection types and fine details that can be found in real scenes. In contrast, we find that direct camera view radiance fields are more robust to difficult surfaces in real settings, although the reconstructions still present artifacts for reflective scenes. In this paper, we seamlessly bring together reflected view and camera view radiance fields into a novel unified radiance field for representing 3D real scenes accurately in the presence of reflections. Our method is robust for reconstructing both real challenging scenes and highly reflective surfaces.

The proposed method, named UniSDF, performs superior to or on par with respective state-of-the-art methods which are tailored for a specific scene type. UniSDF can be applied to any type of dataset, ranging from DTU [1], Shiny Blender [44], Mip-NeRF 360 dataset [3], to Ref-NeRF real dataset [44], leading to the overall best performance. It demonstrates the capability to accurately reconstruct complex scenes with large scale, fine details and reflective surfaces as we see in Fig. 1.

In summary, we propose a novel algorithm that learns to seamlessly combine two radiance fields with a learnable weight field while exploiting the advantages of each representation. Our method produces high quality surfaces in both reflective and non-reflective regions.

2 Related Works

Multi-view stereo (MVS). Many traditional [39, 48] and learning-based [49, 16, 46, 45] MVS methods first estimate multi-view depth maps and then reconstruct the surface by fusing depth maps in a post-processing step. As the core step, depth estimation is mainly based on the photometric consistency assumption across multiple views. However, this assumption fails for glossy surfaces with reflections, and thus MVS methods cannot reconstruct them accurately.

Neural radiance fields (NeRF). As a seminal method in view synthesis, NeRF [28] represents a scene as a continuous volumetric field with an MLP, with position and camera view direction as inputs, and renders an image using volumetric ray-tracing. Since NeRF is slow to train, some methods [30, 42, 8] use voxel-grid-like data structures to accelerate training. Many follow-up works apply NeRFs to different tasks, e . g ., sparse-view synthesis [54, 31, 43], real-time rendering [9, 34, 18, 53], 3D generation [33, 23, 7] and pose estimation [24, 41, 59]. For the 3D reconstruction task, there are many methods [32, 50, 47, 55, 13, 21, 37, 26, 35] integrating NeRF with signed distance functions, a common implicit function for geometry. Specifically, they transform SDFs back to volume density for volume rendering. However, we observe that they are unable to reconstruct shiny / reflective surfaces since NeRF's camera view direction parameterization for the color prediction does not accurately model reflective parts of the scene.

NeRFs for reflections. To render reflective appearance, [40, 5, 57, 58] extend NeRF and decompose a scene into physical components with strong simplifying assumptions, e . g ., known lighting [40] or no self-occlusion [5, 57]. Recently, Ref-NeRF [44] reparameterizes the appearance prediction with separate diffuse and reflective components by using the reflected view direction, which improves the rendering of specular surfaces. As a result, recent works [51, 22, 25, 14, 27] adopt this representation to reconstruct glossy surfaces. While leading to strong view-synthesis for reflections, we find that reflected view radiance field approaches often lead to overly smooth reconstructions with missing details and that their optimization is not stable on real-world scenes. In contrast to existing methods with a single radiance field, we propose to seamlessly combine reflected view and camera view radiance fields into a novel unified radiance field with learnable weight, which is robust for reconstruction in challenging scenes with reflective surfaces. The recent preprint Factored-NeuS [12] also uses camera view and reflected view radiance fields. It separately supervises the rendered colors of two radiance fields with ground-truth color, instead of learning a weight field to combine them like ours. We find that our approach to combine two radiance fields with learnable weight is simpler to train and leads to better reconstruction. Other recent methods [17, 56, 52] use weight to compose the colors from camera view radiance field(s) to render reflections, similarly to us. [17, 56] can only handle planar reflections, e . g ., mirrors, and require ground-truth masks of reflective objects to supervise the weight. In contrast, our method can handle non-planar reflective objects, e . g ., spheres. As we learn the weight to compose camera view and reflected view radiance fields without supervision , our method does not require additional input and can be trained from only RGB images. MS-NeRF [52] uses multiple volume density fields and how to reconstruct the underlying surface is undefined. In contrast, we use a single SDF field to represent geometry and can hence directly extract the iso-surface from it.

3 Method

In this section, we first review the basic elements of NeRF [28]. We then describe the architecture and training strategy of our method.

3.1 NeRF Preliminaries

In NeRF [28], a 3D scene is represented by mapping a position x and ray direction d to a volumetric density σ and color c using MLP. For a pixel in the target viewpoint and its corresponding ray r = o + t d , distance values t i are sampled along the ray. The density σ i is predicted by a spatial MLP that receives the position x as input, while the directional MLP that predicts the color c i uses the bottleneck vector b ( x ) from the density MLP and the view direction d as input. The final color C is rendered as:

where α i = 1 -exp( -σ i δ i ) is opacity, δ i = t i -t i -1 is the distance between adjacent samples, and T i = ∏ i -1 j =1 (1 -α j ) is the accumulated transmittance. The model is trained by minimizing the loss between the predicted and ground truth color:

Note that Mildenhall et al. [28] use a single-layer directional MLP and thus often describe the combination of NeRF's spatial and view dependence MLPs as a single MLP.

SDF

MLP

𝑓

𝐱

𝐝

𝛄

Reflection

𝐛

𝐧

𝐂𝒄𝒂𝒎

Weight MLP

𝐂𝒓𝒆𝒇

𝐖

𝐂

Eq.8

𝝎𝒓

𝐱

𝐱

𝐱

𝐝

𝑓 ௥௘௙

𝑓 ௖௔௠

𝑓 ௪

Radiance MLP

Radiance MLP

𝑑

𝐧′

3.2 UniSDF

Given a set of known images of a scene that potentially contains reflective surfaces, our goal is to optimize a neural implicit field and reconstruct the scene with high fidelity and geometric accuracy. We propose UniSDF, a method that enables us to seamlessly combine camera view radiance fields and reflected view radiance fields to reconstruct both (a) non-reflective surfaces, diffuse reflective surfaces and complex surfaces with both reflective and non-reflective areas as well as (b) highly specular surfaces with a well defined and detailed reflected environment. Our pipeline is shown in Fig. 2. We generate two radiance fields that are parameterized by camera view directions or reflected view directions and combine them at the pixel level using a learned rendered weight.

Volume rendering the SDF. We represent the scene geometry using a signed distance field (SDF), which defines the surface S as the zero level set of SDF d :

To better reconstruct large-scale scenes, we follow Mip-NeRF 360 [3] and transform x into a contracted space with the following contraction:

For volume rendering, we compute the volume density σ ( x ) from the signed distance d ( x ) as: σ ( x ) = α Ψ β ( d ( x )) , where Ψ β is the cumulative distribution function of a zero-mean Laplace distribution with learnable scale parameter β > 0 . The surface normal at x can be computed as the gradient of the signed distance field: n = ∇ d ( x ) / ||∇ d ( x ) || .

Hash Encoding with iNGP. To accelerate training and improve reconstruction of high-frequency details, we use iNGP [30] to map each position x to a higher-dimensional feature space. Specifically, the features { γ l ( x ) } from the pyramid levels of iNGP are extracted with trilinear interpolation and then concatenated to form one single feature vector γ ( x ) , which is passed to the SDF MLP.

Camera View & Reflected View Radiance Fields. In contrast to most existing methods [28, 2, 44] that use a single radiance field, we propose to combine a camera view radiance field and a reflected view radiance field to better represent reflective and non reflective surfaces.

We follow NeRF [28] for representing our camera view radiance field c cam , which is computed from features defined at each position and the camera view direction:

Color ofreflected

Color ofcamera

Learned weight

Composed color

Normal

view radiance field

view radiancefield

where b is the bottleneck feature vector from SDF MLP, n is the normal at x and d is the camera view direction. Similarly to recent works [50, 47], we notice that using surface normals as input leads to better quality.

We represent the reflected radiance field c ref with an MLP f ref as:

where ω r is the reflected view direction around the normal n . In Ref-NeRF [44], it is shown that for BRDFs under a limited set of conditions, view-dependent radiance is a function of ω r only. Unlike Ref-NeRF, which uses separate diffuse and specular components, we only use the specular component, leading to a simpler architecture. Additionally, we observe that using separate diffuse and specular components can lead to optimization instabilities resulting in geometry artifacts (see supp. mat. for details).

The main difference between two radiance fields is the view directional input of the MLP. As shown in Fig. 3, our method mainly uses the reflected view radiance field to represent highly specular reflections such as the tree reflections in the garden spheres or the environment reflection on the sedan car. The camera view radiance field is used to represent more diffuse reflections.

Learned composition. We compose two radiance fields using a learnable weight field in 3D. Specifically, we use an MLP f w to learn the weight values w :

We compose the signals at the pixel level. We first volume render W , C ref , C cam following Eq. 1. We then compose the colors for each pixel as follows:

In Fig. 3, the weight W detects reflections well and assigns high weight to reflected view radiance field in reflective regions. The surface normals show that our model accurately reconstructs both reflective and non-reflective surface geometry.

Motivation of composing radiance fields. Disambiguating the influence of geometry, color and reflection is an ill-posed problem in 3D reconstruction from images. NeRF-based methods [47, 50, 21] with camera view radiance field show their robustness in real-world scenes [1], while having difficulty with reflections [44, 14]. Ref-NeRF based methods [25, 14, 51] with reflected view radiance field usually perform well under restricted conditions, e . g ., highly specular objects [44], while we experimentally find their performance degrades in real-world scenes, e . g ., NeRO [25] and Ref-NeuS [14] on DTU [1] (Tab. 1), BakedSDF [51] on Mip-NeRF 360 dataset [3] (Fig. 4). Therefore, to extend theoretically justified Ref-NeRF representation with robust scene representations,

we propose to exploit the advantages of two radiance fields by combining them with learnable weight. Moreover, since each type of radiance field is specialized for different levels of reflection strength and complexity, we observe that the reconstructed geometries while using the two types of radiance are often complementary (Fig. 5). In our method, we explicitly intertwine the radiance fields in 3D to continuously determine and use the most adapted parametrization for each surface area.

3.3 Training and Regularization

Coarse-to-fine training. We observe that directly optimizing all the features in our multi-resolution hash grid leads to overfitting of training images, in particular to specular appearance details, which in turn results in incorrect geometry as we show in Fig. 7 (a). We observe that this model tends to fake specular effects by embedding emitters inside the surface exploiting the numerous learnable features in the hash grid. Therefore, we propose to instead optimize the hash grid features in a coarse-to-fine fashion, similarly to [21, 37], to avoid overfitting and promote smoother and more realistic surfaces. Specifically, we start with L init coarse pyramid levels in the beginning of training, and introduce a new level with higher resolution every T 0 training fraction (see implementation details in Sec. 4.1).

Regularization. Following prior works [50, 47], we use an eikonal loss [15] to encourage d ( x ) to approximate a valid SDF:

To promote normal smoothness, we constrain the computed surface normal n to be close to a predicted normal vector n ′ . n ′ is predicted by the SDF MLP and normalized. We use the normal smoothness loss L p [44] as:

We also use the orientation loss L o from Ref-NeRF [44] to penalize normals that are 'back-facing', using:

Full loss function. The full loss function L includes the color loss L color of composed color C and the regularizations, which is written as follows:

4 Experiments

4.1 Experimental Settings

Datasets. We extensively evaluate our method on four different types of datasets. The DTU dataset [1] is an indoor object-centric dataset with ground truth point clouds. Following prior works [50, 47], we use the same 15 scenes for evaluation. The Shiny Blender dataset [44] contains six different shiny objects that are rendered in Blender under conditions similar to the NeRF dataset. The Mip-NeRF 360 dataset is proposed in [3] and contains complex unbounded indoor and outdoor scenes captured from many viewing angles. We further evaluate on the three large-scale scenes with reflections that are introduced in Ref-NeRF [44], which consists of the scenes 'sedan', 'garden spheres' and 'toycar'. For simplicity, we name these 3 scenes the 'Ref-NeRF real dataset'.

Implementation details. Based on the Mip-NeRF 360 codebase [29], we implement our method in Jax [6] with the re-implementation of VolSDF [50] and iNGP [30]. In our iNGP hierarchy of grids and hashes, we use 15 levels from 32 to 4096, where each level has 4 channels. For coarse to fine training, we set L init = 4 and T 0 = 2% . Similar to mip-NeRF 360 [3], we use two rounds of proposal sampling and then a final NeRF sampling round. Following Zip-NeRF, we penalize the sum of the mean of squared grid/hash values at each pyramid level with a loss multiplier as 0.1. Our models are all trained on 8 NVIDIA Tesla V100-SXM2-16GB GPUs with a batch size of 2 14 . We train 25k steps on DTU / Shiny Blender and 100k steps on Mip-NeRF 360 / Ref-NeRF real datasets, which takes 0.75h and 3.50h respectively. See the supplement for more details.

Baselines. We compare our method to state-of-the-art volumetric implicit methods in surface reconstruction [47, 11, 13, 21, 51, 25, 14, 22, 12] and view synthesis [3, 44, 4]. Neuralangelo [21] and Zip-NeRF [4] are hash grid-based state-of-the-art methods for reconstruction and view synthesis, respectively. Tailored for handling reflections, [25, 14, 22, 12] are top performing methods for reconstructing and rendering objects with reflective surfaces. BakedSDF [51] is a top performing method for reconstructing high quality mesh of unbounded scenes with reflective surfaces.

To further evaluate the effectiveness of our method, we propose two custom baselines, named 'CamV' and 'RefV'. Using the same backbone as our method, 'CamV' uses only the camera view radiance field, while 'RefV' uses only the reflected view radiance field following Ref-NeRF [44]. Note that for both baselines, we also use our coarse-to-fine training strategy to improve performance.

4.2 Evaluation Results

DTU. We evaluate the reconstruction quality on DTU dataset [1]. Following prior works, we extract the mesh at 512 resolution. For Neuralangelo [21], we report the reproduced results from the official implementation. As shown in Tab. 1, Geo-NeuS [13] performs best on DTU and our method outperforms the remaining methods. Geo-NeuS heavily relies on supervision from accurate SfM point cloud and photometric consistency constraint to achieve top performance, while our method uses rendering loss only like Neuralangelo [21]. For reflective regions, the SfM reconstruction is inaccurate for supervision [14] and the multi-view photometric consistency is not guaranteed. We show that Geo-NeuS performs worse on Shiny Blender [44] in Tab. 2. Besides, we observe that NERO [25] and Ref-NeuS [14] perform worse than their baseline NeuS [47]. Though these methods perform well on objects with strong reflections, e . g ., Shiny Blender [44], they are not robust in general real-world scenes without strong reflections.

Shiny Blender. We summarize the rendering metrics, mean angular error (MAE) of normals and accuracy (Acc) of mesh in Tab. 2. Our method performs best in PSNR and on par with ENVIDR [22] in SSIM, LPIPS and MAE. Note that ENVIDR additionally uses an environment MLP, which we do not require, to improve rendering and reconstruction. Besides, we find ENVIDR unrobust in real-world scenes with geometry artifacts on both reflective and non-reflective surfaces, shown in Fig. 1. For mesh quality, our method performs on par with Ref-NeuS [14] in Acc, while performing much better than it on DTU [1] (Tab. 1). This demonstrates the roubustness of our method in various types of scenes. Moreover, our method explicitly outperforms Geo-NeuS [13], Neuralangelo [21] and Zip-NeRF [4] in all metrics.

bicycle (BakedSDF)

bicycle(Ours)

officebonsai(BakedSDF)

officebonsai(Ours)

Mip-NeRF 360 dataset. As shown in Tab. 2, our method performs on par with Zip-NeRF [4] in rendering. Note that Zip-NeRF focuses on view synthesis, while we focus on surface reconstruction. Compared with BakedSDF [51] and Neuralangelo [21], our performance is much better in all metrics. As shown in Fig. 4, our method reconstructs more complete surfaces and better details, while BakedSDF shows hole artifacts and struggles to reconstruct fine geometric details.

Ref-NeRF real dataset. As shown in Tab. 2, evaluation on this dataset is challenging for all methods. Our method outperforms Ref-NeRF [44] and Neuralangelo [21] in SSIM and LPIPS, Zip-NeRF [4] in PSNR and SSIM, and BakedSDF [51] in LPIPS. For surface reconstruction, Neuralangelo [21] cannot reconstruct reflective spheres well as shown in Fig. 1, while our method accurately reconstructs the smooth surface of the reflective spheres and the fine details on the statue.

Summary of evaluation. The four datasets that we evaluate on include various scene types, ranging from object-level to unbounded scenes, with and without reflections. While some methods perform best on specific datasets, e . g ., Geo-NeuS [13] on DTU and Zip-NeRF [4] on Mip-NeRF 360 dataset [3], we show their performance degrades on other types of datasets, e . g ., Shiny Blender [44]. In contrast, our method shows competitive performance on all datasets and performs best overall (see averaged metrics in Tab. 2). This demonstrates the robustness of our method to various scene types.

Custom baselines comparison. We compare our method with our two custom baselines, CamV and RefV, on the DTU [1], Mip-NeRF 360 [3], and Ref-NeRF real [44] datasets. As shown in Tab. 3, our method outperforms the two baselines in all metrics on all three datasets. Besides, CamV mostly outperforms RefV, while RefV fails on one scene in DTU. This shows that the camera view radiance field is usually more robust than the reflected view radiance field, although this method does not reconstruct the geometry of reflective regions well.

Fig. 5 shows a qualitative comparison, where RefV reconstructs smooth surface for the reflective back window but has artifacts on the side for 'sedan', while CamV fails to reconstruct accurate surfaces because of the reflections. On the 'toycar' scene, RefV fails to reconstruct the correct geometry, while CamV reconstructs shiny surfaces better while showing artifacts on the hood. For RefV, we sometimes observe optimization issues with separate diffuse and specular components, where the specular component may be blank throughout training and the diffuse component (w./o. directional input) wrongly represents the view-dependent appearance with incorrect geometry (see supp. mat. for details). By coupling two difference radiance fields continuously in 3D, our method represents the appearance and geometry better than the baselines that only use a single radiance field, leading to higher-quality reconstructed surfaces.

RefV

CamV

Ours

helmet

coffee

teapot

ball

bicycle

kitchenlego

w./o.coarse-to-fine

Ours

w./o.learned composition

Ours

(a)

(b)

Mesh visualization. In Fig. 6, we visualize our meshes on Shiny Blender [44] and Mip-NeRF 360 dataset [3]. Our method can accurately reconstruct the surfaces of reflective objects as well as large-scale scenes with fine geometric details.

4.3 Ablation Study

Coarse-to-fine training. As shown in Fig. 7 (a), the reconstructions of 'sedan' [44] contain artifacts on the specular window and hood without training in a coarse-to-fine manner. With all feature pyramid grids activated in the beginning, the hash grid backbone can easily overfit to the specular effects with wrong geometry.

Unification of radiance fields. In Tab. 3, we show that composing two radiance fields can consistently improve performance. In this ablation, we justify the effectiveness of unification with learnable weights. Since the main difference of two radiance fields is the view directional input, we design a baseline with a single radiance field that takes both camera view d and reflected view ω r as inputs. Structurally, this baseline has the same input information as our radiance fields and weight field. As shown in Fig. 7 (b), our method performs better in reconstruction, while the baseline cannot reconstruct reflective surfaces well on both scenes.

5 Conclusion

In this paper, we have presented UniSDF, a novel algorithm that learns to seamlessly combine radiance fields for robust and accurate reconstruction of complex scenes with reflections. We find that camera

view radiance fields, e . g ., NeRF, are robust to complex real settings but cannot reconstruct reflective surfaces well, while reflected view radiance fields, e . g ., Ref-NeRF, can effectively reconstruct highly specular surfaces but struggle in real-world settings and to represent other types of surfaces. By adaptively combining camera view and reflected view radiance fields with a learnable weight field, our method significantly outperforms the baselines with either single radiance field. Together with a hash grid backbone to accelerate training and improve reconstruction details, our method performs superior to or on par with state-of-the-art methods, tailored for handling reflections or not, in reconstruction and rendering on different types of scenes, ranging from object-level to unbounded scenes, with and without reflections.

Acknowledgement. We would like to thank Dor Verbin, Peter Hedman, Ben Mildenhall and Pratul P. Srinivasan for feedback and comments.

References

Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-scale data for multiple-view stereopsis. IJCV , 2016. 2, 5, 6, 7, 8, 14, 15, 16, 19, 21

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields. In ICCV , 2021. 4

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. MipNeRF 360: Unbounded anti-aliased neural radiance fields. In CVPR , 2022. 2, 4, 5, 6, 7, 8, 9, 14, 15, 16, 18, 19, 20, 21

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. ZipNeRF: Anti-aliased grid-based neural radiance fields. In ICCV , 2023. 2, 7, 8, 14, 15, 16, 18, 19, 20

Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In ICCV , 2021. 3

James Bradbury, Roy Frostig, Peter Hawkins, Matthew J. Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. 6

Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometryaware 3d generative adversarial networks. In CVPR , 2022. 3

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensorRF: Tensorial radiance fields. In ECCV . Springer, 2022. 2, 3

Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. MobileNeRF: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In CVPR , 2023. 3

Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques , pages 303-312, 1996. 1

François Darmon, Bénédicte Bascle, Jean-Clément Devaux, Pascal Monasse, and Mathieu Aubry. Improving neural implicit surfaces geometry with patch warping. In CVPR , 2022. 7, 16

Yue Fan, Ivan Skorokhodov, Oleg Voynov, Savva Ignatyev, Evgeny Burnaev, Peter Wonka, and Yiqun Wang. Factored-neus: Reconstructing surfaces, illumination, and materials of possibly glossy objects. arXiv preprint arXiv:2305.17929 , 2023. 3, 7, 16

Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-Neus: Geometryconsistent neural implicit surfaces learning for multi-view reconstruction. NeurIPS , 35, 2022. 3, 7, 8, 14, 15, 16, 18

Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ying-Cong Chen. Ref-neus: Ambiguityreduced neural implicit surface learning for multi-view reconstruction with reflection. arXiv preprint arXiv:2303.10840 , 2023. 2, 3, 5, 7, 14, 15, 16, 18

Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099 , 2020. 6

Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In CVPR , 2020. 1, 2

Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Zhang. Nerfren: Neural radiance fields with reflections. In CVPR , 2022. 3

Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In ICCV , 2021. 3

Michael Kazhdan and Hugues Hoppe. Screened Poisson surface reconstruction. ACM Transactions on Graphics (ToG) , 32(3):1-13, 2013. 1

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR , 2015. 14

Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In CVPR , 2023. 2, 3, 5, 6, 7, 8, 14, 15, 16, 18, 19, 20

Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar. ENVIDR: Implicit differentiable renderer with neural environment lighting. arXiv preprint arXiv:2303.13022 , 2023. 2, 3, 7, 18

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3D: High-resolution text-to-3d content creation. In CVPR , 2023. 3

Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. BARF: Bundle-adjusting neural radiance fields. In ICCV , 2021. 3

Yuan Liu, Peng Wang, Cheng Lin, Xiaoxiao Long, Jiepeng Wang, Lingjie Liu, Taku Komura, and Wenping Wang. Nero: Neural geometry and brdf reconstruction of reflective objects from multiview images. arXiv preprint arXiv:2305.17398 , 2023. 2, 3, 5, 7, 16, 18

Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang. SparseNeuS: Fast generalizable neural surface reconstruction from sparse views. In ECCV , 2022. 3

Li Ma, Vasu Agrawal, Haithem Turki, Changil Kim, Chen Gao, Pedro Sander, Michael Zollhöfer, and Christian Richardt. Specnerf: Gaussian directional encoding for specular reflections. In CVPR , 2024. 3

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021. 2, 3, 4, 18

Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, Peter Hedman, Ricardo Martin-Brualla, and Jonathan T. Barron. MultiNeRF: A Code Release for Mip-NeRF 360, Ref-NeRF, and RawNeRF, 2022. 6, 20

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. SIGGRAPH , 2022. 2, 3, 4, 6, 14, 17

Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR , 2022. 3

Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV , 2021. 3

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022. 3

Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. MERF: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics (TOG) , 42(4):1-12, 2023. 3

Yufan Ren, Fangjinhua Wang, Tong Zhang, Marc Pollefeys, and Sabine Süsstrunk. Volrecon: Volume rendering of signed ray distance functions for generalizable multi-view reconstruction. arXiv preprint arXiv:2212.08067 , 2022. 3

Gernot Riegler and Vladlen Koltun. Stable view synthesis. In CVPR , 2021. 1

Radu Alexandru Rosu and Sven Behnke. PermutoSDF: Fast multi-view reconstruction with implicit surfaces using permutohedral lattices. In CVPR , 2023. 2, 3, 6

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR , 2016. 20

Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In ECCV , 2016. 1, 2

Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR , 2021. 3

Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison. iMAP: Implicit mapping and positioning in real-time. In ICCV , 2021. 3

Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR , 2022. 2, 3

Prune Truong, Marie-Julie Rakotosaona, Fabian Manhardt, and Federico Tombari. SPARF: Neural radiance fields from sparse and noisy poses. In CVPR , 2023. 3

Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-NeRF: Structured view-dependent appearance for neural radiance fields. In CVPR , 2022. 2, 3, 4, 5, 6, 7, 8, 9, 14, 15, 16, 17, 18, 19, 20, 21

Fangjinhua Wang, Silvano Galliani, Christoph Vogel, and Marc Pollefeys. IterMVS: Iterative probability estimation for efficient multi-view stereo. In CVPR , 2022. 2

Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Marc Pollefeys. PatchmatchNet: Learned multi-view patchmatch stereo. In CVPR , 2021. 1, 2

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021. 2, 3, 5, 6, 7, 14, 16

Qingshan Xu and Wenbing Tao. Multi-scale geometric consistency guided multi-view stereo. CVPR , 2019. 2

Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. MVSNet: Depth inference for unstructured multi-view stereo. In ECCV , 2018. 1, 2

Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. NeurIPS , 34, 2021. 2, 3, 5, 6

Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T Barron, and Ben Mildenhall. BakedSDF: Meshing neural sdfs for real-time view synthesis. arXiv preprint arXiv:2302.14859 , 2023. 1, 2, 3, 5, 7, 8, 14, 15, 16, 18, 19

Ze-Xin Yin, Jiaxiong Qiu, Ming-Ming Cheng, and Bo Ren. Multi-space neural radiance fields. In CVPR , 2023. 3

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV , 2021. 3

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In CVPR , 2021. 3

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction. NeurIPS , 35, 2022. 3

Junyi Zeng, Chong Bao, Rui Chen, Zilong Dong, Guofeng Zhang, Hujun Bao, and Zhaopeng Cui. Mirror-nerf: Learning neural radiance fields for mirrors with whitted-style ray tracing. In Proceedings of the 31st ACM International Conference on Multimedia , 2023. 3

Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR , 2021. 3

Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (ToG) , 40(6), 2021. 3

Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys. NICE-SLAM: Neural implicit scalable encoding for SLAM. In CVPR , 2022. 3

A Appendix / supplemental material

In the supplementary, we first discuss more experimental details, including dataset and implementation details. Second, we discuss BakedSDF [51] in more detail. We show that BakedSDF is often not stable and we discuss how we finetune this method to improve performance on several scenes. Third, we summarize the detailed evaluation results on DTU [1], Shiny Blender [44], Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets. We also qualitatively compare with the state-of-the-artmethods [13, 51, 21, 14, 4]. Fourth, we include more ablation study. Fifth, we discuss the comparison with two custom baselines, RefV and CamV, in more detail. Finally, we discuss the potential societal impacts and limitations of our method.

B Experimental Settings

B.1 Dataset Details

We use the same data splits as prior works for fair comparison. On DTU [1], following [47, 13], we use all the images for surface reconstruction. On Shiny Blender [44], Mip-NeRF 360 [3], and Ref-NeRF real [44] datasets, we follow the official protocol and use the official training / testing split for training and testing.

B.2 Implementation Details

Network Architecture. In addition to the iNGP [30] structure that we have introduced in the main paper, we further discuss the details of the MLP architectures. Specifically, the SDF MLP f has 2 layers with 256 hidden units and outputs the bottleneck feature vector b with size 256. The two radiance MLP f cam , f ref have 4 layers with 256 hidden units. Besides, the weight MLP f w has a single layer with 256 hidden units.

Recall that following Mip-NeRF 360 [3], we use two rounds of proposal sampling and then a final NeRF sampling round. The proposal sampling is used to bound the scene geometry and recursively generate more detailed sample intervals, while the final NeRF sampling is used to render the final set of intervals into an image. We set the number of samples for these 3 sampling rounds as 64, 32, 32 for the object-level DTU [1] and Shiny Blender [44], and 64, 64, 32 for unbounded Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets.

In Sec. 3 of the main paper, we mainly introduce model details of the final NeRF sampling round, where the color is rendered, for simplicity. Thus, we introduce the details for proposal sampling rounds here. Specifically, the proposal sampling rounds only have a SDF MLP, i . e .no radiance MLP and weight MLP, since color is not rendered in these rounds. Moreover, the two proposal sampling rounds share a SDF MLP, which is different from the SDF MLP in the NeRF sampling round. Contrary to Zip-NeRF [4] that uses a distinct iNGP for each sampling round, we use a single iNGP that is shared by all sampling rounds. We find that this produces similar performance as using multiple iNGPs but explicitly simplifies the model.

Loss Function. In the loss function (Eq. 12 in the main paper, which is for the final NeRF sampling round), we set λ 1 = 10 -4 and λ 3 = 10 -3 . Moreover, we set λ 2 = 10 -4 for Shiny Blender [44], and λ 2 = 10 -3 for DTU [1], Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets. For proposal sampling rounds, we replace L color with L prop, the proposal loss described in Mip-NeRF 360 [3].

Training Details. For training, we use the Adam [20] optimizer with β 1 = 0 . 9 , β 2 = 0 . 999 , ϵ = 10 -6 . We warm up the learning rate in the first 2% iterations and then decay the it logarithmically from 5 × 10 -3 to 5 × 10 -4 .

C BakedSDF

In our experiments, we find that the optimization of BakedSDF [51] is sensitive and often fails completely on Shiny Blender [44] and Ref-NeRF real dataset [44], as shown in Fig. 8.

Image

Normal

BakedSDF only uses eikonal loss, L eik, for regularization, where the corresponding loss weight is set to 0.1 by default. We experimentally find that decreasing the eikonal loss weight can stabilize the training and thus carefully tune it for each scene. For Shiny Blender [44], we set the eikonal loss weight as 10 -2 for 'toaster', 'helmet' and 'coffee', and 10 -1 for 'ball'. Unfortunately, we could not find the best eikonal loss weight for 'car' and 'teapot' scenes since the training does not produce reasonable geometry. For completeness, we report the rendering metrics on these two scenes with eikonal loss weight as 10 -2 . For Ref-NeRF real dataset [44], we set the eikonal loss weight as 10 -2 for 'sedan' and 'toycar', and 10 -5 for 'garden spheres'.

D Detailed Evaluation Results

Tab. 4, Tab. 5, Tab. 6 and Tab. 7 contain the detailed metrics for each individual scene on DTU [1], Shiny Blender [44], Mip-NeRF 360 [3] and Ref-NeRF real [44] datasets respectively.

We qualitatively compare with state-of-the-art methods [21, 51, 4] on Shiny Blender, Mip-NeRF 360 and Ref-NeRF real datasets in Fig. 9. Our method is robust and demonstrates competitive performance on various scene types, ranging from object-level to unbounded scenes, with and without reflections.

Shiny Blender. In addition to the MAE error for evaluating the surface normal accuracy, we also evaluate the mesh quality. Though Chamfer Distance includes both accuracy and completeness , Ref-NeuS [14] points out that the ground-truth meshes of Shiny Blender [44] are double-layered, which results in many redundant points in the ground truth. Therefore, we follow Ref-NeuS [14] and only evaluate the accuracy (Acc) of reconstructed meshes. Besides, since ground-truth meshes of 'ball' and 'teapot' are unavailable, we only evaluate on the remaining four scenes following Ref-NeuS.

As shown in Fig. 10, since the objects in Shiny Blender are highly reflective, our method automatically assigns high weights for the reflected view radiance field in most regions. Though our learned weight is not supervised, it can successfully detect the reflective surfaces. Besides, our reflected view radiance field represents the highly specular reflections of the surrounding environment, which is similar to the observation in Fig. 3 of the main paper.

We further qualitatively compare the surface normals with Geo-NeuS [13], Neuralangelo [21] and Ref-NeuS [14] in Fig. 11. For 'teapot' scene, Geo-NeuS fails to reconstruct the smooth surface of the teapot. Neuralangelo reconstructs incorrect surface elements under the object. Ref-NeuS reconstructs overly smooth surfaces without fine details. For 'ball' scene, Geo-NeuS produces slight artifacts on the surface, while Neuralangelo reconstructs lots of incorrect surface elements inside the ball. In contrast, our method reconstructs the surfaces more accurately on both scenes.

E Custom Baselines Comparison

In the main paper, we have compared our method with two custom baselines, CamV and RefV, both quantitatively (Tab. 3) and qualitatively (Fig. 5).

15.25dB

19.71dB

25.91dB

29.13dB

21.65dE

34.14dB

27.11dB

29.97dB

32.52JB

21.90dB

22.40dB

23.05dB

22.53dB

Ground Truth

Zip-NeRF

BakedSDF

Ours

In Fig. 12, we further visualize the qualitative results on scan 37 of DTU [1]. On the one hand, RefV reconstructs holes on the objects with or without reflections, which is similar to the artifacts that BakedSDF [51] shows in Fig. 4 of the main paper. On the other hand, though the surface is a little noisy, CamV reconstructs shiny objects relatively well. Note that in Fig. 5 of the main paper, CamV also reconstructs the shiny surfaces of 'toycar' well, despite having some small artifacts. Since scan 37 of DTU and 'toycar' of Ref-NeRF real dataset mainly contain reflective surfaces that are less specular, we can infer that camera view radiance field can handle less specular reflections to some extent.

As shown in Fig. 13, we sometimes observe that RefV has optimization issues with separate diffuse and specular components. Specifically, the specular component may be empty throughout training,

Color of reflected

Color ofcamera

viewradiance field

Learned weight

Composed color

Normal

Ground truth

Geo-NeuS

Ref-NeuS

Ours

while the diffuse component represents both view-dependent and non view-dependent appearance. Since diffuse component depends only on the 3D position x , the view-dependence is represented with incorrect geometry, as shown in the results of 'toycar' in Fig. 13. We believe that this issue may be related to the high frequency signals that iNGP [30] can encode. In RefV, the diffuse component is parameterized by the feature vector γ from iNGP, which is capable of representing very high frequency signal. Therefore, the diffuse component may take advantage of the high capacity of the iNGP representation to model the view-dependent appearance with geometry only, leading to an incorrect reconstruction. This is especially true for small-scale scenes with relatively simple view-dependent appearance, e . g ., 'toycar'.

F Ablation Study

Normal. Recall that we use normal n , the gradient of the signed distance field, for computing reflected view direction ω r and loss function Eq. 11. In contrast, Ref-NeRF [44] uses predicted normal n ′ instead. In this ablation study, we follow Ref-NeRF [44] and use the predicted normal n ′ for computing reflected view direction ω r and loss function Eq. 11. As shown in Tab. 8, our method performs better than the baseline in rendering. We visualize the surface normal in Fig. 14. Our method successfully reconstructs the reflective surfaces, while the baseline reconstructs artifacts on them.

G Potential Societal Impacts

Positive impact. Our method can accurately reconstruct complex scenes with reflections, which is a challenging task for existing methods. The accurate reconstruction can be used for many downstream applications, e . g ., robotics and creating 3D experiences for augmented/virtual reality.

Negative impact. Similar to most existing volumetric implicit methods [28, 3, 4, 21, 44], our method needs to be individually trained for each scene. Though the training is fast (3.50h on Mip-

NeRF 360 [3] and Ref-NeRF real [44] datasets), we use 8 NVIDIA Tesla V100-SXM2-16GB GPUs simultaneously, which consumes much energy.

Diffuse

Specular

Normal

w./predicted normal

Ours

w./predictednormal

Ours

H Limitations

In this work, we mainly focus on robust surface reconstruction of scenes with reflections. Our method cannot be used for editing tasks such as relighting. Besides, similar to most NeRF methods [3, 21, 4], our method requires the estimated poses from SfM, e . g ., COLMAP [38], as input for real-world scenes. However, SfM needs to perform feature matching, which is based on multi-view photometric consistency, for pose estimation. As we discuss in the main paper, multi-view photometric consistency is not guaranteed with reflective surfaces and thus SfM may produce inaccurate poses, which may lead to inaccurate reconstruction and rendering. In addition, our method cannot accurately reconstruct the reflective surfaces with sparse input views. It is challenging to determine reflections in this case because of the ambiguity. Note that both Shiny Blender [44] and Ref-NeRF real dataset [44] carefully provide dense 360 degree views around the reflective objects so that it is easier to detect the view-dependent reflective appearance.

I Licenses for Existing Assets

code:

multinerf [29]: Apache License 2.0.

datasets:

DTU [1]: we do not find the license.

Shiny Blender [44]: coffee (CC-0 license), toaster (CC-BY license), car (CC-0 license), helmet (CC-0 license).

Mip-NeRF 360 dataset [3]: CC-BY license.

Ref-NeRF real dataset [44]: it is captured by the authors of Ref-NeRF [44], we do not find the license.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to L9-19 (abstract) and L46-57 (introduction).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Sec. H of supplementary.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We discuss the details of experiments in Sec. 4.1 and Sec. B, including network architecture, training details and dataset details.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The datasets used in this paper are all publicly available. We are unable to immediately release the code, but plan it for the future.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify the experimental details in Sec. 4.1 and Sec. B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We extensively evaluate our method on 4 datasets, which contain 33 scenes in total, and our method needs to be trained for each scene individually with 8 GPUs. Due to the limited computing resources and time, reporting error bars would be too computationally expensive for us.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: As discussed in Sec. 4.1, our models are all trained on 8 NVIDIA Tesla V100-SXM2-16GB GPUs. We train 25k steps on DTU / Shiny Blender and 100k steps on Mip-NeRF 360 / Ref-NeRF real datasets, which takes 0.75h and 3.50h respectively.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We carefully checked the Code of Ethics and can confirm that our research conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential societal impacts in Sec. G of supplementary.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original owners of codes and datasets that we use in Sec. 4.1. We mention their licenses in Sec. I of supplementary.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UniSDF, an improved NeuS architecture capable of reconstructing photorealistic reflective surfaces and robustly working in real-world scenes. The method models reflective color and base color as separate MLPs, using learned weights to blend them and obtain the final colors. Qualitative and quantitative results on several datasets demonstrate that the method surpasses other baselines and results in fewer geometric artifacts.\n\nSTRENGTHS:\nThe method proposes a new network architecture to better represent the reflective radiance field, although most of the concepts are derived from previous works. There are some original contributions, such as the blending between two fields inspired by NeRFRen, which is extended to non-planar surfaces, and the coarse-to-fine training from Neuralangelo, used here to reduce the ambiguity of reflections. Therefore, while the method is novel, its significance is not substantial.\n\nAnother strength is the robust and high-quality reconstruction demonstrated in the paper and supplementary material, which surpasses other baselines. This is valuable since this problem is crucial in real-world 3D reconstruction scenarios.\n\nWEAKNESSES:\nThe biggest weakness is the lack of physical interpretation. In Eq. 8, the final color is a linear combination of the camera view and reflected view radiance fields, resulting in unclear physical meaning for each color component. Although the method focuses on novel view synthesis and geometry reconstruction regardless of the underlying physical components, this lack of physical meaning leads to ambiguity about the assumptions the authors have for the reconstructed scene.\n\nFor example, the model doesn't account for how reflections change with different surface roughness, raising the question of whether it will perform well on surfaces with spatially varying roughness. Additionally, the model does not explicitly trace the reflective ray but instead models the color of the reflective ray through an MLP. This raises concerns about its ability to handle near-field lighting conditions effectively.\n\nFurthermore, I am curious about the differences between Ref-NeRF's equation c = diffuse + specular × tint and Eq. 8 c=(1−w) c_cam+w c_ref. Will these two modeling approaches yield similar performance?\n\nQUESTIONS:\n1. Title: It's unclear from the abstract and introduction where the \"unifying\" concept comes from. \"Unifying scene representation\" suggests combining several scene representations, but theoretically, it is still a NeuS-based representation.\n\t2. Missing Citations: \n\t\ta. MS-NeRF: Multi-space Neural Radiance Field: This paper also presents a similar idea of linearly combining different color fields. \n\t\tb. Several Concurrent Works Regarding Reflective NeRF: \n\t\t\ti. NeRF-Casting: Improved View-Dependent Appearance with Consistent Reflections\n\t\t\tii. SpecNeRF: Gaussian Directional Encoding for Specular Reflections \n\t\tc. Reflection Handling in Gaussian Splatting: \n\t\t\ti. MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections \n\t\t\tii. Spec-Gaussian: Anisotropic View-Dependent Appearance for 3D Gaussian Splatting\n\t3. Illustration of Existing Works: In line 131, I wouldn't call Ref-NeRF[43] as using a single radiance field, as it uses two MLPs to model diffuse color and specular color separately (though the specular MLP is conditioned on the feature of the diffuse MLP)\n\t4. Hyperparameters in Final Loss: The final loss contains several hyperparameters for controlling the strength of each term. In my experience, these weights play an important role. It would be beneficial to conduct small ablations or illustrate how these weights were decided.\n\t5. Input Redundancy: The f_ref and f_cam functions take in both the spatial coordinate x and the bottleneck feature b from the SDF MLP. Given that b is already a spatial feature vector, what's the point of adding x as one of the inputs? This seems a bit redundant.\n\t6. Normals and Orientation Loss: There are two normals being produced: n is the normal from the SDF field, and n′ is the MLP-predicted normal. For Eq. 11, Ref-NeRF uses n′ for orientation loss. It's unclear if this is a typo or if n is actually used for the orientation loss.\n\t7. Volumetric Rendering SDF: In line 124, it would be better to cite which volumetric rendering SDF system is used. I assume it's NeuS.\n\nLIMITATIONS:\nThere are no potential negative society impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of 3D reconstruction in the presence of highly reflective objects. To address the problem, they propose to learn an SDF-based neural representation. Different from prior work, they use two radiance branches in their representation, one conditioned on the camera viewing direction rotated about the normal and one conditioned on the regular viewing direction. The outputs of the two networks are combined using a learned weight to arrive at the final rendered radiance. \n\nThe authors argue that while the reflected direction is better at capturing the details in reflective surfaces, the viewing direction is more robust to real data. \n\nThrough extensive experiments, they show their method is competitive with prior art in both the tasks of NVS and mesh recovery (as measured by Chamfer Distance).\n\nSTRENGTHS:\nI like the problem being attempted in the paper, I also like the insight that the viewing direction conditioning is more robust, while the reflected direction produces better geometry for reflective surfaces in controlled environments. The results are also impressive. \n\nQuality: The ablation studies are well/extensively done. \n\nClarity: The paper is very well written and easy to follow. The figures are all well understandable and the work should be easily reproducible from just the description.\n\nWEAKNESSES:\nSignificance: I think the method is generally quite a straightforward extension of existing work and the paper's contributions are fairly limited. As I see it both the radiance branches have previously existed and the Ref-NeRF parametrization could be considered more “physically based”, the authors forgo a more mathematically solid setup in favour of an increase in performance. \n\nI think the paper would benefit a bit from providing more insight into the reasons for some of the phenomena i.e. why do the weights separate like that for more specular/diffuse regions even though both are conditioned on some sort of parametrization of the viewing direction etc. \n\nI am sympathetic however to the authors that some of these phenomena just occur and might be hard to reason about, but some of the simulated experiments could provide more insight into how the optimization behaves.\n\nQUESTIONS:\n1) Is there a stopgrad on any components of the normal smoothness loss (predicted vs computed normal in Eq 10)? \n\n2) I see there is an ablation on this, but why do the authors think the weight to mix the outputs of the two radiance branches is necessary, why doesn’t this factor just get automatically learned by the two networks? \n\n3) The authors seem to have ran ENVIDR on the Ref-NeRF real dataset (as evidenced by Fig 1?), but do not include the results in Table 2? Why? \n\n4) Have the authors tried passing in $n^\\prime$ instead of $n$ to the radiance network or also calculating $w_r$ about it? If I understand correctly $n^\\prime$ is supposed to be “smoother”, perhaps this would make results more robust for just RefV?\n\nLIMITATIONS:\nLimitations have been adequately discussed in the supplement.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to reconstruct scenes containing both reflective surfaces and non-reflective surfaces with high fidelity. Specifically, it trains a camera view radiance field and a reflected view radiance field separately, combining them by a learnable weight. The method is evaluated on four datasets, which covers different situations including objects and 360 scenes, shiny objects, real-world captures. And the method outperforms or is on par with SOTA on all these datasets, demonstrating its effectiveness.\n\nSTRENGTHS:\n1. Extensive experiments. The method was compared to several SOTA methods and outperforms or is on par with them, showing its ability to reconstruct complicated scenes with reflective surfaces on four datasets. \n2. The paper is well written. And the figures clearly serve for their purposes.\n3. The author did a great job in analyzing the limitations and social impact, which is highly encouraged.\n\nWEAKNESSES:\n1. The method involves two radiance fields and one weight field, and it has to perform volume rendering for colors from both radiance fields and the weights, which is very time-consuming even with implementation of iNGP.\n2. I wonder if you have a possible explanation for why some reflected view radiance field methods is not robust to real-world scene.\n3. I noticed the comparison with Factored-NeuS is only performed on DTU. Since Factored-NeuS is designed to also handle glossy objects, I wonder if how it look like when compared to this paper on Shiny Blender and Ref-NeRF real datasets.\n4. As the authors mention in the section of limitations, this method requires posed images and SfM can fail on reflective surfaces. While this is a common issue, I still wonder how robust this method is to inaccurate camera pose. Even a failure case is fine.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nThe authors listed a quite complete list of limitations. I think some are common issues for many works like requiring camera pose and dense views. With that being said, in the future the authors can consider extending this work to sparse views and how to eliminate the requirement for several fields and volume rendering, making the algorithm more efficient.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new strategy for modelling view-dependent effects in Neural Radiance Field-based scene models. Existing approaches have used networks conditioned on camera view directions, as well as on reflected view directions using surface normals, but this work proposes and validates the idea that both model types should be used simultaneously. This is achieved by employing a learned weight model that mixes the two color predictions, thereby allowing regions where the color is not well explained by the reflection-based model to be handled by the view direction-based model. This is shown through ablations to be a better strategy than either strategy individually, and is also compared to a number of baseline methods which employ reflection-based modelling of view-dependent effects.\n\nSTRENGTHS:\nThis paper presents a simple, yet effective idea that would be easy to incorporate into other works which use reflection-based models. I think a lot of the contribution is in showing such a simple but surprising result which is not obvious to try.\n\nThe quality and clarity of the writeup is high, and the evaluation seems quite thorough in comparing to other relevant works and on relevant datasets. I think the value of the proposed strategy is shown quite clearly by the experiments.\n\nWEAKNESSES:\nThe only notable weakness is that the proposed strategy is a fairly minor deviation from previously proposed methods. However, I think this is largely mitigated by the non-obvious nature of the change and the in-depth experimental evaluation.\n\nQUESTIONS:\nGiven the increasing prevalence of methods like 3DGS which do not use neural networks for modelling view-dependence but rather simpler approaches like spherical harmonics, do you see a way of applying the ideas of this paper to such methods?\n\nLIMITATIONS:\nThe only limitation I see is the reduced interpretability/editability of the model compared to other approaches, but this is addressed by the authors. I see no issues with societal impact beyond generic concerns that are relevant to all scene modelling methods.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}