H2O-SDF: Two-phase Learning for 3D Indoor Reconstruction using Object Surface Fields
Authors: Minyoung Park
Mirae Do
Yeon Jae Shin
Jaeseok Yoo
Jongkwang Hong
Joongrock Kim
Chul Lee
Venue: ICLR 2024 spotlight (2024)
Keywords: 3D reconstruction
Neural implicit surface learning
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: For 3D indoor scene reconstruction, we present a novel two-phase learning approach, which comprises holistic surface learning and object surface learning.

Abstract: Advanced techniques using Neural Radiance Fields (NeRF), Signed Distance Fields (SDF), and Occupancy Fields have recently emerged as solutions for 3D indoor scene reconstruction. We introduce a novel two-phase learning approach, H2O-SDF,  that discriminates between object and non-object regions within indoor environments. This method achieves a nuanced balance, carefully preserving the geometric integrity of room layouts while also capturing intricate surface details of specific objects. A cornerstone of our two-phase learning framework is the introduction of the Object Surface Field (OSF), a novel concept designed to mitigate the persistent vanishing gradient problem that has previously hindered the capture of high-frequency details in other methods. Our proposed approach is validated through several experiments that include ablation studies.

Published as a conference paper at ICLR 2024
H2O-SDF: TWO-PHASE LEARNING FOR 3D INDOOR RECONSTRUCTION USING OBJECT SURFACE FIELDS
Minyoung Park, Mirae Do; Yeon Jae Shin, Jaeseok Yoo Jongkwang Hong, Joongrock Kim & Chul Lee? Al Lab, CTO Division, LG Electronics, Republic of Korea {minyoung5. park, mirae do, yeonjae shin, jaeseok, yoo jongkwang hong, jurock. kim, clee. leee com
ABSTRACT
Advanced techniques using Neural Radiance Fields (NeRF), Signed Distance Fields (SDF), and Occupancy Fields have recently emerged as solutions for 3D indoor scene reconstruction. We introduce a novel two-phase learning approach, H2O-SDF, that discriminates between object and non-object regions within indoor enviromments. This method achieves nuanced balance, carefully preserving the geometric integrity of room layouts while also capturing intricate surface details of specific objects. cornerstone of our two-phase learning framework the in- troduction of the Object Surface Field (OSF), novel concept designed to mitigate the persistent vanishing gradient problem that has previously hindered the capture of high-frequency details in other methods. Our proposed approach is validated through several experiments that include ablation studies.
INTRODUCTION
The reconstruction of geometry and appearance in 3D indoor scenes using multi-view images has gained significant attention
important field of research in both computer vision and graph-
ics. Recent advancements in this field include techniques using Neural Radiance Fields (NeRF)
(Mildenhall et al., 2020), Signed Distance Fields (SDF) (Wang al.) 2021] Yariv al.) [02TT)
and Occupancy Fields (Oechsle et al.) 2021). Although these methods offer generally promising results, they still exhibit certain limitations. For instance, NeRF struggles to accurately capture object geometry due
inadequate constraint between radiance and geometry (Oechsle et al.) 2021]. Similarly, the performance of neural implicit representation such as SDF and Occupancy Fields significantly deteriorates in indoor environments, which often contain significant propor- tion of low-frequene regions like walls, ceilings, and floors (Guo et al. 2022). To address these shortcomings, subsequent research has incorporated additional constraints in the form of semantic (Guo et al. 2022) or geometric priors, such as depth or normal vectors (Wang et al., 2022a Yu al. 2022 These enhancements, though, have not fully resolved the issues associated with indoor scenes, particularly in reconstructing high-frequency areas, where a "smoothness bias" problem continues to affect the final quality of the results (Liang et al. 2023).
Note that indoor scenes present unique challenges in terms of surface representation and learning. Specifically, they feature a mix of room layouts characterized by low-frequency, large-scale sur- faces alongside various indoor objects that have high- frequurr,,,,,, small-scale surfaces. In technical terms, surfaces related to room layouts, which exhibit smooth characteristics, tend converge more rapidly during the learning process than do complex, multiple object surfaces (Tancik et al.) 2020). The latter is especially difficult to reconstruct, even in advanced stages of learning, due to the van- ishing gradient issue. The recent approach aims to utilize method that back-projects 2D depth information onto a 3D point cloud to guide SDF for preserving high-frequency details multiple object regions (Zhu al. 2023). However, directly guiding the SDF with incomplete geometric priors could negatively impact the quality of the reconstruction.
"qqual contribution. The order of these authors was determined randomly. Corresponning author.

---

Published as a conference paper at ICLR 2024
3D Surfare Recenetnurrim
3D Surfion Resvedaation
3D Surface Rocmmemnation
Olaest Surfao Fidd
(a) Input Images
(b) Neus+Normal
(c) H2O-SDF (Phase I)
(d) H-O-SDF (Phase 2)
Figure 1: Comparison of Reconstruction Results
Motivated by the previous limitations and the inherent complexity of reconstructing indoor envi- ronments, we introduce two-phase learning approach called H2O-SDF that stands for Holistic surface learning to Object surface learning for SDF. The first phase, focuses on global indoor scene geometry, while the second phase zooms in on the intricate geometrical and surface details of indi- vidual objects within the indoor space. For the initial phase, holistic surface learning, we present novel rendering loss re-weighting scheme based on normal uncertainty. This effectively addresses the issues of over-smoothing and discontinuity that arise due conflicting information from surface normals and colors. In the second phase, object surface learning, dedicated to indoor object surface learning, we recognize that rendering- based consistency alone cannot fully recover fine- uve-grained sur- face details of the given objects. To overcome this, we propose new object surface representation called the Object Surface Field (OSF). OSF still leverages SDF to capture object geometries and surfaces but they complement each other in inducing the 3D geometry reconstruction without the need for direct SDF value supervision. It effectively solves vanishing gradient problem. Fur- thermore, we enhance the performance of reconstruction by leveraging an advanced OSF-guided sampling technique. As result, object surface learning achieve more fine-grained capturing of sur- face details of individual objects. Fig. iillustrates the superiority of our approach in comparison to other cxisting methods like NeuS with additional normal priors (Fig. [Tb) vs Fig. [c.d)). Through establish the
comprehensive experiments conducted the ScanNet dataset (Dai et al. 201
superior performance of our method over other saate-of-the-ae solutions.
In summary, main contributions
follows:
We present novel two-phase learning approach, which comprises holistic surface learning and object surface learning. This approach enables us to distinguish effectively between object and non-object regions in indoor scenes. Consequently, we preserve the overarching room layout while concurrently capturing intricate details within specific object areas.
The driver of object surface learning is novel concept known the Object Surface Field (OSF). This concept enhances previous SDF to extract watertight surfaces of objects in 3D space by overcoming the vanishing gradient issue of SDF. We propose targeted sampling strategy for OSF that can further improve the quality of reconstruction, particu- larly in areas requiring high-frequency detail.
Our methods substantially outperform previous approaches in the domain of indoor scene reconstruction. This superiority is empirically validated through extensive experimental studies, including a series ablation tests that demonstrate the efficacy each individual component of our two-phase learning approach.
RELATED WORK
2.1 NEURAL IMPLICIT SURFACE REPRESENTATION
Inspired by the success of NeRF Mildenhall et al. 2020). neural field approach that encodes the geometry and radiance information of 3D coordinates using ML.P has become popular. Subsequent
2

---

Published as a conference paper at ICLR 2024
Phase Holistic Surface Learning
Phase Object Surface Learning
Color
Normal
Object Mask
Uniform Sampling
OSP Guided Sampling Stategy Object Surface Probabliity
weighoed
Re weightzd
2ddsss
Co
Cn
o
Fres
Refinement
Nommal
Uocertaraty
Volume Rendering
SDF OSF Cross Guided
o(x)
MLP
d(x)
Ladart
SDF
OSF
Figure 2: Architecture Overview The main pipeline consists of two phases. During the first phase (green part), learn the global indoor scene geometry through re-weighted Le and Ln based normal uncertainty from an input position x. During the second phase (blue part), we further train the Object Surface Field os fxx using Lade.. that is supervised by 2D object mask; Lad.., that cross-guie cross-guides between OSF os (x) and SDF d(x); and Lref that refines os with p (point cloud). During this process, we conduct OSF-guided sampling strategy (green dot).
works like NeuS (Wang et al. 2021) and VoISDF Yarii et al. 2021) enable more accurate surface reconstruction by representing SDF as implicit function. Based on these studies, recent works(Ful et al. 2022 Wang et al. 2022b) show that the inconsistency between the rendered color depth and the (- O-value surface of the SDF could lead less desirable surface reconstructions in practice. To address this issue, Geo-NeuS (Fu et al. 2022 proposes a method that aligns sparse point clouds extracted from SFM with the zero-level set of SDF. Furthermore, HF-NeuS (Wang et al., (022b) concludes that it is challenging to represent both high-frequency and low-frequency components with a single SDF. To overcome such challenge, a new methodology proposed decompose the SDF into their respective base functions and displacement functions. Although these methods improve reconstruction quality at the object level, they face challenges in reconstructing indoor scenes. The main reason is that indoor scenes include significant proportion of low-frequenc, areas (c.g., walls, floors, ceilings), making difficult to restore them using only color supervision.
2.2 NEURAL 3D RECONSTRUCTION FOR INDOOR SCENES
Recently, other research works focus on inducing smoothness in the reconstruction process to im- prove low-frequency surfaces by incorporating geometric cues, including semantic priors (Guo et al.] 2022), normal priors (Wang et al. (2022a) and depth priors (Yu et al. (002)) that are provided by pre- trained models. Due to the inherent properties the neural network, low-frequency surfaces tend to converge more rapidly than complex multiple-object surfaces characterized by high-frequency sur- faces (Tancik et al., 2020). Moreover, the multiple object surfaces are hard to recover even during the advanced learning stage due to the vanishing gradient problem. To address this issue, I2-DDF proposes a method that back projects 2D depth into a 3D point cloud to guide the SDF explicitly. HelixSurf (Liang al. 2023) shows the inaccuracy of 3D priors and proposes method leverage the benefits of both explicit and implicit methods. In contrast, our proposed method, H3O-SDF, captures all intricate geometrical and surface details of objects while maintaining the smoothness of room layouts by learning object surface field.
3
OUR METHOD
We provide an overview of our approach in Fig. Our H₂O-SDF pipeline consists of two phases: concentrates the global scene geometry, and the second phase, object surface learning (Sec. 3.2 delves into the intricate geometrical and surface details of objects within the indoor scene. This dual-phase learning approach achieves a nuanced balance, carefully preserving the geometric integrity of room layouts while also capturing intricate surface details of specific objects.
the first phase, holistic surface learning (Sec. 3.1
3

---

Published as a conference paper at ICLR 2024
3.1 HOLISTIC SURFACE LEARNING
Holistic surface learning is dedicated to reconstructing the overall geometry, including the smooth room layout, which constitutes a significant portion of the indoor scene and predominantly features low-frequency regions. We represent the scene's geometry as the signed distance function (SDF) d(x) for each spatial position x. Practically, the SDF function is realized through a multi-layer perceptron (MLP). The geometry network is formulated as fg R256), where z is a geometrical feature. The view view-dependent color e is also implemented by an MLP. The color network is articulated as fe : (x E R v E R n (x) E RR, z(x) E RR20)) -> c(x, v) RR, where v is the view direction and n(x) Vd (x) is the spatial gradient of SDF at point x. According to the volume rendering formula, the colors along a ray r are rendered by
IR3 + (d E R. z (x)
C(r) = Σ T; oq c(x;, Va), where N is the number of sampled points along the ray, T; = (1- - a3) represents the accumulated transmittance, and a; discrete opacity, with the opaque density p(t) adhering to the original definition in NeuS (Wang et al. 1021). Similarly, we render the surface normal along a ray by nn() - > Ti Oc ....nn (x:).
[1
exp(- fo p(t)dt) the
However, as highlighted by NeuRIS (Wang et al. 2022ap, there exists an inherent conftict between color and normal information. Normal priors can offer precise geometric guidance in texture-less regions, but color images might provide misleading appearance supervision in those regions due to insufficient visual features. Conversely, in regions with fine details, they exhibit the opposite behav- ior That is, if color and normal losses are uniformly applied all pixels, results in discontinuities in planar regions, and details are overly smoothed. To mitigate this situation, we introduce tech- nique to re- re-weight normal and color loss using normal uncertainty, which exhibits low uncertainty in planar regions and high uncertainty in texture-rich surfaces.
We initially estimate the normal fi(r) and the corresponding uncertainty ur of each pixel using a pre-existing monocular normal estimation model (Bae et al. 2021). Subsequently, we determine the weight of normal loss An and color loss Ae with (3n Up) and (Be- uu,) respectively, where 3n and Be are two trade-off hyperparameters and up is the normal uncertainty of the ray Consequently, planar areas with low uncertainty, such as room layout and large objects, are predominantly influenced by normal rather than color loss, leading more coherent and smoother planes. Conversely, texture- rich areas are more influenced by color, preserving details and maintaining sharpness. The normal and color loss are defined below with each corresponding ground truth n(r) and C(r).
(((r))-(r)(() Σ [ī(r) - n())l, (8n -0--4 ur), Le = Σ El(((r) - C(+) || (9e + u.)) FER
Ln
(1)
FER
We employ the Eikonal loss (rropp et al. 2020 to further regularize the gradients of SDF; Ceik
ΣM (IVd(xx)) - -122
3.2 OBJECT SURFACE LEARNING
Object surface learning is propelled by a novel concept that we introduce as object surface field (OSF). OSF directs SDF to encapsulate small-scale geometry and high-frequency details on objects while preserving the smoothness of room layout. The 2D object surface loss is a preliminary step in obtaining the initial value of OSF. The 3D object surface loss, enables OSF to carefully learn the object surface from SDF, letting SDF to capture sighffreueecy details under the guidance of OSF. To further improve the reconstruction performance, leverage OSF-guided sampling strategy to prioritize object surfaces.
2D Object Surface Loss Object surface field (OSF) essentially represents the probability of object surface os fx() for each spatial point x. It is realized by an MLP fo (x E Rd z(x) E R556) + os fx IR. and the OSF of the ray is rendered by: os fr) Ei= a (x.). As preliminary step in learning OSF, we define 2D object surface loss, L2dess = (r), (r))
where BCE is the binary cross-entropy loss, and the indicator function 11, returns 1 if the object surface exists along the ray, and O otherwise. It is determined by the 2D object mask, which can be easily obtained using praited model (Lambert et al. 2020). L2de.; is utilized to minimize the discrepancy between the predicted OSF of the ray and the given 2D object mask, thereby establishing the initial value of OSF.
4

---

Published as a conference paper at ICLR 2024
3D Object Surface Loss 3D object surface loss encourages SDF capture the high- feghffreuency details of objects by interacting with OSF. As depicted in Fig. aa), while Lzd.., trains OSF to learn the approximate object region, there is still a lack of information from the 2D object mask for OSF to learn the precise object boundary. Also, as it employs volume rendering to accumulate os based on density-based weight, it struggles to provide supervision empty spaces with low density, leading to noisy OSF. Conversely, as Fig. bb) illustrates, the 3D object surface loss enables OSF to learn the object surface meticulously, allowing to provide supervision signal the object surface of SDF. For this, we propose the 3D object surface loss as below:
N
L3dor! N
Σ [L)(r) (x)(x;) (xs ax.))) + (1 (1- (r(r)) oxf(x:)
(2)
where, the scaled sigmoid function (x) +x is a hyperparameter determining the steepness of the function, and N is the number of sampled points along the ray r. This loss functions
differently depending on the existence of the object surface along the ray.
The 3D object surface loss is designed for OSF to adhere to the object surface of SDF, simultaneously learning low probability on the non-object surfaces ef- fectively. When the ray intersects with the object sur- face, the probability of the object surface should alter according the sign of the SDF. In essence, during the traversal of the ray within the zero-level set of the SDF, it is anticipated that the object surface probabil- ity should escalate upon entry and diminish upon exit. It is motivated by ObjectSDDF (Wu al. 2022), which explains the correlation between the SDF and the 3D semantic field using a scaled sigmoid function. Un- like ObjectSDF, where SDF values are directly trans- formed into semantic field values and learned by 2D rendering-bor -baiig loss, our method introduces a distinct object surface field to mutually guide the object sur- face of the SDF.
GT
(a)Crdans
(b)+ Ladont
Figure 3: Comparisons of OSF Compared to using only (a) L2d..j introduction of our (b) L3d.e, enables OSF to represent precise object boundaries. Improvement includes object surfaces (Red) and non- object surfaces (Blue).
When the ray intersects only with the room layout sur- face, the OSF of all points along the ray should exhibit low probability. Thus, Lad.r, is structured to prevent OSF activation on the room-layout surface. In multi-view settings, it can effectively prompt OSF to decrease for occluded space located behind objects as well as the room layout. When the ray intersects with the room layout surface after colliding with the object surface, as the probability of OSF has already reduced, OSF remains unaffected by changes in SDF
Moreover, we observe that the reliability of point clouds extracted from multi-view stereo (MVS) is superior in object areas with abundant visual features, while it is inferior in texture-less areas. As consequence, we propose refinement loss to aid the learning of the OSF with the point clouds. The refinement loss is defined as follows:
Lres
Σ 1x(x)) log(os X,EP,
N
(3)
Here, Nr is the number of points in the point clouds P. from view V,, and (x,x)) denotes whether an input point belongs to an object surface or not, returning a value of I or 0 respectively. This approach enhances the performance of the OSF network, allowing for more precise learning of intricate object regions with thin structures and high occlusion.
Mutual Induction of OSF and SDF In this section, we explain in more detail 1) how SDF directs OSF to learn the surface of the object and 2) how well-leamed OSF enables SDF to capture fighffrequency geometric detail. As depicted in Fig. [aa, during the initial stage of object surface learning, the OSF in regions with high rendering weight (w = aa) are trained to converge towards due to L24.., (Eq. 3.2). Initially, the OSF represents high values not only on the surface but also inside the object. However, ÖLzdor, (x)
) is computed as follows:
aLsdort [xx(x)
os (x), if
(x)
(4)
dosf(x) (x) - 0, (x),, otherwise
5

---

Published as a conference paper at ICLR 2024
Interior
Interior
Intetior
Interior
Interior
(a)
(b)
(c)
(c)
(c)
Surface
d(x)
o, (x)
os osf(x)
DCadort Bosf(x)
Bd(x)
Figure 4: Interaction of OSF and SDF Illustration of (a) the initial status of OSF, (b) the influence of the gradient of Ldd.. with respect to OSF, (c) the case when SDF fails to capture thin structure, (d) the influence of the gradient of L3dmes with respect to SDF, and (e) the final result of OSF and SDF. Interior refers to the region inside an object.
As illustrated in Fig. 4bb, DC3dor, (x) yields negative values only in the red region, when os < (x) < fx). In other words, the OSF is induced to have high values in the region where the SDF indicates that surfaces are nearby, but the OSF has not yet learned about those surfaces. Through the learning process, the OSF is trained to decrease both within and outside the object while increases only near the surface. Consequently, the OSF is guided represent high values exclusively on the object surfaces. The following process describes how the well-traincs OSF encourages the SDF to learn high-freyuency details that have not yet been captured (see Fig. 4c)). Considering the case that a ray crosses a thin structure such as a chair leg, iLadda, (Od(x)) yields the following results:
@Lsdor 7 osf((f( (x)x) (1 (1-60 (x),), Od(x) -- - osf(x) (x) (1 - (x)), otherwise
if ox) (x)
(5)
As illustrated in Fig. dd. the SDF in regions of high OSF are pulled towards the negative side when (x) < (x). This loss generates gradients the thin structure, effectively addressing the vanishing gradient problem. By means of the Eikonal loss, the SDF can be trained to learn high- frequency details while maintaining gradients and water-tight shape. Please refer our appendix (Sec. A.2) for the derivation of the vanishing gradient problem in our base model.
Object Surface Field Guided Sampling Strategy In other approaches such as NeuS, hierarchical sampling relies on the volume rendering weight (w T a). derived from density. However, in typical indoor scenes, expansive planar areas, particularly room layouts, tend stabilize early in the training phase, leaving high-frequency details insufficiently captured. Consequently, density- based sampling over-emphasizes already well-rreconsrructed room layout regions, ignoring points in high-frrquency areas. The application of object surface field (OSF) has mitigated the overlooked density issues in high-frequency regions. There remains a necessity to prioritize object surfaces exhibiting more intricate structures compared to room layouts. To overcome this, we introduce novel sampling strategy, guided by the OSF. After uniformly sampling points along the ray, we iter- atively perform importance sampling the probability distribution calculated from w(x) os (x). This approach lets the network to concentrate more on areas with high density and object surface probability. Consequently, the OSF-guided sampling strategy enhances the overall model more precisely capture high- feggefrryeency details. Details of the OSF-guided sampling strategy are provided the Appendix (Sec. A.2
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Datasets To evaluate the effectiveness of our proposed algorithm in real-world scenarios, we con- duct empirical analysis using ScanNet (Dai et al. 2017). ScanNet is a large-scale dataset, comprising 1613 scenes with 2.5 million views. Each scene contains sequence of frames annotated with cali- brated intrinsic and extrinsic camera parameters, in addition to surface reconstruction. We choose scenes from (uuo al. 2022 and randomly selected additional 8 scenes.

---

Published as a conference paper at ICLR 2024
ManhattanSDF
NeuRIS
MonoSDF
HelixSurf
H₂O-SDF
Ground Truth
Figure 5: 3D Reconstruction Results ScanNet H2O-SDF shows improved reconstruction ability for both room-layout regions (blue box) and fine-grained object regions (red box) compared to other methods. Reconstruction for the remaining scenes visualized in the Appendix (Sec. A.3)
Implementation Details We use the NeuS (Wang et al. 2021) our base model, which comprises a geometry network with an 8-layer MLP and an appearance network with a 4-layer MLP. To learn the object surface field, employ object surface network with 4-layer MLP. For model opti- mization, we adopt the Adam optimizer with an initial learning rate of 2e-4 and then train H2O-SDF with batches of 512 rays. We conducted 40k iterations of training during the holistic surface learn- ing phase, followed by an additional 1 10k iterations during the object surface learning phase. We use the following hyperparameters in our experiments: Se = 1.0, Bn 2.0, desk 0.1, A2d.,, 0.5, Asda,, = 0.5, Aref = 0.1. Our code is implemented with PyTorch (Paszke et al. 2017)) conducting all experiments on single NVIDIA RTX 3090Ti GPU. We provide more implementation details in the Appendix (Sec. A.I).
Compared Methods We compare our H₂O-SDF against other traditional MVS method such as COLMAP (Schönberger et al.) 2016) and neural volume rendering methods, including NeRF Mildennhall et 2020), VolSDF (Yariv et al. (02I). NeuS (Wang et al. 2021). Manhat- tanSDF (Guo et al. (2022) NeuRIS (Wang et al. 2022a), MonoSDF (Yu et al. (022). and Helix-
Surf (Liang et al. (023).
Metrics For a quantitative comparison of 3D surface reconstruction, we utilize five standard met- rics introduced in (Murez al. 2020): Accuracy, Completeness, Precision, Recall, and F-score. Following (Sun et [ we adopt F-score as the comprehensive metric to measure the quality of 3D reconstruction, as it considers both accuracy and completeness.
4.2 COMPARISONS
3D Reconstruction To demonstrate the reconstruction ability of H2O-SDF, we provide qualita- tive and quantitative comparisons with other state-of-the-art methods. Our method demonstrates significantly smoother reconstruction results in low-frequency areas compared to other methods, highlighting the benefits our re-weighting scheme (See Fig. 5). Moreover, thanks to the object surface learning, H₂O-SDF can better represent the fine-grained surface details of object geometries (e.g., chair legs or lamp) compared to other methods. Additionally, as shown Tab. [ our method outperforms previous methods, achieving metric. These results, altogether, demonstrate the effectiveness of our two- wha-ph learning approach by substantially improving the overall quality of the indoor scene reconstruction.
oftt--o-thh-ar art performance in every 3D geometry

---

Published as a conference paper at ICLR 2024
Methods
Acc
Comp4
Prect
Recallf
F-scoref
COLMAP
0.047
0.235
0.711
0.441
0.537
ACMP
0.118
0.081
0.531
0.581
0.555
NeRF
0,735
0.177
0.131
0.290
0.176
NeuS
0.179
0.208
0.313
0.275
0.291
ManhatianSDD
0,053
0.056
0.715
0.664
0,688
MonoSDF
0.035
0.048
0,799
0.681
0.733
HelixSurf
0,038
0.044
0.786
0.727
0.755
H20-SDF
0.799
Table 1: Quantitative Comparison on ScanNet We compare H₂O-SDF against other previous state-of-the-ar methods.
Reference Image
SNU
Uncertainty Map
NeuRIS
H₂O-SDF
Ground Truth
Figure 6: Qualitative Normal Comparisons on ScanNet Normal prior (SNU) and uncertainty map obtained from (Bae al. 2021) is utilized in our re-weight scheme.
Normal Predictions We compare the normal prediction quality of H₂O-SDF against a monocular normal estimation method in (Bae et al. 2021 and other normal prior based reconstruction method in [aan et al.) (2022a). As illustrated in Fig. 6 H₂O-SDF excels at reconstructing both broad, low-frequency regions and fine-grained surface details compared to other methods. More normal comparison results provided in the Appendix (Sec. A.3).
Generalizability To further investigate the applicability and generalization of H₂O-SDF, we con- ducted additional experiments the Replica (Straub et al.) (019) dataset and 7-Scenes (Shotton) jet al. 2013) dataset. As shown in Tab. 2 our method achieves superior 3D reconstruction results, irrespective of any specific domain of application.
Model
Accu.
Comp. 1
Pree.1
Recallf
F-scoce
Model
Accu.
Comp..
Prec.?
Recallf
P-scocef
Min
0,112
0,132
0.351
0.336
0.336
MonhaltanSDP
0.131
0.317
0.841
0.307
0.192
NeaRIS
0.027
MonuSDF
0.158
0.328
0.286
0.262
0.301
MomoSDF
0,020
0.017
0.91
0.933
0.934
Ho DSSDD
0.128
0.129
0.416
0.469
0.447
HaO-SDP
0.016
nJ025
0.983
0.935
0.957
(a) Comparison results for 7-Scenes
(b) Comparison results for Replica
Table 2: Quantitative Comparsion on 7-Scenes and Replica datasets
4.3 ANALYSIS
Ablation Study In this section, to validate the effectiveness of individual components in O-O-SDF, we conduct ablation studies in different settings: (1) NeuS: Base model, (2) Model A: NeuS with normal priors, (3) Model B: the model undergoes training solely through our first stage, Holistic Surface Learning defined in Sec. 3.1, (4) Model C: Model with Object Surface Field (OSF) defined in Sec. 3.2 Fig. Tillustrates how the proposed components enhance reconstruction perfor- mance. Model-A demonstrates the ability to reconstruct low- lo-frrquency areas but still exhibits noise. With the proposed HSL, Model-B yields smoother and more consistent room layout surfaces than Model-A. By incorporating the OSF, we observe that it better captures intricate details within object regions, such stands and chair legs. Finally, incorporating with OGS, our full model more suc-
8

---

Published as a conference paper at ICLR 2024
Mothod NeuS
NeuS
Normal
Roweight OSF
OGS
Acc|
Comp1
Prset
Recall
F-scoref
0.132
0.114
0.376
0.334
0.353
Model A
0.040
0.042
0.794
0.742
0.766
Model B
0.041
0.041
0.802
0.751
0.776
Model
0.039
0.041
0.817
0.762
0.787
1l,0 SDF
0.037
0.039
0.830
0.773
0.800
Table 3: Quantitative Results on the Ablation Study of 12 scene ScanNet. Reweight means re-
weighting scheme for Holistic Surface Learning (Sec. B.Ip. OSF indicates L2dos; + Lidnas + fref for Object Surface Field (Sec. 3.2). OGS means Guidee Sampling strategy
NeuS
Model
Model B
Model C
H₂O-SDF
Ground Truth
Figure 7: Visualization Results of Ablation Study
cessfully facilitates the reconstruction of fine-gr ene-graieed detailed regions. Quantitative results in Tab. demonstrate the effectiveness of H2O-S O-FDF through the progressive addition of different components to the baseline model. More ablation studies provided in the Appendix (Sec. A.3).
Object Surface Learning As shown in Fig. g we conduct additional analysis on the effectiveness of object surface learning in the of thin structure regions. NeuS with normal prior exhibit higher weights in the floor regions characterized by low-frequency surfaces rather than in thin struc- ture regions due the vanishing gradient problem. On the other hand, the introduction of object surface field (OSF) exhibits peak values in the thin structure regions with zero SDF values, as illus- trated in Fig. c). This demonstrates that OSF directs gradients thin structures. By incorporating the OSF-Guided Sampling strategy (OGS), H3O-SDF resolves weight ambiguity more effectively.
b
0.25
Waight
SDF
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
0.2 0.4
Distance
Distance
Distance
(a) Reference Image
(b) NeuS + Normal
(c) HDO-SDF w/o OGS
(d) H₂O-SDF
Figure 8: Analysis Object Surface Learning For visualization, extract the weights distribu- tion and SDF values from a set of points along the emitted ray corresponding to single pixel.
CONCLUSION
We introduce novel two-phase learning approach for 3D indoor reconstruction. The first phase, holistic surface learning, reconstructs the overall geometry of scene, resolving conflicts color and normal information. The second phase, object surface learning, addresses the vanishing gradient problem and enables the capturing of detailed object surface geometry by leveraging the object sur- face field (OSF). This concept is interesting as OSF extends beyond existing 2D priors, functioning as a 3D geometry cue that delivers 3D spatial information to the SDF. Our method has proven to ScanNet. Future work will fo-
achieve ofate-of-the-at reconstruction accuracy and smoothness cus on integrating faster convergence radiance fields (c.g. TensoRF (Chen et al. 2022)) to expedite training and developing applications like scene-editing using OSE.

---

Published as a conference paper at ICLR 2024
REFERENCES
Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the aleatoric uncer- tainty in surface normal estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13137-13146, 2021.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision (ECCV), 2022.
Decai Chen, Peng Zhang, Ingo Feldmann, Oliver Schreer, and Peter Eisert. Recovering fine details for neural implicit surface reconstruction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4330 4330-4339, 2023.
Angela Dai, Angel X Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 201277.
Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-neus: neural implicit surfaces learning for multi-view reconstruction. Advances in Neural Information Processing Systems, 35:3403-3416 2022.
Amos Gropp,
Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric reg- ularization for learning shapes. In Proceedings of Machine Learning and Systems 2020, pp- 3569-3579, 2020.
Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, and Xiaowei Zhou. Neural 3d scene reconstruction with the manhattan-world assumption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5511-5520. 2022.
John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for multi dollin semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2879-28726 2020.
Zhihao Liang, Zhangjin Huang, Changxing Ding, and Kui Jia. Helixsurf: A robust and efficient neural implicit surface learning of indoor scenes with iterative intertwined regularization. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13165- 13174, 2023.
William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. Seminal graphics: pioneering efforts that shaped the field, pp. 347-353. 1998.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich, Atlas: End-to-end 3d scene reconstruction from posed images. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16. pp. 414-431. Springer, 2020.
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5589-5599, 2021.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part III 14, pp- 501-518. Springer, 2016.
10

---

Published as a conference paper at ICLR 2024
Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2930-2937, 2013.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren. Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica indoor spaces. arXiv preprint arXiv: (XXiv:9906005997 2019.
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d reconstruction from monoeular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15597,15607, 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Ken Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping Wang. Neuris: Neural reconstruction of indoor scenes using normal priors. In European Conference on Computer Vision, pp. 139-155. Springer, 2022a.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. Hf-neus: Improved surface reconstruction us- ing high-frequency details. Advances in Neural Information Processing Systems, 35: 15669691 2022b.
Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided op- timization of neural radiance fields for indoor multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5610-5619, 2021.
Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng. Object-compositional neural implicit surfaces. In Computer Vision-ECCV 2022: 17th Euro- pean Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXVII, pp. 197-213. Springer, 2022.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. 2021.
Advances in Neural Information Processing Systems,
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Ex- ploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems, 55:25018-25033 8-25032, 2022.
Jingsen Zhu, Yuchi Huo, Qi Ye, Fujun Luan, Jifan Li, Dianbing Xi, Lisha Wang. Rui Tang, Wei Hua, Hujun Bao, et al. 12-sdf: Intrinsic indoor scene reconstruction and editing via raytracing neural sdfs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12489-12498, 2023.
11

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 ADDITIONAL IMPLEMENTATION DETAILS
Network Architecture Our detailed network architecture is illustrated in Fig. Our method uses three networks to encode the implicit radiance field, implicit signed distance field, and object surface field. The geometry network takes input position mapped by positional encoding (Mildenhall et al. 2020 and outputs SDF values and geometry features.
Object Surface Field Network fa
x
Geometry Network f
SGF
1
256
256
256
Objjct Surfeee Probabiliyy
x
PE(x)
39
256
217
Appearance Network f
256
256
256
256
256
AGB
Input Position
View Direction
Geometry feature
X
Weights
Normal Direction
Output
PE(V)
n
Figure 9: Network Architecture Our network takes as input position x and the view direction v of the sample point, and outputs SDF, color and probability of object surface. Additionally, the input position x and view direction v are mapped by positional encoding PF.)).
Experimental Setting We test our method with 4 scenes as the previous study (Wei al. 2021 Wang et al. 2022a Liang et al., 2023) and an additional 8 scenes, which were randomly selected. For each scene, we uniformly sample frames at rate of one-tenth or one-sixth proportional the video length. These frames are resized in 640 x 480 resolution. We adopted SNU (Bae et al.) 2021) as our normal estimation module, which takes a single image as input normal map and uncertainty map output. To facilitate the re- weighting scheme, normalized the uncertainty map to a range between and Additionally, we take MSeg (Lambert et al. (0200) as semantic segmentation estimation module, employing its officially provided pre-trained model. To create the object mask used object surface learning, we masked the predicted segmentation results by assigning value of 1 to categories related to objects and 0 all other categories.
produces predicted
To evaluate the 3D reconstruction surface quality, we use the Accuracy, Com- pleteness, Precision, Recall, and F-score with threshold of 5em. The definitions of 3D reconstruc- tion metrics are shown in Tab. aa). 'P and Pd are the set of points sampled from the predicted mesh and ground truth mesh. We use four standard metrics evaluate the rendered normal quality, and definitions of normal evaluation metrics are shown in Tab. bb.. n and n* are the predicted normal vector and ground truth normal vector.
Evaluation Metrics
A.2 ADDITIONAL METHOD DETAILS
Training step We train ODODSDF in two phases: Phase holistic surface learning step, incorpo- rating Chot de Le + An Lu Aesk Lesks and Phase object surface learning step, adding Lovi >22d.., L2ddny Aadnas Lidwng + Ares Lref, with an -gFddddded sampling the later stage. The final loss function is Lnot Lobj-
-oob--
12

---

Published as a conference paper at ICLR 2024
Vanishing Gradient Problem In this section, explain the vanishing gradient problem that exists in SDF-based neural representation. The explanation motivated by ISD (Zhu et al. 2023) which is based VolSDF (arri et al. 202I). Unlike the paper, model is based on NeuS Waag
et al. 2021), so we demonstrate the gradient vanishing problem on the NeuS based method.
Suppose the loss for the view-dependent color is function 2 = c(p(d(x))), where is the density and x(x) is a signed distance function (SDF) for point x. The derivative of & with respect to x can
02 de Be 0d (Zhu et al. 2023). The following derivation is to show the derivative of L be written as de 0d öx
vanishes rapidly point x is getting far from the surface.
According to NeuS, for p(x) O, density is defined as
dd, (d(x))
p(x) = dx
(6)
(d((x))
where
L (d((x))) 1 1 s
dd, (d(x)) can be computed by dx
04. 0d
se (x)(x) Od (e- (x)) + 1)2 0x
Od 0x
Note that d follows the characteristic of SDF by Eikonal loss that,
ad 0x
~ 1
(9)
Hence,
d
3e -x.d(ss))
(d(x))
(10)
dx
(e se=xx((x) + 1)2
Therefore, if express p(x) in terms of d(x),
se ex)((x)
(ed()) +
86 sed(((
p(x)
(11)
e=xi((x)
1 1
In this case, the partial derivative of p with respect d can be calculated
Op 8² d)d(()
(12)
ad (e>d(()) 1)
Standard deviation 1/s is trainable parameter that approaches to zero as the network training converges (Wang al. 2021) In other words, is trained with sufficiently large value.
Methods
Definition
Methods
Definition
Accuracy
meanper (ming- p- |lp p' II)
Mean
Σ cos-1
[nnn"] [alla*
Completeness
meang" ep (minper ||p p=
Median
median
cos
[nllar
Precision
meaupep (min ep- p 0.06)
RMSE
A E(ros 2
Recall
mean, e (minpep p 0.05)
[alla
F-score
Precicion Ttreall
deg
A (n.
cos
deg")
(a) Evaluation Metrics for 3D Reconstruction Result
(b) Evaluation Metrics for Rendered Normal Result
Table 4: Evaluation Metrics
13

---

Published as a conference paper at ICLR 2024
0p
0d
-1.0
-0.5
0.0
0.5
1.0
d(x)
Figure 10: Graph of 0p/Od with respect d(x)
Uniform Sampling
Importance Sampling
d(x)
(x)
o8f(x)
NeuS
Top View
H2O-SDF w/o OGS
H-O-SDF
(a) Indoor Scene Scenario
(b) Point Sampling Comparison
Figure 11: Illustration of (a) simplified indoor scene scenario, and (b) comparison of point sampling schemes between the previous method and ours. OGS represents OSF-guided sampling strategy.
The graph of 0p/0d is shown in Fig. 10 and as increases, the width of the peak in the graph narrows. As result, as d(x) gets larger, which means the point x becomes far from the surface, 0p/öd becomes near zero rapidly. It indicates the gradient of loss vanishes fast with increasing d(x).
Object Surface Field Guided Sampling Strategy Details In this section, we describe in detail about object surface field guided point sampling strategy with an example. Fig. IT]a) shows simplified indoor scene scenario in which multiple objects and room layout exist. In this case, ray passes by a green object while hitting an orange object and a room layout (gray).
In the previous methods like NeuS, importance sampling is performed based on volume rendering weight (w = T a) that can be derived from density. However, in this case, we observe the unnec- essary weight generated near the green object surface (See Fig, [T(b) NeuS). It can be explained as a bias problem caused by inconsistency between the volume rendering weight and the SDF implicit surface. Inconsistency happened because there is lack of constraint between the color field and
14

---

Published as a conference paper at ICLR 2024
the geometry field (Chen et al. 2023). As result, due to the bias problem, unnecessary points sampled while thin structure (orange object) is ignored.
However, as shown in Fig. (Tbb) H3O-SDF wo OGS, the introduction of the object surface field (OSF) shows its efficacy in alleviating the bias problem. The OSF learns a tight SDF surface and provides 3D supervision to the object surface and empty space, acting regularizer to geom- etry and volume rendering weight. Fig. S also demonstrates this effect, showing a reduction in unnecessary weight on the empty space.
We observe that the density-based sampling strategy still slightly over-emphasizes the already well- reconstructed room layout regions in the initial stage. To address this issue, propose an OSF- guided sampling strategy to prioritize object surfaces exhibiting more intricate structures compared to room layouts. The probability of sampling computed based on w(x) ox)(x), where is spatial position. As illustrated in Fig. bTbb H2O-SDF, this strategy enhances the effectiveness of the OSF, inducing more densely sampled in regions where object surface probability and volume rendering weight are both high. Especially, it makes the learning process more focused on OSF and SDF collaboratively guiding each other, thereby improving the process of SDF capturing high- frequency detail.
Refinement Loss Function Details In this section, we provide more details of the refinement func- tion loss. The point mask (x,x) in Eq. 5 is defined as:
1, 0,
osf(x,) 2 oxf(x, <
(x()))
(13)
where 0 is hyper- parameeer to estimate whether input points belong to an object surface or not, and we set = 0.5.
A.3 ADDITIONAL EXPERIMENTS
Curves of Standard Deviation We additionally provide the standard deviation curve of the SDF during the learning process. For training the SDF network, NeuS introduces the probability density function to apply the volume rendering method and is defined as follows: (x(x)
se /1= ==x)² Here, 1/s represents the standard deviation of (s(x). During the network training converges, 1/s gradually decreases and tends towards zero, indicating a higher level of certainty and accuracy in the learned SDF representation. As shown in Fig. [2] our method ex- hibits near-zer zero standard deviation values compared to NeuS with normal and segmentation priors. This visualization result demonstrates that our method provides robust geometry cue, enabling the effective learning of SDF.
Standard devistion (1/s)
0.020
Standard deviation (1/s)
0.020
Standard deviation (1/s) (//))
0.0025
0.015
0.015
0.0015
0.000
0.
0.00342
0.010
0.00109
0.00026
0.005
0.005
0.0005
50k
100k
150k
50k
100k
150k
50k
100k 150k
a
(b)
(c)
Figure 12: Curve of Standard Deviation Illustrations of (a) NeuS with normal priors, (b) NeuS with normal and segmentation priors, and (c) our method.
Additional Analysis on the Object Surface Learning To further analyze the effectiveness of our method, we provide additional experimental results that specifically focus on large object regions and layout regions pass through by empty space regions between multiple objects (See Fig. 13). As illustrated in Fig. [)),), NeuS with normal prior exhibits inconsistencies between the SDF im- plicit surface and the rendered surface. This inconsistency suggests the presence of a bias problem, highlighting the fact that simply incorporating geometry priors guide the SDF representation does
15

---

Published as a conference paper at ICLR 2024
b C
a b C
0.15
0.15
0.05
0.05
0.05
Distalpp
0.2
0.6
0.25
0.15
0.10
0.05
0.05
0.05
d
Distance
Distanco
Distance
0.0
(a) Reference Image
(b) NeuS Normal
(c) H3O-SDF w/o OGS
(d) H2O-SDF
Figure 13: Additional Analysis on the Object Surface Learning We followed the same visualiza- tion approach used in Fig. In the case of the 1st row, the ray hits the surface of the sofa and the floor. In the case of the 2nd row, the ray passes through empty space between multiple objects while hitting the surface of the floors.
HSL
OSL
Accu...
Comp
Prec.
Recallf
F-score
NeuS
Normal Reweight
2D
3D
Ref
OGS
0.132
0.114
0.376
0.334
0.353
0.040
0.042
0.794
0.742
0.766
0.041
0.041
0.802
0.751
0.776
0,040
0.041
0.807
0.755
0.779
0.040
0,041
0.809
0.760
0.784
0.039
0.041
0.817
0.762
0.787
0.037
0.039
0.830
0.773
0.800
Table 5: Comprehensive ablation study of each component in H₂O-SDF.
not guarantee the accurate learning of the true SDF In contrast, illustrated in Fig. 13 (c) and (d), our method successfully builds the appropriate connection between the SDF implicit surface and the rendered surface. These results demonstrate the capability of object surface learning to accurately learn and represent the true SDE, surpassing the limitations of using geometry priors alone.
Ablation Study In this section, we report more detailed ablation study results for individual com- ponents in HDO.SDF. The notations used the table are as follows: (1) NeuS Base model, (2) Normal: integration with normal prior, (3) Reweight: re-weighting scheme for Holistic Surface Learning defined Sec. 3.1 (4) 2D: 2D object surface loss L22...r- (5) 3D: object surface loss C3dargs (6) Ref: refinement loss Cref and (7) OGS:
Object Surface Field guided sampling strategy. performance with the integration of each compo- nent within the framework. This incremental improvement highlights the distinct effectiveness and contribution of each component to the overall model's I's efficacy.
Tab. 5 shows the progressive enhancement
Normal Predictions Fig. 14 and Tab. 6 present additional results of normal prediction, which compare the proposed method with other state-of-the. neural implicit surface learning methods. As shown in Fig. 14. our method exhibits superior normal predictions with smoother and more
16

---

Published as a conference paper at ICLR 2024
Reference Image ManhattanSDF
NeuRIS
MonoSDF
HelixSurf
H₂O-SDF
Ground Truth
Figure 14: Additional Normal Prediction Results ScanNet
Methods
Mean
Median
RMSEJ
11.25 o
22.5 º
30 *
ManhattanSDF
19.14
10.82
29.43
52.86
74.4
80.96
HelixSurf
19.3
11.92
28.87
47.19
74.43
82.16
NeuRIS
15.21
8.77
23.19
61.46
79.76
85.42
MonoSDF
14.36
7.23
23.39
66.26
80.82
85.81
H:O-SDF
13.48
6.96
21.73
66.94
81.82
86.90
Table 6: Quantitative Normal Comparisons on ScanNet
precise geometry. Moreover, Tab. Glalso demonstrates that our approach outperformed the existing methods in all normal metrics.
Reconstructions this section, we provide additional qualitative and quantitative results the selected 12 scenes on ScanNet. Results in Fig. 15 show that our proposed method achieves more smooth and fine-grained surface quality than other methods. According to the quantitative results (Tab. 7), method outperforms the existing methods. However, for the scene0603 and scene0616, our method performs slightly worse compared to NeuRIS (Wang et al., (202a)) and MonoSDF (Yu
et al. 2022). This discrepancy be attributed to the absence of ground truth meshes in certain
17

---

Published as a conference paper at ICLR 2024
ScenelfM22.0 (X0
Soenerk01 2,10
Melbod
Acc
Compl
Prec1
Recall
Fascore
Accl
CompL
Prec
Recall
E-score
ManhatianSDF
0.070
0.665
0.608
0.048
0.736
0.707
NeuRIS
0.032
0.046
0.850
0.726
0.783
0.025
0.029
0.920
0.867
0.893
MonoSDF
H₂O-SDP
0.029
0.046
0.873
0.740
0.801
0.024
0.028
0.930
0.875
0.901
Scene0050.00
SoeneC084L00
Method
Acc1
Compl
Prect
Recall?
F-seoref
Acei
CompL
Proef
Recalff
F-score"
ManhattanSDF
0.058
0.059
0.707
0.642
0.675
0.055
0.053
0.639
0.621
0.630
NeuRIS
0.043
0.752
0.695
0.695
MonoSDF
0.041
0.053
0.717
0.625
0.668
0.038
0.030
0.856
0.882
0.870
H₂O-SDF
0.03
0.036
0.856
0.818
0.027
0.029
0.893
0.855
SceneO1 14.0
Secne 00
Metbod
Acc|
Compl
Prect
Recall?
F-scoref
Acci
Compl
Precf
Recallf
F-score7
ManhattanSDF
0.153
0.083
0.513
0.503
0.308
0.105
0.068
0.577
0.588
0.583
NeuRIS
0.063
0.060
0.635
0.600
0.617
0.066
0.057
0.685
0.651
0.668
MonoSDF
0.202
0.469
0.642
0.636
H₂O-SDF
0.063
0.057
0.704
0.682
0.693
0.043
0.042
0.789
0.734
0.760
Scene0580.00
Soenc0603.00
Method
Acc1
Comp1
Prect
Recall?
F-scoref
Acc]
Compl
Precf
Recallf
F-score
ManhattanSDF
0.104
0.062
0.616
0.650
0.632
0.143
0.066
0.583
0.620
0.601
NeuRIS
0.065
0.052
0.671
0.667
0.669
0.039
0.049
0.800
0.693
0.743
ManoSDF
0.0381
0.048
0.756
0.700
0.725
0.154
0.089
0.503
0.494
0.499
H₂O-SDF
0.784
Scene0616..0 00
Sceme06 17.00
Method
Acc.
Comp.
Prect
Recall?
Escoref
Aoc]
Comp.
Prect
Recalff
F-score"
ManhattanSDF
0.072
0.088
0.521
0.431
0.472
0.132
0.051
0.506
0.646
0.568
NeuRIS
0.048
0.057
0.712
0.614
0.639
0.067
0.057
0.731
0.695
0.713
MonoSDF
0.077
0.493
H₂O.SDF
0.040
0.051
0.730
0.615
0.667
0.049
0.047
0.771
0.726
0.748
Scene0625
Scene0721
Method
Acc.
Compl
Preet
Recal
F-score?
Acc|
Comp.
PrecT
Recalff
F-score"
ManhattanSDF
0.056
0.076
0659
0.613
0.635
0.088
0.055
0.675
0.640
0.657
NenRIS
0.027
0.025
0.885
0.882
0.883
0.043
0.041
0.785
0.734
0.759
MonoSDF
0.031
0.036
0.814
0.785
0.799
0.046
0.047
0.723
0.665
0.693
H₂O-SDF
0.019
Table 7: Quantitative Comparisons of Individual Scenes on ScanNet For a fair comparison, we used the mesh provided officially by ScanNet as the ground truth. Therefore, the official results of each model may slightly differ from the results we measured.
Time
ManhattanSDF
NeuRIS
MonoSDF
H₂O-SDF
Training (hours)
5.2
4.5
21.5
4.5
Inference (seconds)
30
21
35
21
Table 8: Comparison of training and inference time
areas, as shown in Fig. 15 (see 5th and 6th row). Moreover, demonstrated in the qualitative results Fig. 15 our method exhibits better reconstruction results for both scenes as well (see 5th and 6th row).
Training and Inference Time In Tab. 8 present comparison of training and inference times against baseline models. Our model demonstrates comparable speed to NeuRIS, outpacing the other models in terms of efficiency.
Object Mesh Extraction To demonstrate the potential applications of our trained object surface field (OSF), we extracted object mesh from our final reconstruction (See Fig. [66]. To extract object mesh we find the intersection of the signed distance function (SDF) and OSF where the SDF is near the zero-level set (ld(x)) < 0a) and OSF is above certain threshold (osf(x)
0.sf). Then, the
intersection of the SDF remains and the rest of the SDF filtered out. Then the mesh extraction
from SDF is accomplished using the Marching Cubes algorithm (Lorensen & Cline 1998). The results show the effectiveness and possibility OSF extracting object meshes.
18

---

Published as a conference paper at ICLR 2024
ManhattanSDF
NeuRIS
MonoSDF
H₂O-SDF
Ground Truth
Figure 15: Additional 3D Reconstruction Results on ScanNet
19

---

Published as a conference paper at ICLR 2024
(a)
(b)
(c)
(d)
(e)
Figure 16: Object Mesh Extraction The results show the object meshes extracted from 5 ScanNet scenes.
20


Reviews:
[{"review_id": "mfpayFMCeT", "reviewer": "Reviewer_93pm", "summary": "This paper proposed a two-phase framework (H2O-SDF) for 3D indoor scene reconstruction. In particular, the proposed method adopts a two-stage method, which consists of one-stage reconstruction for the scene layout followed by a second-stage reconstruction of the objects using NERF. The key contribution is to introduce the concept of the object surface field. The 2D and 3D object surface losses are introduced to estimate the SDF for fine object surface details. The experiments are conducted on ScanNet and show superior results compared with existing methods.", "questions": "- It would be great to elaborate more on the insight of OSF and also compared with existing density functions parameterised with the SDF values. \n- The ablations studies are missing. Please demonstrate all the losses introduced in stage two all contributes to the improvement of the reconstruction. In addition the proposed method leverages the point cloud obtained from MVS. It would be great to show how these priors can influence the final performance.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The method reconstruct the layout and the object separately and achieves very good reconstruction on the details of the objects.\n+ The introduced OSF captures the occupancy of the surface of the 3D object.\n+ The introduced two losses let the SDF captured more surface details.", "weaknesses": "-\t2D object surface loss. Could it be explained as the loss between the rendered object masks and the ground truth masks? It would be great to make it clear that the proposed method actually requires object annotations.\n-\tIt would be great to explain OSF with more details. Based on the description in the paper, it is quite similar to the absolute gradient field of the occupancy values. In particular, Eq. 3 actually enforces the OSF to have large values on the object defined by the 3D points.  In addition, 3D points provide strong prior on the details of the shapes. It would be great to provide the ablations study of using the point cloud with MVS images or not.\n-\tExperiments on ablations studies. It is not clear to the reviewer what model A, B, C are. It would be great to provide detailed explanations about those models.\n-\tFor the second stage, it would be great to ablate whether all the losses have contributed to the final results. The proposed method adopts more accurate point cloud obtained from MVS images compared with monocular depth estimated from a single image. Those factors should be ablated to demonstrate the performance benefits from OSF and the sampling strategy not from the prior data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0bgFJ3cWE7", "reviewer": "Reviewer_ypMg", "summary": "This paper proposes a two-phase learning approach named H2O-SDF that combines both holistic surface learning and object surface learning, for 3D reconstruction in indoor environments. \n\nThe main contributions are: 1) a two-phase learning framework that balances between the reconstruction of global room geometry and local object details. 2) Introduction of Object Surface Field (OSF), a new concept designed to address the vanishing gradient problem suffered by SDF, which hinders the reconstruction of high-frequency details. The authors also introduce an OSF-guided sampling strategy to prioritize object surfaces in the sampling process.", "questions": "1. It would be interesting to find out to what extent this method relies on pre-trained models and priors, which might limit its application in environments where such models are not easily available.\n\n2. Running time/computational complexity of the proposed method. Please refer to point 2 under Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper tackle an important issue in the field of 3D indoor scene reconstruction — the difficulty of preserving the overall geometry while capturing intricate object details. It introduces a two-phase learning approach, which has not been explored before. \n2. The OSF concept is new and shows promising results in handling the inherent vanishing gradient issue in the learning process.\n3. It is an interesting idea to use normal uncertainty as a guidance to re-weight normal and color loss, to adaptively moderate normal and color losses in both low-texture and texture-rich regions.\n4. The submission appears to be well-organized with its ideas clearly articulated.\n5. Experimental evaluations, together with ablation studies, confirm the effectiveness of H2O-SDF. The results show that the proposed solution outperforms existing state-of-the-art methods.", "weaknesses": "1. The explanation and exposition of some key, novel concepts, such as OSF, L2D_OSF, L3D_OSF, could be more thorough. There is insufficient mathematical detail on the OSF guided sampling strategy (although there is graphical illustration in the appendix A2, the explanation seems to be mostly a repetition of the main body). Strengths of the proposed formulation could be better appreciated by providing more detailed explanations and mathematical insights. \n\n2. Running time: The paper does not provide specific details about the computational complexity or running time of the approach, for both training and inference.  It only states that all experiments were conducted on a single NVIDIA RTX 3090Ti GPU.\n\n3. Comparison with more diverse data (this is more of a suggestion): While the paper compares favorably to state-of-the-art methods on the ScanNet dataset, it would strengthen the paper to include a broader range of data under different conditions, such as different indoor layout complexities, object variations etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qr13yOjDv6", "reviewer": "Reviewer_rsDk", "summary": "The paper proposes a neural 3D indoor reconstruction framework to reconstruct 3D mesh of indoor scenes with a volume rendering framework. The key motivation of this paper is to decouple the learning of the layout and object with two stages. In the first stage, the layout of the scene is trained with an uncertainty-aware rendering loss function on both color and normal prediction. In the second stage, a new term named Object surface field (OSF) is introduced to measure the object occupancy of a 3D point, and authors demonstrate how SDF will facilitate SDF with the presented mutual induction. Extensive experiments on ScanNet have showcased the effectiveness of the proposed framework over different state-of-the-art (SOTA) methods.", "questions": "I appreciate the motivation of the design of this paper, however the use of a model seen on the same dataset limits the value of the proposed method. I would consider to improve my rating if my concerns listed in the weaknesses part can be well addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) The motivation to decouple the learning of layout and object into two stage is straightforward and clear. The layout contains planar areas and the objects may have more high-frequency signals, thus may have different pace of convergence.\n\n(2) The introduction of OSF is novel, and how the OSF can be transformed back to SDF and assist its representation is technically sound.\n\n(3) Experiments on ScanNet have shown the advantages of proposed components of the method.", "weaknesses": "(1) The major concern for me is that of the technical impact of this work is limited by introducing a normal estimation network [1] which is also trained on ScanNet, to provide pseudo groundtruth normal and uncertainty during training. This cannot ensure fairness among baseline comparison and highly constraints the generalizability of the proposed method onto different benchmarks. A fair setting would be replace this network with another model or method which is pretrained on other datasets, or alternatively, test this method onto other indoor datasets such as 7-Scenes. This will significantly improve the fairness and technical impact of this work.\n\n(2) In the supplementary material, authors present that they apply the OSF-based Filtering during reconstruction. I am curious about where does the major improvement of OSF comes from, either the proposed osf loss or the filtering. Authors are expected to conduct ablation study about this to make the contribution more convincing.\n\n(3) Minor: The presentation can be further improved, and there exists noticeable typos in the submission such as Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w1bRSTXEJa", "reviewer": "Reviewer_ZNfu", "summary": "The paper proposes follow-up method to the SDF-based NeRF-like methods for indoor reconstruction, with a focus to improve geometry on objects in the second phase of optimization. The main novelty is the introduction of the auxiliary representation of Object Surface Field (OSF), which is activated on object surfaces. OSF can learned with 2D supervision of instance segmentation, as well as a loss in 3D which jointly constrain the SDF field and OSF field, bringing about zeros of SDF around object surfaces, leading to improved reconstruction on detailed high-frequency object parts. The method is evaluated against baseline SDF-based NeRF-like methods, on scenes including ScanNet and Replica.", "questions": "Please see the Weakness section for questions to address.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "[1] The proposal of using instance segmentation as additional input to the pipeline, as well as designing effective supervision signals with input segmentation. \n\nDespite it is not new to use additional signals to the task (e.g. monocular normal and depth supervision for 2D-3D consistency, sparse points to supervise local SDF values, and using semantics and planar assumptions improve geometry of layouts), the paper is one of the first to demonstrate the usage of instance segmentation to improve fine geometry. More importantly, the paper does so in a non-trivial way, by introducing OSF to explicitly evolve object surfaces using a 2D loss between the input segmentation and OSF as well as a 3D loss between OSF and SDF.\n\n[2] Illustration of the relationship of OSF and SDF (and the gradients), and the use of the OSF to drive SDF.\n\nThe paper provides informative illustration in Fig. 4 and related text on the relationship of OSF and SDF and how does the optimization of OSF loss drive SDF towards zero point around surfaces. The illustration using examples and 1D figures is clear and supports the motivation of the design of OSF.\n\n[3] Extensive evaluation of the proposed method against baseline methods, and on more than one datasets.", "weaknesses": "[1] Clarification on OSF. Despite the good illustration of OSF as mentioned above, extra clarification is urgently needed to explain the motivation of the mathematical form of the 3D OSF loss (Equ. 2), and details in Fig. 4.\n\nSpecifically, despite Fig. 4 explains how the gradients of the 3 loss drives SDF to form a zero points around surfaces, and paper does not provide intuitive explanation on (a) why the various terms of the loss in Equ. 2 are designed as they are, (b) how \\gamma controls the steepness of the function, how it matters and how $\\gamma$ is picked (better with illustrations similar to Fig. 4). Additionally, it is not clear that, between Fig. 4 (a) and (c), why different d(x) lead to identical $\\sigma_\\gamma(x)$. Without clarifying the issues it is difficult to understand why OSF and the losses are designed the way they are, despite being proven effective.\n\n[2] Demonstration of applying the proposed OSF and losses general SDF-like NeRF-based methods. The proposed OSF and losses should theoretically be applicable to all of the baselines methods as simple drop-in, but somehow the paper decides to compare against its own vanilla baseline. Is it possible to apply to other existing methods to better showcase the general nature of the proposed method, and how effective will it be?\n\n[3] Limited scenes to evaluate. The main evaluation is done on ScanNet, with limited qualitative results on Replica. However ScanNet is known to have image quality issues. Why is the method not evaluated and compared on alternative datasets including Replica, Tanks & Temples, etc, as is done in other papers like MonoSDF? Without the additional evaluation, it is difficult to decide the generalization of the method across various indoor scenes.\n\n[4] Additional comparison. One less important thing to add is, potential comparison with I^2-SDF. Despite I^2-SDF is based on additional geometry supervision signals, the goal is aligned with the proposed method, and it showcases similar improvement in fine detailed objects. It would be beneficial to add comparison to I^2-SDF to inspire the discussion on optimal strategy to improve reconstruction of high frequency signals in indoor geometry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)