NeuralPlane: Structured 3D Reconstruction in Planar Primitives with Neural Fields
Authors: Hanqiao Ye
Yuzhou Liu
Yangdong Liu
Shuhan Shen
Venue: ICLR 2025 Oral (2025)
Keywords: 3D Reconstruction
3D Scene Understanding
Scene Abstraction
Neural Rendering
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: NeuralPlane rebuilds indoor scenes as arrangements of planar primitives from multi-view images.

Abstract: 3D maps assembled from planar primitives are compact and expressive in representing man-made environments. In this paper, we present **NeuralPlane**, a novel approach that explores **neural** fields for multi-view 3D **plane** reconstruction. Our method is centered upon the core idea of distilling geometric and semantic cues from inconsistent 2D plane observations into a unified 3D neural representation, which unlocks the full leverage of plane attributes. It is accomplished through several key designs, including: 1) a monocular module that generates geometrically smooth and semantically meaningful segments known as 2D plane observations, 2) a plane-guided training procedure that implicitly learns accurate 3D geometry from the multi-view plane observations, and 3) a self-supervised feature field termed *Neural Coplanarity Field* that enables the modeling of scene semantics alongside the geometry. Without relying on prior plane annotations, our method achieves high-fidelity reconstruction comprising planar primitives that are not only crisp but also well-aligned with the semantic content. Comprehensive experiments on ScanNetv2 and ScanNet++ demonstrate the superiority of our method in both geometry and semantics.

Published as a conference paper at ICLR 2025
NEURALPLANE: STRUCTURED 3D RECONSTRUCTION IN PLANAR PRIMITIVES WITH NEURAL FIELDS
Hanqiao Ye!.2, Yuzhou Liu Yangdong Liu?, Shuhan Shen 1.2+ Schooo of Artificial Intelligence, University of Chinese Academy of Sciences 2 Institute of Automation, Chinese Academy of Sciences
ABSTRACT
3D maps assembled from planar primitives are compact and expressive in rep- resenting man-made environments. In this paper, we present NeuralPlane, novel approach that explores neural fields for multi-view 3D plane reconstruc- tion. Our method is centered upon the core idea of distilling geometric and semantic cues from inconsistent 2D plane observations into a unified 3D neu- ral representation, which unlocks the full leverage of plane attributes. is ac- complished through several key designs, including: 1) a monocular module that generates geometrically smooth and semantically meaningful segments known as 2D plane observations, 2) plane-guided training procedure that implicitly learns accurate 3D geometry from the multi-view plane observations, and 3) a self-supervised feature field termed Neural Coplanarity Field that enables the modeling of scene semantics alongside the geometry. Without relying on prior plane annotations, method achieves high-fidelity reconstruction comprising planar primitives that are not only crisp but also well-aligned with the seman- tie content. Comprehensive experiments strate the superiority of our method in both geometry and semantics. Project page: https: http//eeaaallglaae ane wig.gtttthh.. mrithub.i..
ScanNetv2 and ScanNet+ demon-
INTRODUCTION
While inferring dense geometry such as volumetric grids and meshes from 2D images has been extensively studied (Schönherger et al., 2016; Murez et al., 2020; Wu et al., 2023b), there is grow- ing realization that reconstructs sparse 3D maps composed of self-contained primitives (Xue et al., 2024; Li al., 2024; Kluger et al., 2024; Liu al., 2024). In this paper, we focus on reconstructing structured indoor scenes as configurations of planar primitives. Among common geometric prim- itives such as line segments and edges, planar primitives are particularly essential for describing man-made environments. They are compact yet expressive parametric entities that provide not only strong geometric cues but also rich semantic information. Therefore, 3D maps represented by planar primitives are notably lean and convenient for practical use, and have been successfully deployed in various applications in areas such as robotics (Zhou et al., 2021; Liu et al., 2023) and augmented reality (ARKit, 2024; ARCore, 2024).
The reconstruction of 3D plane maps is conventionally approached by fitting planar primitives to an unordered point cloud or mesh generated from depth sensors multi-view stereo (Yu & Lafarge, 2022). The input geometries of these methods are typically noisy and incomplete, and provide only limited semantic information for robust plane detection. A recent work by Watson et al. (2024) incorporates optimized 3D semantic embeddings into the plane fitting module, i.e., RANSAC (Fis- chler& & Bolles, 1981), which helps discern adjacent planes. However, it still heavily relies on the quality of input geometry, and requires to train from 2D plane annotations. These prior efforts basi- cally decouple 3D plane reconstruction as two independent problems of non-plane-biasce geometry recovery and plane estimation, which bottlenecks the full utilization of plane attributes. Departing from such geometry+RANSAC paradigm, Xie et al. (2022) developed the first learning- model that incrementally reconstructs planes from posed video fragments. However, though supervised by 3D plane annotations, the method struggles with its adaptability to intricate structures.
"oorrsponniig author.

---

Published as a conference paper at ICLR 2025
Input: Posed images
PlanarRocon
Primilives
(Vew inconstroen)
AirPlanes
Plane-Guided Neural Fields Learning
Neural Coplmarity Field
Neanal Parer
Grouping
Ours
RANSAC
Density
Reference
(a) Method Overview
(b) Showcasing Results of 3D Plane Reconstruction
Figure Structured 3D reconstruction in planar primitives with neural fields.
Motivated by previous limitations, as well as the success of neural fields in sparse geometry encod- ing (Ye et al., 2023; Xue al., 2024; Li al., 2024) and 3D segmentation (Ying al., 2024; Kim et al., 2024), we introduce new approach named NeuralPlane. The proposed method maintains consistent neural representation for environmental plane structures from multi-view inconsistent plane observations, which further enables the extraction of high-quality 3D planar primitives.
Fig. (a) provides an overview of our pipeline, which unfolds into three phases: 1) Firstly, we combine two robust vision models into training-free module capable of excavating well-defined plane segments from a single image. Meanwhile, plane parameters of these segments are coarsely estimated, yielding set of locally initialized 3D entities referred to as Local Planar Primitives. 2) Secondly, we design a plane-guided training procedure where the high degree intra-primitive ge- ometric regularity is explored to optimize the density field. This can effectively mitigate the issue of poor surface reconstruction when encoding the scene geometry by volume density, however, we recognize that the density field alone is insufficient for extracting semantically well-aligned planar primitives. Thereby, we further propose Neural Coplanarity Field, a self-supervised feature field that involves high-level semanties. This field utilizes local planar primitives for inter-primitive reason- ing within a contrastive learning framework, and can accurately capture the coplanar relationships between different regions. 3) Finally, on top of the learned neural representation, two-stage plane extraction process: before geometrically fitting planar primitives, the neural repre- sentation is firstly grouped into isolated regions based on the learned coplanarity features, wherein module called Neural Parser is introduced to facilitate the discrimination of plane instances.
conduct
Fig. (b) showcases the superiority of NeuralPlane over existing state-of-the-arts (Xie et al., 2022; Watson et al., 2024), where our method faithfully rebuilds scene layouts as sets of crisp planar primitives while preserving fine-grained semantics. In summary, our key contributions are:
We present NeuralPlane, a novel approach for multi-view 3D plane reconstruction that eschews plane annotations. enjoys the synergy between geometry and semantics in the context of neural fields and achieves leading performance in extensive experimental studies.
To capture accurate plane locations, self-prompting mechanism that integrates two robust vision models is first proposed excavate 2D plane regions, where strong plane regularity is then applied to guide the optimization of the implicitly encoded geometry.
We enhance plane discrimination by incorporating semanties: Neural Coplanarity Field acts as the core driver of involving the semantic separation via contrastive learning, while Neural Parser introduced to jointly model the learned coplanar relationships.
RELATED WORK
2.1
MULTI-VIEW PLANE RECONSTRUCTION
The basis of 3D plane reconstruction from multi-view images lies in the detection of planar regions from either RGB inputs inferred dense geometries. Despite impressive progress in single
2

---

Published as a conference paper at ICLR 2025
view plane detection (Yang & Zhou, 2018; Liu et al., 2019; Yu et al., 2019), unified multi-view plane reconstruction remains challenging due to severe cross-view inconsistencies. Several attempts have been made to locally recover 3D planes from limited number of images by predicting plane correspondences (Jin et al., 2021; Agarwala et al., 2022; Tan et al., 2023) or stereo matching (Liu al., 2022). However, scaling multi-view scenarios for scene-level reconstructions is still prob- lematic. More relevant to our work, PlanarRecon (Xie et al., 2022), the first learning-based method, reconstructs 3D plane maps by incrementally detecting, tracking, and fusing 3D planes from posed video fragments. Chen et al. (2023) draw upon neural fields to detect planar primitives from RGB- D sequences. AirPlanes (Watson et al., 2024) adopts the conventional two-stage pipeline of fitting planes to non-plane-biased geometry, where the learned 2D plane priors are lifted into 3D for robust plane distinction.
2.2 HIGHER- LEVEL MAPPING VIA NEURAL RENDERING
Neural rendering has emerged a powerful technique that allows distilling various 2D observa- tions from multiple views into 3D neural fields. It has greatly reshaped a wide range of 3D scene reconstruction and understanding tasks. Recent works (Ye et al., 2023; Xue et al., 2024: Li al., 2024) find the neural field favorable for implicitly encoding sparse parametric primitives such as line segments and edges. Object-compositinaal neural representations have also been extensively explored for object-centric mapping (Yang et al., 2021; Yu et al., 2022a; Wang et al., 2023; Park et al., 2024). Besides, there are advances in constructing semantic maps with neural fields by dis- tilling high-dimensional features (Kobayashi et al., 2022; Kerr et al., 2023; Zhu al., 2024) or contrastive learning (Fan et al., 2023; Bhalgat et al., 2023; Kim et al., 2024). Our study falls within the exploration of neural fields for parametric primitives. However, instead of focusing solely on geometry, we further incorporate semantics to build the neural representation at a higher level.
3 METHOD: NEURALPLANE
Given a set of posed images, our goal is to recover the underlying scene structure using a collec- tion of 3D planar primitives. To this end, we first process the input images to acquire monocular plane priors known as local planar primitives (Sec. 3.1). Then, we introduce our neural scene rep- resentation, which is optimized under the plane guidance (Sec. 3.2). Once optimized, the neural representation is made explicit by our plane extraction algorithm (Sec. 3.3).
3.1 GENERATING LOCAL PLANAR PRIMITIVES
Essentially, a local planar primitive / = (M C I, πT) is defined as a plane segment M of a single image 1, associated with plane parameters π [n, o under the world space. Here, n E R3 and E R are respectively the unit normal vector and the offset of the plane from the origin. Although remarkably advanced single-view plane recovery models (Tan et al., 2021; Shi et al., 2023) can be employed to estimate such local planar primitives, we will show that purely local yet robust image priors are sufficient to achieve satisfactory performance, which eliminates the need to deliberately train a model on plane annotations.
Plane Segments from Monocular Priors. We assume the plane segment M of a local planar prim- itive to be an image region that (1) has consistent normal directions and (2) is geometrically contin- uous. To excavate such plane segments, we draw inspiration from Mazur et al. (2024) repurpose a pre-trained monocular normal predictor (Bae et al., 2021) and the powerful Segment Anything Model (SAM) (Kirillov et al., 2023). Specifically, we start by applying K-means clustering (Lloyd, 1982) to the predicted normal map. obtaining low-frequency regions with each having similar nor- mal directions. Subsequently, SAM's automatic mask generator is employed to further segment these smooth regions by selecting the smallest mask for each prompt. This sver-segmentation strat- egy produces semantically meaningful segments, while approximately enforcing the geometric con- tinuity. Finally, masks exceeding a specified size threshold are chosen the desired plane segments.
Initializing Local Plane Geometry, For each local planar primitive P. we initialize its plane normal with n, which denotes the average of predicted normals within the plane segment. To estimate the plane offset o, we propose to use sparse 3D keypoints that are often available as a by by-product of centturefromm-oone (SfM) systems. Specifically, the plane offset initialized by minimizing the
3

---

Published as a conference paper at ICLR 2025
Coplanarity Feature: plp)
(a) Geometry
es
Network
Density: a(p)
000
Color: c(p,
Pa
Crgh
Volume Rendering
Pig
Explicit Normal Regularizatien for candure sigles of rigs Gorena
(b) Semantics Branch
pull
Pecode-eepti Supersision
Gruk Neural Parser
Chureene
f
Rigartite Matcinng
shared
Phendnrllephh Duar MC) MID 5)
Rendering PDB 1210 700000
Figure Plane-guided neural fields learning. The overall training scheme consists of two branches: (a) the geometry branch (Sec. 3.2.1) translates plane constraints into intra-primitive regularization terms, i.e,, Lnoemal and de-dephi (b) the semantics branch (Sec. 3.2.2) performs inter- primitive reasoning via contrastive learning, and introduces query-based learning module (Neural Parser) to jointly model the captured coplanar relationships.
total distance to the 3D keypoints {Pı...., Pm detected in M:
0 = arg min
Wi
d2. where d, = || Ps + v112. w/
veR
The weight u, = 1/a. indicating the reliability of p., is approximated using the average reprojection error é; across views where p; visible. Although error-prone, the results are sufficient provide effective geometric guidance for our neural representation, which will be detailed below.
3.2 LEARNING NEURAL REPRESENTATION UNDER PLANE GUIDANCE
Note that local planar primitives are generated with minimal dependence on viewpoint, leading to severe inconsistency across views. We acknowledge that is non-trivial to explicitly establish cor- respondences and merge them into consistent 3D plane reconstruction. As a consequence, we pro- pose fuse them implicitly using neural fields. The overall training scheme is illustrated in Fig. 2, which consists of two major branches: 1) the geometry branch (Sec. 3.2.1) focuses learning ac- curate plane locations, while 2) the semantics branch (Sec. 3.2.2) captures coplanar relationships between different regions from conflicting 2D plane segments.
3.2.1 GEOMETRY BRANCH
NeuralPlane implicitly encodes geometry volume density. To alleviate the well-known shupe-radiance ambiguity (Zhang et al., 2020), we translate strong plane constraints provided by prepared local planar primitives into regularization terms.
Preliminaries. Standard NeRF (Mildenhall et al., 2020) represents a scene as a continuous volume density function R + [0, 1] and continuous view-dependent color function (R3,
[0, 1J3. According to the volume rendering formula, given a ray emanating from camera center o. with direction v, its expected termination point computed by integrating N sampled points {p: Oe tz = 1,...,N.t, < t++1} along the ray:
N
h(w),, h(l.) = T(1))((4) = [Ia-a(())) all:).
The piccewise-consse function h(t,) is approximation of the rendering probability density func- tion (PDF) to weight the contribution of each sampled point. The opacity value a(t,) is computed (1.) exp(-o(p:)A/).
4

---

Published as a conference paper at ICLR 2025
Explicit Normal Regularization. To ensure normal consistency on planar surfaces, normal regu- larizer Loorall is introduced to penalize the deviation of the NeRF-derived surface normal á from n (initialized in Sec. 3.1). We estimate ñ in algebraic manner, using triplet of rays sampled from the plane segment MM. For each ray triplet T {ra(t) = Og + tw: [ randomly drawn from Pa. render its expected termination points { by eq. (2), and estimate the normal of the plane passing through:
sign(s v1)
where (x22 21) x (23 21).
||
Here, the direction of n is adjusted point towards the camera (same no)) by checking the angle between & and U1. Then, the normal loss with respect to Pa is defined as:
narraal (o; Pa) = Ey ||1 - á T na
(4)
Pseudo-Depth Supervision. While the normal regularization induces smoothness in planar regions, alone is insufficient for achieving high geometric accuracy. To further guide the density field learning, we introduce plane-derived pseudo-depth as extra supervision. The pseudo-depth D is computed as the ray distance from the camera centre o, to the corresponding P: D = -oo + o cos s. where ip the angle between n and the ray direction v. Since defectively estimated n and o lead to noisy D, we follow DS-NeRF (Deng et al., 2022) to model the depth label of ray as a random variable normally distributed around D with variance g=1; each ray drawn from Pa. we minimize the KI divergence between the pseudo-depth distribution (1; D7, 3-1) and the piecewise- conseatt rendering PDF (1):
(t; D.88"),, For
Cp-eeph(o, n,; Pa) E,.-P. DKr. [N(t; D-
(5)
In addition, enable gradient descent over the estimated plane parameters T. (i.e.. Th and o.), using gradient signals from Lp-depch to jointly refine the inaccurate local geometry.
3.2.2 SEMANTICS BRANCH
As highlighted by AirPlanes (Watson al., 2024), semantics are crucial in 3D plane reconstruction since ideal planar primitives need to be semantically self-contained. For instance, though geomet- rically coplanar, a closed door should be regarded as a different planar primitive from the wall that encloses it. However, it is intractable for purely geometric methods, e.g. RANSAC, to resolve such semantic conflicts. To address this, we retrofit the neural field with plane-level semantics.
Neural Coplanarity Field. We start by introducing Neural Coplanarity Field (NCF) Φ R33 + Rd. which outputs a d-dimensional feature Φ(p) over any 3D location p. The coplanarity between two rays is defined as the probability that their expected termination points lie the same plane struc- ture. Here, we aim to measure such coplanarity us- ing the similarity between their rendered features, which we call the coplanarity feature: f(x) MN h(p.))
We train NCF through the margin-based contrastive learning paradigm (Chopra et al., 2005) where the objective comprises two terms: the intra-primet primiiiieee pull loss Cpall and the inter-primitive push loss Lpush-
Considering two rays from the same local planar primitive Pa. e.g., Ti and rj. their coplanarity fea- tures constitute pair of positive samples and are pulled close by minimizing the 2 distance:
(a) Ref. View (b) Ground Truth (c) Rendered
Figure 3: Visualization of NCF
Cqun (p: Pa) E (m - f fm,;))12.
Next, we define the primitive-level coplanarity feature of P. as: f. fr-P.(())) The push loss Epush is computed over them, as find that it can reduce computation overhead and accelerate
5

---

Published as a conference paper at ICLR 2025
convergence. Note that local planar primitives generated through over-segmentation, and Cpush is computed across views, which indicates that two primitives in a batch do not necessarily belong to distinct planar structures. This ambiguity prompts us incorporate geometry: fa and f. form pair of negative samples only if the corresponding plane parameters Ta. T. are sufficiently far from each other. More formally, the push loss with margin m is formulated as:
Fpsst ( Pa. Pa) = -olo--o01 or [|,.-nal <t..] Rm - llf - Soll2),
where 1p.1 is the indicator function, and for in are pushing thresholds w.r.t the offset and normal.
As shown Fig. (c), we visualize the rendered feature maps using PCA (Hastie al., 2009) to demonstrate what is learned by NCF. The reference views and ground truth are also provided for comparison. The results shows that the proposed NCF can effectively capture the coplanar relation- ships between different regions with clear boundaries, considering both geometry and semantics.
To obtain instance centroids and subsequently decompose the learned feature field into instances, one can apply unsupervised clustering methods to thousands of rendered features af- ter training (Bhalgat et al., 2023). However, instead of this post-processing, we propose a learning- based module called Neural Parser concurrently learn of semantic prototypes during training. Each prototype is assumed to be the centroid of an isotropic Gaussian distributed cluster of copla- narity features, which represents semantically isolated subregion.
Neural Parser.
As illustrated in Fig. 2, we are motivated by NEAT (Xue et al., 2024) to adopt a query-based ar- chitecture, where Np semantic prototypes {p; HR,,, are predicted from N2 Icarnable queries (q: E ] (randomly initialized) through simple feed-forward network (FFN) 0 Rd, - Rd. During each training iteration, the rendered primitive-level features {f) are utilized to supervise the module. Specifically, we first apply DBSCAN (Ester et al., 1996) to these noisy features to obtain Ne centroids {u, E Rd), N. Then, the bipartite matching is performed between the predicted pro- totypes {pi)N, and centroids {MiINN using efficient Hungarian algorithm (Jonker & Volgenant, 1987). The cost matrix is defined as the L distance between each pair. With the optimal permuta- tion of assignment t determined, our objective is to keep prototypes apart while further minimizing the distance between each centroid and its matched prototype:
Cap 0 (q): = Σ i
ReLU(m - llp, -Pill)) + ma P7(1)12,
where min(N No), p; 0(g:).
Overall Training Scheme. The NeuralPlane framework is trained by assembling all loss terms:
L = (a)) + EP,, [A1 Enoemal (0; P.) + A2 C2 (o, π& P.) + A>a P.))
+ E (P pppss (c; P,. P,)] + Cap (0, (q): [p]),
where Ligh is the standard NeRF photometric loss, and the set P denotes all local planar primitives. We set the balancing parameters Å₁ X2, Xs to 0.01, 0.1 and 0.5, respectively.
3.3 EXTRACTING GLOBAL 3D PARAMETRIC PLANES
Once the implicit neural representation is optimized to encode both the geometry and semantics of the scene, straightforward 3D plane extraction method is employed make the representation explicit. The process involves five steps: (1) render sufficient termination points and their copla- narity features from training views; (2) assign the label of the nearest semantic prototype to each point based on its rendered coplanarity feature; (3) group these points according to their labels; (4) parametrically 3D planes to each point group using RANSAC; (5) project the inliers onto each fitted plane and triangulate the rasterized projection into mesh, which represents the final planar primitive. We refer the reader to Appendix A. for more details.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Datasets. To evaluate NeuralPlane's s ability to reconstruct 3D plane maps, we conduct experiments on 12 challenging real-world indoor scenes: 8 scenes from ScanNety2 (Dai et al., 2017) and scenes

---

Published as a conference paper at ICLR 2025
Reference
PlanarRecon
SimpleReecon+Aill AirPlanes
MomaSDF+RANSCC
NeuralPlane
Reference
PlanarRecon
FineRecom+
ObjectSDF++ +RANSAC
NeuralPlane
Figure 4: Qualitative results on ScanNety2. For each scene, the first row presents the top view of the entire room, and the second displays the details viewed from given pose. Only the best two geometry+RANG, RANNSC methods are included for comparison. NeuralPlane reconstructs clean planar structures with fine details and coherent semantics.
from ScanNet++ (Yeshwanth et al., 2023). Following standard practice (Xie et al., 2022), we adopt scripts provided by PlaneRCNN (Liu et al., 2019) to generate ground-truth 3D planes for evaluation.
Baselines. Our NeuralPlane evaluated against the learning-bast multi-view plane reconstruction approach PlanarRecon (Xie et al., 2022), as well wide variety of +RANSC methods.
The geometry of the latter inferred either by learning-basee based Multi View Stereo (MVS) methods, including SimpleRecon (Sayed al., 2022) and FineRecon (Stier et al., 2023), by representative indoor neural surface reconstruction methods (Wang et al., 2021) listed as follows: (1) Manhat- tanSDF (Guo et al., 2022), which improves the reconstruction quality by enforcing the Manhattan- world assumption to planar regions like walls and floors; (2) NeuRIS (Wang et al., 2022), which utilizes predicted normal priors adaptively guide the optimization of neural representations; (3) MonoSDF (Yu et al., 2022b), which uses both learned monocular depth and normal priors; (4) ObjectSDF++ (Wu et al., 2023a), which extends MonoSDF by incorporating dense instance anno- tations for high-fidelity coject-compositional surface reconstruction. We also include the traditional surface reconstruction method (Schönberger al., 2016; Kazhdan & Hoppe, 2013) implemented in COLMAP (Schönberger & Frahm, 2016). Given meshes produced by these methods, we run Sequential RANSAC or its variant, namely AirPlanes (Watson et al., 2024), to partition them into planar primitives. For all data-driven components, we use the official pre-trained models.
Metrics. We evaluate the geometric quality introduced by Bozic et al. (2021). Among these metrics, adopt Chamfer Distance and F-score with a threshold 5 cm as the comprehensive metrics for comparison. To assess how well the reconstructed planar primitives align with the ground-truth conventional segmentation metrics (Arbeláez et al., 2011; I; Tan et al., 2021; Shi et al., 2023): rand index (RI), variation of information (VOI) and segmentation covering (SC).
plane reconstruction utilizing standard 3D metrics
semantics, further employ three
Implementation Details. NeuralPlane is implemented in Nerfstudio (Tancik et al., 2023) on top of Nerfacto, unified approach in the literature of NeRF. Considering the lack of texture in typical

---

Published as a conference paper at ICLR 2025
Quantitative results on ScanNetv2.. Top-3 results are highlighted as first, second and third. At the top, also report the quality of raw geometries in 7 baseline methods, using official scene reconstructions as the ground truth. is tested with its MLP variant. produces a set of seman- tically decomposed reconstructions, and RAq.AA is executed on each of them independently.
Table I:
Geometty
Segmentation
Method
Accu. L
Comp.
Chamfer |
F-score
RI1
VOL.
COLMAP (Schönberger ot al., 2016)
18.35
13:40
15.88
40.5
42.4
41.2
RicoleRecon (Sayed al., 2022)
6.74
5.55
6.15
59.2
58.8
59.0
FineRecon (Stier 2023)
6.12
4.19
5.16
68.6
72.9
70.6
Man (Guo al., 2022)
9,95
7.51
8.73
50.4
50.9
50.6
NeuRtS (Wang et al., 2022)
10:23
5.69
7,96
62.0
64.8
63.2
MomoSDFI (Yu al., 20226)
5,28
5.07
5.18
69.7
69.7
69.7
ObjoctSDF=s1 (Wu al., 2023g)
8.82
5.25
7.03
68.9
60.9
COLMAP
19.32
13.37
16.34
40.6
40.9
40.6
0.928
3.91
0.154
SimpleRecon
6,24
6,00
6.12
$7.7
52.6
54.9
0.949
2.65
0,272
FineRccon
5.20
69.1
64.9
66.7
0.41
0.276
MonhattanSDF
RANNAR
9.37
8.70
9.04
51.0
50.8
0.900
0.281
NeuR1S
9.87
6.35
8.11
59.6
59.3
59.3
0.945
2.57
0.293
MonoSDF
5,91
5.43
5.67
65.9
66.1
63.9
0.945
2,38
0,333
ObjeciSDF++*
8.41
6.88
58.0
68.8
62.8
0.952
2.32
0.334
SimpleRecon
5.43
6,60
6,01
39.2
51.5
55.1
0.944
2.51
0.341
FineRecon
AirPPlanes (Wat-
4.93
5.95
5.44
70.8
62.2
66.2
0.947
2.43
0,310
ManhatlanSDF
son al., 2024)
9.46
9.30
9.38
52.7
50.2
$1.3
0.940
2.61
0,315
NouRIS
6.05
7.38
6.71
56.7
61.0
0.943
2.55
0,291
MonoSDF
4.57
6.33
5,45
72.2
62.0
66.6
0.948
2.38
0.346
PlanarRecon (Xie 2022)
7.74
0.909
0.265
N Ren
5,38
4.65
5.02
67.6
70.0
68.7
0.949
2.37
0.364
NeuralPlane
4.92
4.27
4.59
70.5
71.9
71.2
0.955
2,25
0.376
indoor scenes, we use the COLMAP toolbox (Sarlin et al., 2019) that supports LoFTR (Sun al., 2021). to export dense 3D keypoints for the initialization of local planar primitives.
We train NeuralPlane for 4k iterations with batches of 8192 rays across all scenes. For the first 1k iterations, we nullify Lp-depth and Lpua on local planar primitives that observe fewer than 50 3D keypoints. After the density field holistically optimized to a decent state, we re-estimate the plane offsets of these nullified primitives using rendered depth, globally activate the overall loss eq. (9), and enable the local geometry refinement. We list several key hyperparameters used in our main experiments: the dimension of coplanarity feature d = 4, the pushing thresholds (lor (n) (8 cm, cos 10)), and the number of semantic prototypes N, = 32. A fixed set of hyperparameters is employed across all test scenes. Preprocessing local planar primitives takes around 2 to 5 minutes, followed by about minutes for training on a single NVIDIA RTX 3090 GPU.
4.2 EVALUATION
ScanNetv2. We select 8 challenging scenes with diverse layouts to benchmark the adaptability of various methods. To demonstrate the superiority of our local planar primitives generation method introduced in Sec. 3. we include a variant denoted by NeuralPlane PlaneRecTR, where the ini- tial local planar primitives are instead produced by PlaneRecTR (Shi al., 2023). As reported in Tab. 1, NeuralPlane excels existing methods across all metrics. For geometry+RANSAC methods, we observe that AirPlanes (Watson et al., 2024) in combination with FineRecon (Stier et al., 2023) or MonoSDF (Yu et al., 2022b) exhibits impressive results in Precision but suffers a significant drop in Recall. This indicates that incorporating learned plane embeddings can effectively avoid the undesirable fusion of adjacent planes and thus better fit the input geometry (high precision). However, the involvement of plane embeddings during RANSAC also incurs the risk of missing planes, resulting in more false negatives (low recall). In contrast, NeuralPIane strikes good bal- ance between Precision and Recall. In addition, we observe that independently applying RANSAC to each decomposed mesh of ObjectSDF++, which is optimized with ground-truth instance anno- tations, yields improved geometry and segmentation. However, NeuralPlane achieves even better segmentation with only machine-predicee masks. See Fig. 4 for qualitative evaluation.
8

---

Published as a conference paper at ICLR 2025
Table 2:
Quantitative results on ScanNet++. Top-3 results are highlighted as first, second and third. is tested using its Multi-Res Grids variant.
Geometry
Segmentation
Accu.
Comp. | Chamfer
R1
VO1
COLMAP
24.42
14.53
19.47
47.4
44.2
45.4
0.920
3.91
0.160
SimpleRecon
11.03
8.54
9.79
43.9
45.8
44,6
0.936
3.17
0.195
FineRecon
RANSAC
4.70
5,36
80.3
71.1
0.929
2.79
0.252
ManhattanSDF
8.72
8.67
8.70
56.3
$7.7
0.928
3.06
0.248
NeuRIS
5.35
4.33
4.84
81.4
80.4
80.9
0.941
2.46
0.315
MonoSDFt
5.59
4.60
5.09
77.0
78.5
77.7
0.939
2.47
0.288
SimpleRecon
7.97
9.84
8.91
46.5
42.1
44.1
0.931
2.90
0.219
FineRecon
0.941
2.66
0.277
ManhattanSDF
+AirPlanes
7.35
9.84
8.59
57.9
52.0
54.5
0.935
2.85
0.267
NeuRIS
2.99
6.38
4.69
87.3
73.8
79.9
0.943
2.53
0.287
MonoSDFt
3.21
7.38
5.29
82.8
67.6
74.2
0.935
2.69
0.264
PlanarRecon
7.91
14.29
53.1
0.231
NeuralPlane PlaneRecTR
6.03
6.17
71.1
69.1
0.939
2.72
0.301
NeuralPlane
4.33
4.87
4.60
80.8
78.7
79.7
0.950
2.38
0.356
Reference View
PlanarRecon
NerPlne
NeuRIS+RANSAC MomsSDF=+RANAAC
NeuralPlane
Figure 5: 3D plane reconstruction results on ScanNet++.
ScanNet++. We further conduct experiments 4 scenes from ScanNet+ (Yeshwanth al., 2023), noting that no pre-trained model adopted our method or baselines has encountered similar scene layouts before. These scenes challenging due the significant absence of textures. We observe that low-texture regions generate normal priors that provide NeuRIS with relatively precise geomet- ric guidance, but result in sparse and misleading SIM keypoints for our method. Despite this, as reported in Tab. 2, our approach still achieves competitive geometric performance. Moreover, our method consistently demonstrates remarkable segmentation performance, preserving fine-grained and coherent semantics (see qualitative results in Fig. 5). It is also worth noting that in our method, geometry directly represented by volume density, which allows for 40 training speedup com- pared to other neural implicit methods. A more detailed comparison of reconstruction efficiency is provided in Appendix A.4.
Table 3: Ablation study proposed components. NCF and NP denote the proposed neural coplanarity field and neural parser module, respectively. The best results are bold.
Components
Geometry
Segmentation
Method
Lnumaal
Cp depth
Refine.
NCF
NP
Chamfer
F-score
RI f
VOI
SC
Nerfacto
19.61
17.3
0.903
4.66
0.119
Model
4.96
68.1
0.951
2.27
0.371
Model
f
10.01
41.9
0.935
2.98
0.305
Model
v
/
V
/
5.02
66.0
0.944
2.44
0.358
Model D
Y
4.98
66.1
0.940
2.59
0.352
Model
4.91
69.8
0.947
2.40
0.364
NeuralPlane
4.59
0.955
2.25
0.376

---

Published as a conference paper at ICLR 2025
4.3 ABLATION STUDY
Ablating Components. To verify the merits of each component in NeuralPlane, we conduct abla- tions using the scenes from ScanNetv2. Quantitative results are listed in Tab. 3, where Nerfacto is our backbone with default settings and Model E, the neural parser is replaced by post-clustering simi- lar to ContrastiveLift (Bhalgat et al., 2023). Nerfacto and models A through all struggle with degraded reconstruction quality due to insufficient geometric guidance. Model D disregards high-level semantics during the plane extraction, resulting in significant se- mantic conflicts as depicted in Fig. (d). NeuralPlane excels all these models on all metrics, demonstrating that geometry and semantics are tightly entangled in the task, and that the combination in method ap- pears notable synergy.
(a) Ground Truth
(b) Model (wlo Cp
Model C (w/o Refine.)
(d) Model (w/o NCF)
Ablating Number of Semantic Prototypes. Remem- ber that the number of semantic prototypes N, is heuristically determined hyperparameter in neural parser and we opt for NN 32 across all scenes in our main experiments. We assess the effect of varying V on the reconstruction results and observe that both ge- ometry and segmentation deteriorate as Np continues to increase (see Fig. 7 (a)). This could be explained by the inherent ambiguity in defining what constitutes single plane structure: in Fig. 7 (b), we illustrate that by controlling Np. one can adjust the level of de- tail (LOD), while the level with Nº 32 is potentially closer to the ground truth used for evaluation.
(c) using PlaneRecTR
(L) NeuralPlane
Low
High
Error map:
Figure 6: Qualitative ablation on compo- nents. The upper part shows the recon- structed 3D planes in different colors. The lower part presents the error map.
2.5
70
Piscace ft
2.4
Reference View
N₂ = 128
N₂
69
2.3
68
100
of Prototypes: Ng
N₂
N₂
Ny=1 (w/ NCF)
(a) Number of Semantic Prototypes
(b) Different Levels of Detail
Figure 7: Ablation study on number of semantic prototypes. (a) There is minor decline in performance as N₂ grows. (b) A case shows that the level of detail is controlled by Np.
CONCLUSION
Reconstructing man-made environments arrangements of planar primitives demands not only ge- ometric fidelity but also semantic alignment. this paper, we introduced NeuralPlane, a novel plane reconstruction pipeline that learns accurate 3D locations and instance-level semantics of plane struc- tures from multi-view plane observations implicitly using neural fields. We presented an annotation- free module that incorporates strong single image priors to identify well-defined 2D plane segments, and demonstrated how these segments can drive the joint modeling of scene geometry and seman- ties. Our method achieves remarkable performance in delivering high-quality planar primitives with fine details and coherent semantics. Besides, our approach shows the potential in adjusting the level of detail, which we believe is practical and will provide inspiration for future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to thank Yuxin Cao and Qiran Qian for their feedback on drafts. This work was supported in part by the Beijing Natural Science Foundation (No. L223003), the National Natural Science Foundation of China (No. U22B2055, 62273345 and 62402495), and the Key R&D Project in Henan Province (No. 231111210300)
REFERENCES
Samir Agarwala, Linyi Jin, Chris Rockwell, and David F. Fouhey. PlaneFormers: From sparse view planes to 3D reconstruction. In ECCV, pp. 192-209. 2022. 3
Pablo Arbeláez, Michael Maire, Charless C. Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. (3(5))898-916 2011. 7, 18
ARCore. Fundamental Concepts: Environmental Understanding. In Google for Developer: Aug-
mented Reality Essentials, Accessed: 2024-08-23.
ARKit. Placing content on detected planes. In Apple Developer Documentation, Accessed: 2024- 08-23.
Eduardo Arnll,, Jamie Wynn, Sara Vicente, Guillermo Garcia-Hernando, Áron Monszpart, Vic- Adrian Prisacariu, Daniyar Turmukhambetov, and Eric Brachmann. Map-free visual relocal- ization: Metric pose relative to a single image. In ECCV, pp. 690-708, 2022. 20
Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Estimating and Exploiting the Aleatoric Un- certainty in Surface Normal Estimation. In ICCV, pp. 13117-13126, 2021.
Yash Sanjay Bhalgat, Iro Laina, Joao Henriques, Andrea Vedaldi, and Andrew Zisserman. Con- trastive lift: 3D object instance segmentation by slow-fast contrastive fusion. In NeurlPS, 2023. 3, 6, 10, 19
Aljaz Bozic, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Niessner. TransformerFusion: Monocular RGB scene reconstruction using transformers. In NeurIPS, 2021.
Zheng Chen, Qingan Yan, Huangying Zhan, Changjiang Cai, Xiangyu Xu, Yuzhong Huang, Weihan Wang, Ziyue Feng, Lantao Liu, and Yi Xu. PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields, arXiv preprint arXiv:2401.0087. 2023.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In CVPR, pp. 539-546, 2005.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In CVPR, pp. 5828- 5839, 2017. 6, 17
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised NeRF: Fewer Views and Faster Training for Free. In CVPR, pp. 12882-12891. 2022. 5. 16
Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. SuperPoint: Self-supervised interest point detection and description. In CVPRW, pp. 224-236. 2018. 20
Johan Edstedt, Qiyu Sun, Georg Bökman, Marten Wadenbäck, and Michael Felsberg, RoMa: Robust dense feature matching. In CVPR, pp. 19800, 2024. 20
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, pp. 2.1..31.22 1996. 6
Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. NeRF-SOS: Any-view self-supervised object segmentation complex scenes. In ICL.R. 2023.

---

Published as a conference paper at ICLR 2025
Martin A. Fischler and Robert C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381 1981. 1
Haoyu Guo, Sida Peng, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, and Xiaowei Zhou. Neural 3D scene reconstruction with the manhattan-world assumption. In CVPR, pp. 5511-5520, 2022. 8, 20
T. Hastie, Tibshirani, and J.H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, volume 2. Springer, 2009. 6
Linyi Jin, Shengyi Qian, Andrew Owens, and David F. Foubey. Planar surface reconstruction from sparse views. In ICCV, pp. 12971-12980. 2021.
R. Jonker and A. Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing. 38(4):325-340, 1987. 6
Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction, ACM Trans. Graph., 32(3):29:1-299 12:9113, 2013.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. LERF: Language embedded radiance fields. In ICCV, pp. 19672-19682. 2023. 3
Chung Min Kim, Mingxuan Wu, Justin Kerr, Ken Goldberg, Matthew Tancik, and Angjoo Kanazawa. GARField: Group Anything with Radiance Fields. In CVPR, pp. 21530-21539. 2024. 2, 3, 17
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. In ICCV, pp. 3992-4003. 2023. 3, 15
Florian Kluger, Eric Brachmann, Michael Ying Yang, and Bodo Rosenhahn. Robust shape fitting for 3D scene abstraction. T-PAMI, (6009666366663366 2024.
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing NeRF for editing via feature field distillation. In NewrlPS, 2022.
Lei Li, Songyou Peng, Zehao Yu, Shaohui Liu, Rémi Pautrat, Xiaochuan Yin, and Marc Pollefeys. 3D neural edge reconstruction. In CVPR, pp. 21219-21229, 2024. 2,
Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys. LightGlue: Local feature matching at light speed. In ICCV, pp. 17581-17592. 2023. 20
Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz. PlaneRCNN: 3D plane detection and reconstruction from single image. In CVPR, pp. 4445 4454. 2019. 3, 7, 17
Jiacheng Liu, Pan Ji, Nitin Bansal, Changjiang Cai, Qingan Yan, Xiaolci Huang, and Yi Xu. PVaneMVS: 3D plane reconstruction from multi-view stereo. In CVPR, pp. 8655-8665, 2022.
Yuzhou Liu, Lingje Zhu, Xiaodong Ma, Hanqiao Ye, Xiang Gao, Xianwei Zheng, and Shuhan Shen. PolyRoom: Room-aware transformer for floorplan reconstruction. ECCV, pp. 322-339. 2024.1
Zheng Liu, Xiyuan Liu, and Fu Zhang. Efficient and consistent bundle adjustment on lidar point clouds. IEEE Trans. Robotics, 39(6):4366 4386,,43866611 2023. 1
Stuart P. Lloyd. Least squares quantization PCM. IEEE Trans. Inf. Theory, 28(2):129-136, 1982.
Hidenobu Matsuki, Riku Murai, Paul H. Kelly, and Andrew J. Davison. Gaussian Splatting SLAM. In CVPR, 2024. 20
Kirill Mazur, Gwangbin Bae, and Andrew Davison. SuperPrimitive: Scene Reconstruction at Primitive Level. In CVPR, pp. 2024.
12

---

Published as a conference paper at ICLR 2025
Marina Meila. Comparing clusterings: An axiomatic view. In ICML, pp. 577-584. 2005. 18
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, pp. 405-44 421. 2020.
Zak Murez, Tarrence As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich, Atlas: End-to-end 3D scene reconstruction from posed images. In ECCV, pp. 414- 431. 2020.
Liangliang Nan and Peter Wonka. PolyFit: Polygonal surface reconstruction from point clouds. In ICCV, pp. 2372-2380. 2017. 20
Minyoung Park, Mirae Do, YeonJae Shin, Jaeseok Yoo, Jongkwang Hong, Joongrock Kim, and Chul Lee. H2O-SDF: Two-phase Learning for 3D Indoor Reconstruction using Object Surface Fields. In ICLR, 2024. 3, 19
Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From Coarse to Fine: Robust Hierarchical Localization Large Scale. In CVPR, pp. 17708-12717 2019. 8
Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, and Clément Godard. SimpleRecon: 3D reconstruction without 3D convolutions. In ECCV, pp. 1-19, 2022. 7, 8, 17
Johannes L. Schönberger, Enliang Zheng. Jan-Michael Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In ECCV, pp. 501-518. 2016. 1, 7,8,
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-mottie revisited. In CVPR, pp. 4104 1104-4113. 13, 2016. 7
Jingja Shi, Shuaifeng Zhi, and Kai Xu. Pe:::RecTR: Unified Query Learning for 3D Plane Recov- cry from a Single View. In ICCV, pp. 9377-9386, 2023. 3, 7,
Noah Stier, Anurag Ranjan, Alex Colburn, Yajie Yan, Liang Yang, Fangchang Ma, and Baptiste Angles. FineRecon: Dwpth-aware Feed-forward Network for Detailed 3D Reconstruction. In ICCV, pp. 18423-18432. 2023. 7, 17
Jia Ming Sun, Hong Shen, Yuang Wang, Hu Jun Bao, and Xiao Wei Zhou, LoFTR: Detector-Free Local Feature Matching with Transformers. In CVPR, pp. 8918 8927. 2021. 8, 20
Bin Tan, Nan Xue, Song Bai. Tianfu Wu, and Gui-Song Xia. PlancTR: Seucture-Guider Trans- formers for 3D Plane Recovery. In ICCV, pp. 4186-4195, 2021. 3,7
Bin Tan, Nan Xue, Tianfu Wu, and Gui-Song Xia. NOPE-SAC: Neural one-plane RANSAC for sparse-view planar 3D reconstruction. T-PAMI, 45(12):15233-1544 2023. 3
Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A Modular Framework for Neural Radiance Field Development. In ACM SIGGRAPH, pp. 1-12, 2023. 7
Bing Wang, Lu Chen, and Bo Yang. DM-NeRF: 3D scene geometry decomposition and manipula- tion from 2D images. In ICLR. 2023.
Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping Wang. NeuRIS: Neural Reconstruction of Indoor Scenes Using Normal Priors. In ECCV, pp. 139-155. 2022. 8, 19, 20
Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurlPS, 2021. 7
Jamie Watson, Filippo Aleotti, Mohamed Sayed, Zawar Qureshi, Oisin Mac Aodha, Gabriel Bros- tow, Michael Firman, and Sara Vicente. AirPlanes: Accurate plane estimation via 3D-consistent embeddings. In CVPR, pp. 5270-5280, 2024. 1, 2, 3, 5, 8, 18, 19
13

---

Published as a conference paper at ICLR 2025
Qianyi Wu, Kaisiyuan Wang, Kejie Li. Jianmin Zheng, and Jianfei Cai. ObjectSDF++: Improved Compet-Compositional Neural Implicit Surfaces. In ICCV, pp. 21707-21717. 2023a. 7,
Tong Wu, Jiaqi Wang, Xingang Pan, Xudong XU, Christian Theobalt, Ziwei Liu, and Dahua Lin.
Voxurf: Voxel-based efficient and accurate neural surface reconstruction. In ICLR, 2023b.
Yiming Xie, Matheus Gadelha, Fengting Yang. Xiaowei Zhou, and Huaizu Jiang. PlanarRecon: Realtime 3D Plane Detection and Reconstruction from Posed Monocular Videos. In CVPR, pp- 6209-6216 2022.
3, 7, 8, 18
Nan Xue, Bin Tan, Yuxi Xiao, Liang Dong, Gui-Song Xia, Tianfu Wu, and Yujun Shen. NEAT: Distilling 3D Wireframes from Neural Attraction Fields. In CVPR, pp. 19968-19977. 2024. 1,2, 3, 6
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In ICCV, pp. 13759-13768. 2021.
Fengting Yang and Zihan Zhou. Recovering 3D planes from a single image via convolutional neural networks. ECCV, pp. 87-103, 2018. 3
Yunfan Ye, Renjiao Yi, Zhirui Gao, Chenyang Zhu, Zhiping Cai, and Kai Xu. NEF: Neural edge fields for 3D parametric curve reconstruction from multi-view images. In CVPR, pp. 2023. 2,3
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. ScanNet++: A high- fidelity dataset of 3D indoor scenes. ICCV, pp. 12-22, 2023. 7,
Haiyang Ying, Yixuan Yin, Jinzhi Zhang, Fan Wang, Tao Yu, Ruqi Huang, and Lu Fang. Om- niSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning. In CVPR, pp. 20612-20622. 2024. 2
Hong-Xing Yu, Leonidas J. Guibas, and Jiajun Wu. Unsupervised discovery of object radiance
fields. In ICLR. 2022a. 3
Mulin Yu and Florent Lafarge. Finding good configurations of planar primitives in unorganized point clouds. In CVPR, pp. 6357-6366. 2022. 1, 18
Zehao Yu, Jia Zheng, Dongze Lian, Zihan Zhou, and Shenghua Gao. Single-image piece- piece-wise planar reconstruction via associative embedding. CVPR, pp. 1029-1037. 2019.
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction. In NeurIPS. 2022b. 7, 8, 17, 19, 20
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. NeRF++: Analyzing and improving neural radiance fields, arXiv preprint arXiv:2010.07492, 2020. 4
Lipu Zhou, Shengze Wang. and Michael Kaess. r-LSM: LiDAR smoothing and mapping with planes. In ICRA, pp. 5751-5757, 2021.
Siting Zhu, Guangming Wang, Hermann Blum, Jiuming Liu, Liang Song, Mare Pollefeys, and Hesheng Wang- SNI-SLAM: Semantic neural implicit SLAM. In CVPR, pp. 21167-21177. 2024.
Zi Han Zhu, Songyou Peng. Viktor Larsson. Wei Wei Xu, Hu Jun Bao, Zhao Peng Cui. Martin R. Oswald, and Mare Pollefeys. NICE-SLAM: Neural Implicit Scalable Encoding for SLAM. In CVPR, pp. 12776-12786 2022. 20
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
In Appendix A.I, provide additional details about the method, including the procedures of gen- erating 2D plane segments and extracting 3D parametric planes. Next, we present further implemen- tation details, including the network architecture and training configurations (Appendix A.2). Then, detailed descriptions on the data processing steps, implementation details of baseline methods and the evaluation protocol are provided Appendix A.3, More experimental evaluation and qualitative results can be found in Appendix A.4. Finally, in Appendix A.5, we discuss the limitations of our method and anticipate future research trajectories.
A.1 ADDITIONAL METHOD DETAILS
2D Plane Segments from Monocular Priors. In Sec. 3.1, when identifying plane regions from single image, we first apply K-means clustering (K=8) to its estimated normal map, and select the largest 6 clusters initial masks (see Fig. 8 (c) ). which correspond to low-frequency regions. We then utilize the zero-shot capacities of the Segment Anything Model (SAM) to generate all pos- sible plane instances by prompting with 256 query points evenly sampled from the initial masks. SAM-generated masks are filtered using post-processing techniques including stability checks and non-maximal suppression (NMS) introduced by Kirillov al. (2023). After eliminating unstable and duplicate masks, we only keep the smallest mask for each query. The overlaps between the SAM-generated masks and the initial masks, expected to geometrically continuous and smooth,
represent the desired 2D plane segments (see Fig. 8 (d) ). We further remove noisy 2D plane seg- ments whose pixel areas are less than 0.4 % of the image size.
(A) Image
(b) Normal Map
Initial Masks
(d) 2D Plane Segments
PlaneRecTR
(1 Groand Truth
Figure 8: 2D plane segment generation. We cluster (b) monocular normal priors for (c) initial masks, and then enforce geometric continuity by semantic over-segmentatton for (d) 2D plane seg- ments. (e) Single-view plane segmentation results by PlaneRecTR (Shi et al., 2023). (f) Ground truth rendered from 3D plane annotations.
Compared to the PlancRecTR's predictions in Fig. 8 (e) and the ground truth in Fig. 8 (f), our method exhibits over-segmentatinn; in other words, tends to split single plane into multiple segments due to SAM's fine-graines visual understanding capability.
Pseudo-Depth Supervision. In Sec. 3.2.1, plane-derived aseudo-depth is introduced as extra geo- metric guidance. The expected pseudo-depth D of a ray r from local planar primitive P is computed base on the plane parameters π [n, o) (defined in world space). More precisely, let the expected termination point of be denoted by o + De, where o, is the camera center and v is the unit
15

---

Published as a conference paper at ICLR 2025
direction vector. Since r is assumed to terminate at the planar surface æ, have
n (0z + -D) +0= 0,
(10)
which can be rewritten as where is the angle between n and u.
D 10+ n 0.)) (n,
-(0 + n0.)) COS
(11)
Motivated by DS-NeRF (Deng et al., 2022), the pseudo-depth loss defined in eq. (5) encourages the termination distribution of ray to match the pseudo-depth distribution. Different from the paper, allow gradients to flow through the pseudo-depth distribution, thereby jointly optimizing and Here, the KL divergence is computed as:
/ N(t N(t: log
NW D h(t)
DEL DW(t
(12)
D,
Alj:
(13)
h(t;)
where (t; D, 1) denotes the normal distribution that are used to model the noisy pscudo-depth
variable d, and h(t) is the piccewise-constant approximation of the rendering probability density function defined in eq. (2). Then, we have:
[RI[W(t; D ([[((()]
Σ exp (BB(1) D)2
3(1
D)2
(14)
log
log h(t,))
Sty.
The variance parameter 3-1 is fixed to 0.005 across all rays.
3D Parametric Plane Extraction. After training, in Sec. 3.3, we design a straightforward algorithm to extract 3D parametric planes from the learned neural representation. More specifically, for given local planar primitive P, randomly sample 128 rays from its 2D plane segment and render their termination points. The normal of P is also assigned to each termination point as its local normal. To render the coplanarity features more efficiently, instead of sampling and integrating features along the rays, we simply use features queried at the termination locations as the ray-level coplanarity features. To group these termination points into different subregions, we first find the closest semantic prototype to each coplanarity feature by measuring the Euclidean distance in feature space. If more than half of the features share the same nearest semantic prototype, then we assign the label of that prototype to all the points. Otherwise, this local planar primitive is considered ambiguous and discarded as an outlier. We continue sampling until the number of successfully labeled points exceeds 2 x 106.
After grouping these points by their labels, we sequentially fit 3D planes to each group using RANSAC. A point is considered an inlier if: (1) the angle between its normal and the normal hypothesis is less than 20", and (2) the distance from the point to the plane is less than 0.08 m.
Next, given a collection of fitted plane parameters and inliers, transform them into triangulated planar surfaces by:
(1) Downsample the inliers using voxel grid filtering with a leaf size cm.
Project the downsampled points onto the fitted plane and rasterize the 2D projection map into regular binary grid with a resolution of cm.
(3) Traverse the binary grid and triangulate by connecting occupied neighbors, which produces 2D mesh.
(4) Backproject the 2D mesh into the original 3D space to obtain a 3D mesh which represents the final real-world planar structure.
16

---

Published as a conference paper at ICLR 2025
A.2 ADDITIONAL IMPLEMENTATIN DETAILS
Network Architecture. The neural coplanarity field is configured as separate output head along- side the default Nerfacto model (see Fig. 9). The coplanarity features are encoded by a 12-level hashgrid with resolutions sampled from 16 to 256, and a 4-layer MIP with hidden dimensions of 256 and ReLU activation. To speed up the training process, we follow GARField (Kim et al., 2024) to first render the hash value and then feed it into the coplanarity feature MLP. In addition, we con- strain coplanarity features to a unit hypersphere. The feed-forward network employed in Neural Parser is 3-layer MLP with both input and hidden dimensions of 8 and ReLU activation.
Scene Contraction contract(p)
Hash Encoding
p
Density MLP
Concat
Volume Rendering
V
PE(
Color MLP
c(p. v)
Feature MLP
Normalization
fx)
Nerfacto
Neural Coplanarity Field
Figure 9: Network architecture.
Training Details. Before training, camera poses are centralized without scale adjustment, and we disable joint pose optimization using the provided poses. During each training iteration, we ran- domly select 128 local planar primitives with probabilities weighted by the sizes of corresponding 2D plane segments. In each local planar primitive, we uniformly sample n = 8192 128 64 rays, with N=48 points sampled per ray. We further randomly generate 64 triplets and 64 pairs of rays to obtain explicit surface normals for Loormal defined in eq. (4) and positive samples for Lpull defined in eq. (6), respectively. The push margin in eq. (7) is set to 1.5 for the first 1k iterations, and is later increased to 2. In Neural Parser, the DBSCAN epsilon is fixed to 0.2. We optimize plane parameters of local planar primitives using Adam optimizer with exponential decay schedule from initial learning rate of 1 x 10-3 to x 10
A.3 MORE DETAILS ON EXPERIMENTS
A.3.1 DATASETS
In our experiments, two publicly available datasets, ScanNety2 (Dai et al., 2017) and Scan- Net++ (Yeshwanth et al., 2023), used evaluate method they offer surface reconstruc- tions and densely annotated instance-level semantics that can be further processed (Liu et al., 2019) obtain the ground truth. Specifically, we sample 8 scenes from the official validation set of Scan- Netv2 and scenes from ScanNet++. All frames are resized to resolution of 640 x 480. For each scene from ScanNetv2, eigheighth of the frames are uniformly selected for reconstruction, while in ScanNet++, equally spaced frames are downsampled according to the number of registered frames, leaving 150~279 frames per scene, Tab. 4 lists more details of the selected scenes.
A.3.2 IMPLEMENTATION DETAILS ON BASELINE METHODS
All the baseline methods are implemented following the official codebases and instructions. FineRe- con (Stier et al., 2023) recovers accurate surfaces by introducing a depth guidance strategy, and here use the multi-view depth predicted by SimpleRecon (Sayed et al., 2022) for such additional prior. Besides, as pointed by Yu et al. (2022b), the MLP architecture inherently robust to motion blur and noisy camera poses due to its "smoothness bias". Thus, adopt the MLP variant of MonoSDF
17

---

Published as a conference paper at ICLR 2025
Table Details of selected scenes. For each scene, downsample the video a lower frame rate. Down. Rate denotes the downsample rate. Area is the surface area of ground-truth mesh
Scene II)
Type
# of Frames
Down. Rate
Area (m²)
0084_00
bathroom
247
12.5 %
42.68
0164_03
kitchen
140
12.5 %
28.97
ScanNerv2
0217_00
bedroom
156
12.5 %
40.65
0316_00
lounge
97
12.5 %
26.56
0356_00
hedroom
170
12.5 %
27.77
0427
conference room
165
12.5 %
24.11
0488 01
kitchen
137
12.5 %
38.53
0568
lounge
207
12.5
71.49
ScanNett+
16659a3107
conference room
181
1.8 %
133.38
31a2c91c43
bathroom
184
2.4%
31.72
7hc286c1h6
bathroom
150
3.3 %
25.13
303745abe7
office
279
4.4 %
63.43
for experiments on ScanNetv2, while in ScanNet++ with high-quality RGB captures, we use the Multi-Res. Grid variant for faster convergence.
The Sequential RANSAC employed in baselines and our method is all adopted from Planar- Recon (Xie al., 2022), with thresholds specifically fine-tuned for each method. We render per- pixel depth map for each RGB keyframe using geometries given by various baseline surface recon- struction methods so as combine them with AirPlanes (Watson al., 2024).
A.3.3 EVALUATION PROTOCOL
As several methods such as FineRecon (Yu & Lafarge, 2022) can predict geometry for unseen re- gions, we mask out these regions via visibility check, only preserving the observed areas for fair evaluation.
Tab. 5 (a) lists the metrics in geometry, where P and I' point clouds evenly sampled from the predicted and ground-trume truth mesh each consisting of 200 000 points. To assess how well the plane reconstruction aligns with the ground-truie trrth semantic information, we follow the evaluation protocol of PlanarRecon (Xie et al., 2022), reporting segmentation quality using three typical metrics. These metrics are listed in Tab. 5 (b) and were originally used for comparing partitions (Meila, 2005; Arbeláez et al., 2011). The ground-truth clustering C* is the partition of the ground-truth mesh into sets Ci, Cz,.. Cxx-, where C represents the collection of vertices of the i-th ground-truth planar primitive. The predicted clustering C is obtained by partitioning ground-truth vertices according to the plane IDs of their nearest neighbors in the predicted mesh. The total number of vertices in the ground-truth mesh is denoted by No.
Rand Index represents the probability that C and C agree on the clustering of randomly selected pair of vertices, where Naisagree defined the number of pairs that are clustered differently by C and C
Variation of Information, the sum of the entropies of C and C* minus the mutual informa- tion between them, measures the distance between two partitions by information difference.
Segmentation Covering is the average of mutual coverings between C and C*, where the covering C(C* + (C) is computed as:
C(C + C) Nc
CC|-
max IoU Ecc
CEC
A.4 MORE EXPERIMENTAL RESULTS
A Ablating Feature Dimension of NCF. We investigate the performance of our proposed NeuralPlane when using different feature dimensions d. Specifically, evaluate the effect of varying d from
18

---

Published as a conference paper at ICLR 2025
Table Definitions of metrics. The quality of 3D plane reconstruction is evaluated by both geom- etry and segmentation metrics.
(a) Evaluation Metrics for Geometry:
Evaluation Metries for Segmentation.
Metric
Definition
Metric
Definition
Accuracy
meaäpep (minly+ ep Ip p I1a)
No 1) NNnoppo
Completeness
meaällg ep (minper112 p (1a)
Index
No(Nv -
Chamfer
(Accuracy Completeness) /2
Variation of Information
H(C)
Precision (S)
meanger (mingrer- [p pp 0.05) 100
Recall (%)
meanprer (minperllep lpp p 0.03) x 100
C(C -0+
Segmentation Covering
F-score (%)
2 x Precision Recall/ (Precision Recall)
128 using F-score, VOI, and SC metrics on ScanNetv2 dataset. As illustrated in Fig. 10, the performance of NeuralPlane is almost optimal when d is larger than 2. Similar results are also observed by ContrastiveL.ift (Bhalgat et al., 2023) and AirPlanes (Watson al., 2024). light of this, we simply choose d 4 in our experiments for trade-off between performance and efficiency.
F-score ?
0.38
68
VOL
0.38
10 of Dimensions: d
100
10
100
of Dimensions:
(a) Geometry
(b) Segmentation
Figure 10: Impact of different numbers of feature dimension.
Ablating DBSCAN Epsilon. To supervise the neural parser module, our proposed method uses DBSCAN first cluster the noisy primitive-level features rendered during each iteration. The study on the impact of varying the DBSCAN epsilon is presented in Fig. 11. The results indicate that the performance of NeuralPlane is close optimal when the epsilon is in the range of 0.2 0.5. A higher cluster epsilon leads to coarse-grained feature discrimination, while a lower epsilon may result in over-segmentation. However, thanks the robust mechanism of point grouping and pa- rameter estimation introduced in Sec. 3.3, no significant performance deterioration is observed even DBSCAN is ineffective (with the cluster epsilon set to 2).
F-score
2.3
2.2
0
0.2
0.5
1.5
0.2
0.5
1.5
DBSCAN epsilon
DBSCAN epsilon
(a) Geometry
(b) Segmentation
Figure 11: Performance on ScanNety2 with varying DBSCAN epsilon.
A Sensitivity Analysis of SIM Geometry. Incorporating geometric cues, such estimated monoc- ular depth and normal maps, into neural scene representations is widely adopted to improve the performance of indoor 3D surface reconstruction (Yu et al., 2022b; Wang et al., 2022: Park al., 2024). Likewise, NeuralPlane requires additional geometric priors for the initialization local planar primitives. We find that, although sparse and noisy, the readily available SfM point cloud
19

---

Published as a conference paper at ICLR 2025
is sufficient to provide good initialization. To further analyze the robustness of our approach to SfM geometry of different quality levels, we utilize different SfM point clouds triangulated from correspondenees estimated with various feature matching techniques. Results in Tab, 6 demonstrate that dense matching methods, e.g., LoFTR (Sun et al., 2021), are more favorable for our method, as sparse matching methods, e-g., SuperPoint (DeTone et al., 201 8) + LightGlue (Lindenberger al., 2023), are struggling with low-texture regions. Besides, our approach can consistently benefit from the development of more advanced feature matcher, e.g., RoMa (Edstedt al., 2024).
Table 6: Performance of NeuralPlane initialized with different SIM keypoints.
Geometry
Segmentation
Method
Chamfer
F-score
RI
VOI
SC f
w/o SfM Geometry
10.13
40.2
0.925
3.08
0.293
SuperPoint + LightGlue
5.90
65.4
0.942
2.49
0.345
LoFTR (Indoor)
4.57
72.8
0.956
2.24
0.380
LoFTR (MegaDepth) < main paper
4.59
71.2
0.955
2.25
0.376
RoMa (Indoor)
4.62
73.2
0.956
2.12
0.400
Evaluation on Reconstruction Efficiency. In contrast to learning-basse based MVS methods that can be implemented online at interactive speeds, our proposed NeuralPlane a relatively time-consuming method that requires per-scene optimization. However, our method is much more efficient than other offline neural surface reconstruction methods. Tab. 7 presents comparison of average execution time against baseline models on ScanNetv2.
Table 7: Comparison of time consumption across neural implicit methods.
Pre- tre-processing (min)
Training (h)
Inference (min)
ManhattanSDP (Guo al., 2022)
44.5
5.6
2.3
NeuRIS (Wang et al., 2022)
0.5
4.2
1.2
MonoSDF (MLP) (Yu et al., 2022b)
7.5
0.7
NeuralPlane
2.9
0.1
0.5
Additional Qualitative Results. In Figs. 12 and 13, we visualize the entire 3D plane reconstruction on ScanNetv2 and ScanNet++ datasets, drawing comprehensive comparisons against hate-of-the-an methods. Moreover, we provide close-up views rendered from reference poses in Fig. 14.
Out-of-Domain Experiment. While work primarily focuses indoor scenarios, we also con- ducted experiments on two small-scale outdoor scenes from the Niantic MapFree dataset (Arnold al., 2022). The results in Fig. 15 highlight the potential of our method to generalize to outdoor environments.
A.5 LIMITATIONS AND FUTURE WORK
Failure Cases. Fig. 16 illustrates several common failures, showcasing issues such (a) missing structures and (b) under-segmeetattion, both of which stem from inaccuracies in 2D plane observa- tions. The method is also not equipped handle (c) non-planar surfaces. Besides, manually setting the number of semantic prototypes limits the adaptability of our method; for instance, excessively large value may lead to (d) ver-seementttion.
NeuralPlane at its core is integrating noisy 2D plane observations into unified 3D neural represen- tation, but it may fail recover from catastrophic errors such as severe inaccuracy in mono-normal estimation and SfM geometry, which we did not consider. Meanwhile, the method is currently restricted to compact environments and better suited to indoor settings. Complex and large-scale scenes continue to pose many challenges, including (1) the need for large model capacities, and (2) the presence of massive non-plana clutter, which we leave for future research. Intersection anal- ysis (Nan & Wonka, 2017) integrating other cues such occlusion edges for more structured reconstruction is another interesting research topic. It is also promising to implement our method in an online and efficient SLAMM framework (Zhu et al., 2022; Matsuki et al., 2024).
20

---

Published as a conference paper at ICLR 2025
RANSAC
AirPlanes
Reference Mesb
NeuralPlane
SimpleRecon
FineRecon
SimpleRecon
FineRecon
Gromad Truth
PlanarRecon
NeaRIS
Db
NeuRIS
MonoSDF
Reference Meah
NeuralPlane
SimpleRecon
FineRecou
SimpleRecon
FineRecon
Ground Truth
PlanarRecon
NeuRIS
ObjectSDF++
NeuRIS
MonoSDF
Reference Mesh
NeuralPlane
SimpleRocon
FineRocon
SimpleReoon
FineRocon
PlanarRecon
ObjectSDF++
MonoSDDF
Reference Mesh
NeuralPlane
SimpleRecoa
FineRecon
SimpleRecon
FincRecon
Ground Truih
PlanarRecon
NeuRIS
ObjectSDF
NeuRIS
MonoSDF
Reference Mesh
NeuralPlane
SimpleRecon
FineRecon
SimpleRecon
FineRecon
Ground Truih
PlanaRRecon
NeuRIS
Db
NeuRIS
MonoSDF
Figure 12: More qualitative comparisons on ScanNetv2.
21

---

Published as a conference paper at ICLR 2025
RANSAC
AirPlanes
Reference Mesh
NeuralPlane
SimpleRecon
FineRocon
SimpleRecoa
FineRecon
Ground Truth
PlanarRecon
NeuRIS
MonoSDF
NeuRIS
MonoSDF
Reference Mesh
NeuralPlane
SimpleRecon
FineRecon
SimpleRecon
FineRecon
Ground Truth
NouRIS
MonoSDF
NeuRIS
Mo
Reference Mesh
NeuralPPaane
SimpleRecom
FineRecon
SimpleReccn
FineRevon
Ground Truth
PlanatRecon
NeuRIS
MonoSDF
NeuRIS
MonoSDF
Reference Mesh
NeuralPlane
SimpleRecon
FineRecon
SimpleRecon
FineReson
Ground Truth
PlanarRecom
NeuRIS
MonoSDF
NeuRIS
MonoSDF
Figure 13: More qualitative comparisons on ScanNet++. Ceilings and occluding walls have been filtered for better visualization.
22

---

Published as a conference paper at ICLR 2025
Reference View
PlanarRecon
FR + AirPlanes
NeuralPlane
Ground Truth
Figure 14: Close-ups of qualitative results.
23

---

Published as a conference paper at ICLR 2025
Figure 15: Our method is applicable to outdoor environments. As no ground-truth reconstruc- tion is available, we recommend viewing videos in the supplementary for more comprehensive evaluation.
(a) Missing structures
(b) Under-segmeetationn
(c) Non- Non-planar surfaces
(d) sverrseementttion
Figure 16: Failure Cases. In each case, from left to right: reference image, reconstruction, and the ground truth. (a) Planes nearing the optical axis, as well as intricate structures, are difficult to detect in 2D and thus likely be absent from the final reconstruction. (b) The failure to seman- tically differentiate between two adjacent planes in certain views could lead to under-segmentation. (c) Non-planar surfaces are forcibly fitted into planes. (d) Sver-segmentatiin due too large value of semantic prototypes.
24


Reviews:
[{"review_id": "AzpQE4njSc", "reviewer": "Reviewer_FCSv", "summary": "This paper presents a neural 3D reconstruction system on 3D plane reconstruction of indoor scenes. Inspired by the recent success of neural radiance field and image foundation models (SAM2), this paper presents a multi-view 3D reconstruction pipeline leveraging these techniques. The training scheme consists of three phases: Initializing plane segments and parameters -> optimizing a neural feature field for plane-specific feature representation (encouraged by a list of geometry-guided loses) -> plane extraction by grouped features and RANSAC. Experiments are conducted extensively on two representative indoor datasets: ScanNet and 7-scenes and are compared with a group of competitve baseline methods.", "questions": "I am impressed by the technical contribution made by authors for this work. However, there also exist a few major concerns for me at this time which discourage me to grant this paper a higher value. \n\nPlease try to address my concerns listed in the weakness part. I will accordingly consider to improve my overall rating if the concerns are well solved.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The unique advantage of the system is the association of geometry and semantic features for plane reconstruction problem, which requires both geometry-aware perception and semantic-aware grouping.\n2. The paper leverages foundation models to achieve plane segment initializaing and utilize geometry-driven losses to optimize the system. There is no groundtruth plane segmentation or geometry label required. \n3. Overall the paper and diagrams are well written and presented.\n4. The experimental comparison and thorough and convincing on both plane geometry reconstruction and segmentation.", "weaknesses": "1. I think the major advantage of this paper is the unsupervised learning paradigm on the plane reconstruction problem. However, since both ScanNet and ScanNet++ has groundtruth images, this unique advantages seems not to be fully enjoyed and reflected. So I suggest authors to apply the proposed system on some outdoor scenes containing plane structures (such as autonomous driving dataset or street view datasets), to verify the adaptability of the method.\n \n2. I have tested PlaneRecon in my previous projects. It can incrementally reconstructs planes in an online and real-time manner. Besides, it can trains over multiple scenes and directly test without any test-time optimization. Although its precision should fall behind than the neural reconstruction papers, the generalizability and speed is higher than the proposed method. I am curious on whether the paper has such potential to tackle these limitations.\n\n3. Missing a few related works: (1) Recovering 3d planes from a single image via convolutional neural networks (2) PlaneMVS: 3D Plane Reconstruction From Multi-View Stereo (3) Single-image piece-wise planar 3d reconstruction via associative embedding.\n\n4. On quantitative evaluation, it is unclear that what is the groundtruth is here. Does it stand for the groundtruth planar part only or the entire mesh? As a plane reconstruction paper, I think the former one should be more reasonable. If so, the first several methods listed in Table 1 cannot directly be compared with the proposed method. Authors should make it clearer on this part.\n\n5. Is the proposed method robust to the hyperparameters listed in the paper especially (1) to,tn and m in the push loss during training and (2) the parameters selected in RANSAC during plane fitting? Some ablation studies on hyper-parameter robustness are expected to make the method more generalizable across most indoor scenes, since the geometric scale and semantic distribution can have large variance among scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ntJC0h3rcE", "reviewer": "Reviewer_6wEq", "summary": "This paper presents NeuralPlane, a method for reconstructing 3D scene plane primitives via neural fields without GT plane labelling. The method is divided into three main stages: firstly, it combines pre-trained normal prediction and SAM to generate initial 2D planar segments and estimates their 3D parameters using SfM keypoints. Secondly, it optimises two neural fields, a density field based on planar geometric constraints, and a coplanar neural field that understands the semantic relationships between regions . The neural coplanar field is followed by a neural parser module that helps to model the learned coplanar relations. Finally, the optimised neural representations are converted into explicit 3D planes through point sampling, feature-based clustering and RANSAC fitting. The method is evaluated on the ScanNetv2 and ScanNet++ datasets, and the results show that it outperforms both the learning-based method and the Geometry+RANSAC method in terms of geometric and semantic metrics.", "questions": "Since it is a complex system paper, making it hard to reproduce, will the code be publicly avaliable?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and easy to understand.\n- The proposed method does not require ground-truth plane annotations, as it can learn effectively from noisy monocular model outputs.\n- The method demonstrates SOTA performance and achieves clean plane segmentation results.", "weaknesses": "- The proposed method involves numerous hyperparameters, including balancing parameters for loss, the number of semantic prototypes, and parameters listed in Lines 850-863.\n- As a complex system, it is important to discuss and present failure cases to help readers understand the method’s limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jKV1xjORXr", "reviewer": "Reviewer_1CZJ", "summary": "The paper presents NeuralPlane, a novel approach to 3D plane reconstruction that utilizes neural fields for generating structured 3D maps from multi-view images without the need for plane annotations. The method emphasizes two main aspects: geometry and semantics. Key contributions include:\n1. Monocular Plane Segmentation: A monocular module extracts geometrically smooth (based on off-the-shelves Surface Normal Predictor) and semantically meaningful 2D plane observations (based on Segmenet Anything Model). \n2. Plane-Guided Neural Representation: The model utilizes these 2D segments to train a neural field that captures accurate 3D plane locations. A surface normal regularization and pseudo-depth regularization terms are proposed.\n3. Neural Coplanarity Field: This self-supervised feature field enables semantic consistency within the 3D reconstruction by grouping planar regions that share coplanar relationships. A contrastive loss is proposed to distinguish between planes with similar geometric properties but different semantic properties. \nThe method demonstrates superior performance on ScanNetv2 and ScanNet++ datasets, indicating its effectiveness in indoor environments.", "questions": "1. Eqn 4: n_i is not defined, is it the surface normal of the selected local planar primitive? Please explicitly define n_i in the text or equation, and confirm if it refers to the surface normal of the local planar primitive.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Novelty in Combining Geometry and Semantics: The method’s approach to merging geometry with semantic information through a neural coplanarity field is innovative, enhancing the semantic consistency of 3D reconstructions. A complex system with multiple stages is proposed to estimate/reason the local planar regions and the associated parameters, to associate the planes in 3D using radiance field, to resolve semantic conflicts using Neural Coplanarity Field.\n2. High-Quality Reconstruction: Experimental results indicate that NeuralPlane achieves fine-grained and coherent plane reconstructions, outperforming existing methods in most of the metrics. \n3. Efficiency: NeuralPlane’s volume density representation allows for faster training compared to implicit methods, an important practical advantage.\n4. Extensive ablation studies: the authors include sufficient ablation studies to support the effectiveness of the proposed modules.\n5. Good writing: the paper is clearly written that the technical details are clearly presented.", "weaknesses": "1. Complexity of Methodology: The proposed method involves several stages, including monocular plane segmentation, neural coplanarity field training, and plane extraction. In each stage, there are several submodules and I believe there are some hyperparameters decisions in each stage. While effective, this complexity (especially the combination of submodules and associated parameters) may impact its scalability to larger scenes or generalizability. For example, K-means clustering on predicted normal map, mask size threshold in SAM, thresholds to form negative pairs in Neural Coplanarity Field, Loss balancing parameters, etc. It would be great if the authors could share the insights on the impact of these hyperparameter settings on different scenes. For example, is a universal set of hyperparameters applied to all the test scenes? It would be great if the authors could provide a sensitivity analysis or ablation study on key hyperparameters across different scenes. This would help clarify how robust the method is to parameter changes and whether a universal set of parameters is feasible.\n\n2. Dependence on Initial 2D Plane Segmentation Quality: As the regularizations are based on the quality of initial local plane geoemtry, the method’s success can depend on the quality of 2D plane segments obtained from monocular priors, which may introduce inaccuracies in challenging environments for monocular predictors. As the authors mentioned , the local planar primitives can result in severe inconsistency across views (Line 194). \n\n3. Over-Segmentation Issue: As highlighted in the paper, the Segment Anything Model (SAM) tends to over-segment planes, resulting in multiple smaller plane segments for a single surface. Although this is managed in the training process, it may require further refinement to avoid segmentation inconsistencies in complex scenes. From the visualization on the GitHub page, it seems to me that some of the small segments usually resulted in inaccuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TGTuOicnT6", "reviewer": "Reviewer_YwVB", "summary": "This paper proposed a framework called NeuralPlane to reconstruct 3D indoor scenes as planar primitives from posed 2D images. The author first employs 2D prior models to generate local planar primitives, then uses the geometric and semantic priors to guide the NeRF-style reconstruction learning. Finally, a decoding algorithm is designed to extract the global explicit plane mesh from the learned neural field.\nExtensive experiments are conducted to evaluate the performance on ScanNetv2 and ScanNet++ datasets.", "questions": "* What's the computation complexity of this work? Like the GPU memory usage and training time? How's it compared to related works?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposed a comprehensive 3D reconstruction framework with plane primitives. \n\n* Compared to some of the similar works that only focus on detecting the planes, this work also utilizes the detected planes to guide neural field learning.\n\n* The presentation of this paper is clear and concise. \n\n* The qualitative and quantitative results look promising.", "weaknesses": "* This work involves a lot of submodules, especially the neural field with geometric, semantic, and coplanar features could be computationally expensive.\n\n* A concurrent work is very relevant to this paper and could be discussed in the related works section:\nChen, Zheng, et al. \"PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields.\" arXiv preprint arXiv:2401.00871 (2023).\n\n* The text illustration and conceptual figure of Neural Parser could be improved, current version is not clear enough and easy to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)