NeuralPlane: Structured 3D Reconstruction in Planar Primitives with Neural Fields

Authors: Hanqiao Ye, Yuzhou Liu, Yangdong Liu, Shuhan Shen
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3d reconstruction, 3d scene understanding, scene abstraction, neural rendering

Abstract:
3D maps assembled from planar primitives are compact and expressive in representing man-made environments. In this paper, we present **NeuralPlane**, a novel approach that explores **neural** fields for multi-view 3D **plane** reconstruction. Our method is centered upon the core idea of distilling geometric and semantic cues from inconsistent 2D plane observations into a unified 3D neural representation, which unlocks the full leverage of plane attributes. It is accomplished through several key designs, including: 1) a monocular module that generates geometrically smooth and semantically meaningful segments known as 2D plane observations, 2) a plane-guided training procedure that implicitly learns accurate 3D geometry from the multi-view plane observations, and 3) a self-supervised feature field termed *Neural Coplanarity Field* that enables the modeling of scene semantics alongside the geometry. Without relying on prior plane annotations, our method achieves high-fidelity reconstruction comprising planar primitives that are not only crisp but also well-aligned with the semantic content. Comprehensive experiments on ScanNetv2 and ScanNet++ demonstrate the superiority of our method in both geometry and semantics.

Decision: accept (oral)
Reviewer scores: 8, 6, 10, 8

Reviews and discussion:
<thread>
creation date:
1729933044207

thread_type:
official_review

<thread_object>
creator: reviewer_ywvb

<content>
summary:
This paper proposed a framework called NeuralPlane to reconstruct 3D indoor scenes as planar primitives from posed 2D images. The author first employs 2D prior models to generate local planar primitives, then uses the geometric and semantic priors to guide the NeRF-style reconstruction learning. Finally, a decoding algorithm is designed to extract the global explicit plane mesh from the learned neural field.
Extensive experiments are conducted to evaluate the performance on ScanNetv2 and ScanNet++ datasets.

strengths:
This paper proposed a comprehensive 3D reconstruction framework with plane primitives. 

* Compared to some of the similar works that only focus on detecting the planes, this work also utilizes the detected planes to guide neural field learning.

* The presentation of this paper is clear and concise. 

* The qualitative and quantitative results look promising.

weaknesses:
* This work involves a lot of submodules, especially the neural field with geometric, semantic, and coplanar features could be computationally expensive.

* A concurrent work is very relevant to this paper and could be discussed in the related works section:
Chen, Zheng, et al. "PlanarNeRF: Online Learning of Planar Primitives with Neural Radiance Fields." arXiv preprint arXiv:2401.00871 (2023).

* The text illustration and conceptual figure of Neural Parser could be improved, current version is not clear enough and easy to follow.

questions:
* What's the computation complexity of this work? Like the GPU memory usage and training time? How's it compared to related works?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618030868

thread_type:
official_review

<thread_object>
creator: reviewer_1czj

<content>
summary:
The paper presents NeuralPlane, a novel approach to 3D plane reconstruction that utilizes neural fields for generating structured 3D maps from multi-view images without the need for plane annotations. The method emphasizes two main aspects: geometry and semantics. Key contributions include:
1. Monocular Plane Segmentation: A monocular module extracts geometrically smooth (based on off-the-shelves Surface Normal Predictor) and semantically meaningful 2D plane observations (based on Segmenet Anything Model). 
2. Plane-Guided Neural Representation: The model utilizes these 2D segments to train a neural field that captures accurate 3D plane locations. A surface normal regularization and pseudo-depth regularization terms are proposed.
3. Neural Coplanarity Field: This self-supervised feature field enables semantic consistency within the 3D reconstruction by grouping planar regions that share coplanar relationships. A contrastive loss is proposed to distinguish between planes with similar geometric properties but different semantic properties. 
The method demonstrates superior performance on ScanNetv2 and ScanNet++ datasets, indicating its effectiveness in indoor environments.

strengths:
1. Novelty in Combining Geometry and Semantics: The method’s approach to merging geometry with semantic information through a neural coplanarity field is innovative, enhancing the semantic consistency of 3D reconstructions. A complex system with multiple stages is proposed to estimate/reason the local planar regions and the associated parameters, to associate the planes in 3D using radiance field, to resolve semantic conflicts using Neural Coplanarity Field.
2. High-Quality Reconstruction: Experimental results indicate that NeuralPlane achieves fine-grained and coherent plane reconstructions, outperforming existing methods in most of the metrics. 
3. Efficiency: NeuralPlane’s volume density representation allows for faster training compared to implicit methods, an important practical advantage.
4. Extensive ablation studies: the authors include sufficient ablation studies to support the effectiveness of the proposed modules.
5. Good writing: the paper is clearly written that the technical details are clearly presented.

weaknesses:
1. Complexity of Methodology: The proposed method involves several stages, including monocular plane segmentation, neural coplanarity field training, and plane extraction. In each stage, there are several submodules and I believe there are some hyperparameters decisions in each stage. While effective, this complexity (especially the combination of submodules and associated parameters) may impact its scalability to larger scenes or generalizability. For example, K-means clustering on predicted normal map, mask size threshold in SAM, thresholds to form negative pairs in Neural Coplanarity Field, Loss balancing parameters, etc. It would be great if the authors could share the insights on the impact of these hyperparameter settings on different scenes. For example, is a universal set of hyperparameters applied to all the test scenes? It would be great if the authors could provide a sensitivity analysis or ablation study on key hyperparameters across different scenes. This would help clarify how robust the method is to parameter changes and whether a universal set of parameters is feasible.

2. Dependence on Initial 2D Plane Segmentation Quality: As the regularizations are based on the quality of initial local plane geoemtry, the method’s success can depend on the quality of 2D plane segments obtained from monocular priors, which may introduce inaccuracies in challenging environments for monocular predictors. As the authors mentioned , the local planar primitives can result in severe inconsistency across views (Line 194). 

3. Over-Segmentation Issue: As highlighted in the paper, the Segment Anything Model (SAM) tends to over-segment planes, resulting in multiple smaller plane segments for a single surface. Although this is managed in the training process, it may require further refinement to avoid segmentation inconsistencies in complex scenes. From the visualization on the GitHub page, it seems to me that some of the small segments usually resulted in inaccuracy.

questions:
1. Eqn 4: n_i is not defined, is it the surface normal of the selected local planar primitive? Please explicitly define n_i in the text or equation, and confirm if it refers to the surface normal of the local planar primitive.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670032348

thread_type:
official_review

<thread_object>
creator: reviewer_6weq

<content>
summary:
This paper presents NeuralPlane, a method for reconstructing 3D scene plane primitives via neural fields without GT plane labelling. The method is divided into three main stages: firstly, it combines pre-trained normal prediction and SAM to generate initial 2D planar segments and estimates their 3D parameters using SfM keypoints. Secondly, it optimises two neural fields, a density field based on planar geometric constraints, and a coplanar neural field that understands the semantic relationships between regions . The neural coplanar field is followed by a neural parser module that helps to model the learned coplanar relations. Finally, the optimised neural representations are converted into explicit 3D planes through point sampling, feature-based clustering and RANSAC fitting. The method is evaluated on the ScanNetv2 and ScanNet++ datasets, and the results show that it outperforms both the learning-based method and the Geometry+RANSAC method in terms of geometric and semantic metrics.

strengths:
- This paper is well-written and easy to understand.
- The proposed method does not require ground-truth plane annotations, as it can learn effectively from noisy monocular model outputs.
- The method demonstrates SOTA performance and achieves clean plane segmentation results.

weaknesses:
- The proposed method involves numerous hyperparameters, including balancing parameters for loss, the number of semantic prototypes, and parameters listed in Lines 850-863.
- As a complex system, it is important to discuss and present failure cases to help readers understand the method’s limitations.

questions:
Since it is a complex system paper, making it hard to reproduce, will the code be publicly avaliable?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714242084

thread_type:
official_review

<thread_object>
creator: reviewer_fcsv

<content>
summary:
This paper presents a neural 3D reconstruction system on 3D plane reconstruction of indoor scenes. Inspired by the recent success of neural radiance field and image foundation models (SAM2), this paper presents a multi-view 3D reconstruction pipeline leveraging these techniques. The training scheme consists of three phases: Initializing plane segments and parameters -> optimizing a neural feature field for plane-specific feature representation (encouraged by a list of geometry-guided loses) -> plane extraction by grouped features and RANSAC. Experiments are conducted extensively on two representative indoor datasets: ScanNet and 7-scenes and are compared with a group of competitve baseline methods.

strengths:
1. The unique advantage of the system is the association of geometry and semantic features for plane reconstruction problem, which requires both geometry-aware perception and semantic-aware grouping.
2. The paper leverages foundation models to achieve plane segment initializaing and utilize geometry-driven losses to optimize the system. There is no groundtruth plane segmentation or geometry label required. 
3. Overall the paper and diagrams are well written and presented.
4. The experimental comparison and thorough and convincing on both plane geometry reconstruction and segmentation.

weaknesses:
1. I think the major advantage of this paper is the unsupervised learning paradigm on the plane reconstruction problem. However, since both ScanNet and ScanNet++ has groundtruth images, this unique advantages seems not to be fully enjoyed and reflected. So I suggest authors to apply the proposed system on some outdoor scenes containing plane structures (such as autonomous driving dataset or street view datasets), to verify the adaptability of the method.
 
2. I have tested PlaneRecon in my previous projects. It can incrementally reconstructs planes in an online and real-time manner. Besides, it can trains over multiple scenes and directly test without any test-time optimization. Although its precision should fall behind than the neural reconstruction papers, the generalizability and speed is higher than the proposed method. I am curious on whether the paper has such potential to tackle these limitations.

3. Missing a few related works: (1) Recovering 3d planes from a single image via convolutional neural networks (2) PlaneMVS: 3D Plane Reconstruction From Multi-View Stereo (3) Single-image piece-wise planar 3d reconstruction via associative embedding.

4. On quantitative evaluation, it is unclear that what is the groundtruth is here. Does it stand for the groundtruth planar part only or the entire mesh? As a plane reconstruction paper, I think the former one should be more reasonable. If so, the first several methods listed in Table 1 cannot directly be compared with the proposed method. Authors should make it clearer on this part.

5. Is the proposed method robust to the hyperparameters listed in the paper especially (1) to,tn and m in the push loss during training and (2) the parameters selected in RANSAC during plane fitting? Some ablation studies on hyper-parameter robustness are expected to make the method more generalizable across most indoor scenes, since the geometric scale and semantic distribution can have large variance among scenes.

questions:
I am impressed by the technical contribution made by authors for this work. However, there also exist a few major concerns for me at this time which discourage me to grant this paper a higher value. 

Please try to address my concerns listed in the weakness part. I will accordingly consider to improve my overall rating if the concerns are well solved.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732116483733

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response by Authors

comment:
We are deeply grateful to the reviewers for dedicating their time and effort to the reviewing process.
We are pleased to note that the reviewers find the presentation of the paper clear and concise (FCSv, 6wEq, 1CZJ, YwVB), and acknowledge the novelty of our work in associating geometry and semantics (FCSv, 1CZJ, YwVB) as well as delivering high-quality reconstruction without requiring plane annotations (FCSv, 6wEq, 1CZJ, YwVB).

We also notice that the reviewers' major concerns mainly lie in the **complexity of methodology** (6wEq, 1CZJ, YwVB) and its **robustness to hyperparameters** (FCSv, 6wEq, 1CZJ, YwVB).
Accordingly, during the rebuttal period, we measured the average training time and peak GPU memory consumption of each proposed module, and
conducted more detailed ablation studies on the ScanNetv2 dataset, with a particular focus on the analysis of hyperparameter sensitivity.
These additional measurements and experimental results presented below serve as general responses to the reviewers' shared concerns.
We hope these efforts will comprehensively evaluate the performance of our work and provide further clarity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553761547

thread_type:
official_comment

<thread_object>
creator: area_chair_zulq

<content>
title:
Please read the rebuttal and reply

comment:
Dear Reviewers,

Thanks again for serving for ICLR, the discussion period between authors and reviewers is approaching (November 27 at 11:59pm AoE), please read the rebuttal and ask questions if you have any. Your timely response is important and highly appreciated.

Thanks,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes to method for multi-view 3D plane reconstruction. The main idea is to utilize foundational models to provide prior (normal, segmentation, etc) and then employ neural fields to learn a plane field that is aware of both geometry and scene semantics. The propose method presents an unified framework for both geometry and semantics, as well as a way to exploit strong prior from other models. During rebuttal, the major weakness raised by the reviewers is the complexity of the proposed method, which includes multiple modules with a handful of hyperparameters to tune. Other weaknesses include low efficiency due to the neural fields, missing related works, dependency of foundational models and paper presentation. The authors have actively addressed the concerns and provide supporting materials. After rebuttal, all reviewers suggest to accept the paper.