Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain Model

Authors: Yaxuan Huang, Xili Dai, Jianan Wang, Xianbiao Qi, Yixing Yuan, Xiangyu Yue
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: layout reconstruction, holistic 3D representation, large 3D model.

Abstract:
Room layout estimation from multiple-perspective images is poorly investigated due to the complexities that emerge from multi-view geometry, which requires muti-step solutions such as camera intrinsic and extrinsic estimation, image matching, and triangulation. However, in 3D reconstruction, the advancement of recent 3D foundation models such as DUSt3R has shifted the paradigm from the traditional multi-step structure-from-motion process to an end-to-end single-step approach.
To this end, we introduce Plane-DUSt3R, a novel method for multi-view room layout estimation leveraging the 3D foundation model DUSt3R. Plane-DUSt3R incorporates the DUSt3R framework and fine-tunes on a room layout dataset (Structure3D) with a modified objective to estimate structural planes. By generating uniform and parsimonious results, Plane-DUSt3R enables room layout estimation with only a single post-processing step and 2D detection results. 
Unlike previous methods that rely on single-perspective or panorama image, Plane-DUSt3R extends the setting to handle multiple-perspective images. Moreover, it offers a streamlined, end-to-end solution that simplifies the process and reduces error accumulation.
Experimental results demonstrate that Plane-DUSt3R not only outperforms state-of-the-art methods on the synthetic dataset but also proves robust and effective on in the wild data with different image styles such as cartoon. Our code is available at: https://github.com/justacar/Plane-DUSt3R

Decision: accept (poster)
Reviewer scores: 10, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730018632946

thread_type:
official_review

<thread_object>
creator: reviewer_urzg

<content>
summary:
This work propose Plane-DUSt3R to reconstruct room layout from sparse views with unknown poses. The DUSt3R is retrained to have the amodal perception to see through the occluder and predict floors, walls, ceilings pointmap. The unknown camera parameters can then be estimated from the predicted point map. The outcome from Plane-DUSt3R is then integrated with the other prediction from single-view layout model (e.g., plane instance masks), which is then formulated as a minimum cut problem to produce the final room layout.

strengths:
+ This work introduce state-of-the-art geomety prediction model, DUSt3R, into the classical layout estimation task, solving an underexplored aspect with sparse views and unknown poses.
+ The problem formulation in Sec.3.1 is clear and fluent, which unifies previous room layout estimator in a single pipeline and help readers understand how the propose method augments the existing pipeline.
+ A new benchamrk with some reasonable baselines is constructed for this task.

weaknesses:
- In Table2, the performance by the proposed method on Co3Dv2 and RealEstate10K are missing. It's reasonable that DUSt3R and MASt3R perform worse as they didn't trained on the synthetic Structured3D dataset. What about the generalized performance of the proposed Plane-DUSt3R on the other real-world dataset?

- It's a pity that the proposed Plane-DUSt3R is only trained and mainly evaluated on the synthetic Structured3D dataset. There are many existing resource with multiview and room layout annotation on the 360 panorama domain, e.g., ZInD[1] has multiview layout on unfurnished rooms, MP3DLayout[2] has single-view layout annotation but the source Matterport3D also offer nearby views. Gather more data by projecting these 360 to perspective can make this work much stronger.
  - [1] Zillow Indoor Dataset: Annotated Floor Plans With 360º Panoramas and 3D Room Layouts
  - [2] Manhattan Room Layout Reconstruction from a Single 360° image: A Comparative Study of State-of-the-art Methods

- The qualitative comparison in Fig.6 and Fig.8 is unfair to the baseline. In the fourth column, the baseline "Noncuboid+MASt3R" is presented without texture. We can then easily spot one big issue that some of the walls are duplicated and misaligned. However, in the final column, the layout wireframe from the proposed method is hiding by the rgb point cloud, making it difficult to judge if the similar issue happend to the proposed method as well.

- Too few qualitative results. On the in-domain Structured3D dataset, only two are provided in Fig.6. The additional three in supp's Fig.9 also has two duplicated scenes from Fig.6. In application of room tour and room demo, the final visual outcome is what we really care about while the improvement on number is sometimes hard to interpret how it affect the final visual. In addition, some more visualization, espeically on the failure cases, may can provide some hint for future work to further improve.

questions:
Seems that Sec3 only covers camera poses. Is the camera intrinsic also estimated as in DUSt3R or assumed to be known in this work?

How the accuracy vary with the number of input views? Is the performance scalable with more input views?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674396048

thread_type:
official_review

<thread_object>
creator: reviewer_oarr

<content>
summary:
This paper introduces Plane-DUSt3R, a method that leverages the DUSt3R framework to estimate structural planes from multi-view images by finetuning on a room layout dataset (Structure3D). This approach allows for room layout estimation with just a single post-processing step and 2D detection results, handling multiple-perspective images. Experimental results show that Plane-DUSt3R outperforms existing methods on synthetic datasets and is robust across various real-world image styles, including cartoons.

strengths:
1. The major strength of this paper is the first time addressing 3D room layout estimation from multi images by using large reconstruction model s (i.e., DUST3R). 
2. To support using DUST3R, this paper proposed Plane-DUSt3R to estimate structural planes from multi-view images by only require a single post-processing step.
3. Plane-DUSt3R achieves the best performance comparing with the baselines.

weaknesses:
1. I think the major contribution of the performance boost should go to Dust3R architecture and its pretrained weight, as it is a strong prior model that gives stable and faithful 3D points and camera poses output for Plane-DUSt3R.

2. The key model of this paper is only the post-processing step of extracting layout planes, which is, technically speaking, quite incremental.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708510779

thread_type:
official_review

<thread_object>
creator: reviewer_h8ec

<content>
summary:
In this work authors propose sparse-view layout reconstruction pipeline that combines existing single view layout methods such as [1] and sparse view 3D reconstruction method Dust3R [2], to generate more accurate 3D layout from sparse images. 

Authors retrained [1] to more accurately detect 2D premitives. They also retrain Dust3R, to only predict layout-plane pointmaps, essentially ignoring foreground furniture. They call this new model Plane-Dust3R. Authors train their PlaneDust3R method on synthetic multi-view Structure3D dataset. And compare accuracy of multiple components of their pipeline, on this synthetic dataset. 

From 2D primitives and 3D plane pointmaps, authors design post-processing pipeline that converts them to 3D plane equations, and their relationships, resulting in full 3D layout representations. 


[1] Yang, Cheng, et al. "Learning to reconstruct 3d non-cuboid room layout from a single rgb image." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[2] Wang, Shuzhe, et al. "Dust3r: Geometric 3d vision made easy." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

strengths:
Proposed sparse multi-view layout method is first of its kind. Authors make use of existing methods and retrain them to make them work for their task. 

Overall pipeline is shown to work better than naively combining existing methods such as NonCuboid + Mast3R / NonCuboid + GT Pose. 

Overall paper is well written, is easy to follow. There are many components in the pipeline, but authors explain them in an understandable manner.

weaknesses:
The big limitation of method is that it is only shown to work on synthetic dataset. While availability of real multi-view layout dataset is scarse, there exists many multi-view 3D datasets (such as ScanNet, ARKit Scene) etc. These datasets have ground-truth depth maps and camera poses that can be used for pseudo ground-truth generation and the proposed method can be compared against other layout reconstruction methods on such dataset. Authors do provide qualitative results on some real-world data, but it is not enough to assess accuracy of method on the real data. If authors can show effectiveness of the method on real data this method can be promising.

Second, I am not convinced as to why PlaneDust3R is needed. If authors take regular Dust3R and use off-the-shelf semantic segmentation to segment walls, floors, and ceilings and use only those points to generate layout (by combining this with NonCuboids + f3 post-processing proposed in the paper), that might also be enough. Retraining method only on synthetic dataset such as Structure3D might reduce their accuracy on the real-world dataset.

questions:
In Table-2, other datasets results are not useful since this method does not have numbers for those. I would suggest authors to keep results for only structure 3D.

3D precision and recall should be reported at different threshold. I think 15deg threshold is pretty high. Maybe authors should show results at 5deg, 15deg, and 30deg angular thresholds, and similar translation thresholds. This will give more comprehensive view of the methods performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731155617522

thread_type:
official_review

<thread_object>
creator: reviewer_6jgc

<content>
summary:
The paper proposes an extension of DUSt3R to room layout reconstruction by retraining and using additional processing steps. Results demonstrate superior performance.

strengths:
- The method is sound with great results outperforming other methods.
- This is the first method for unposed sparse view room layout reconstruction, especially when the views are not overlapping. 
- The whole idea is interesting, especially to use structural plane depth map and metric scale for DUSt3R. 
- The evaluation is great, especially the design of baselines.
- The paper is well-written and was a pleasure to read.

weaknesses:
- Evaluation on more datasets would be more interesting, e.g. CAD-Estate dataset with structural elements annotations.

questions:
- Figures seem to have low resolution since they are most likely represented as jpg/png images. I recommend using vector graphics instead for better visualisations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732176573705

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We appreciate your valuable feedback and your recognition of our work’s novelty and performance. We noticed that there are common concerns about only training and evaluating on one synthetic dataset, and a desire to see quantitative results on some real-world data. We have carefully considered your suggestion and conducted additional experiments on the CAD-Estate dataset. Our choice of CAD-Estate is motivated by several reasons: 1. Data Quality and Scale: CAD-Estate provides room layouts from 2246 videos, offering a larger and more diverse evaluation base compared to alternatives like ScanNet layout [1] (293 views). 2. Multi-view Completeness: 360-panorama datasets primarily offer rotational variations, while CAD-Estate provides multiple views with both rotational and translational differences for each scene. 3. Practical Considerations. CAD-Estate is already well-established and has a much more similar annotation style to Structured3d. We believe the evaluation results conducted on this dataset are representative.

For evaluation settings, despite CAD-Estate's similar annotation style, some notable differences exist. Our method and Structured3D operate under the assumption of a single floor, single ceiling, and multiple walls configuration. CAD-Estate, however, presents more complex scenarios, including multiple ceiling levels (particularly in attic rooms) and interconnected rooms through open doorways (whereas Structured3D treats doorways as complete walls). To ensure a fair comparison, we carefully selected a subset of CAD-Estate data that aligns with  Structured3D's annotation style. Our final evaluation dataset consists of 100 scenes containing 469 images, with each scene containing 2 to 10 images. 

We report performance using both 2D metrics (IoU and pixel error) and 3D metrics (precision and recall). While CAD-Estate's label classes include ["<ignore>", "wall", "floor", "ceiling", "slanted"], we only focus on wall, floor, and ceiling classes. We utilize the dataset's provided intrinsic parameters for reprojection during the evaluation. Results are reported for both "Noncuboid + GT pose" and "Plane-DUSt3R (metric)". Please refer to our supplementary materials for qualitative results.

| Method |re-IoU (%)↑ | re-PE (%)↓ |3D-Precision (%)↑ |3D-Recall (%)↑ |
|--------|----------|----------|---------|---------|
| Noncuboid + GT pose on cad-estate | 55.99 | 20.33 |15.59|**30.28**|
| Ours (metric) on cad-estate | **63.14** | **15.15** | **22.58**|26.55|


Our method achieves performance in re-loU (63.14% vs 55.99%) and re-PE (15.15% vs 20.33%), along with higher 3D precision (22.58% vs 15.59%) at 15 degrees and 0.2m thresholds. The relatively low precision scores can be attributed to our model's bias from training on Structured3D, where most adjacent walls are predominantly orthogonal and ceilings are typically horizontal. In contrast, CAD-Estate contains more diverse architectural features, including non-orthogonal walls and slanted ceilings. While the baseline method shows higher 3D recall, this is likely due to its tendency to generate duplicated planes and its utilization of ground-truth pose information. Additionally, some performance discrepancies may be influenced by some annotation inconsistencies in CAD-Estate's 3D triangle meshes.

[1] https://github.com/vevenom/ScanNet-Layout
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a pipeline for multi-view room layout estimation, leveraging the 3D foundation model DUSt3R. Experiments on both synthetic and real-world datasets demonstrate its improved performance and generalization.

Main strengths of this paper are as below:

- The paper presents a new pipeline for multi-view room layout estimation, simplifying traditionally complex multi-view geometry into a single-step solution using the 3D foundation model DUSt3R.

- Experiments on synthetic and real-world datasets validate its improved performance and generalization.

- The paper is well-organized and offers comprehensive experimental results, making it accessible and providing valuable insights.

Needing more experiments were raised by the reviewers. Also, the new part different from the previous methods need more detailed depictions. Please revise the paper according to the discussions before submitting the final version.