Unposed Sparse Views Room Layout Reconstruction in the Age of Pretrain Model
Authors: Yaxuan Huang
Xili Dai
Jianan Wang
Xianbiao Qi
Yixing Yuan
Xiangyu Yue
Venue: ICLR 2025 Poster (2025)
Keywords: layout reconstruction
holistic 3D representation
large 3D model.
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Room layout estimation from multiple-perspective images is poorly investigated due to the complexities that emerge from multi-view geometry, which requires muti-step solutions such as camera intrinsic and extrinsic estimation, image matching, and triangulation. However, in 3D reconstruction, the advancement of recent 3D foundation models such as DUSt3R has shifted the paradigm from the traditional multi-step structure-from-motion process to an end-to-end single-step approach.
To this end, we introduce Plane-DUSt3R, a novel method for multi-view room layout estimation leveraging the 3D foundation model DUSt3R. Plane-DUSt3R incorporates the DUSt3R framework and fine-tunes on a room layout dataset (Structure3D) with a modified objective to estimate structural planes. By generating uniform and parsimonious results, Plane-DUSt3R enables room layout estimation with only a single post-processing step and 2D detection results. 
Unlike previous methods that rely on single-perspective or panorama image, Plane-DUSt3R extends the setting to handle multiple-perspective images. Moreover, it offers a streamlined, end-to-end solution that simplifies the process and reduces error accumulation.
Experimental results demonstrate that Plane-DUSt3R not only outperforms state-of-the-art methods on the synthetic dataset but also proves robust and effective on in the wild data with different image styles such as cartoon. Our code is available at: https://github.com/justacar/Plane-DUSt3R

Published as a conference paper at ICLR 2025
UNPOSED SPARSE VIEWS ROOM LAYOUT RECON- STRUCTION IN THE AGE OF PRETRAIN MODEL
Yaxuan Huang'
Xili 2
Jianan Wang2 Xianbiao Qi Yixing Yuan
Xiangyu Yue
Hong Kong Center for Construction Robotics, The Hong Kong University of Science and Technology
The Hong Kong University of Science and Technology (Guangzhou)
AAstribot *Intellifusion Inc.
Multimedia Lab (MMLab) and SHIAE, The Chinese University of Hong Kong
In-domain
Out-of-domain
ser
In-the-add
Input
Layout estimation
Input
Layout estimation
Figure 1: We present novel method for estimating room layouts from a set of unconstrained indoor images. Our approach demonstrates robust generalization capabilities, performing well on both in-
the-wild datasets (Zhou et al. 2018) and out-of-domain cartoon (Weber et al. 2024) data.
ABSTRACT
Room layout estimation from due to the complexities that emerge from multi-view geometry, which requires muti-step solutions such as camera intrinsic and extrinsic estimation, image matching, and triangulation. However, in 3D reconstruction, the advancement of recent 3D foundation models such as DUSt3R has shifted the paradigm from the traditional multi-step structure-from-moott approach. To this end, we introduce DlaneRDUSt3R, novel method for multi- view room layout estimation leveraging the 3D foundation model DUSt3R. Plane- DUSt3R incorporates the DUSt3R framework and fine-tunes on a room layout dataset (Structure3D) with a modified objective to estimate structural planes. By generating uniform and parsimonious results, Plane-DUS(32 enables room layout estimation with only single post-processing step and 2D detection results. Un- like previous methods that rely single persspetrsee or panorama image, Plane- DUSt3R extends the setting to handle multiple-perse perspective images. Moreover, it offers streamlined, end-to-end solution that simplifies the process and reduces er- ror accumulation. Experimental results demonstrate that Plane-DUSt3R not only outperforms state-of-the-ar methods on the synthetic dataset but also proves ro- bust and effective on in the wild data with different image styles such as cartoon. Our code is available at: https:/githubb.co//jsaaaaae haaam//uutaaaraaanDDaaaa PlaneenSnneRR
images is poorly investigated
process to an end-to-end single-step
INTRODUCTION
3D room layout estimation aims to predict the overall spatial structure of indoor scenes, playing crucial role in understanding 3D indoor scenes and supporting wide range of applications. For example, room layouts could serve as a reference for aligning and connecting other objects in indoor environment reconstruction (Nie al. 2020). Accurate layout estimation also aids robotic path planning and navigation by identifying passable areas (Mirowski et al.) 2016). Additionally, room
Equal contribution, Corresponding author

---

Published as a conference paper at ICLR 2025
layouts essential in tasks such augmented reality (AR) where spatial understanding is critical. Therefore, 3D room layout estimation has attracted considerable research attention with continued
development of datasets (Zheng et al., 2020 Wang et al.) 2022) and methods (Yang et al. 2022
Stekovic et al. 2020 Wang et al. 2022) over the past few decades.
Methods for 3D room layout estimation (Zhang al. 2015 Hedau et al. 2009 Yang et al. 2019)
initially relied on the Manhattan assumption with a single perspective or panorama image as input. Over time, advancements (Stekovic al. 2020) have relaxed the Manhattan assumption to accom- modate more complex settings, such as the Atlanta model, or even no geometric assumption at all. Recently, Wang et al. (2022) introduced a "ieuw""view" approach, capturing single room with two panorama images, marking the first attempt to extend the input from a single image to multiple im- ages. Despite this progress, exploration in this direction remains limited, hindered by the lack of well-annotated multi-view 3D room layout estimation dataset.
Currently, multi-view datasets with layout annotations are very scarce. Even the few existing datasets, such as Structure3D (Zheng et al. 2020), provide only a small number of perspective views (typically ranging from to 5). This scarcity of observable views highlights a critical issue: wide- baseline sparse-view structure from motion (SfM) remains an open problem. Most contemporary multi-view methods (Wang et al., 2022 Hu et al. 2022) assume known camera poses or start with noisy camera pose estimates. Therefore, solving wide-basche aade-baseline sparse- view SiM would significantly advance the field of multi-view 3D room layout estimation. The recent development of large-scale training and improved model architecture offers potential solution. While GPT-3 (Brown) 2020) and Sora (Brooks et al. (2044) have revolutionized NLP and video generation, DUS(3R (Wang et al. 2024) brings a paradigm shift for multi-view 3D reconstruction, transitioning from multi-step SIM process to an end-to-end approach. DUSt3R demonstrates the ability to reconstruct scenes from unposed images, without camera intrinsic/extrinsic or even view overlap. For example, with two unposed, potentially non-overiapping views, DUSt3R could generate 3D pointmap while inferring reasonable camera intrinsic and extrinsic, providing an ideal solution to the challenges posed by wide-baseline sparse-view SfM in multi-view 3D room layout estimation.
In this paper, we employ DUSt3R to tackle the multi view 3D room layout estimation task. Most single-view layout estimation methods (Yang et al.] (022) follow two-step process: 1) extracting 2D & 3D information, and 2) lifting the results to a 3D layout with layout priors. When extending this approach to multi-view settings, additional step is required: establishing geometric primitive correspondence across multi-view before the 3D lifting step. Given the limited number of views in existing multi-view layout datasets, this correspondenc-sttaliihhi a sparse-view SfM problem. Hence, incorporating a single-view layout estimation method with DUSt3R to handle multi-view layout estimation is natural approach. However, this may introduce a challenge: independent plane normal estimation for each image fails to leverage shared informa- tion across views, potentially reducing generalizability to unseen data in the wild. To this end, we adopt DUSt3 to solve correspondence establishement and 3D lifting simultaneously. which jointly predict plane normal and lift 2D detection results to 3D. Specifically, we modify DUSt3R esti- mate room layouts directly through dense 3D point representation (pointmap). focusing exclusively on structural surfaces while ignoring occlusions. This achieved by retraining DUSt3R with the objective to predict only structural planes, the resulting model is named Plane-DUSt3R However, dense pointmap representation is redundant for room layout, as a plane can be efficiently represented by its normal and offset rather than large number of 3D points, which may consume significant space. To streamline the process, we leverage well-established off-the-shelf 2D plane detector to guide the extraction of plane parameters from the pointmap. We then apply post-processing to ob- tain plane correspondences across different images and derive their adjacency relationships.
step essentially becomes
Compared to existing room layout estimation methods, our approach introduces the first pipeline capable of unposed multi-view (perspective images) layout estimation. Our contributions can be summarized as follows:
1. We propose an unposed multi-view (sparse-view) room layout estimation pipeline. To the best of our knowledge, this is the first attempt at addressing this natural yet underexplored setting in room layout estimation.
2. The introduced pipeline consists of three parts: 1) 2D plane detector, 2) a 3D information prediction and correspondence establishment method, Plane-DUStaB and 3) a post- processccns algorithm. The 2D detector was retrained with SOTA results the Structure3D dataset (see Ta-
2

---

Published as a conference paper at ICLR 2025
ble 3). The Plane-DUStSy achieves 5.27% and 5.33% improvement in RRA and mAA metries, for the multi-view correspondence task compared to state-of-the-ai methods (see Table
3. In this novel setting, we also design several baseline methods for comparison to validate the advantages of our pipeline. Specifically, we outperform the baselines by 4 projection 2D metrics pipeline not only performs well
and 1 3D metric respectively (see Table [). Furthermore, on the Structure3D dataset (see Figure 6). but also generalizes effectively to in-the-wild datasets (Zhou et al. 2018) and scenarios with different image styles such cartoon style (see Figure
2 RELATED WORK
Layout estimation. Most room layout estimation research focuses single-perspective image inputs. Stekovic et al. (2020) formulates layout estimation as a constrained discrete optimization problem to identify 3D polygons. Yang et al. (2022) introduces line-plane constraints and connec- tivity relations between planes for layout estimation, while [Sun et al. (019)) formulates the task as predicting 1D layouts. Other studies, such Zou al. (2018), propose to utilize monocular 360-degree panoramic images for more information. Several works extend the input setting from single panoramic to multi-view panoramic images, e.g. |Wann al.] (2022) and (Hu al. (2022). However, there is limited research addressing layout estimation from multi-view RGB perspective images. Hooward-ennins et al. (2019) detects and regresses 3D piece-wise planar surfaces from series of images and clusters them to obtain the final layout, but this method requires posed images. The most related work is Jin al. (20TT) which focuses on a different task: reconstructing indoor scenes with planar surfaces from wide- baselae, unposed images. It is limited to two views and requires an incremental stitching process to incorporate additional views.
Holistic scene understanding. Traditional 3D indoor reconstruction methods are widely appli- cable but often lack explicit semantic information. To address this limitation, recent research has increasingly focused on incorporating holistic scene structure information, enhancing scene under- standing by improving reasoning about physical properties, mostly centered single-perspective images. Several studies have explored the detection of 2D line segments using learming- based de- tectors (Zhou et al.) 2019; Pautrat al.] 2021 Dai et al., 2022). However, these approaches of- ten struggle to differentiate between texture-based lines and structural lines formed by intersecting planes. Some research has focused on planar reconstruction to capture higher-level information (Liu et al. 2018 Yu et al., 2019 Liu et al. 2019). Certain studies (Huang et al., 2018 Nie et al.) 2020 Sun et al. 2021) have tackled multiple tasks alongside layout reconstruction, such as depth estima- tion, object detection, and semantic segmentation. Other works operate on constructed point maps; for instance, Yue et al. (2023) reconstructs floor plans from density maps by predicting sequences of room corners to form polygons. SceneScript (Avetisyan et al. 2024) employs large language models to represent indoor scenes as structured language commands.
Multi-view pose estimation and reconstruction. The most widely applied pipeline for pose es- timation and reconstruction on a series of images involves SfM (Schönberger & Frahm 2016) and MVS (Schönberger et al., 2016)), which typically includes steps such as feature mapping, finding cor- respondences, solving triangulations and optimizing camera parameters. Most mainstream methods build upon this paradigm with improvements on various aspects of the pipeline. However, recent works such as DUSt3R (Wang et al., 2024) and MASt3R (Leroy et al. (2024) propose a reconstruc- tion pipeline capable of producing globally-aligne achieved by casting the reconstruction problem as a regression of pointmaps, significantly relaxing input requirements and establishing simpler end-to-end paradigm for 3D reconstruction.
pointmaps from unconstrained images. This is
3 METHOD
In this section, we formulate the layout estimation task, transitioning from single-view to a multi- view scenario. We then derive our niew layout estimation pipeline as shown in Figure 2 (Sec- tion 3.1. Our pipeline consists of three parts: 2D plane detector f1, 3D information prediction and correspondence establishment method Plane-DUSta f2 (Section B.2), and a post- processing algorithm fa (Section 3.3)
3

---

Published as a conference paper at ICLR 2025
2D Detection f
Post-procens processingsnng
Single-view Reconstruction
Input
p.l
Single-tee view pianes with 3D parameters (nomalBoltse)
Plane Reconstruction
Reconstruction
Bindview
Plane pointmaps Careea Poses
P. P.L.J.W
Figure 2: Our multi-view room layout estimation pipeline. It consists three parts: 1) 2D plane detector f1. 2) 3D information prediction and correspondence establishment method Plane- DUSt3 f2, and 3) post-preee post-processing algorithm fs.
3.1 FORMULATION THE MULTI-VIEW LAYOUT ESTIMATION TASK
We begin by revisiting the single-vier view layout estimation task and unifying the formulation of exist- ing methods. Next, we extend the formulation from single-view to multiple-view setting, providing detailed analysis and discussion focusing the choice of solutions. Before formulating the lay- out estimation task, we adopt the "geometric primitives relationships" representation from Zheng et al. (2020) to model the room layout.
Geometric Primitives.
The scene layout could be represented set of planes (P1, P2 ...] in 3D space and their corresponding 2D projections {P1, P2, ... in images. Each plane is parameterized by its normal E 5 and offset d. For 3D point E IR3 lying on the plane, we have n x + d = 0.
Planes:
Lines Junction Points: 3D space, two planes intersect at a 3D line, three planes inter- sect 3D junction point. We denote the set of all 3D lines/junction points in the scene as {L1, L2 ...)/(JI,, ...} and their corresponding 2D projections as (1, (1,,2,...1) //ii....//ii.................... images.
Relationships.
Plane/Line relationships: An adjacent matrix Wp/N Wi E 10 is used to model the relation-
ship between planes/lines. Specifically, W, (i.ji)) if and only if P, and P, intersect along line; otherwise, (i,jj = 0. Similarly to plane relationship, Wii,j) = and only if L, and
Ly intersect at certain junetion, otherwise, Wi(i,))
The pipeline of single-view view layout estimation methods (Lu et al. 2019 Yang et al. 2018 Stekovic et al. 2020) can formulated as:
2022 Liu et al.
(2D,3D) 83 (P. (P
where fi is function that predicts 2D and 3D information from the input single view. Gener- ally speaking, the final layout result (P, L, W) can directly inferred from the outputs fi- However, eπrоrs arising from fi usually adversely affect the results. Hence, refinement step that utilizes prior information about room layout is employed to further improve the performance. There- fore, fs typically encompasses post-processing and refinement steps where the post-procens processcnns step generates an initial layout estimation, and the refinement step improves the final results.
For instance. Yang al. (2022) chooses the HRnet network (Wang et al., 2020) fi backbone to extract 2D plane p. line 1, and predict 3D plane normal n and offset from the input single view. After obtaining the initial 3D layout from the outputs of f1, the method reprojects the 3DD

---

Published as a conference paper at ICLR 2025
line to 2D line 1 on the image and compares it with the detected line 1 from fi- fs minimizes the error I1ž - 11l3 to optimize the 3D plane normal. In other words, it uses the better-detected 21) line to improve the estimated 3D plane normal. In contrast, SStekovic et al.] (2020) uses a different approach: its fi predicts a 2.5D depth map instead of 2D line / and uses the more accurate depth results to refine the estimated 3D plane normal. Among the works that follow the general framework I (Liu et al. 2019 2018), Yang et al. (2022) stands out as the best single-view perspective image layout estimation method without relying on the Manhattan assumption. Therefore, we present its formulation in equation (2) and extend it to multi-view scenarios.
f1 (p,l,n,d) fo (P.L,J.W)
In room layout estimation from unposed multi-view images, two primary challenges aris: 1) camera pose estimation, and 2) 3D information estimation from multi-view inputs. Camera pose estimation particularly problematic given the scareity of annotated multi-view layout dataset. Thanks to the recent advancements in 3D vision with pretrain model, this challenge could be effectively bypassed: DUS(3R (Wang et al., (024)) has demonstrated the ability to reconstruct scenes from unposed images without requiring camera intrinsic extrinsic, and even without overlap between views. Moreover, the 3D pointmap generated from DUSt3R can provide significantly improved 3D information, such as plane normal and offset, compared single-view methods (Yang et al., 2022) (see Table [ of experiment section). Therefore, DUSt3R represents a critical advancement in extending single- view layout estimation multí-view scenarios. Before formulating the multi-view solution, we first present the key 3D representation of DUSt3R: the pointmap X and the camera pose T. The camera pose T obtained through global alignment, as described the DUSt3R (Wang al. 2024)).
Given a set of RGB images IZI...., ZI, E R xx captured from distinct viewpoints of the same indoor scene, we associate each image I₁ with a canonical pointmap X; E #x x3 The pointmap represents one-to-one mapping from each pixel (u, v) in the image to a corresponding 3D point in the world coordinate frame: (u, v) E R + X (u, v) E IR3.
Pointmap X:
Camera Pose T: Each image I, is associated with a camera-to-world pose T. E SE(3).
Now, the sparse-view layout estimation problem can be formulated as shown in equation
[I, I2,...
fi.jz (p,1, X, fs (P,L,,,W).
(3)
In this work, we adopt the HRnet backbone from Yang et al. 2022 as ft. In the original DUSt3R (Wang et al. 2024) formulation, the ground truth pointmap X represents the 3D coordi- nates of the entire indoor scene. contrast, we are interested plane pointmap X that represents the 3D coordinates of structural plane surfaces, including walls, floors, and ceilings. This formula- tion intentionally disregards occlusions caused by non-structural elements, such as furniture within the room. Our objective is to predict the scene layout pointmap without occlusions from objects, even when the input images contain occluding elements. For simplicity, any subsequent reference X in this paper refers to the newly defined plane pointmap X We introduce Plane-DUS(3R as f2 and directly infer the final layout via fa without the need for any refinement.
3.2 fe: PLANE-BASED DUST3R
The original DUS(3R outputs pointmaps that capture all 3D information in a scene, including furni- ture, wall decorations, and other objects. However, such excessive information introduces interfer- ence when extracting geometric primitives for layout prediction. such as planes and lines. To obtain structural plane pointmap X, we modify the data labels from the original depth map (Figure (a)) to the structural plane depth map (Figure44(b)) and then retrain the DUSt3R model. This updated objective guides DUS13R to predict the pointmap of the planes while ignoring other objects. The original DUSt3R does not guarantee output at metric scale, so we also trained a modified version of Plane-DUS(3R that produces metric-scale results.
Given set of image pairs P {(I,I,) i * 5.1 < i,j each image pair, the model comprises two parallel branches. As shown in Figure B the detail of the architecture be found in Appendix A The regression loss function is defined as the
I # for
5

---

Published as a conference paper at ICLR 2025
VIT
Transformer
Regression
Weights
Cress Attention
VIT Encoder
Transformer Decoder2
Regression Heed2
Plane pointmaps In common coordinate frame
Figure 3: Plane-DUSt3R architecture remains identical to DUSt3R. The transformer decoder and regression head are further fine-tuned on the occlusion ceeusiioo-eeeeeeeee depth map (see Figure 4
scale-invariant Euclidean distance between the normalized predicted and ground-trute pointmaps: (v, = 1 " where view v E (1,2) and is the pixel index. The scaling factors and 2 represent the average distance of all corresponding valid points to the origin. In addi- tion, by incorporating the confidence loss, the model implicitly learns to identify regions that are more challenging to predict.
DUSt3R (Wang et al., (024). the confidence loss defined as: fcont = Eve (1.2) ΣiEDo C. frege (v, i) - a log C; where De C (1,..., , H} (I,...:We W} are sets valid pixels on which the ground truth defined.
Structural plane depth map. The Structure3D dataset provides ground truth plane normal and offset, allowing us to re-render the plane depth map at the same camera pose (as shown in Figure 4 We then transform the structural plane depth map D2 to pointmap X in the camera coordinate frame v. This transformation given by X the camera intrinsic matrix. Further details of this transformation can be found in Wang al. (2024).
1. K KiD i.j p f.j [.j]
jD D DT where K E R
In the multi-view setting, scale variance is required, which differs from DUSt3R. To accommodate this, we modify the regression loss to bypass normalization for the predicted pointmaps when the ground-truth pointmaps are metric. Specifically, we set 2 == 2 whenever the ground-truth metric, resulting the following loss function fregr(U; i) = XX x1113/2.
Metric-scale.
(a) The original DUS(3R depth map.
(b) The Plane-DUSt3R depth map.
Figure 4: The (a) original DUSt3R depth map and (b) occlusion removed depth map.
3.3 fa:
In this section, introduce how combine the multi-view plane pointmaps X and 2D detec- tion results p, to derive the final layout (P, L, W). For each single view I,, we can in- fer a partial layout result (P., L, J,, W.) = (X. 1', from the single view pointmaps X; and 2D detection results p','' through post-process algorithm 91 camera coordinate. Then, boreesooneence-estabiie
and merging algorithm combines all partial results to get the final layout (P, L, J. W} g2( PI, J,, W.....).
Single-view room layout estimation gi. For an image I,, 91 mainly addresses two tasks: 1) lifting 2D planes to 3D camera coordinate space with 3D normal from pointmap X; and 2) inferring the wall adjacency relationship. We follow the post- post-processing procedure in Yang et al. (2022) but with two improvements. First, the plane normal and offset d are inferred from X, instead of directly regressed by network fi- The points from X, which belong to same plane are used to calculate and d. Second, with the better 3D information pointmap X,
better infer pseudo wall

---

Published as a conference paper at ICLR 2025
66
(a) Projected Lines
(b) Rotated Lines
(c) Aligned Lines
(d) Correspondance
Figure 5: (a) Planes are projected onto the x-z plane as 2D line segments. (b) The scene rotated so that line segments are approximately horizontal or vertical. (c) Line segments are classified and aligned to either horizontal or vertical. (d) Merged planes are shown, with segments belonging to the same plane indicated by the same color and index.
adjacency through the depth consisteney of inferred plane intersection L (inferred from 2D plane p) and predicted line region L (extracted from the region of X:). In our experiments, the depth consistency tolerance 6 set to 0.005.
Multi-view room layout estimation 92- Based on the results of gl, 92 uses the global alignment of DUSt3R (refer to appendix A) to get the camera pose T; for each image I, Then, we can transform all partial layout results ((P). L J1, Wi)....) to the same coordinate space. In this coordinate space, we establish correspondence for each plane, then merge and assign a unique ID to them.
Since we allow at most one floor and one ceiling detection per image, we simply average the param- eters from all images to obtain the final floor and ceiling parameters. As for walls, we assume all walls are perpendicular to both the floor and ceiling. To simplify the merging process, we project all walls onto the x-z plane defined by the floor and ceiling. This projection reduces the problem to a 2D space. making easier identify and merge corresponding walls. Figure illustrates the entire process of merging walls. Each wall in an image denoted as one line segment, as shown in Figure 5a We then rotate the scene that all line segments are approximately horizontal or vertical, as depicted in 5b In Figure each line segment is classified and further rotated to be either horizontal or vertical, based on the assumption that all adjacent walls are perpendicular each other.
Figure 5d shows the final result after the merging process. The merging process could be regarded as a classical Minimum Cut problem. In Figure [dd] all line segments can be regarded as node in graph, two nodes have a connection if and only if they satisfy two constraints. 1) they belong to the same categories (vertical or horizontal). 2) they do not appear in the same image. 3) they are not across with the other category of node. Finally, the weight of each connection is settled as the Euclidean distance of their line segment centers. Based on this established graph, the merging results are the optimal solution of the minimum cut this graph. The detail of the merging process can be found in Algorithm|] of Appendix.
EXPERIMENTS
4.1 SETTINGS.
Dataset. Structured3D Zheng al. 2020) is synthetic dataset that provides large collection of photo-realistic images with detailed 3D structural annotations. Similar to Yang et al. (002)). the dataset divided into training, validation, and test sets at the scene level, comprising 3000, 250, and 250 scenes, respectively. Each scene consists of multiple rooms, with each room containing to images captured from different viewpoints. To construct image pairs that share similar visual content, retain only rooms with at least two images. Within each room, images paired to form image sets. Ultimately, we obtained 11,836 image pairs for the training set and 11,030 image pairs for the test set. For validation, assess all rooms from the validation set. For rooms that only have one image, we duplicate that image to form image pairs for pointmap retrieval. In the subsequent inference process, retain only one pointmap per room.
Training details. During training, we initialize the model with the original DUSt3R checkpoint. We freeze the encoder parameters and fine-tune only the decoder and DPT heads. Our data augmentation
7

---

Published as a conference paper at ICLR 2025
Figure 6: Qualitative comparison on the Structure3D test set. From left to right: (1-3) input views of the scene, (4) result by baseline method (Noncuboid+MASt3R) shown in wireframe only, and (5) our method's 's result visualized with predicted point clouds. Please refer to the appendix for more complete results.
strategy follows the same approach DUSt3R, using input resolution of 512 x 512. We employ the A optimizer (Loshchilov & Hutter 2017) with a cosine learning rate decay schedule, starting with base learning rate of lc-4 and a minimum of le-6. The model is trained for 20 epochs, including warm-up epochs, with a batch size of 16. We train two versions Plane-DUStar one with metric-scale loss and the other without it. We name the metric-scale one as Plane-DUSt3R (metric) and the other one as Plane-DUSt3R.
Evaluation. Following the task formulation in equation (3) evaluation protocol consists of three parts to assess f1, /2, and the overall performance, respectively.
For the 2D information extraction module f1, we use the same metric as Yang et al. 2022) for comparison: Intersection over Union (IoU), Pixel Error (PE), Edge Error (EE), and Root Mean Square Error (RMSE).
For the multi-view information extraction module f2, we report the Relative Rotation Accuracy (RRA) and Relative Translation Accuracy (RTA) for each image pair to evaluate the relative pose error. We use threshold of r 15 to report RTA @ 15 and RRA@ @ 15 (The comprehensive results of different thresholds can be seen in Table of Appendix C.I). Additionally, we calculate the mean Average Accuracy (mAA30), defined as the area under accuracy curve of the angular differences at min(RRA 30, RTA 30).
Finally, for evaluating the overall layout estimation, we employ 3D precision and 3D recall of planes metrics. A predicted plane is considered matched with a ground truth plane if and only if the angular difference between them less than and the offset difference is less than O. 15m. Each ground truth plane can be matched only once.
Baselines. As this work is the first attempt at 3D room layout estimation from multi-view perspective images, there are no existing baseline methods for direct comparison. Therefore, we design two reasonable baseline methods. We also compare our and fz with other methods of the same type.
Since we use Noncuboid (Yang et al. 2022) as our f1, we not only compare it against the baselines from their paper (Liu et al.) 2019] Stekovic et al. 2020), but also retrain with better hyper- parameters obtained through grid search.
compare it to recent data-driven image matching DUSt3R (Wang ct al. 2024) and MASt3R (Leroy et al. 2024).
For /2 (Plane-DUSt3R),
Finally, for the overall multi-view layout baseline, we design two methods: 1) Noncuboid with ground truth camera poses and 2) Noncuboid with MASt3R. this context, we introduce the fusion of MASt3R (Leroy et al. 2024) and NonCuboid (Yang et al., 2022) as our baseline method. MAS(3R further extends DUS(3R, enabling it to estimate camera poses at a metric scale from sparse image inputs. We employ the original NonCuboid method to obtain single-view layout reconstruction. Next, we utilize the predicted camera poses to unify all planes from their respective camera poses into common coordinate system. For instance, we designate the coordinate system of the first image as the world coordinate system. We then perform the same operation as described in Sec 3.3 to achieve the final multi-view reconstruction. The Noncuboid with ground truth camera
8

---

Published as a conference paper at ICLR 2025
Figure 7: Birdview of multi-view 3D planes aligned to the same coordinate. The first row shows cases of our pipeline results after post-pree post-proeessing step. The second row is the results of Non- cuboid+MASt3R. Line segments of the same color indicate that they belong to the same plane.
Table I: Quantitative results on Structure3D dataset.
Method
re-1oU(%)
re-PE(%)
re-EEL
re-RMSE
3D-precisiont %
3D-recall y7
Noncuboid MASGR
74,31
8.57
12.72
0.4831
37.00
43.39
Noncuboid pose
7.97
46.96
50.59
Ours (metric)
75.34
8.60
10.83
0.4388
48.9%
45.35
(aligned)
poses is introduced ablation study to eliminate the effects of inaccurate pose estimation. The experimental setup is the same as the Noncuboid with MASt3R pipeline, except for the use of ground truth camera poses instead of poses estimated by MASt3R.
4.2 MULTI-VIEW ROOM LAYOUT ESTIMATION RESULTS
In this section, we compare our multi-view layout estimation pipeline with two baseline methods, both qualitatively and quantitatively. Additionally, we conduct experiments verify the effective- ness of our pipeline components fi 2D detector and f2 Plane-DUSt3R.
Layout results comparison. Table and Figure pressen quantitative and qualitative comparisons of our pipeline with two baseline methods. Ours (metric) and Ours (aligned) in Table referr to the methods from our pipeline using Plane-DUSt3P (metric) and Plane-DUSt respectively. The first 4 metries (re-IoU, re-PE, re-EE, and re-RMSE) are calculated similarly to their 2D counterparts (IoU, PE, EE, and RMSE), except that the predicted 2D results are reprojected from the estimated multi-view 3D layout. Compared with the baseline methods. Plane-Do -UUStR achieves superior 3D plane normal estimations compared to Noncuboid's 's single-view plane normal estimation, even when using ground truth camera pose (Noncuboid GT pose). Figure 7 further demonstrates that Plane- DUSt3R could predict accurate and robust 3D information with sparse-view input.
Table 2: Comparison with data-driven image matching approaches.
RealEstate 1OK
Structured3D
CAD-estate
Methods
mAA@34
RRA@ 15 RTA@15 mAA@:0 30 RRA@15 RTA@15 mAA@30
(8) DUS:AR Wann et al.
2024
61.2
89.44
85.00
76.13
99.88
84.82
76.38
MASIER (Lery el 2024
76.4
92.94
89.77
85.34
99.94
99.00
95.29
(b) Plane-DUSt3R (metric)
98.21
96.66
90.67
94.61
70.52
61.48
(aligned)
information prediction and corresponeen--estabiisded method Plane-DUSt3R f2. Ta- ble 2 shows the comparison results of our Plane-DUS(3R (part (b) in Table 2). recent popular data- driven image matching approaches (part (a) in Table 2 in RealEstate 10K (Zhou et al. 2018). Struc- ture3D (Zheng al. 2020), and CAD-Estate (Rozumnyi et al. 2023) datasets. Note that in part(b), results on RealEstate IOK dataset are not provided since our model is specifically designed to predict room structural elements, while RealEstate 10K contains outdoor scenes that may lead to prediction

---

Published as a conference paper at ICLR 2025
Table 3: 2D detectors comparison on Structure dataset.
Method
ToU %
PE(%)
EEJ
RMSE,
Planar R-CNN (Liu et 2019)
79.64
7.04
6.58
0.4013
Rac (Stekovic al. 2020
76.29
8.07
7.19
0.3865
Noncubord Yang et al. 2022
79.94
6.40
6.80
0.2827
Noncuboid (re-trained)
80.18
6.13
6.41
0.2631
Figure 8: Qualitative comparison on in mnldhe-wild data Zhou et al., 2018). From left to right: (1-3) three imput views of the scene, (4) layout reconstruction by baseline method (Noneuboid-MAA(3R) shown in wireframe only, and (5) our method's s result visualized with predicted point clouds.
failures. Instead, we utilize CAD-Estate Estate dataset, which is derived from RealEstate with addi- tional room layout annotations, as more suitable benchmark for comparison. The results of parts (a) and (b) on three datasets show the advancements of MASt3R, not only in traditional multi-view datasets (RealEstate 1 BINK,,AAD-taaaa could get better performance compared to the previous SOTA MASt3R. One arguable point is that Plane-DUSt3R obviously better since it is fine-tuned on Strucutre3D. That is the message we want to convey. DUSGRM/ASSR are the SOTAs in both multi- view and saprse-view camera pose esti- mation tasks. After our improvements (section 3.2) and fine-tuning, Plane-DUSt3R could get 5.33 points better the sparse-view layout dataset.
but also sparse-vies view dataset (Structure3D), Plane-DUSt3K
Comparison 2D detectors (fi). We retrain the Noncuboid method with a more thorough hyper- parameter grid search, resulting in improved version. Table compares its results with other baseline methods from Yang et al. (2022).
Comparison of various input views. We experiment the impact of different number of input views in Appendix C.2. The results in Table 5 show general improvement trend as the number of views increases.
4.3 GENERALIZABILITY TO UNKNOWN AND OUT-Or-DOMAIN DATA
Figure and 12 also demonstrate the generalizability of our pipeline. It not only performs well on the testing set of Structure3D (Figure 6). but also generalizes well to new datasets, such as RealEsttae 10K (Figure shows examples from this dataset). Furthermore, our pipeline proves fective even with data the wild shown in appendix in Figure 11 We also experiment pipeline on the dataset CAD-estate (see Appendix for details).
CONCLUSION
This paper introduces the first pipeline for multi-view layout estimation, even in sparse-view set- tings. The proposed pipeline encompasses three components: 2D plane detector, a 3D information prediction and correspondence establishment method, and a post-processing algorithm. As the first comprehensive approach to the multi-view layout estimation task, this paper provides a detailed analysis and formulates the problem under both single-view and multi-view settings. Additionally, we design several baseline methods for comparison to validate the effectiveness of our pipeline. both 2D projection and 3D metrics. Fur- thermore, our pipeline not only performs well on the synthetic Structure3D dataset, but generalizes effectively to in-the-will wild datasets and scenarios with different image styles such the cartoon style.
Our approach consistently outperforms the baselines
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is partially supported by the Hong Kong Center for Construction Robotics (Hong Kong ITC-sponsored InnoHK center), the National Natural Science Foundation of China (Grant No. 62306261), CUHK Direct Grants (Grant No. 4055190), and The Shun Hing Institute of Advanced Engineering (SHIAE) No. 8115074,
REFERENCES
Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun- Yi Yang, Samir Aroudj, Su- vam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Campbell Orme, et al. Scene- script: Reconstructing scenes with autoregressive structured language model. arXiv preprint arXiv:2403.13064 2024.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https::/opeeaatt com/ s - simuladto
video-generation--een
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 2020.
Xili Dai, Haigang Gong. Shuai Wu, Xiaojun Yuan, and Yi Ma. Fully convolutional line parsing.
Neurocomputingg 506:1-11, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 0rXiv:2010.11990 2020.
Varsha Hedau, Derek Hoiem, and David Forsyth. Recovering the spatial layout of cluttered rooms. In 2009 JEEE 12th international conference on computer vision, pp. 1849-1856. IEEE, 2009.
Henry Howard-Jenkins, Shuda Li, and Victor Prisacariu. Thinking outside the box: Generation of unconstrained 3d room layouts. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part I 14, pp- 432-448, Springer, 2019.
Zhihua Hu, Bo Duan, Yanfeng Zhang, Mingwei Sun, and Jingwei Huang. Mvlayoutnet: 3d lay- out reconstruction with multi-view panoramas. Proceedings of the 30th ACM International Conference on Multimedia, pp. 1289-1298, 2022.
Siyuan Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, and Song-Chun Zhu. Holistic 3d scene parsing and reconstruction from a single rgb image. In Proceedings of the European conference on computer vision (ECCV), pp. 187-203. 2018.
Linyi Jin, Shengyi Qian, Andrew Owens, and David F Fouhey. Planar surface reconstruction from sparse views. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 12991-13000, 2021.
Vincent Leroy. Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. arXiv preprint arXiv:2406. 09756, 2024.
Chen Liu, Jimei Yang, Duygu Ceylan, Ersin Yumer, and Yasutaka Furukawa. Planenet: Piece- wise planar reconstruction from a single rgh image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2579-2588, 2018
Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz. Planerenn: 3d plane de- tection and reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4450-4459, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17110 2017.
11

---

Published as a conference paper at ICLR 2025
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv:1611.0367. 2016.
Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang. To- talddunderstanding:: Joint layout, object pose and mesh reconstruction for indoor scenes from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 55 55-64, 64, 2020.
Rémi Pautrat, Juan-Tin Ting Lin, Viktor Larsson, Martin R Oswald, and Mare Pollefeys. Sold2: Self- supervised occlusion-awer -wwrre line description and detection. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 11368 1378,, 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12179-12188, 2021.
Denys Rozumnyi, Stefan Popov, Kevis-Kokitsi Maninis, Matthias NieBner, and Vittorio Ferrari. Estimating generie 3d room structures from 2d annotations, 2023. URL https://arxiv: org/abs 2306. 09077
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from motion revisited. In Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2016.
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
Sinisa Stekovic, Shreyas Hampali, Mahdi Rad, Sayan Deb Sarkar, Friedrich Fraundorfer, and Vin- cent Lepetit. General 3d room layout from a single view by render-and-compare In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XVI 16, pp. 187-203. Springer, 2020.
Cheng Sun, Chi-Wei Hsiao, Min Sun, and Hwann-Tzon Chen. Horizonnet: Learning room layout with Id representation and pano stretch data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1047-1056, 2019.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Hohonet: 360 indoor holistic understanding with latent horizontal features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Haiyan Wang, Will Hutchcroft, Yuguang Li, Zhiqiang Wan, Ivaylo Boyadzhiev, Yingli Tian, and Sing Bing Kang. Psmnet: Position-aware stereo merging network for room layout estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8616-8625, 2022.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolutin representation learn- ing for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43 (10):3349-333 3364, 2020.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697-207000 2024.
Ethan Weber, Riley Peterlinz, Rohan Mathur, Frederik Warburg, Alexei A. Efros, and Angjoo Kanazawa. Toon3d: Seeing cartoons from new perspective. In arXiv, 2024.
Cheng Yang, Jia Zheng, Xili Dai, Rui Tang. Yi Ma, and Xiaojun Yuan. Learning to reconstruct 3d non-cuboid room layout from a single rgb image. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2534-2543 2022.
12

---

Published as a conference paper at ICLR 2025
Shang- Ta Yang, Fu-En Wang, Chi-Han Peng. Peter Wonka, Min Sun, and Hung- Kuo Chu. Dula-net: A dual-pre projection network for estimating room layouts from a single rgb panorama. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3363-3372, 2019.
Zehao Yu, Jia Zheng, Dongze Lian, Zihan Zhou, and Shenghua Gao. Single-image piece-wise planar 3d reconstruction associative embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1029-1037, 2019.
Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Connecting the dots: Floorplan reconstruction using two-level queries. Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 845-854, 2023.
Yinda Zhang, Fisher Yu, Shuran Song, Pingmei Xu, Ari Seff, and Jianxiong Xiao. Large-scale scene understanding challenge: Room layout estimation. In CVPR Workshop, volume 3, 2015.
Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In Computer Vision-ECCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pp. 519-535. Springer, 2020.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint /800: 1805.80000000 09870, 2018.
Yichao Zhou, Haozhi Qi, and Yi Ma. End-to-end wireframe parsing. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 962-971, 2019.
Chuhang Zou, Alex Colburn, Qi Shan, and Derek Hoiem. Layoutnet: Reconstructing the 3d room layout from single rgh image. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2051-2059, 2018.
13

---

Published as a conference paper at ICLR 2025
A DUST3R DETAILS
Given a set of RGB images {1, Ig...., .
In ) E R we first pair them to create a set of image pairs P = {L,, 1,) i i j. i S n). For each image pair (L, Lj) E P., the model estimates two point maps X j.s. along with their corresponding confidence maps C.,,, CJ Specifically, both pointmaps are expressed in the camera coordinate system of I,, which implicitly accomplishes dense 3D reconstruction.
The model consists of two parallel branches, shown in Fig each branch responsible for pro- cessing one image. The two images are first encoded a Siamese manner with weight-sharing ViT encoder(Dosovitskiy
al. 2020) produce latent features F1, F2: F, Encoder T.)- Next, F1, F2 are fed into two Identical decoders that continuously share information through cross- attention mechanisms. By leveraging cross-attention mechanisms, the model able learn the relative geometric relationships between the two images. Specifically, for each encoder block:
G DecoderBlock,, (G1, G2,1-1),
(4)
2 DecoderBlock,, G2
where G1.0 := Fi, G2.0 Fg. Finally, the DPT(Ranfil et al. 2021) head regresses the pointmap and confidence map from the concatenated features of different layers of the decoder tokens:
1,1, C1.1 Head, (G1,0 G1.1 ..., G1,B)
(5)
X2,1, C2.1 = Head? (G,,0, G2,1,..,, G2,B)
where B is the number of decoder blocks. The regression loss function is defined as the scale- invariant Euclidean distance between the normalized predicted and ground trruh pointmaps:
1 x
Legr (v, i) =
(6)
where v E 21, and is the pixel index. The scaling factors and E represent the average distance of all corresponding valid points to the origin. The original DUSt3R couldn't guarantee output at metric scale, so also trained a modified version of Plane-DUS(3R that produces metric- scale results. The key change we made was setting == Σ. By introducing the regression loss in confidence loss, the model could implicitly learn how to identify regions that are more challenging to predict compared to others. Same as in DUSt3R (Wang et al. 2024):
Lcont Σ
Σ
G. (v, - a log C.
(E(1,2) iED"
To obtain the ground-truth pointmaps X we first transform the ground truth depthmap D Hx into a pointmap X" express in the camera coordinate of e by X
K D jD.j Disl" with camera intrinsic matrix K 33x3 Then we obtain X by X TTA(") with Ti, To E 3x34 the camera-to-world poses and h being the homo- geneous transformation.
Global Alignment For global alignment, we aim to assign a global pointmap and camera pose for each image. First, the average confidence scores of each pair images are utilized as the similarity scores. A higher value of confidence implies a stronger visual similarity between the two images. These scores are employed to construct a Minimum Spanning Tree, denoted as g(V, E), where each vertex V corresponding indicates that images I, and I share significant visual content. We aim to find globally aligned point maps(x" E " x3) and a transformation T. E 3 than transform the prediction into the world coordinate frame. To do this, for each image pair maps _ X and their confidence maps C Cm.. For simplicity, we use the annotation X. : n. m., := X mm Since n. and X m., are in the same coordinate frame, T. := T, should align both point maps with the world-coordinate. We then solve the following optimization problem:
image in the input set and each edge
(n, m) E E
(n, m) E È we have two point
HW ΣΣΣ EEE VEE ==1 where V E e means v can be either n or m for the pair e and o, is a positive scaling factor. To avoid the trivial solution where a. 0, we ensure that Ie ae
x" arg min x.P.a
1Ix -..T.o.T.X
14

---

Published as a conference paper at ICLR 2025
B fs ALGORITHM
The goal of multi-view layout estimation is similar to that of single-view: we need to estimate 3D parameters for each plane and determine the relationships between adjacent planes. However, in multi-view setting, we must ensure that each plane represents unique physical plane in 3D space. The main challenge in multi-view reconstruction is that the multiple images, causing duplication. Our task is to identify which planes correspond to the same physical plane across different images and merge them, keeping only one representation for each unique plane.
physical plane may appear in
Since we allow at most one floor and one ceiling detection per image, we simply average the param- eters from all images to obtain the final floor and ceiling parameters. As for walls, we assume all walls are perpendicular both the floor and ceiling. To simplify the merging process, we project all walls onto the x-Z plane defined by the floor and ceiling. This projection reduces the problem to space, making easier identify and merge corresponding walls. Figure S illlustratss the entire line segment, shown in Figure 5a We then rotate the scene so that all line segments are approximately horizontal or vertical, as depicted in 5b In Figure 5c each line segment classified and further rotated to be either horizontal or vertical, based on the assumption that all adjacent walls are perpendicular to each other.
process of merging walls. Each wall in image is denoted
Algorithm Merge Plane
Require:
vertical lines, horizontal lines 1: Sort perticalLéness by x-axis value Initialize clusters with the first segment.
3: for each segment in vertical do
4: found + False
5: for each cluster in clusters do
6: if tines.image_id cluster centterrimoe_iee iid then
continue
8: end
9: if distance(lâne, Sbuster.centroiie procinniyy then
10: if overlap(linee, .cnlrrddc) oorrhap then
Insert line into cluster found + True break end if
12:
13:
14:
15:
if intersect(Lnne, cluster, horizontalLines, margin) then
16:
Insert line into cluster
17:
found <- True break
18: end
19: end if
20: end for
21:
end for
22: found- = False then
23:
Create new cluster with line
24:
Append the cluster clusters
25:
end
Ensure: Clusters
C ADDITIONAL QUANTITATIVE RESULTS
C.1 PERFORMANCE UNDER VARYING THRESHOLDS
To give complete view of the performance of the Plane-DUSEaS method, present the results under various threshold settings (the threshold both camera's 's translation and rotation) in Table 4
15

---

Published as a conference paper at ICLR 2025
Table Quantitative results with different thresholds on Structure3D dataset.
Threholl(TTansttttinn & Rotation)
3D-precisiont %1)
3D-recall
0.1m,
34.11
31.66
0.15m, 10"
48.37 52.63
0.2m, 15"
64.64
59.53
0.4m, 30
82.75
76.13
Table 5: Quantitative results with different input views on Structure3D dataset.
Input
re-ToUi
re-PE(%)
re-FF.
re-RMSE
3D-rrecision (+)
3D-recall( %
2
75.02
8.72
8.70
0.4148
53.19
42.60
3
75.29
8.53
8.56
0.3596
54,43
47.97
75.55
8.39
8.55
0.3463
54.91
49.44
5
75.57
8.35
8.59
0.3422
55.02
49.59
C.2 PERFORMANCE UNDER INPUT VIEWS
The impact of varying input views on performance is presented in Table For each input views, we select this number of views from the 5 views as input and only test on rooms that have all 5 views to climinate potential bias from room complexity variations. The results show a general improvement trend as the number of views increases.
D ADDITIONAL QUALITATIVE RESULTS
Figure 9 showcases more visualization of our method on the Structured3D dataset, while Figure I0 presents failed cases. To demonstrate real-world applicability, we present results on in-the-wild images in Figure 11 and Figure 12
E EVALUATION RESULT ON CAD-ESTATE DATASET
We conducted additional evaluation the CAD Estate dataset Rozumnyi al. (2023). CAD- Estate is derived from RealEstate10K datase(Zhou et al. (208) and provides generic 3D room lay- outs from 2D segmentation masks. Due differences in annotation standards between CAD-Estate and Structured3I), we selected subset of the original data that aligns with our experimental setup. Our method and Structured3D assume single floor, single ceiling, and multiple walls configura- tion. In contrast, CAD-Estate includes scenarios with multiple ceilings (particularly in attic rooms) and interconnected rooms through open doorways, whereas Structured3D treats doorways as com- plete walls. To ensure a fair comparison, we sampled 100 scenes containing 469 images that closely match Structure3D's annotation style. Each scene contains 10 images.
Since CAD-Fstate only provides 2D segmentation annotations without 3D information, we report performance using 2D metries: loU and pixel error. While CAD-Estate's label classes include ["ignore", "wall", "floor", "'celling,, "slanted"), we only focus on wall, floor, and ceiling classes. We utilize the dataset's provided intrinsic parameters for reprojection during evaluation. Results are reported for both "Noncuboid + GT pose" and "Plane-DUS(3R (metric)" in Table 6 our results in Figure 13 and Figure 14
We visualize
Table 6: Quantitative results with on CAD-estate dataset.
Method
re-IoU((S)
re-PE(%))
Noncubusd + GT pose (metric)
55.99
20.33
16

---

Published as a conference paper at ICLR 2025
Figure 9: Qualitative results Structure3D testing The 5-th column is result visualized with pointcloud, the last column is the result shown in pure wireframe
Figure 10: Failed case on Structure3D testing set. The first 4 columns are input views, the 5-th col- umn is our result visualized with pointcloud, the last column is the result shown in pure wireframe.
17

---

Published as a conference paper at ICLR 2025
Figure 11: We provide qualitative results on in-the-wild data.
BAROK
Figure 12: We provide qualitative results on out domain cartoon data
Weber et al. 2024
(b)
Figure 13: Visualization results from the CAD-Estate dataset. (a) Input views are shown in the top row, followed by C'AD-Estate's ground-truth segmentation in the middle row, and predicted segmentation the bottom row. (b) Our 3D reconstruction results displayed with point clouds (top row) and wireframe renderings (bottom row).
(b)
Figure 14: Visualization results from the CAD-Estate dataset. (a) Input views are shown in top row, followed by CAD-Estate's ground-truth segmentation in the middle row, and predicted segmentation the bottom row. (b) Our 3D reconstruction results displayed with point clouds (top row) and wireframe renderings (bottom row).
18


Reviews:
[{"review_id": "MXPoOevepl", "reviewer": "Reviewer_6JGC", "summary": "The paper proposes an extension of DUSt3R to room layout reconstruction by retraining and using additional processing steps. Results demonstrate superior performance.", "questions": "- Figures seem to have low resolution since they are most likely represented as jpg/png images. I recommend using vector graphics instead for better visualisations.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The method is sound with great results outperforming other methods.\n- This is the first method for unposed sparse view room layout reconstruction, especially when the views are not overlapping. \n- The whole idea is interesting, especially to use structural plane depth map and metric scale for DUSt3R. \n- The evaluation is great, especially the design of baselines.\n- The paper is well-written and was a pleasure to read.", "weaknesses": "- Evaluation on more datasets would be more interesting, e.g. CAD-Estate dataset with structural elements annotations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9GGM9neF1C", "reviewer": "Reviewer_H8Ec", "summary": "In this work authors propose sparse-view layout reconstruction pipeline that combines existing single view layout methods such as [1] and sparse view 3D reconstruction method Dust3R [2], to generate more accurate 3D layout from sparse images. \n\nAuthors retrained [1] to more accurately detect 2D premitives. They also retrain Dust3R, to only predict layout-plane pointmaps, essentially ignoring foreground furniture. They call this new model Plane-Dust3R. Authors train their PlaneDust3R method on synthetic multi-view Structure3D dataset. And compare accuracy of multiple components of their pipeline, on this synthetic dataset. \n\nFrom 2D primitives and 3D plane pointmaps, authors design post-processing pipeline that converts them to 3D plane equations, and their relationships, resulting in full 3D layout representations. \n\n\n[1] Yang, Cheng, et al. \"Learning to reconstruct 3d non-cuboid room layout from a single rgb image.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.\n\n[2] Wang, Shuzhe, et al. \"Dust3r: Geometric 3d vision made easy.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "questions": "In Table-2, other datasets results are not useful since this method does not have numbers for those. I would suggest authors to keep results for only structure 3D.\n\n3D precision and recall should be reported at different threshold. I think 15deg threshold is pretty high. Maybe authors should show results at 5deg, 15deg, and 30deg angular thresholds, and similar translation thresholds. This will give more comprehensive view of the methods performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Proposed sparse multi-view layout method is first of its kind. Authors make use of existing methods and retrain them to make them work for their task. \n\nOverall pipeline is shown to work better than naively combining existing methods such as NonCuboid + Mast3R / NonCuboid + GT Pose. \n\nOverall paper is well written, is easy to follow. There are many components in the pipeline, but authors explain them in an understandable manner.", "weaknesses": "The big limitation of method is that it is only shown to work on synthetic dataset. While availability of real multi-view layout dataset is scarse, there exists many multi-view 3D datasets (such as ScanNet, ARKit Scene) etc. These datasets have ground-truth depth maps and camera poses that can be used for pseudo ground-truth generation and the proposed method can be compared against other layout reconstruction methods on such dataset. Authors do provide qualitative results on some real-world data, but it is not enough to assess accuracy of method on the real data. If authors can show effectiveness of the method on real data this method can be promising.\n\nSecond, I am not convinced as to why PlaneDust3R is needed. If authors take regular Dust3R and use off-the-shelf semantic segmentation to segment walls, floors, and ceilings and use only those points to generate layout (by combining this with NonCuboids + f3 post-processing proposed in the paper), that might also be enough. Retraining method only on synthetic dataset such as Structure3D might reduce their accuracy on the real-world dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7axQZHceCY", "reviewer": "Reviewer_oARR", "summary": "This paper introduces Plane-DUSt3R, a method that leverages the DUSt3R framework to estimate structural planes from multi-view images by finetuning on a room layout dataset (Structure3D). This approach allows for room layout estimation with just a single post-processing step and 2D detection results, handling multiple-perspective images. Experimental results show that Plane-DUSt3R outperforms existing methods on synthetic datasets and is robust across various real-world image styles, including cartoons.", "questions": "None", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The major strength of this paper is the first time addressing 3D room layout estimation from multi images by using large reconstruction model s (i.e., DUST3R). \n2. To support using DUST3R, this paper proposed Plane-DUSt3R to estimate structural planes from multi-view images by only require a single post-processing step.\n3. Plane-DUSt3R achieves the best performance comparing with the baselines.", "weaknesses": "1. I think the major contribution of the performance boost should go to Dust3R architecture and its pretrained weight, as it is a strong prior model that gives stable and faithful 3D points and camera poses output for Plane-DUSt3R.\n\n2. The key model of this paper is only the post-processing step of extracting layout planes, which is, technically speaking, quite incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tSE3q7is7f", "reviewer": "Reviewer_UrzG", "summary": "This work propose Plane-DUSt3R to reconstruct room layout from sparse views with unknown poses. The DUSt3R is retrained to have the amodal perception to see through the occluder and predict floors, walls, ceilings pointmap. The unknown camera parameters can then be estimated from the predicted point map. The outcome from Plane-DUSt3R is then integrated with the other prediction from single-view layout model (e.g., plane instance masks), which is then formulated as a minimum cut problem to produce the final room layout.", "questions": "Seems that Sec3 only covers camera poses. Is the camera intrinsic also estimated as in DUSt3R or assumed to be known in this work?\n\nHow the accuracy vary with the number of input views? Is the performance scalable with more input views?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ This work introduce state-of-the-art geomety prediction model, DUSt3R, into the classical layout estimation task, solving an underexplored aspect with sparse views and unknown poses.\n+ The problem formulation in Sec.3.1 is clear and fluent, which unifies previous room layout estimator in a single pipeline and help readers understand how the propose method augments the existing pipeline.\n+ A new benchamrk with some reasonable baselines is constructed for this task.", "weaknesses": "- In Table2, the performance by the proposed method on Co3Dv2 and RealEstate10K are missing. It's reasonable that DUSt3R and MASt3R perform worse as they didn't trained on the synthetic Structured3D dataset. What about the generalized performance of the proposed Plane-DUSt3R on the other real-world dataset?\n\n- It's a pity that the proposed Plane-DUSt3R is only trained and mainly evaluated on the synthetic Structured3D dataset. There are many existing resource with multiview and room layout annotation on the 360 panorama domain, e.g., ZInD[1] has multiview layout on unfurnished rooms, MP3DLayout[2] has single-view layout annotation but the source Matterport3D also offer nearby views. Gather more data by projecting these 360 to perspective can make this work much stronger.\n  - [1] Zillow Indoor Dataset: Annotated Floor Plans With 360º Panoramas and 3D Room Layouts\n  - [2] Manhattan Room Layout Reconstruction from a Single 360° image: A Comparative Study of State-of-the-art Methods\n\n- The qualitative comparison in Fig.6 and Fig.8 is unfair to the baseline. In the fourth column, the baseline \"Noncuboid+MASt3R\" is presented without texture. We can then easily spot one big issue that some of the walls are duplicated and misaligned. However, in the final column, the layout wireframe from the proposed method is hiding by the rgb point cloud, making it difficult to judge if the similar issue happend to the proposed method as well.\n\n- Too few qualitative results. On the in-domain Structured3D dataset, only two are provided in Fig.6. The additional three in supp's Fig.9 also has two duplicated scenes from Fig.6. In application of room tour and room demo, the final visual outcome is what we really care about while the improvement on number is sometimes hard to interpret how it affect the final visual. In addition, some more visualization, espeically on the failure cases, may can provide some hint for future work to further improve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)