Estimating Generic 3D Room Structures from 2D Annotations
Authors: Denys Rozumnyi
Stefan Popov
Kevis-kokitsi Maninis
Matthias Nießner
Vittorio Ferrari
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: room layout
3D reconstruction
dataset construction
dataset
optimization
TL;DR: We annotate 2246 generic 3D room layouts just from 2D segmentation masks, which are easy to annotate for humans.

Abstract: Indoor rooms are among the most common use cases in 3D scene understanding. Current state-of-the-art methods for this task are driven by large annotated datasets. Room layouts are especially important, consisting of structural elements in 3D, such as wall, floor, and ceiling. However, they are difficult to annotate, especially on pure RGB video. We propose a novel method to produce generic 3D room layouts just from 2D segmentation masks, which are easy to annotate for humans. Based on these 2D annotations, we automatically reconstruct 3D plane equations for the structural elements and their spatial extent in the scene, and connect adjacent elements at the appropriate contact edges. We annotate and publicly release 2246 3D room layouts on the RealEstate10k dataset, containing YouTube videos. We demonstrate the high quality of these 3D layouts annotations with extensive experiments.

Estimating Generic 3D Room Structures from 2D Annotations
Denys Rozumnyi
Stefan Popov' Kevis-Kokitsi Maninis' Vittorio Ferrari'
Matthias Nießner*
GGoole Research
2ETH Zurich
Technical University of Munich
Abstract
Indoor rooms are among the most common use cases in 3D scene understanding. Current state-of- the-art methods for this task are driven by large annotated datasets. Room layouts are especially important, consisting of structural elements in 3D, such wall, floor, and ceiling. However, they are difficult annotate, especially pure RGB video. We propose a novel method to produce generic 3D room layouts just from 2D segmentation masks, which are easy annotate for humans. Based on these 2D annotations, we automatically reconstruct 3D plane equations for the structural elements and their spatial extent in the scene, and connect adjacent elements at the appropriate contact edges. We annotate and publicly release 2246 3D room layouts on the RealEstate 10k dataset, containing YouTube videos. We demonstrate the high quality of these 3D layouts annotations with extensive experiments.
1 Introduction
Estimating 3D structures from video is one of the key tasks in computer vision. Many methods have been proposed, including Mouuttrr-frrm-MMoiiee [49] and Multi- View Stereo [50] [S] pipelines. In particular, indoor room furniture 33 43 and layout important in recent years.
65 56 estimation is becoming more
The key ingredient state-of-the-arr methods are large annotated datasets for training. A variety of real 7 [] and synthetic [72] [6] (27 datasets with annotated 3D objects are available. Some datasets even offer 3D room layout, which is defined as a set of 3D structural elements such as wall, floor, and ceiling. Synthetic datasets 772 16 27 typically provide 3D room layouts along with 3D objects. In contrast, most real datasets focus only on 3D object annotations 77, 55 57
There are just a few real datasets with annotated 3D room layouts. However, they require the images/videos to be acquired with special devices or sensors such as RGB-D cameras or panoramic
captures 4 55 77 76 10 65]. This limits the range of scenes they can cover. Moreover, these datasets often offer only simple room structures such as Cuboid 77 76 19 66) or Manhattan 65 77
layouts commonly-avainue RGB videos. A naive system for annotating such 3D room layouts would ask human annotators to draw layouts in 3D. However, this extremely hard task, which would require very high abstraction and 3D representation abilities from the annotators, as well as very advanced user interface. Instead, we propose method annotate 3D room layouts that much easier for human annotators: draw an amodal segmentation mask for each structural element in 2D, and also approximately mark its visible parts in 2D (Fig. [T). This reduces the sophisticated target 3D task simply drawing segmentation masks in 2D, which is common annotation process used to create 2D datasets 39 32 74] Moreover,
In this paper propose method for annotating generic 3D
Work done while the author was at Google.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Input
Visible
Structure
3D Layout
Figure 1: From 2D annotations 3D room layouts. Given an RGB video, we ask human annotators to draw amodal segmentation masks and visible parts of each structural element, e.g. wall, floor. Then, our method automatically derives high-quality 3D layouts from these 2D annotations.
all annotation is performed each video frame independently without requiring the annotator to provide correspondences among video frames.
The simplification of the annotator task is enabled by shifting much of the work to an automatic method we propose, capable to derive room layouts from these 2D annotations. This method estimates 3D plane equation for each structural element, as well as a finite spatial extent that captures all parts of the plane that are in the camera's field-of-view at any time during the video. We estimate all elements jointly, and connect adjacent elements at the right contact edges in 3D, matching their shared edge as observed in 2D annotations.
Using the proposed approach, we annotate 2246 scenes from the RealEstate 10k 75 dataset that contains YouTube videos of indoor scenes. The rooms are complex and cover generic types, not limited to Cuboid/Manhattan. They can even be composite, such as two rooms connected by a door or a staircase. The method also works when the video does not show the full room (a common case). We evaluate the quality of the produced 3D layout annotations on RealEstate 10k in terms of Reprojection Intersection-over Union (IoU) to the input ground-truth 2D segmentation masks, and on ScanNet [I]. which enables measuring 3D depth errors as it features ground-truth depth maps. find the 3D layouts to be highly accurate, with only about 20cm depth error (out meter wide scenes), and high Reprojection IoU around 0.9. We also manually inspected the 3D layouts, finding that we reconstruct correctly about 98% of all structural elements that are actually in the scene. We have released the dataset publicly at htttts: //gi::hb//tbbb com haom/googleeeoo wonggle-reeearoe research/ cad- estate.
Related work
Datasets of 3D indoor scenes. Many datasets with 3D indoor scenes were proposed (see Table 1
Synthetic datasets have ground-truth 3D room layouts by construction, e.g. Structured3D [72] 3D-
FRONT [16 InteriorNet (27]. House3D [60]. SUNCG [53), SceneNet (38). OpenRooms
However, the imagery is not real, causing domain gap when training models on them 59 22
Many datasets of real images have only 3D objects annotated, without structural elements (see Table D.. few real datasets with 3D room layouts were introduced, but they heavily rely on special acquisition devices sensors to create the 3D annotation of the scene, which limits the range of scenes they can cover. For instance, Zillow Indoor Dataset [T], Realtor360 [65], and PanoContext [70] annotate layouts on 360" panoramic images. SceneCAD [4] was acquired with an
RGB-D sensor. MatterportLayout 77 and LayoutNet (76) require both a depth sensor and panoramic images.
Most of these datasets cover rooms with simple layouts, that are either: (a) Cuboid 70 76
19
66 or
(b) Manhattan 665 77 (all structural elements are aligned with canonical coordinate system), and the methods to construct them not work more complex room struetures. Some datasets (10) construct generic room layouts (like ours), but are limited by the data capturing process (see above).

---

Dataset
Data type
#Scenes
#lmages
Objects
Layout
Type
72
16
60
53
synthetic
varies
varies varies
G
2
44
57
29
RGB
varies
21
62
52
RGB-D
varies
varies
Replica
555
RGB-D video
18
7
G
SceneCAD
RGB-D video
1,151
G
MatterportLaayon
70
RGB-D pano image
2,295
2,295
M
LayoutNet 76
RGB-D pano image
571
571
C
Zillow Indoor 10
RGB pano video
1,524
71,474
G
Realtor360 665
RGB pano image
2,573
2,573
M
PanoContext 70
RGB pano image
700
700
C
Ours
RGB video
2,246
375,677
G
Table I: Related datasets with 3D room structures. We distinguish layout types: Cuboid (C), Manhattan (M), Generic (G). We propose the first dataset with generic 3D room layouts that were annotated without any special acquisition devices sensors and only from an RGB video.
In contrast to the above works, we propose a method for annotating generic 3D room layouts on commonly-available RGB videos. Morcover, annotation process easy for human annotators, involving only drawing in 2D. With it, we annotate 2246 RGB videos from the RealEstate OK dataset (75), which release publicly.
Layout estimation methods. A variety of methods for 2D/3D room layout estimation have been proposed. Some methods estimate from dense 3D point clouds derived from RGB-D fusion 40
18] and some from a single panoramic RGB image [1 65 56 63
76.. Also, many 3D room layout) from a single RGB 64). They are trained on datasets with such 66 Most methods make some additional assumptions such as rooms. This family of methods could benefit from our
methods estimate a 2D room layout (i.e. 2D projection
image [25 31
73
69 54 19
20
13
14
3
2D) room layouts annotations
Cuboid 25 31
73
19 20 or Manhattan
40
new dataset.
Machine-assisted annotation. Many datasets in computer vision have been generated using muuchine-assistee annotations. Among them are object segmentation with bounding boxes [47 26). scribbles [30] [5, extreme points [37]. language models [48], and even localized captions 42 We believe ours is the first method for machine-assisted 3D room layout annotation.
3 Method
We want to estimate the layout of the room in input RGB video. A layout is set of structural elements in 3D, each with an instance index from one of the following classes: Floor, Ceiling, Wall, Slanted (could be wall, ceiling, etc.), Door, and Window. Each structural element is represented by a 3D plane with a finite spatial extent capturing all parts that are the camera's 's field-of-view at any time during the video. This spatial extent is independent of room furniture, which means structural elements should estimated even behind it. Finally, the spatial extents of neighboring elements are expected to be connected at the right contact edges in 3D.
Our method inputs video frames (J....., IN). ). along with their camera intrinsics and extrinsics, e.g. computed by COLMAP (49] The main outputs are the 3D plane equations (pI,....PM of each structural element and a triangular 3D mesh denoting their spatial extent.
In the annotation stage, annotators draw an amodal segmentation mask for each structural element in 2D, and also approximately mark its visible parts in 2D, for each frame independently from a small subset of input frames (every 30th, Sec. 3.1). Then, we propose an automatic pipeline to generate 3D room layouts from these 2D annotations (Fig, 22.. First, we track points on visible parts and assign point tracks to structural elements (Sec. B.2). Then, we estimate 3D plane equation for each structural element by minimizing a joint loss function that combines terms for fitting the 3D geometry induced by its associated point tracks, matching the edges between structural elements marked in the 2D annotations, and encouraging vertical walls to be perpendicular the floor and ceiling (Sec. 3.3 Next, estimate the spatial extent of each plane union of the associated
3

---

sinadural elemants
parts
Human annotators
Result: mesh
visible parts
par-olement visible parts
points sampled find correspondences, relebel
eslimale infinile plane equations
unproject segmeniations. compule union, out planes
structural elementss
Machine
points on the edges
Pipeline overview. For each selected video frame independently, humans annotate segmentation masks for structural elements and their visible parts. We track 2D points on visible parts and use the 3D geometry induced by them to estimate 3D plane equations for the structural elements. Finally, we estimate the spatial extent of each structural element union of unprojected 2D annotations across all frames.
Figure 2:
plane's amodal segmentation over frames (Fig. 2 bottom box, rightmost; Sec, 3.4 automatically measure reconstruction quality, and we retain only videos where it is high (Sec. 3.5
). Finally, we
3.1 Human 2D annotations
2D Structural elements annotation. Annotators draw an amodal segmentation mask for each structural element as a polygon, i.e. including its entire surface even if occluded by furniture (covering classes Floor, Ceiling, Wall, Slanted, Door, Window; Fig. 2). Moreover, also ask annotators to draw occlusion edges. An occlusion edge a curve of arbitrarily shape separating two structural elements that do not meet in 3D along this edge, e.g. when wall occludes part of another wall behind it (Fig. top row, occlusion edges between olive and dark red walls).
2D Visible parts annotation. We additionally ask annotators mark the visible parts of all structural elements approximately in selected frames (Fig. 2 Its main purpose is to guide the process of establishing correspondences between annotations across frames (Sec. 3.2), which in turn helps estimating 3D plane equations (Sec. 3.3). Annotators are not required to label all visible pixels accurately, but to focus on regions with non- non-uniform texture including grating. shadows, flat objects near the surface like posters, paintings, etc. As a rule of thumb, they should exclude objects protruding more than 3 em from the plane surface, furniture, glossy surfaces (e.g. the glass frame of picture), and artificially added overlays (e.g. captions).
3.2 2D Point tracking
key component of our method is tracking 2D points sampled on the visible parts of structural elements. We sample points with low-discrepancy Sobol sampling (51), and then track them over time with RAFT [58] This process results in many point tracks. track o; spans multiple frames f. each with 2D point of-
Structural elements correspondences. Since video frames are annotated independently, labels of the same structural elements do not necessarily match across frames. We solve this problem by establishing correspondences based on the point tracks. For each pair of consecutive annotated frames, we create a correspondence matrix between each pair of structural elements they contain. An entry (i,j) represents number of tracked points that have label i the first frame and label in the second frame. Then, we compute the optimal matching between structural elements by the Hungarian method [24]. This way, the instance indices are consistent over time, with each index representing the same structural element across all video frames.

---

Inputs
Structure
Visible
Tracked points
Edges
Final layout
Figure 3: 3D room layout reconstruction on an example scene. Given an input video and manual 2D annotations of structural elements and their visible parts, we combine point tracks fitting, edge matching. and perpendicularity constraints generate a 3D room layout.
Assigning 2D point tracks to structural elements. The process just described also determines which structural element each point track belongs to. Formally, the process assigns structural element index j to each track 24, which we denote by a(t) = j. This will be useful below, as we try fit a 3D plane based on the evidence from the point tracks assigned to it.
3.3 Estimating 3D plane equations
Each structural element is represented by its 3D plane equation and spatial extent. The equation p has 4 parameters representing the plane's 3D normal and offset from the origin. To estimate it, we optimize the parameters of all planes jointly, so as to minimize a loss involving three terms. First, the 3D plane should fit well the 3D geometry induced by the tracked points assigned that structural element (Fitting point tracks loss). Second, the edges between structural elements in the 2D annotations should correspond the intersections of their corresponding 3D planes (Edge loss). Third, vertical walls should be perpendicular to the floor and ceiling (Perpendicularity loss).
Fitting point tracks loss. Consider 2D point track 2; assigned to a structural element j a(t). Now also consider candidate plane equation pj for that structural element, determining the plane in 3D space. We can now unproject a 2D point æf from the track into straight line in 3D space, given the camera parameters for frame / (Fig. 4 This line will intersect the candidate plane at 3D point x! We denote this process of unprojection and plane intersection as X unproj (o. Pj. f), and refer "unprojecting point =!f onto plane p,".
In practice, all 2D points o!f within a single track xa correspond the same point in 3D space. Hence, we define loss that prefers plane equation Pj such that the 3D unprojection lines for each point within a point track intersect that plane the same point:
Iraa Σ
|unproj(s!, PP..() X.ll
[a,] E, unproj(æf ps- f) is the average all unprojected points for the track; the sum over ff runs over all frames where track x, is visible; and |xy is the number of such frames (i.e. the track length). Basically, the loss penalizes the distance of each frame's unprojections to their mean. The complete loss runs over all point tracks t:
where X, =
T
CT
Σ
PP(o), | X₂
Note how this includes all point tracks that are assigned to various different planes, as marked in att).
Edge loss. This loss encourages the intersection of two structural elements in 3D to project to the edge between their 2D annotations. other words, an edge between two structural elements 2D annotations unprojects to the intersection of the corresponding 3D planes. The core constraint can be phrased formally as: when the same 2D point is unprojected onto two different 3D planes, the resulting 3D points will be equal only if the two 3D planes intersect at that 3D point.
5

---

P3
X
Figure 4: Unprojecting 2D points onto 3D plane. Each point 2/ (green) in a point track is unprojected onto its assigned plane Py as a 3D point X (blue). This point is the intersection of the unprojected line (blue) and the 3D plane. If we had the ideal 3D plane, all these lines would intersect at the same 3D point. In practice, we plane equation by minimizing a loss on the distance of X to their mean X₂ (red).
To compute the loss, we first extract 2D edge points eg that lie at the boundary between two structural elements in annotated frame. For each edge point es, denote the frame where it lies by f(k), and the indices of the two structural elements connects by ilk) and jkk). It is defined as LE
1 E
umproj (eee f(.)) - unproj ck. Ps.R)
where E is the total number of edge points across all frames and all structural elements.
Perpendicularity loss. Since we know that vertical walls are perpendicular to the floor and ceiling, we minimize the cosine of the angle between 3D planes associated to these structural elements (slanted structural elements are not affected by this loss). The set of pairs of Wall and Ceiling/Floor is denoted by P. where (w, E P. w. [1, Then, the perpendicularity loss is defined as
Cp
Σ (Nc
cos <(pw, Pe)
Joint loss. The overall loss is weighted sum of all loss terms: L +agLe apLp.
Optimization. We minimize the joint loss with ADAM 23 with a learning rate of 0.1. The plane equations are initialized with ones, and normalized so that the normal direction part is of unit length. The loss is minimized for at most 100k iterations. Optimization is stopped early if there is no improvement for the last 500 iterations. The hyperparameters are set manually to ag = 0.1 and ap = 0.1. The number of samples for point tracking is computed by requiring the average distance between neighboring sampled points to exactly 30 pixels.
3.4 Estimating the 3D spatial extent of planes
At this point, the plane equations of all structural elements are known. As those equations define infinite planes, now cut them to determine their spatial extent, defined by the union of all parts that are in the camera field-of- of-v-ewwvww-- at any time during the video (even pieces occluded by furniture). We do not extrapolate beyond what the video shows, and do not strive for closed room models. We represent spatial extent with a triangular mesh, where each triangle has a label indicating to which structural element it belongs.
Polygon union. As the first step, we determine the complete spatial extent of the plane, integrated across all video frames. Usually, no single frame shows the whole extent, with different frames showing different (typically overlapping) pieces. Therefore, we compute the complete spatial extent as the union of extents each frame. To achieve this, we unproject the polygon annotations of structural element to 3D using the unproj(-) function of Sec. S.3. Since unprojecting 2D polygons from different frames onto the same 3D plane results coplanar polygons 3D, we directly compute their union (Fig.) 2 bottom box, right).

---

before
after
Figure Spatial extent refinement, before (top) and after (bottom). We cut hanging walls extending outside the room boundary, and fill in the holes between neighboring planes (blue).
Refinement. As shown in Fig. (top), the resulting mesh contains artifacts, e.g. hanging wall extending outside the room boundary and small holes between structural elements. Moreover, structural elements never intersect precisely in the 3D space. To address this shortcoming, we introduce refinement procedure, which ensures that structural elements intersect precisely in the 3D space. We extend each plane's unified polygon towards its neighboring planes until they intersect in 3D. If such extension does not increase the size of the polygon by more than 10%, is accepted. This procedure fills in the holes between neighhoring 3D polygons. Finally, for each pair of intersecting 3D polygons, find their intersection line and the smallest part it smaller than 10% of the total area. As shown in Fig. 5 the room layout becomes more visually pleasing after refinement.
Doors and windows. Annotators are asked to label all doors and windows separately unless the door is open, in which case the next room must be annotated. We assign each door and window a structural element index, with which shares most of border. Then, the door and window segmentation masks are used for spatial extent calculation of their assigned structural elements. Finally, we also calculate spatial extent of each door and window and include them in the final reconstructed mesh.
3.5 Automatic quality control
The 3D estimation method in Sec. 3.1 to 3.3 can sometimes fail. This typically happens in the presence of large annotator errors or insufficient camera motion. To ensure high quality, we introduce automatic quality control mechanism.
We measure Reprojection IoU by rendering the estimated 3D layout with the given camera parameters for each video frame. Such rendering resembles the input 2D structural elements annotation. We compute the similarity between the rendering of each structural element and its annotation as Intersection-over- Union (IoU), and average over all structural elements and frames in a video (e.g. Fig. 3 comparing structural elements annotations to rendered final layouts).
Since our 3D estimation method is stochastic due to randomly sampled point tracks and stochastic optimization, each run gives different results. We automatically select the best run for each video after R = 100 repetitions, based on Reprojection loU. We then discard all videos with reprojection IoU below 0.8, keeping only high quality reconstructions.
4 Experiments
use our method from Sec. 3 to annotate the RealEstate 10k dataset [75] (RE10k), which gathered videos of indoor scenes from YouTube. A total of 21 human raters annotated 3743 of those videos. After keeping only high quality ones (S5c.3.3). this results 2246 3D layouts in our dataset. Each video lasts from to 10 seconds, with 6.5 seconds average, recorded at 30 frames per second. We annotate one frame per second to avoid redundancy, but use all frames for point tracking (Sec. 3.3)
Finally, for evaluation purposes, also annotate 50 scenes from the ScanNet dataset [T
In the following subsections, we present evaluation measures based on ground-truth 2D segmentations and 3D depth (Sec. 4.1). evaluate the quality of our annotations ScanNet [1] and RE10k according
7

---

RETOK
ScanNet
11
Runs
ToUT
ToU>
Ours (full method)
100
0.89
0.90
0.22
Ours (no quality control)
100
0.83
0.85
0.30
depth)
ScanNet 1
(depth)
Ours (no quality control)
30
0.81
0.84
0.33
Pr
Re
loUt
PrW
Re 1
loUt
Ours (no quality control)
1
0.72
0.79
0.36
Ours 88.5 98.0 0.89
83.8 91.2 0.90
Ours (no quality control)
10
0.90
0.84
0.33
No perpend. loss Lp
10
0.80
0.83
0.34
No edge loss LE
10
0.72
0.84
0.33
No points tracking Cr
10
0.34
Fail
Fail
Table 2: Left Top row: our main results, with our full method. Rows 2-4: ablation removing the automatic quality control filter and reducing the number of runs R. Rows 5-8: ablation removing parts of the loss. Right Manual inspection on 50 randomly selected scenes from RE10k and 20 from ScanNet. We report precision and recall of whether a structural element is successfully reconstructed.
to those measures (Sec. 4.2). and additionally also by manual inspection (Sec. evaluate a state-of-the-art 2D layout estimation method [31] on our dataset (Sec. 4.4p
4.3
Finally, we
4.1 Evaluation measures
Reprojection IoU. We first measure quality based on the consistency of our reconstructed layouts with the input 2D annotations, as defined in Sec. 3.5
Depth error. As a second measure, we compute the average absolute depth error e. This is computed only on the visible parts of structural elements, as other parts might contain furniture occluding them. We produce depth maps by rendering our 3D room layout at each frame, and then compare them to the ground ttuth depth maps. These are available for ScanNet [T], which was acquired with an RGB-D sensor, but not for RE10k, which contains pure RGB videos.
4.2 Results
Full method on ScanNet. We validate the quality of our reconstructions on the ScanNet dataset [ (50 random scenes), based on the ground-truth depth maps released with it. These have been acquired with structured light scanner and are therefore high quality. As Table 2 (left, top row) shows, the average depth error is 0.22 meters. To put this in context: walls are 2.7 meters tall and 3 meters long on average, and rooms in ScanNet have large spatial extent of 7 meters average. Hence, depth error of only 0.22 meters represents high quality. This is further confirmed by a very high average Reprojection IoU of 0.90 ScanNet.
Full method on RE10K. Our full method achieves a high Reprojection IoU of 0.89 on RE10k (Table 2 left, top row). This indicates high quality 3D reconstructions recovering most structural elements and their spatial extent correctly.
Ablation study on 1E10 and ScanNet. Table (left, rows 2-8) show several ablations. In row 2 we turn off the quality control filter of Sec. 3.5 This results is significantly worse Reprojection IoU and depth error, demonstrating that our quality control filter is working well. From row 2 to gradually reduce the number of runs R. Both Reprojection IoU and depth error get continuously worse with fewer and fewer runs. The reconstruction quality is substantially higher at R = 100 than for just single run, demonstrating the value of having multiple runs and of our run selection criterion (Sec. 3.5). Also note how in all these experiment better IoU always corresponds to better depth error on ScanNet, confirming that Reprojection IoU is good indicator of reconstruction quality.
Finally, we ablate components our joint loss (Sec. 3.3 in rows 5-8. The fitting point tracks loss Lr (2) is the most important one since the optimization fails completely (results in NaNs gradients) on ScanNet scenes (row 8). On the few scenes from RE10k where optimization converged, Reprojection IoU is very low (0.34 on average). Disabling the edge loss LE 3 leads to significantly lower IoU

---

Figure 6: 3D Room layout reconstruction examples. For each example scene, we show one or two of the input annotations overlaid on the video frame (top), and the final reconstructed layout (bottom).
and larger depth error (row 7), compared to the using the complete loss (row 5). The perpendicularity loss Lp has a smaller impact on overall performance (row 6).
4.3 Manual inspection
We assess here the quality of our 3D layouts by manual inspection of 50 randomly selected scenes from RE10k and 20 from ScanNet. For each scene, we measure (1) the total number of ground-truth structural elements in the scene; (2) the number of elements successfully reconstructed (recall); (3) the number spurious elements incorrectly created (enabling to measure precision). A structural element is considered successfully reconstructed if the plane normal is correct and the spatial extent consistent with the 2D annotations (up to 10% error both cases). Note that humans cannot verify the correct depth of each plane, as this is perceptually hard to judge. As Table 2 (right) shows, quality is excellent with a recall 98% and precision of 89% on RE10k.
4.4 Automatic layout estimation our dataset
Most methods for 3D room layout estimation require special inputs, such e.g. RGB-D fusion 40 18), and therefore cannot run on our dataset. Some methods input single RGB image,
they only estimate a 2D room layout (i.e. 2D projection of 3D room layout), e.g. 25 31
73
69

---

LSUN dataset 66
Hedau dataset 19
Our dataset
Pixel Error (%)4
Pixel Error (%)
Pixel Error (% 4
Hedau et al. 19
24.23
21.20
Mallya et al.
B5
16.71
12.83
DeLay [2]
10,63
9.73
CFILE 45
7.57
8.67
Zhang el al. [68
6.58
12.70
ST-PIO 71
5.29
6.60
Lin et al. [1 (baseline)
6.25
7.41
26.3
Table 3: Baseline method. We train and evaluate the method [31] on our dataset, following the train
and test split. We compare existing datasets [66 [9] and other layout estimation methods.
While our dataset offers 3D layout annotations, we evaluate here 2D estimation method 31 on it. This allows us to get a sense of the difficulty our dataset and to position relative to other existing datasets.
54 19 20
13
14
34,
33
We train and evaluate the method [1] on our dataset. To this end, we render our 3D layouts using the camera poses of the video frames, and we run [1 each frame independently. In Table 3 we report results of [3] on our dataset, and of various methods on two existing datasets [66] 19T. The method [3I] performs at the state-of-the-an on the existing datasets, with low eтror around 6% - 7%. Instead, it performs much worse on our dataset (26%), demonstrating offers harder challenge than the previous ones.
5 Limitations
Our 3D room structures only contain surfaces that are observed in the videos. We do not produce geometry for the unobserved parts. For example, if floor is partially visible behind wall, only produce geometry for what we can observe from the video. Ideally, we would also like to generate plausible geometry for the unobserved parts.
Another limitation that we reconstruct only planar surfaces. Our current method cannot handle curved surfaces, which sometimes can occur in upscale apartments.
6 Conclusion
We proposed novel way to annotate 3D room layouts from only 2D annotations. We annotate 2246 general 3D room layouts dataset of real-estate RGB videos from YouTube, which have publicly released. Extensive evaluations confirmed the high quality of this dataset.
Acknowledgements. We thank Prabhanshu Tiwari and Mohd Adil for coordinating the annotation process, and all the annotators who have contributed to creating the dataset.
10

---

References
[1] Firikur Agustsson, Jasper R. R. Uijlings, and Vittorio Ferrari. Interactive full image segmentation by considering all regions jointly. In CVPR, June 2019.
[2] Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann. Objectron:
A large scale dataset of object-centric videos in the wild with pose annotations. In CVPR, pages 7822-7831,
June 2021.
[3] Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, and Matthias Niessner.
Scan2cad: Learning cad model alignment in rgh-d scans. In CVPR, June 2019.
[4] Armen Avetisyan, Tatiana Khanova, Christopher Choy. Denver Dash, Angela Dai, and Matthias Nießner.
Scenecad: Predicting object alignments and layouts rgb-d scans. ECCV, August 2020.
[5] Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jiebo Luo, and Tsuhan Chen. icoseg: Interactive co-
segmentation with intelligent scribble guidance. In CVPR, pages 3169-3176, 2010.
[6] Ricardo Cabral and Yasutaka Furukawa, Piecewise planar and compact floorplan reconstruction from images. In CVPR, pages 628 635, 2014.
[7] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments.
3DV, 2017.
[8] Xiaoxue Chen, Hao Zhao, Guyue Zhou, and Ya-Qin Zhang. Pq-transformer: Jointly parsing 3d objects and layouts from point clouds. IEEE Robotics and Automation Letters, 2022.
[9] Jasmine Collins. Shubham Goel, Kenan Deng, Achleshwar Luthra, Leon Xu, Erhan Gundogdu, Xi Zhang, Tomas F Yago Vicente. Thomas Dideriksen, Himanshu Arora, Matthieu Guillaumin, and Jitendra Malik. Abo: Dataset and benchmarks for weal-world 3d object understanding. CVPR, 2022.
[10] Steve Cruz, Will Hutchcroft, Yuguang Li, Naji Khosravan, Ivaylo Boyadzhiev, and Sing Bing Kang- Zillow indoor dataset: Annotated floor plans with 360" panoramas and 3d room layouts. In CVPR, pages
2133-2143, June 2021.
[11] Angela Dai. Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 3d reconstructions of indoor scenes. In CVPR, 2017.
Scannet: Richly-annotate
[12] Saumitro Dasgupta. Kuan Fang. Kevin Chen, and Silvio Savarese. Delay: Robust spatial layout estimation
for cluttered indoor scenes. In CVPR, pages 616-624, 2016.
[13] Luca Del Pero, Joshua Bowdish, Daniel Fried, Bonnic Kermgard, Emily Hartley, and Kobus Barnard.
Bayesian geometric modeling of indoor scenes. In CVPR, pages 2719-2726, 2012.
[14] Luca Del Pero, Joshua Bowdish, Bonnie Kermgard, Emily Hartley, and Kobus Barnard, Understanding bayesian rooms using composite 3d object models. In CVPR, pages 153- 53--160,220 2013.
[15] Arda Duzceker, Silvano Galliani, Christoph Vogel, Pablo Speciale, Mihai Dusmanu, and Marc Pollefeys. Deepvideomvs: Multi-view stereo on video with recurrent spatio-temporal fusion. In CVPR, pages
15324-15333, June 2021.
[16] Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In ICCV,
pages
2021.
[17] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao.
3d-future: 3d furniture shape with texture. IJCV pages 1-25, 2021.
[18] Huan-ang Gao, Beiwen Tian, Pengfei Li, Xiaoxue Chen, Hao Zhao, Guyue Zhou, Yurong Chen, and Hongbin Zha. From semi-supervised to onni-supervised room layout estimation using point clouds. IEEE Robotics and Automation Letters, 2023.
[19] Varsha Hedau, Derek Hoiem, and David Forsyth. Recovering the spatial layout of cluttered rooms. In
ICCV, ICCV, pages 1849 1856, 2009.
[20] Varsha Hedau, Derek Hoiem, and David Forsyth. Recovering free space of indoor scenes from a single
image. In CVPR, pages 2807-2814, 2012.
[21] Binh-Son Hua, Quang-HHieu Pham, Due Thanh Nguyen. Minh-Khoi Tran, Lap-Fai Yu, and Sai-Kit Yeung. Scenenn: A scene meshes dataset with annotations. In 3DV, 2016.
[22] Haoshuo Huang. Qixing Huang, and Philipp Krahenbuhl. Domain transfer through deep activation matching. In ECCV, September 2018
[23] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and
Yann LeCun, editors, ICLR, 2015.
[24] H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 1955.
[25] C. Lee, V. Badrinarayanan, T. Malisiewicz, and A. Rabinovich. Roomnet: End-to-on end room layout estimation. In ICCV, pages 4875 4884, Los Alamitos, CA. USA, oct 2017. IEEE Computer Society. [26] Victor Lempitsky, Pushmeet Kohli, Carsten Rother, and Toby Sharp. Image segmentation with bounding box prior. In ICCV, pages 277-284, 2009.
[27] Wenbin Li, Sajad Sacedi, John McCormac, Ronald Clark, Dimos Tzoumanikas, Qing Ye, Yuzhong Huang, Rui Tang, and Stefan Leutenegger. Interiornet: Mega-seale multi-sensor photo-realistic indoor scenes dataset. In BMVC, 2018
[28] Zhengqin Li, Ting- Wei Yu, Shen Sang, Sarah Wang, Meng Song, Yuhan Liu, Yu-Ying Yeh, Rui Zhu, Nitesh Gundavarapu, Jia Shi, Sai Bi, Slong-Xing Yu, Zexiang Xu, Kalyan Sunkavalli, Milos Hasan, Ravi Ramamoorthi, and Manmohan Chandraker. Openrooms: An open framework for photorealistic indoor scene datasets. In CVPR, pages 7190-7199, June 2021.
11

---

[29] Joseph J Lim. Hamed Pirsiavash, and Antonio Torralba. Parsing ikea objects: Fine pose estimation. In ICCV. pages 2992-2999, 2013.
[30] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR. pages 3159-3167, 2016.
[31] Hung Jin Lin, Sheng- theng--Wii Huang. Shang- Hong Lai, and Chen-Kuo Chiang. Indoor scene layout estimation from a single image. In ICPR, 2018.
[32] Tsung-Yi Yi Lin, Michacl Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, ECCV, pages 740-755, Cham, 2014. Springer International Publishing.
[33] Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz. Planerenn: 3d plane detection and reconstruction from single image. In CVPR, June 2019.
[34] Chen Liu, Jimei Yang, Duygu Ceylan. Ersin Yumer, and Yasutaka Furukawa. Planenet: Piece-wise planar reconstruction from single rgb image. In CVPR, June 2018.
[35] Arun Mallya and Svetlana Lazebnik. Learning informative edge maps for indoor scene layout prediction. In ICCV, pages 936 944, 2015.
[36] K. Maninis, S. Popov, M. Niesner, and V. Ferrari. Vid2cad: Cad model alignment using multi-view
constraints from videos. JEEE TPAMI, +5(01):1320-1322 jan 2023.
[37] K.-K. Maninis, S. Caclles, J. Pont-Tuset, and Van Gool. Deep extreme cut: From extreme points to object segmentation. In CVPR, 2018.
[38] John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J.Davison. Scenenet rgb-d: Can 5m synthetic images beat generie imagenet pre-training on indoor segmentation? ICCV, 2017.
[39] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Whang--hann Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild.
In CVPR, 2014.
[40] Srivathsan Murali, Pablo Speciale. Martin R. Oswald, and Mare Pollefeys. Indoor scan2bim: Building
information models of house interiors. IROS, pages 6126-6133, 2017. 2017.
[41] Giovanni Pintore. Marco Agus, and Enrico Gobbetti. Atlantanet: Inferring the 3d indoor layout from single 360 image beyond the manhattan world assumption. In ECCV, page 432-448, Berlin, Heidelberg, 2020. Springer-Verle
[42] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting
vision and language with localized narratives. In ECCV,, 2020.
[43] Stefan Popov.
Bauszat, and Vittorio Ferrari. Corenet: Coherent scene reconstruction from a single
rgb image. In ECCV, 2020.
[44] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.
In ICCV, 2021.
[45] Yuzhuo Ren, Shangwen Li. Chen Chen, and C.-C. Jay Kuo. A coarse-to-fine indoor layout estimation (efile) method. In Shang- Hong Lai, Vincent Lepetit, Ko Nishino, and Yoichi Sato, editors, ACC,, pages 36-51, Cham, 2017. Springer International Publishing.
[46]
Stephan R. Richter, Hassan Abu Alhaija, and Vladlen Koltun. Enhancing photorealism enhancement.
IEEE TPAMI, 1502)1170- 2023.
[47] Carsten Rother, Vladimir Kolmogorov. and Andrew Blake, "grabcut": Interactive foreground extraction
using iterated graph cuts. ACM TOG, 23(3):309-314, aug 2004. Christian Rupprecht, Iro Laina, Nassir Navab, Gregory D. Harger, and Federico Tombari. Guide me:
[48]
Interacting with deep networks. CVPR, 2018.
[49] Johannes Lutz Schönberger and Jan-Michael F'rahm. Structure- monectue-from--oe motion revisited. In CVPR, 2016. Johannes Lutz Schönherger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016.
from-motion
[50]
[51] LM Sobol'. On the distribution of points in a cube and the approximate evaluation of integrals. USSR Computational Mathematics and Mathematical Physics, 7(4):86-112 12, 1967.
[52] Shuran Song, Samuel P. Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, pages 567-576. 2015.
[53] Shuran Song, Fisher Yu, Andy Zeng. Aneel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from single depth image. CVPR, 2017.
[54] Sinisa Stekovic, Shreyas Hampali, Mahdi Rad, Sayan Deb Sarkar, Friedrich Fraundorfer, and Vincent Lepetit. General 3d room layout from single view by render-and-comparr. In Andrea Vedaldi. Horst Bischof, Thomas Brox, and Jan Micchal Frahm, editors, ECCV, pages 187-203, Cham, 2020. Springer International Publishing.
[55] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June You. Yuyang Zou, Kimberly Leon. Nigel Carter, Jesus Briales. Tyler Gillingham, Elias Mueggler, Luis Pesqueira. Manolis Savva. Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arv:: 006607 2019.
[56] Cheng Sun, Chi- Wei Hsiao, Min Sun, and Stwann-Tzong Chen. Horizonnet: Learning room layout with 1d representation and pano stretch data augmentation. In CVPR, June 2019.
12

---

[57] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In
CVPR, 2018.
[58]
Zachary Teed and Jía Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, page
402-419, Berlin, Heidelherg, 2020. Springer-Verde
[59]
Acuna, Mark Brophy, V. Jampani. Cem Anil, Thang To. Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. CVPRW, pages 1082- 0082-10828, 2018
Jonathan Tremblay. Aayush Prakash,
[60]
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. ICLR, 2018.
[61] Fei Xia, Amir R. Zamir, Zhi-Yang Yang He. Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson Env:
real- weal-world perception for embodied agents. In CVPR. IEEE, 2018.
[62] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection
in the wild. In WACV. pages 75-82, 2014.
[63] Jiu Xu, Bjöm Stenger, Tommi Kerola, and Tony Tung. Pano2cad: Room layout from single panorama
image. In WACV, pages 044-362.222 2017.
[64] Chenggang Yan, Biyao Shao, Hao Zhao, Ruixin Ning, Yongdong Zhang, and Feng Xu. 3d room layout estimation from a single rgb image. JEEE Transactions on Multimedia, (3(11)301443000 3024,, 2020. [65] Shang-Ta Yang, Fu-En Wang. Chi-Han Peng, Peter Wonka, Min Sun, and Hung kuo Chu. Dula-net: projeppoooction network for estimating room layouts from a single rgh panorama. CVPR, pages 3358-3367, 2018.
[66]
Fisher Yu, Yinda Zhang. Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03365, 2015. Sergey Zakharov, Rares Ambrus, Vitor Campanholo Guizilini, Wadim Kehl, and Adrien Gaidon. Photo- realistic neural domain randomization. In ECCV, 2022.
[67]
[68] Weidong Zhang. Wei Zhang. Kan Liu, and Jason Gu. Learning predict high-quality edge maps for room layout estimation. IEEE Transactions Multimedia, 19(5):33-9433 2017.
[69] Weidong Zhang, Wei Zhang, and Yinda Zhang. Geolayout: Geometry driven room layout estimation based depth maps of planes. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, [70] Yinda pages 632-648, Cham, 2020. Springer International Publishing.
ECCV,
Shuran Song, Ping Tan, and fianxiong Xiao. Panocontext: A whole-room 3d context model for panoramic scene understanding. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, ECCV, pages 6666, Cham, 2014. Springer International Publishing.
[71] Hao Zhao, Ming Lu, Anbang Yao, Yiwen Guo, Yurong Chen, and Li Zhang. Physics inspired optimization on semantic transfer features: An alternative method for room layout estimation. In CVPR, pages
2017.
[72] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In ECCV, 2020.
[73] Wenzhao Zheng, Jiwen Lu, and Jie Zhou. Structural deep metric learning for room layout estimation. In ECCV, pages 735-751, Cham, 2020. Springer International Publishing.
[74] Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pages 5122-5130, 2017.
[75]
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification:
Learning view synthesis using multiplane images. SIGGRAPH, 37, 2018.
[76] Chuhang Zou, Alex Colburn, Qi Shan, and Derek Hoiem. Layoutnet: Reconstructing the 3d room layout
from a single rgb image. In CVPR, pages 2051 2051-2059, 2018
[77] Chuhang Zou, Jheng- Wei Su, Chi-Han Peng. Alex Colburn, Qi Shan, Peter Wonka, Hung- Kuo Chu, and Derek Hoiem. Manhattan room layout reconstruction from single 360 image: A comparative study of state-of-the-art methods. IJV,, 12)(5): (00()):411-111 1410-1431, 2021.
13


Reviews:
[{"review_id": "uLaiuyZtT1", "reviewer": "Reviewer_42Nb", "summary": "", "questions": "", "limitations": "The authors do not discuss the potential limitations of their dataset in the paper. As I pointed out in the previous section, in my opinion, the dataset created migh have some weaknesses. \n\nThe authors did not comment in societal negative impacts. As far as I checked, I think their work does not have a societal impact.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* The method for generating the ground truth is reasonable and well explained. \n* The ablation of the method is solid and evidences that, although not perfect, the method offers high quality layouts.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2MseSqdiFX", "reviewer": "Reviewer_WKsm", "summary": "", "questions": "", "limitations": "None.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ The method is sound. I have been thinking for a while about the same idea: using pytorch optimizers to get 3D planes on multiview images.\n+ A large-scale evaluation demonstrates that the metholody's scalability.\n+ This paper comes from Google and Niesser's group, which is not a technical issue but potentially an advantage. This may (re-)trigger interests in this important field that I've been working on for years.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "tMvuZX6Af2", "reviewer": "Reviewer_aGuE", "summary": "", "questions": "", "limitations": "Yes in the supplemental.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This is a novel and smart way to annotate 3D scene layout. Many researchers have tried to annotate 3D from 2D annotations, but the proposed approach finds a good perspective.\n- The annotation quality is high, which is validated by humans and on ScanNet. This is critical to a 3D dataset from 2D annotations.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Ei2tb5LyqQ", "reviewer": "Reviewer_ujhv", "summary": "", "questions": "", "limitations": "There is no limitation section in the paper. ", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The proposed method is the first for machine-assisted 3D room layout annotation. They can generate the 3D layouts of indoor scenes using 2D RGB videos, without 3D data.\n\n- The method can automatically generate 3D layouts from 2D annotations of 2D structural elements annotation and 2D visible parts annotation.\n\n- The experiments on the reprojection of 3D layouts to 2D validate the reconstruction accuracy of proposed datasets.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "UTPHi2xWeY", "reviewer": "Reviewer_ChJp", "summary": "", "questions": "", "limitations": "1. First, it would be nice to see a 3D layout estimation baseline established using your data -- it seems to be a rule of thumb when presenting a new dataset. At least, I would recommend preparing a training/validation/test split and provide data loaders (e.g., in PyTorch). In Related Work, you mention a number of layout estimation methods, claiming that these methods could benefit from the use of your dataset: while being quite evident, this statement needs though to be supported with an empirical evidence.\n\n2. Next, you mention that only a small part of frames should be annotated to obtain a decent quality. This is a good finding, however, is causes plenty of questions to arise. Have you performed a study to determine the minimal ratio of frames that should be annotated? How the quality of layouts depend on the ratio of annotated frames? Is there a certain threshold, where the quality reaches a plateau and does not improve with the increase of the ratio?\n\n3. In addition, your algorithm requires camera intrinsics and extrinsics alongside annotated RGB inputs; it is claimed that camera parameters might be computed by COLMAP. For ScanNet experiments, do you use (precise) camera parameters provided in the dataset? Or do you assume there are only RGB data is available, and estimate camera poses and intrinsics by yourselves? If you have not conducted such a study, I would definitely advise you doing so, since the scenario with only RGB videos available might highlight the potential and the limitations of your approach in the best possible way.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "*First, as I understand, your contribution is two-fold: first, you describe an algorithm for producing 3D layouts from 2D annotations, second, you present a dataset of layouts created by this algorithm. So, I assume that concerns about both the process (algorithm) and the result (dataset) are both legit, and review your work accordingly.*\n\n**Significance.** The contribution is undisputable, considering the fact that there are no other approaches of creating a 3D room layout dataset from RGB videos only. Besides, no other layout dataset shares the same characteristics, so there are no direct competitors to the presented dataset.\n\n**Relevance for a broad community.** This is a quiet niche work, serving for specific purposes, which hardly can be re-conceptualized and used for another tasks rather than 3D layout estimation. \n\n**Quality.** This work seems to be thoughtful and solid, performed at a relatively high technical level, with much attention to details. Overall, it makes a good impression, while I personally find it a bit lacking of motivation behind certain design choices.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper received overall positive reviews. Reviewers generally regard the method for generating the ground truth 3D room structures novel and sound and the ablation study solid. They initially raised some concerns: dataset motivation, how this dataset compares with other dataset like Zillow Indoor, failure case analysis, release of the annotation tool, algorithm advantage, etc. The rebuttal was persuasive and addressed most concerns. The final ratings unanimously recommend acceptance. The AC checked the paper, rebuttal, and review comments, and recommends accepting the paper.
Decision: Accept (Poster)