SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Runs	RE10k	ScanNet [11]
IoU↑	IoU↑	ε↓
Ours (full method)	100	0.89	0.90	0.22
Ours (no quality control)	100	0.83	0.85	0.30
Ours (no quality control)	30	0.81	0.84	0.33
Ours (no quality control)	1	0.72	0.79	0.36
Ours (no quality control)	10	0.90	0.84	0.33
No perpendicular. loss LP	10	0.80	0.83	0.34
No edge loss LE	10	0.72	0.84	0.33
No points tracking LT	10	0.34	Fail	Fail
RE10k (no depth)	ScanNet [11] (depth)
Pr%↑	Rc%↑	IoU↑	Pr%↑	Rc%↑	IoU↑
Ours	88.5	98.0	0.89	83.8	91.2	0.90
Table 2: Left Top row: our main results, with our full method. Rows 2-4: ablation removing the automatic quality control filter and reducing the number of runs  $R$ . Rows 5-8: ablation removing parts of the loss. Right Manual inspection on 50 randomly selected scenes from RE10k and 20 from ScanNet. We report precision and recall of whether a structural element is successfully reconstructed.
to those measures (Sec. 4.2), and additionally also by manual inspection (Sec. 4.3). Finally, we evaluate a state-of-the-art 2D layout estimation method [31] on our dataset (Sec. 4.4).
4.1 Evaluation measures
Reprojection IoU. We first measure quality based on the consistency of our reconstructed layouts with the input 2D annotations, as defined in Sec. 3.5.
Depth error. As a second measure, we compute the average absolute depth error  $\epsilon$ . This is computed only on the visible parts of structural elements, as other parts might contain furniture occluding them. We produce depth maps by rendering our 3D room layout at each frame, and then compare them to the ground-truth depth maps. These are available for ScanNet [11], which was acquired with an RGB-D sensor, but not for RE10k, which contains pure RGB videos.
Full method on ScanNet. We validate the quality of our reconstructions on the ScanNet dataset [11] (50 random scenes), based on the ground-truth depth maps released with it. These have been acquired with a structured light scanner and are therefore high quality. As Table 2 (left, top row) shows, the average depth error  $\epsilon$  is 0.22 meters. To put this in context: walls are 2.7 meters tall and 3 meters long on average, and rooms in ScanNet have large spatial extent of 7 meters on average. Hence, a depth error of only 0.22 meters represents high quality. This is further confirmed by a very high average Reprojection IoU of 0.90 on ScanNet.
Full method on RE10k. Our full method achieves a high Reprojection IoU of 0.89 on RE10k (Table 2, left, top row). This indicates high quality 3D reconstructions recovering most structural elements and their spatial extent correctly.
Ablation study on RE10k and ScanNet. Table 2 (left, rows 2-8) show several ablations. In row 2 we turn off the quality control filter of Sec. 3.5. This results is significantly worse Reprojection IoU and depth error, demonstrating that our quality control filter is working well. From row 2 to 4 we gradually reduce the number of runs  $R$ . Both Reprojection IoU and depth error get continuously worse with fewer and fewer runs. The reconstruction quality is substantially higher at  $R = 100$  than for just a single run, demonstrating the value of having multiple runs and of our run selection criterion (Sec. 3.5). Also note how in all these experiment better IoU always corresponds to better depth error on ScanNet, confirming that Reprojection IoU is a good indicator of reconstruction quality.
Finally, we ablate the components of our joint loss (Sec. 3.3) in rows 5-8. The fitting point tracks loss  $\mathcal{L}_T$  (2) is the most important one since the optimization fails completely (results in NaNs gradients) on ScanNet scenes (row 8). On the few scenes from RE10k where optimization converged, Reprojection IoU is very low (0.34 on average). Disabling the edge loss  $\mathcal{L}_E$  (3) leads to significantly lower IoU
Figure 6: 3D Room layout reconstruction examples. For each example scene, we show one or two of the input annotations overlaid on the video frame (top), and the final reconstructed layout (bottom).
and larger depth error (row 7), compared to the using the complete loss (row 5). The perpendicularity loss  $\mathcal{L}_P$  (4) has a smaller impact on overall performance (row 6).
4.3 Manual inspection
We assess here the quality of our 3D layouts by manual inspection of 50 randomly selected scenes from RE10k and 20 from ScanNet. For each scene, we measure (1) the total number of ground-truth structural elements in the scene; (2) the number of elements successfully reconstructed (recall); (3) the number of spurious elements incorrectly created (enabling to measure precision). A structural element is considered successfully reconstructed if the plane normal is correct and the spatial extent is consistent with the 2D annotations (up to a  $10\%$  error in both cases). Note that humans cannot verify the correct depth of each plane, as this is perceptually hard to judge. As Table 2 (right) shows, quality is excellent with a recall of  $98\%$  and precision of  $89\%$  on RE10k.
4.4 Automatic layout estimation on our dataset
Most methods for 3D room layout estimation require special inputs, such as e.g. RGB-D fusion [40, 4, 8, 18], and therefore cannot be run on our dataset. Some methods input a single RGB image, but they only estimate a 2D room layout (i.e. 2D projection of a 3D room layout), e.g. [25, 31, 73, 69,
LSUN dataset [66]	Hedau dataset [19]	Our dataset
Pixel Error (%)↓	Pixel Error (%)↓	Pixel Error (%)↓
Hedau et al. [19]	24.23	21.20	-
Mallya et al. [35]	16.71	12.83	-
DeLay [12]	10.63	9.73	-
CFILE [45]	7.57	8.67	-
Zhang et al. [68]	6.58	12.70	-
ST-PIO [71]	5.29	6.60	-
Lin et al. [31] (baseline)	6.25	7.41	26.3
Table 3: Baseline method. We train and evaluate the method [31] on our dataset, following the train and test split. We compare to existing datasets [66, 19] and to other layout estimation methods.
54, 19, 20, 13, 14, 34, 33]. While our dataset offers 3D layout annotations, we evaluate here a 2D estimation method [31] on it. This allows us to get a sense of the difficulty of our dataset and to position it relative to other existing datasets.
We train and evaluate the method [31] on our dataset. To this end, we render our 3D layouts using the camera poses of the video frames, and we run [31] on each frame independently. In Table 3, we report results of [31] on our dataset, and of various methods on two existing datasets [66, 19]. The method [31] performs at the state-of-the-art on the existing datasets, with a low error around  $6\% - 7\%$ . Instead, it performs much worse on our dataset  $(26\%)$ , demonstrating it offers a harder challenge than the previous ones.
Our 3D room structures only contain surfaces that are observed in the videos. We do not produce geometry for the unobserved parts. For example, if a floor is partially visible behind a wall, we only produce geometry for what we can observe from it in the video. Ideally, we would also like to generate plausible geometry for the unobserved parts.
Another limitation is that we reconstruct only planar surfaces. Our current method cannot handle curved surfaces, which sometimes can occur in upscale apartments.
We proposed a novel way to annotate 3D room layouts from only 2D annotations. We annotate 2246 general 3D room layouts on a dataset of real-estate RGB videos from YouTube, which we have publicly released. Extensive evaluations confirmed the high quality of this dataset.
Acknowledgements. We thank Prabhanshu Tiwari and Mohd Adil for coordinating the annotation process, and all the annotators who have contributed to creating the dataset.