SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Quantitative comparison of floorplan reconstruction on Structured3D dataset with real-scale measurements (0.1m). The best results are in bold font.
Method	Room↑	Corner@0.1 ↑	Angle@0.1 ↑	Delta@0.1 ↓
Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
HEAT [9]	0.969	0.94	0.954	0.434	0.436	0.435	0.417	0.419	0.418	0.017	0.017	0.017
RoomFormer [24]	0.977	0.965	0.971	0.637	0.612	0.624	0.608	0.585	0.596	0.029	0.027	0.028
Ours	0.991	0.978	0.984	0.652	0.602	0.626	0.648	0.598	0.622	0.004	0.004	0.004
Box IoU loss calculates the GIoU loss [6]  $\mathbb{L}_{giou}$  for all slicing boxes with ground truth slicing box  $\bar{b}_i$ :
$$
\mathcal {L} _ {b i o u} ^ {(i)} = \mathbb {1} _ {\left\{\sigma^ {*} (i) \leq N \right\}} \mathbb {L} _ {g i o u} \left(b _ {i}, \bar {b} _ {\sigma (i)}\right) \tag {9}
$$
Thus, the final loss  $\mathcal{L}_{btotal}$  of the box transformer  $\mathcal{T}_{box}$  is defined as below:
$$
\mathcal {L} _ {b t o t a l} = \sum_ {i = 1} ^ {N} \lambda_ {3} \mathcal {L} _ {b c e n t e r} ^ {(i)} + \lambda_ {4} \mathcal {L} _ {b p r o b} ^ {(i)} + \lambda_ {5} \mathcal {L} _ {b c o r n e r} ^ {(i)} + \lambda_ {6} \mathcal {L} _ {b i o u} ^ {(i)} \tag {10}
$$
where  $\lambda_{\{3,\dots,6\}}$  are the hyperparameters for weighting the loss functions.
Dataset. We conducted experiments on a large-scale indoor synthetic dataset called Structured3D [25]. Following the approach in [9, 24], we divided the complete dataset into 2991 training samples, 250 validation samples, and 241 test samples, taking into account the ignore list specified in the HEAT [9] codebase. By employing the pre-processing method described in Section 3.1, we obtained a density map of size  $256 \times 256$  for each sample, with pixel values normalized to the range of  $[0, 1]$ .
Baselines. We compare our method with the following competing methods: Floor-SP [8] - scores taken from [24], MonteFloor [22] - scores taken from [24], HEAT [9] - scores based on their official pre-trained weights, RoomFormer [24] - scores based on their official pre-trained weights.
Evaluation metrics. We follow the Room/Corner/Angle evaluation method from MonteFloor and HEAT to measure the quality of floorplan reconstruction. However, the existing metrics suffer from a scale-variant nature as they calculate results based on a normalized  $256 \times 256$  pixel space, without considering the actual size of each floorplan. Consequently, this leads to a more lenient evaluation for floorplans with more rooms or larger sizes. Therefore, we propose the use of real-scale measurements, such as meters, as the threshold for Corner/Angle evaluation. For instance, Corner@0.1 denotes the measurement with a threshold of 0.1 meter.
We found that the current corner evaluation solely focuses on the distance to the ground truth corner, disregarding the shape connectivity between corners. This approach can lead to scenarios where there are multiple "correct" corners but inadequate connectivity (Figure 1(a)), resulting in a low Angle score. Hence, we additionally introduce the Delta metric that measures the global structure of the
Table 3: Ablation study on loss function of box transformer. The best results are in bold font.
Method	Room↑	Corner↑	Angle↑
Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
w/o Box IoU loss	0.981	0.964	0.972	0.756	0.774	0.765	0.740	0.758	0.749
w/o corner mask	0.988	0.975	0.981	0.882	0.821	0.850	0.871	0.811	0.84
Ours	0.991	0.978	0.984	0.889	0.821	0.854	0.878	0.812	0.844
Table 4: Ablation study on post-processing. The best results are in bold font.
Method	Room↑	Corner↑	Angle↑
Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Baseline	0.991	0.978	0.984	0.889	0.821	0.854	0.878	0.812	0.844
cluster w/ GT room corner	0.994	0.987	0.99	0.894	0.831	0.861	0.884	0.822	0.852
w/o gap filling	0.991	0.977	0.984	0.887	0.815	0.849	0.875	0.805	0.839
w/o cell complex rounding	0.97	0.956	0.963	0.745	0.745	0.745	0.72	0.72	0.72
w/ gap filling + rounding	0.991	0.978	0.984	0.889	0.821	0.854	0.878	0.812	0.844
floorplan by computing the discrepancy between Corner and Angle scores. Thus, a lower Delta score indicates a better shape for the resulting floorplans.
Implementation details. We implemented our model in PyTorch and trained our center transformer  $\mathcal{T}_{center}$  on 2 NVIDIA V100s for 2 days. The box transformer  $\mathcal{T}_{box}$  was trained on 8 NVIDIA V100s for 1.9 days. We use the Adam optimizer with  $b_{1} = 0.9$  and  $b_{2} = 0.999$ . The learning rates of two transformers and the ResNet-50 are 2.5e-4, and 1e-5, and we train the center transformer for 5000 epochs, box transformer for 16000 epochs with batch size 128 and 123 for each GPU. We empirically set  $\lambda_{1} = 1, \lambda_{2} = 1$  in Equation 5,  $\lambda_{3} = 1, \lambda_{4} = 1, \lambda_{5} = 5., \lambda_{6} = 1$  in Equation 10,  $\gamma = 2$  in Equation 4, and  $\bar{M} = 25, \bar{N} = 50$  in Section 3.3.
4.2 Floorplan reconstruction performance
Quantitative comparison. As shown in Table 1, our SLIBO-Net outperforms all the baselines in the overall F1 score. Compared to RoomFormer, which has the best F1 score among all baselines, our model improves the F1 score of Room by  $1.3\%$  and Angle by  $3.6\%$ , while maintaining a comparable precision of Corner. Since RoomFormer fails to generate good room shapes (i.e., low Angle score), we round its results by our cell complex, denoted as RoomFormer*. Compared to RoomFormer*, our model achieves better F1 scores in all metrics. It is worth highlighting that our model exhibits a substantial performance boost of  $5.2\%$  in the Delta F1 score when compared to RoomFormer. This advancement indicates our model's ability to generate more plausible and realistic floorplans.
Real-scale evaluation. As shown in Table 2, our model attains the highest scores across most of the metrics, including Room, Corner, Angle, and Delta, with a particular emphasis on the Delta scores. This improvement can be attributed to the incorporation of cell complex regularization, which effectively enhances our model's capabilities under more stringent evaluation conditions.
Qualitative comparison. The qualitative results are shown in Figure 6. Compared to HEAT, our model can reconstruct the correct floorplan structure and preserve the wall thickness, while HEAT fails to reconstruct the walls. Compared to RoomFormer, our model preserves all structural details without any self-intersection, whereas RoomFormer generates some self-intersections that are semantically implausible. Please refer to the supplementary for additional comparisons.
The effectiveness of loss functions. In this experiment, we evaluate the necessity of two loss functions. First, we remove the Box IoU loss (Equation 9) in our total loss function (Equation 10) of the box transformer. Second, we eliminate the corner mask by setting  $w_{\sigma(i)}$  to 1 in Equation 8.
Table 5: Quantitative comparison and computational efficiency by varying number of tokens in box transformer. The best results are in bold font.
# tokens	Time (sec./epoch)↓	Room↑	Corner↑	Angle↑
Training	Inference	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
50	38.36	6.923	0.991	0.978	0.984	0.889	0.821	0.854	0.878	0.812	0.844
100	42.84	7.243	0.974	0.959	0.967	0.846	0.808	0.827	0.831	0.792	0.811
200	50.92	7.620	0.966	0.950	0.958	0.827	0.782	0.805	0.805	0.775	0.790
300	59.97	7.850	0.962	0.950	0.956	0.812	0.765	0.789	0.793	0.769	0.781
Table 6: Quantitative comparison on the mean-shift clustering method. Rows 4 to 7 present mean-shift clustering results using various bandwidth parameters. The best results are in bold font.
Method	Room↑	Corner↑	Angle↑
Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Ours	0.991	0.978	0.984	0.889	0.821	0.854	0.878	0.812	0.844
Bandwidth=1	0.979	0.986	0.982	0.881	0.821	0.850	0.870	0.811	0.840
Bandwidth=1.2	0.982	0.983	0.983	0.883	0.821	0.851	0.872	0.811	0.841
Bandwidth=1.25	0.986	0.982	0.984	0.887	0.823	0.854	0.876	0.813	0.844
Bandwidth=1.3	0.986	0.979	0.982	0.885	0.819	0.851	0.874	0.810	0.841
As shown in Table 3, we obtain the best performance with the Box IoU loss and the corner mask. The Box IoU loss significantly boosts our performance, which agrees with previous work [20]. Our slicing box representation is compatible with this Box IoU loss, unlike the corner representation. We attribute this Box IoU loss as one of the key factor for surpassing other methods in score.
The necessity of each step in post processing. In this experiment, we examine two aspects of our floorplan reconstruction method. First, we investigate how the accuracy of room center estimation affects the performance. Second, we evaluate the impact of gap filling and cell complex rounding in our post-processing step in Section 3.4. The upper block of Table 4 shows that using the ground truth room center for clustering the slicing boxes slightly improves the performance. The lower block of Table 4 shows that applying gap filling and cell complex rounding achieves the best performance. We observe that cell complex rounding significantly enhances the performance, indicating that cell complex regularization is a crucial component in our post processing.
The impact of the number of tokens. In this experiment, we examine how the number of tokens in the box transformer influences the computational efficiency and the final accuracy, as shown in Table 5. We observe that both the training time and inference time increase significantly as we increase the number of tokens. On the other hand, the accuracy also drops as we use more tokens. The main reason is that more tokens require more training time and introduce more dummy tokens, which makes it harder to reach the same accuracy within the same training time. These results confirm the importance of using a reasonable number of tokens to represent the floorplan, as discussed in Section 1.
The effectiveness of center transformer. In this experiment, we evaluate the necessity of our center transformer. We compared our results with another method that directly employs mean-shift clustering of the room centers estimated by the box transformer. As shown in Table 6, the conventional clustering approach experiences numerical fluctuations because of the varying bandwidth parameter. Overall, our method surpasses the performance of the mean-shift clustering approach.
We propose SLIBO-Net, a transformer-based method for floorplan reconstruction from 3D point clouds. Our method uses a novel floorplan representation that slices the floorplan into boxes, which reduces the tokens and improves the room shape supervision. We further use cell complex derived from the 3D point cloud as regularization and post-process to enhance the local details. We advocate a
3D point cloud
Density map
HEAT [9]
Roomformer [24]
Figure 6: Qualitative evaluations on Structured3D. Visual comparisons with competing methods.
Results
Figure 7: Failure cases. Left: our method struggles in reconstructing non-Manhattan layouts. Right: ground truth. One possible approach would be to extend the cell complex and post-processing to include non-Manhattan lines.
scale-independent metric that measures the local details better. Our method produces more meaningful and realistic floorplans and achieves state-of-the-art performance on various evaluation metrics.
Limitations. Our method is limited by its inability to handle non-Manhattan layouts (see Figure 7). Moreover, our network relies on a two-branch framework that constrains the reconstruction performance by the room center transformer.
Future work. Our method has 7 hyper-parameters in our loss functions that have not been thoroughly tuned. We intend to optimize these parameters to improve the performance. Additionally, we would like to integrate the cell complex into the box representation to avoid the need for the rounding step. However, this would require a major change in the data format for our current method, so it needs further investigation. Finally, we aim to make the post-processing more capable of handling non-Manhattan layouts.
Broader impact. This work significantly improves the efficiency and accuracy of floorplan reconstruction, which can benefit various applications in scene understanding, robotics, VR/AR, and construction industry. However, it also poses some risks, as the reconstruction is biased through training data and other factors.
The project was funded in part by the National Science and Technology Council of Taiwan (110-2221-E-007-061-MY3, 110-2221-E-007-060-MY3, 112-2425-H-007-002-). We gratefully acknowledge Taiwan Web Service Corporation (TWSC) for their generous contribution of cloud GPU computing resources to support our research.
[19] Sebastian Ochmann, Richard Vock, Raoul Wessel, and Reinhard Klein. Automatic reconstruction of parametric building models from indoor point clouds. Computers & Graphics, 54:94-103, 2016. ISSN 0097-8493. doi: https://doi.org/10.1016/j.cag.2015.07.008. Special Issue on CAD/Graphics 2015.
[20] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658-666, 2019.
[21] Victor Sanchez and Avideh Zakhor. Planar 3d modeling of building interiors from point cloud data. In 2012 19th IEEE International Conference on Image Processing, pages 1777-1780, 2012. doi: 10.1109/ICIP.2012.6467225.
[22] Sinisa Stekovic, Mahdi Rad, Friedrich Fraundorfer, and Vincent Lepetit. MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans. In International Conference on Computer Vision (ICCV 2021), Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV, pages 16034-16043, Online, Canada, October 2021. URL https://hal.science/hal-03482218.
[23] Xuehan Xiong, Antonio Adan, Burcu Akinci, and Daniel Huber. Automatic creation of semantically rich 3d building models from laser scanner data. Automation in Construction, 31:325-337, 2013. ISSN 0926-5805. doi: https://doi.org/10.1016/j.autcon.2012.10.006. URL https://www.sciencedirect.com/science/article/pii/S0926580512001732.
[24] Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[25] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photorealistic dataset for structured 3d modeling. In Proceedings of The European Conference on Computer Vision (ECCV), 2020.