Title: SLIBO-Net: Floorplan Reconstruction via Slicing Box Representation with Local Geometry Regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: deep learning

layout reconstruction
Abstract: This paper focuses on improving the reconstruction of 2D floorplans from unstructured 3D point clouds. We identify opportunities for enhancement over the existing methods in three main areas: semantic quality, efficient representation, and local geometric details. To address these, we presents SLIBO-Net, an innovative approach to reconstructing 2D floorplans from unstructured 3D point clouds. We propose a novel transformer-based architecture that employs an efficient floorplan representation, providing improved room shape supervision and allowing for manageable token numbers. By incorporating geometric priors as a regularization mechanism and post-processing step, we enhance the capture of local geometric details. We also propose a scale-independent evaluation metric, correcting the discrepancy in error treatment between varying floorplan sizes. Our approach notably achieves a new state-of-the-art on the Structure3D dataset. The resultant floorplans exhibit enhanced semantic plausibility, substantially improving the overall quality and realism of the reconstructions. Our code and dataset are available online.

SLIBO-Net: Floorplan Reconstruction via Slicing Box Representation with Local Geometry Regularization

Abstract

This paper focuses on improving the reconstruction of 2D floorplans from unstructured 3D point clouds. We identify opportunities for enhancement over the existing methods in three main areas: semantic quality, efficient representation, and local geometric details. To address these, we presents SLIBO-Net, an innovative approach to reconstructing 2D floorplans from unstructured 3D point clouds. We propose a novel transformer-based architecture that employs an efficient floorplan representation, providing improved room shape supervision and allowing for manageable token numbers. By incorporating geometric priors as a regularization mechanism and post-processing step, we enhance the capture of local geometric details. We also propose a scale-independent evaluation metric, correcting the discrepancy in error treatment between varying floorplan sizes. Our approach notably achieves a new state-of-the-art on the Structured3D dataset. The resultant floorplans exhibit enhanced semantic plausibility, substantially improving the overall quality and realism of the reconstructions. Our code and dataset are available online 1 .

1 Introduction

Reconstructing 2D floorplans from unstructured 3D point clouds has received significant attention in recent years as it facilitates various applications in scene understanding, robotics, VR/AR, etc. Prior works [24, 8, 9, 22, 14] typically employ a series of operations to address the problem at hand. The raw 3D point cloud is initially projected along the gravity axis (or z-axis) to generate a 2D density map. This density map is then fed into a deep neural network to either extract room masks [8, 22] or regress room corners [9, 14], followed by an optimization process to extract room polygons. In contrast to the two-stage framework, Yue et al. [24] introduced the RoomFormer model, an end-to-end trainable, transformer-type architecture that directly predicts room polygons using two-level queries. Although deep learning approaches have exhibited impressive performance in floorplan reconstruction, we have identified the following opportunities for improvement.

Semantic quality. The plausibility or semantic quality of reconstructed floorplans heavily relies on the accuracy of estimated room corners and masks. Insufficient supervision of the room shape or fragile room representations can result in artificial room shapes, as demonstrated in Figure 1(a), that are not plausible.

Efficient representation. One main challenge is designing a representation that requires only a reasonable number of tokens in the transformer architecture and can handle a variable number of rooms, each with a variable number of corners. For example, the naive approach of predicting the layout room by room, corner by corner does not require too many tokens but is too difficult to

(a)Artificial Corners

(b)Density Map

(c)Ground Truth

train. To address this challenge, we employ a novel floorplan representation that slices the floorplan into a set of boxes by scanning it in a top-down manner (Figure 3). This representation offers two key advantages. Firstly, it yields a manageable number of slicing boxes, making it suitable for the transformer architecture as each slicing box can be represented by a query token. Secondly, these slicing boxes provide improved supervision of the room shape by utilizing Intersection over Union (IoU) loss [15]. Furthermore, this representation naturally leads to higher semantic quality, enhancing the overall accuracy and realism of the reconstructed floorplans.

Local details. We observed that deep learning approaches excel in predicting the overall room structure but struggle to capture local geometric details such as wall thickness due to image resolution limitations (Figure 1(b)). We incorporate prior knowledge about geometric details derived from the input 3D point cloud to address this limitation. This knowledge is encoded as cell complexes, which serve multiple purposes. It acts as a regularization mechanism during the training phase and is also utilized as a post-processing step to enhance the accuracy of the reconstructed floorplans.

Evaluation. The metrics employed in previous studies are scale-dependent, resulting in differential treatment of errors between large and small floorplans. To illustrate this, consider starting with a floorplan of a specific size. Adding additional rooms to the floorplan alters its error measurement, leading to a more lenient evaluation of incorrect corner predictions. This behavior is counter-intuitive and primarily masks a drawback of neural networks in scaling to larger inputs. Hence, we advocate for adopting a scale-independent metric that mitigates this issue.

In summary, we make the following contributions: 1) We propose a novel transformer architecture for floorplan reconstruction that only requires a manageable number of tokens. 2) We propose integrating geometric priors as regularization during training and as a post-process to improve local geometric details. 3) We advocate the use of a new metric that is scale independent and that better measures if local details are correctly reconstructed. 4) We produce more semantically meaningful floorplans that do not suffer from implausible geometric details.

2 Related Work

Optimization-based methods. Early work mostly assumed the task to be based on finding 3D planes from the point clouds using traditional techniques such as Hough transform [16, 2], plane sweeping/plane fitting [4, 23, 17, 12, 18, 21], and leveraging histograms of point cloud positions [7]. Higher-level structured objects such as rooms could then be assembled from the recovered planes. For larger buildings with many rooms, the optimization became increasingly difficult to solve. Researchers then developed graph-based data structures [10, 5, 11, 8, 22] to help organizing the reconstructed sub-models (e.g., corners, walls, objects, and rooms) and guide the optimization to follow certain semantic, topological constraints. In particular, Ikehata et al. [11] used hierarchical tree graphs of reconstructed sub-models, such as objects, walls, and rooms, to guide the 3D reconstructions. Bassier and Vergauwen [3] extracted highly-detailed BIM-compatible objects from point clouds by plane fitting and RANSAC. In the same spirit of extracting more semantic and realistic 3D models, Ochmann et al. [19] proposed methods to extract parametric 3D building models from input point clouds. A key advantage was that their reconstructed floorplans are volumetric - i.e., having wall thickness. Our results shared the same idea. Finally, our cell complex-based constraints was inspired

(a)InputPoint Cloud

(b)Segmented Clusters

(c)Wall Clusters

(d)Wall Segments

(e)Cell Complex

by [13], in which they proposed an optimization-based approach to register multiple 3D scans of multi-story, multi-room buildings.

Deep learning methods. More recent methods began to leverage neural networks to solve the floorplan reconstruction problems. Earlier methods were hybrid methods that utilized both optimization and neural networks. Floor-SP [8] assembled room segments detected by Mask R-CNN [1] to form full floorplans by sequentially solving shortest path problems. MonteFloor [22] later solved the room assembly problem by Monte-Carlo Tree-Search (MCTS). FloorNet [14] took a point cloud and RGB images as inputs, utilized a triple-branch neural network to generate pixel-wise predictions on floorplan geometry and semantics, and then used an integer programming to generate the final floorplans in vector-graphics format.

Later, fully neural network-based methods (which are end-to-end trainable) were being developed and delivered state-of-the-art performances. HEAT [9] was a method to reconstruct planar graphs with underlying geometric structures from 2D raster image inputs. It had two main applications building segmentation in aerial images and floorplan reconstruction from 2D density maps distilled from point clouds. It utilized an end-to-end trainable edge classification neural network for the task. Recently, RoomFormer [24] proposed a single-stage, end-to-end trainable neural network to directly predict floorplans from a given point cloud. Their main innovation was to encode a floorplan as variable-size set of polygons, which are variable-length sequences of ordered vertices. This design enabled efficient handling by transformers. Our method is an improvement upon it.

3 Methodology

3.1 Pre-processing

In the data preprocessing stage, we adopt a similar process as in previous works [8, 9, 14, 22] to project the raw 3D point cloud along the gravity axis (or z-axis), resulting in a 2D density map in the aerial view (or x-y plane). It is important to note that while this density map provides adequate visual information for describing the overall structure (e.g., rooms and walls) of the floorplan, it faces challenges in capturing geometric details like wall thickness due to the limitation of image resolutions.

Encoding geometric priors. We extract wall segments from 3D point cloud data as geometric priors for floorplan reconstruction. We project these wall segments as parametric lines onto the x-y plane, forming a "cell complex" representation. Figure 2 shows the steps of computing a cell complex from a 3D point cloud. First, we segment the point cloud by normals and proximity (Figure 2(b)). Next, we fit planes for each cluster and select those orthogonal to gravity as wall clusters (Figure 2(c)). After that, we project points onto planes and group collinear line segments as wall segments (Figure 2(d)). Finally, we convert wall segments into parametric lines and remove non-axis aligned lines to get the cell complex (Figure 2(e)). After obtaining the cell complex representation, we follow the strategy described in Section 3.4, where we adjust our ground truth by rounding it to the cell complex representation for training.

1

1

1

1

3.2 Floorplan representation

One key challenge in our work was designing a suitable data structure for floorplans. The representation should work with the transformer architecture and only require a reasonable number of tokens. One possible representation would be to represent each grid cell of the cell complex as a query token, but this would require a large number of tokens, e.g., 2655 for our dataset. Another possible representation is to encode a floorplan as a set of room polygons. However, we observed that predicting rooms as sequences of corners may lead to artificial room shapes as shown in Figure 1(a). We propose to employ a novel floorplan representation that slices the floorplan into a set of boxes by scanning the floorplan along the vertical axis (or y-axis). We denote this box set as B = { b i } i M =1 , where b i = { b j i } N m j =1 and N m is the number of boxes b j i in the m th room. Examples of the slicing boxes representation are depicted in Figure 3. The major advantage of this representation lies in its ability to maintain a manageable number of slicing boxes and query tokens ( ≤ 50 in our dataset). However, each floorplan may consist of a varying number of rooms, with each room containing a different number of slicing boxes. This poses a challenge in clustering the slicing boxes to their respective rooms. The unordered nature of the rooms, combined with the variable number of rooms and slicing boxes, makes it difficult to assign room IDs for the classification task. To tackle this challenge, we transform the original box classification problem into a 2D coordinate regression problem. Specifically, we employ one network to predict the room center for each slicing box, and another network to predict all the room centers from the density map. We compute the room center coordinate as the bounding box center for each room in the floorplan, represented as C = { c i } i M =1 , where M denotes the number of rooms and c i ∈ [ -1 , 1] × [ -1 , 1] . This allows us to cluster the slicing boxes based on the predicted room centers.

3.3 Network architecture

Figure 4 illustrates the SLIBO-Net's architecture, comprising two primary components: the center transformer T center and the box transformer T box . The center transformer T center is responsible for predicting the coordinates of all room centers based on the density map, while the box transformer T box predicts both the slicing boxes and their corresponding room center coordinates. Subsequently, a post-processing step is implemented to merge the estimated slicing boxes into a complete floorplan with distinct room layouts, utilizing the predicted room center points. Each component are elaborated in the next subsections.

Center transformer. First, we feed a 256 × 256 density map into a ResNet-50 feature extractor and generate one 16 × 16 feature map. Following the encoder-decoder transformer architecture, we feed the extracted feature maps into the transformer encoder block. The output tokens of the transformer encoder block are then used for cross attention in the transformer decoder block. In order to query all the room centers from the density map, we use ¯ M learnable embeddings as the input tokens of the transformer decoder. The output tokens of the transformer decoder are further processed using two auxiliary MLP heads. Specifically, we use: (i) room center MLP head F center room that outputs the coordinates of rooms' bounding box center C room = { c room i } ¯ i M =1 ; and (ii) room prob MLP head F prob room that outputs the probabilities P room = { p room i } ¯ i M =1 representing the probability of valid room center.

Box transformer. As for the box transformer T box , we follow the same architecture of center transformer T center except the number of input tokens of the transformer decoder and the output MLP heads. We query all the slicing boxes from the density map via ¯ N learnable embeddings. All the output tokens of the box transformer T box are further processed using two auxiliary MLP heads.

(a) CenterTransformer

Positional Encoding

256

256

MLP

Room Center

Transformer

Transformer

Encoder

25

Decoder

25-

Feature Backbone

MLP

Room Prob

Post-process

(b)Box Transformer

Positional Encoding

256

256

Room Center

MLP

Density Map

Transformer

Encoder

50

Transformer

Decoder

→50

Feature Backbone

MLP

We use (i) box coord MLP head F coord box that outputs six scalars at once. The first two scalars represent the corresponding room's center coordinates C box = { c box i } ¯ N i =1 and the remain four scalars represent the upper-left and lower-right corners of slicing boxes B = { b i } ¯ N i =1 ; and (ii) box prob MLP head F prob box that outputs the probabilities P box = { p box i } ¯ N i =1 representing the probability of valid slicing boxes.

Bipartite matching. Note that our neural network would always predict 25 room centers ( C room ) and 50 slicing boxes, which generally would be more than the ground-truth room centers C room and slicing boxes B , respectively. Thus, following DETR [6], we conduct a set-based bipartite matching between predictions and targets in the center transformer T center and the box transformer T box . Since we use the same strategy in both of the transformers, we explain this step using the center transformer T center as an example. Our goal is to optimize a bipartite matching objective on a permutation function σ ( · ) : Z + → Z + which maps ¯ M prediction indices to M targets:

where d 1 ( · , · ) represents L 1 distance between predicted and target room center coordinates, d 2 () represents the focal loss between predicted and target room center probabilities, and ✶ {·} is an indicator function. Finally, we use the Hungarian Matching algorithm to calculate the minimal cost σ ∗ that maps ¯ M predicted room centers to M target room centers and ¯ N predicted slicing boxes to N target slicing boxes.

3.4 Post-processing

Here, we describe the post-processing steps in Figure 5 to reconstruct the floorplan using the slicing boxes B and room centers C room obtained in the previous step by the SLIBO-Net.

The first step in the post-processing phase is to cluster the slicing boxes B into rooms. Given the predicted room centers C room from the center transformer T center and the predicted slicing boxes C box corresponded rooms' centers C box as the input, we can cluster the slicing boxes B into different rooms by assigning the C box to the nearest room centers C room from the center transformer T center .

After clustering the boxes into different rooms, we fill the gaps between the bottom and top of any two adjacent boxes in the same room with a box whose height is the vertical distance between them and whose width is the intersection length of them.

Finally, we union together the boxes for each room to form a room area A r . We then overlay A r on the cell complex, which allows us to calculate the intersection ratio for each room per cell. By assigning each cell to the room with the largest intersection ratio, we can obtain a room representation by tracing the boundary formed by the cell groups. For non-connected components within a single room, we choose the component with the largest area. A cell remains empty if no room has more than 50% overlap.

Post-process

Room Centers

Slicing Boxes

Clustering

Gap Filling

Rounding

3D Point Cloud

Cell Complex

(a) Clustered Boxes

(b) Gap Filled Boxes

(c)Rounded Cells

Result

3.5 Loss functions

We first elaborate on the loss functions we use for the center transformer T center based on the optimal permutation σ ∗ from the bipartite matching. For the center transformer T center , we use the index set { i ; σ ∗ ( i ) ≤ M } representing the best matching prediction indices. For the coordinate regression, we use the smooth L 1 loss L smooth ( · , · ) which is defined as follow:

where X is the prediction and Y is the label.

Room center loss calculates the smooth L 1 loss with ground truth room center c room :

Room prob loss calculates the focal loss for room probability of being valid:

where λ 1 and λ 2 are the hyperparameters for weighting the loss functions.

Next, we describe the loss functions we compute for the box transformer T box based on the optimal permutation σ ∗ from the bipartite matching. For the box transformer T box , we use the index set { i ; σ ∗ ( i ) ≤ N } representing the best matching prediction indices.

Box room center loss calculates the smooth L 1 loss for each slicing box room center with ground truth room center c box :

Box prob loss calculates the focal loss for each slicing box probability of being valid:

✶ where b corner i represented all 4 corners of a slicing box. We assume a conversion from the original upper-left and lower-right representation of the slicing boxes b i . We set w σ ( i ) to 1 if and only if the corner b corner σ ( i ) equals any corner in the floorplan; otherwise, it is 0.

Box IoU loss calculates the GIoU loss [6] L giou for all slicing boxs with ground truth slicing box b i :

Thus, the final loss L btotal of the box transformer T box is defined as below:

where λ { 3 ,..., 6 } are the hyperparameters for weighting the loss functions.

4 Experiments

4.1 Experimental settings

Dataset. We conducted experiments on a large-scale indoor synthetic dataset called Structured3D [25]. Following the approach in [9, 24], we divided the complete dataset into 2991 training samples, 250 validation samples, and 241 test samples, taking into account the ignore list specified in the HEAT [9] codebase. By employing the pre-processing method described in Section 3.1, we obtained a density map of size 256 × 256 for each sample, with pixel values normalized to the range of [0 , 1] .

Baselines. We compare our method with the following competing methods: Floor-SP [8] - scores taken from [24], MonteFloor [22] - scores taken from [24], HEAT [9] - scores based on their official pre-trained weights, RoomFormer [24] - scores based on their official pre-trained weights.

Evaluation metrics. We follow the Room/Corner/Angle evaluation method from MonteFloor and HEAT to measure the quality of floorplan reconstruction. However, the existing metrics suffer from a scale-variant nature as they calculate results based on a normalized 256 × 256 pixel space, without considering the actual size of each floorplan. Consequently, this leads to a more lenient evaluation for floorplans with more rooms or larger sizes. Therefore, we propose the use of real-scale measurements, such as meters, as the threshold for Corner/Angle evaluation. For instance, Corner @0 . 1 denotes the measurement with a threshold of 0.1 meter.

We found that the current corner evaluation solely focuses on the distance to the ground truth corner, disregarding the shape connectivity between corners. This approach can lead to scenarios where there are multiple "correct" corners but inadequate connectivity (Figure 1(a)), resulting in a low Angle score. Hence, we additionally introduce the Delta metric that measures the global structure of the

floorplan by computing the discrepancy between Corner and Angle scores. Thus, a lower Delta score indicates a better shape for the resulting floorplans.

Implementation details. We implemented our model in PyTorch and trained our center transformer T center on 2 NVIDIA V100s for 2 days. The box transformer T box was trained on 8 NVIDIA V100s for 1.9 days. We use the Adam optimizer with b 1 =0.9 and b 2 =0.999. The learning rates of two transformers and the ResNet-50 are 2.5e-4, and 1e-5, and we train the center transformer for 5000 epochs, box transformer for 16000 epochs with batch size 128 and 123 for each GPU. We empirically set λ 1 = 1 , λ 2 = 1 in Equation 5, λ 3 = 1 , λ 4 = 1 , λ 5 = 5 ., λ 6 = 1 in Equation 10, γ = 2 in Equation 4, and ¯ M = 25 , ¯ N = 50 in Section 3.3.

4.2 Floorplan reconstruction performance

Quantitative comparison. As shown in Table 1, our SLIBO-Net outperforms all the baselines in the overall F1 score. Compared to RoomFormer, which has the best F1 score among all baselines, our model improves the F1 score of Room by 1 . 3% and Angle by 3 . 6% , while maintaining a comparable precision of Corner. Since RoomFormer fails to generate good room shapes (i.e., low Angle score), we round its results by our cell complex, denoted as RoomFormer ∗ . Compared to RoomFormer ∗ , our model achieves better F1 scores in all metrics. It is worth highlighting that our model exhibits a substantial performance boost of 5 . 2% in the Delta F1 score when compared to RoomFormer. This advancement indicates our model's ability to generate more plausible and realistic floorplans.

Real-scale evaluation. As shown in Table 2, our model attains the highest scores across most of the metrics, including Room, Corner, Angle, and Delta, with a particular emphasis on the Delta scores. This improvement can be attributed to the incorporation of cell complex regularization, which effectively enhances our model's capabilities under more stringent evaluation conditions.

Qualitative comparison. The qualitative results are shown in Figure 6. Compared to HEAT, our model can reconstruct the correct floorplan structure and preserve the wall thickness , while HEAT fails to reconstruct the walls. Compared to RoomFormer, our model preserves all structural details without any self-intersection, whereas RoomFormer generates some self-intersections that are semantically implausible. Please refer to the supplementary for additional comparisons.

4.3 Ablation Studies

The effectiveness of loss functions. In this experiment, we evaluate the necessity of two loss functions. First, we remove the Box IoU loss (Equation 9) in our total loss function (Equation 10) of the box transformer. Second, we eliminate the corner mask by setting w σ ( i ) to 1 in Equation 8.

As shown in Table 3, we obtain the best performance with the Box IoU loss and the corner mask. The Box IoU loss significantly boosts our performance, which agrees with previous work [20]. Our slicing box representation is compatible with this Box IoU loss, unlike the corner representation. We attribute this Box IoU loss as one of the key factor for surpassing other methods in score.

The necessity of each step in post processing. In this experiment, we examine two aspects of our floorplan reconstruction method. First, we investigate how the accuracy of room center estimation affects the performance. Second, we evaluate the impact of gap filling and cell complex rounding in our post-processing step in Section 3.4. The upper block of Table 4 shows that using the ground truth room center for clustering the slicing boxes slightly improves the performance. The lower block of Table 4 shows that applying gap filling and cell complex rounding achieves the best performance. We observe that cell complex rounding significantly enhances the performance, indicating that cell complex regularization is a crucial component in our post processing.

The impact of the number of tokens. In this experiment, we examine how the number of tokens in the box transformer influences the computational efficiency and the final accuracy, as shown in Table 5. We observe that both the training time and inference time increase significantly as we increase the number of tokens. On the other hand, the accuracy also drops as we use more tokens. The main reason is that more tokens require more training time and introduce more dummy tokens, which makes it harder to reach the same accuracy within the same training time. These results confirm the importance of using a reasonable number of tokens to represent the floorplan, as discussed in Section 1.

The effectiveness of center transformer. In this experiment, we evaluate the necessity of our center transformer. We compared our results with another method that directly employs mean-shift clustering of the room centers estimated by the box transformer. As shown in Table 6, the conventional clustering approach experiences numerical fluctuations because of the varying bandwidth parameter. Overall, our method surpasses the performance of the mean-shift clustering approach.

5 Conclusions

We propose SLIBO-Net, a transformer-based method for floorplan reconstruction from 3D point clouds. Our method uses a novel floorplan representation that slices the floorplan into boxes, which reduces the tokens and improves the room shape supervision. We further use cell complex derived from the 3D point cloud as regularization and post-process to enhance the local details. We advocate a

3D point cloud

Density map

HEAT [9]

Roomformer [24]

Ours

Ground truth

I

I

1

1

1

1

Results

GroundTruth

scale-independent metric that measures the local details better. Our method produces more meaningful and realistic floorplans and achieves state-of-the-art performance on various evaluation metrics.

Limitations. Our method is limited by its inability to handle non-Manhattan layouts (see Figure 7). Moreover, our network relies on a two-branch framework that constrains the reconstruction performance by the room center transformer.

Future work. Our method has 7 hyper-parameters in our loss functions that have not been thoroughly tuned. We intend to optimize these parameters to improve the performance. Additionally, we would like to integrate the cell complex into the box representation to avoid the need for the rounding step. However, this would require a major change in the data format for our current method, so it needs further investigation. Finally, we aim to make the post-processing more capable of handling non-Manhattan layouts.

Broader impact. This work significantly improves the efficiency and accuracy of floorplan reconstruction, which can benefit various applications in scene understanding, robotics, VR/AR, and construction industry. However, it also poses some risks, as the reconstruction is biased through training data and other factors.

Acknowledgments and Disclosure of Funding

The project was funded in part by the National Science and Technology Council of Taiwan (110-2221E-007-061-MY3, 110-2221-E-007-060-MY3, 112-2425-H-007-002-). We gratefully acknowledge Taiwan Web Service Corporation (TWSC) for their generous contribution of cloud GPU computing resources to support our research.

References

Waleed Abdulla. Mask r-cnn for object detection and instance segmentation on keras and tensorflow. https://github.com/matterport/Mask_RCNN , 2017.

Antonio Adan and Daniel Huber. 3d reconstruction of interior wall surfaces under occlusion and clutter. In 2011 International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission , pages 275-281, 2011. doi: 10.1109/3DIMPVT.2011.42.

Maarten Bassier and Maarten Vergauwen. Unsupervised reconstruction of building information modeling wall objects from point cloud data. Automation in Construction , 120:103338, 2020. ISSN 0926-5805. doi: https://doi.org/10.1016/j.autcon.2020.103338.

Angela Budroni and Jan Boehm. Automated 3d reconstruction of interiors from point clouds. International Journal of Architectural Computing , 8:55-73, 01 2010. doi: 10.1260/1478-0771.8.1.55.

Ricardo Cabral and Yasutaka Furukawa. Piecewise planar and compact floorplan reconstruction from images. In 2014 IEEE Conference on Computer Vision and Pattern Recognition , pages 628-635, 2014. doi: 10.1109/CVPR.2014.546.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , pages 213-229. Springer, 2020.

Kunal Chelani, Chitturi Sidhartha, and Venu Madhav Govindu. Towards automated floorplan generation. In Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing , ICVGIP 2018, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450366151. doi: 10.1145/3293353.3293430. URL https://doi.org/10.1145/3293353.3293430 .

Jiacheng Chen, Chen Liu, Jiaye Wu, and Yasutaka Furukawa. Floor-sp: Inverse cad for floorplans by sequential room-wise shortest path. In The IEEE International Conference on Computer Vision (ICCV) , 2019.

Jiacheng Chen, Yiming Qian, and Yasutaka Furukawa. Heat: Holistic edge attention transformer for structured reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3866-3875, June 2022.

Yasutaka Furukawa, Brian Curless, Steven M. Seitz, and Richard Szeliski. Reconstructing building interiors from images. In 2009 IEEE 12th International Conference on Computer Vision , pages 80-87, 2009. doi: 10.1109/ICCV.2009.5459145.

S. Ikehata, H. Yang, and Y. Furukawa. Structured indoor modeling. In 2015 IEEE International Conference on Computer Vision (ICCV) , pages 1323-1331, Los Alamitos, CA, USA, dec 2015. IEEE Computer Society. doi: 10.1109/ICCV.2015.156.

Minglei Li, Peter Wonka, and Liangliang Nan. Manhattan-world urban reconstruction from point clouds. In ECCV , 2016.

Gahyeon Lim and Nakju Doh. Automatic reconstruction of multi-level indoor spaces from point cloud and trajectory. Sensors (Basel) , 2021.

Chen Liu, Jiaye Wu, and Yasutaka Furukawa. Floornet: A unified framework for floorplan reconstruction from 3d scans. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 , pages 203-219, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01231-1.

Jiachen Liu, Yuan Xue, Jose Duarte, Krishnendra Shekhawat, Zihan Zhou, and Xiaolei Huang. End-to-end graph-constrained vectorized floorplan generation with panoptic refinement. In ECCV , 2022.

Josep Lladós, Jaime López-Krahe, and Enric Martí. A system to understand hand-drawn floor plans using subgraph isomorphism and hough transform. Mach. Vision Appl. , 10(3):150-158, aug 1997. ISSN 0932-8092. doi: 10.1007/s001380050068. URL https://doi.org/10.1007/s001380050068 .

Aron Monszpart, Nicolas Mellado, Gabriel J. Brostow, and Niloy J. Mitra. Rapter: Rebuilding man-made scenes with regular arrangements of planes. ACM Trans. Graph. , 34(4), jul 2015. ISSN 0730-0301. doi: 10.1145/2766995. URL https://doi.org/10.1145/2766995 .

Liangliang Nan and Peter Wonka. Polyfit: Polygonal surface reconstruction from point clouds. In Proceedings of the IEEE International Conference on Computer Vision , pages 2353-2361, 2017.

Sebastian Ochmann, Richard Vock, Raoul Wessel, and Reinhard Klein. Automatic reconstruction of parametric building models from indoor point clouds. Computers & Graphics , 54:94-103, 2016. ISSN 0097-8493. doi: https://doi.org/10.1016/j.cag.2015.07.008. Special Issue on CAD/Graphics 2015.

Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 658-666, 2019.

Victor Sanchez and Avideh Zakhor. Planar 3d modeling of building interiors from point cloud data. In 2012 19th IEEE International Conference on Image Processing , pages 1777-1780, 2012. doi: 10.1109/ ICIP.2012.6467225.

Sinisa Stekovic, Mahdi Rad, Friedrich Fraundorfer, and Vincent Lepetit. MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans. In International Conference on Computer Vision (ICCV 2021) , Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV, pages 16034-16043, Online, Canada, October 2021. URL https://hal.science/hal-03482218 .

Xuehan Xiong, Antonio Adan, Burcu Akinci, and Daniel Huber. Automatic creation of semantically rich 3d building models from laser scanner data. Automation in Construction , 31:325-337, 2013. ISSN 0926-5805. doi: https://doi.org/10.1016/j.autcon.2012.10.006. URL https://www.sciencedirect. com/science/article/pii/S0926580512001732 .

Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photorealistic dataset for structured 3d modeling. In Proceedings of The European Conference on Computer Vision (ECCV) , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a learning-based framework on floorplan reconstruction from point cloud. To tackle with the complex polygons emerge widely in floorplan, the polygon is decomposed to non-overlapping, vertical-slicing boxes as regression target of the networks. A bbox transformer and a room transformer are designed for bbox regression and room regression, respectively. A few post-processing steps are needed to intersect the two branches and integrate the bboxes into rooms. Extensive experiments are conducted on Structured3D dataset to verify its performance.\n\nSTRENGTHS:\n(1) The idea of dividing complex rooms into bboxes is novel and can effectively reduce the difficulty of learning the polygon corners from scratch.\n(2) The overall pipeline is clear and straightforward, and well presented.\n(3) The key challenges of the problem are well displayed in the introduction section, making it easier for readers to follow and future researchers to investigate on.\n\nWEAKNESSES:\n(1) Bbox assignment: the paper assigns the slicing bboxes to rooms by nearest distance search. However, this may not always be the case when some irregular polygons (the center living room shown in Figure 7 left). Some bboxes may be incorrectly matched with other rooms when there exists some neighboring rooms whose center is more adjacent. I wonder if there are more well-suited ways for bbox integration.\n(2) Bbox corner regression: to represent a bbox, the paper learns to regress both the center and two corners (4 scalars). But there exists redundancy between their coordinates. Regressing height and width is enough if the bbox center is given. Please demonstrate the reason to regress both corners other than learning (h, w).\n(3) A very recent work on floorplan generation (https://arxiv.org/abs/2207.13268) also employs the bbox (room) iou loss to enforce the correct structural regularity. Although it is a different problem, it is good to cite and discuss this paper.\n\nQUESTIONS:\nOverall I think the paper is novel and technically sound, with a clear pipeline for future research to follow. So currently I lean toward accepting this paper. However, I still expect authors’ feedback on my concerns listed in weaknesses. For question 1, I think a possible choice is to learn an assignment matrix during training between bboxes and rooms to encourage the matching process to be learned in a data-driven manner instead of only relying on center distances.\n\nLIMITATIONS:\nAuthors have discussed their limitations on non-Manhattan floorplan reconstruction. It is indeed a challenging problem and may need some specified design to be proposed in the future.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SLIBO-Net, a novel approach for reconstructing 2D floorplans from unstructured 3D point clouds. By employing a transformer-based architecture and an efficient floorplan representation, the method improves room shape supervision and manages token numbers effectively. The proposed approach achieves state-of-the-art results on the Structure3D dataset, generating floorplans with enhanced semantic plausibility and realistic reconstructions.\n\nSTRENGTHS:\nTechnical Contribution: The authors introduce a novel approach for reconstructing 2D floorplans, tackling the challenges associated with semantic quality, efficient representation, and local geometric details. The proposed solutions are well-designed, showcasing a clear advancement over existing methods.\n\nPresentation: The paper is well-structured and clearly presents the motivations, methodology, and experimental results. The technical concepts and terminologies are adequately defined and explained.\n\nEvaluation: The authors conducted a thorough experimental evaluation, demonstrating the effectiveness of their proposed method. The comparisons with existing approaches and the state-of-the-art highlight the benefits of the proposed approach. The proposed scale-independent evaluation metric addresses a common limitation and ensures a fair assessment of performance across different floorplan sizes.\n\nContributions: The paper provides several notable contributions, including a novel transformer architecture for floorplan reconstruction, the integration of geometric priors for improved local geometric details, the adoption of a scale-independent evaluation metric, and the generation of semantically meaningful floorplans without self-intersections. These contributions offer valuable insights for future research.\n\nFurthermore, the supplementary section of the paper introduces a valuable contribution in the form of a new labeled dataset called Gibsonlayout, which augments the Gibson dataset with annotated 2D floor-plans. The potential release of this dataset to the research community would be highly beneficial for advancing the field and promoting further research on the problem at hand. The availability of such a dataset would facilitate comparative evaluations, enable benchmarking, and encourage the development of more robust and accurate solutions.\n\nWEAKNESSES:\nThe authors have effectively addressed a range of limitations in their paper, including the difficulties associated with handling non-Manhattan scenes and the seemingly arbitrary choice of the many hyperparameters. However, it is notable that the significant improvement in performance can be largely attributed to the Box IoU function, as acknowledged by the authors themselves. From a practical standpoint, this suggests that the additional technical contributions and associated complexity may have limited value for real-world applications.\n\nQUESTIONS:\nThe description, design, and usage of the cell complex as presented in the paper may not be immediately intuitive or clear. It requires multiple readings to fully grasp how it is utilized in the final stages of the proposed method. To improve the clarity of this aspect, it would be advantageous for the authors to provide more explicit explanations and visual aids. For instance, incorporating a diagram that illustrates how and when the cell complex is employed would greatly enhance understanding and ensure a smoother comprehension of its role in the overall pipeline.\n\nThe authors' motivation in the proposed approach is heavily influenced by the transformer-based representation and the desire for a reasonable number of tokens. However, the impact of the selected number of tokens on the final performance, as well as the size and speed of the network, remains unclear. Conducting an ablation study that explores the relationship between the number of tokens and these factors would greatly enhance the effectiveness of the work. By systematically analyzing the effects of varying token counts, the authors can provide valuable insights into the trade-offs between representation size, computational efficiency, and model performance. Including such an ablation study would further strengthen the paper and provide a more comprehensive understanding of the proposed approach.\n\nLIMITATIONS:\nThe authors have done good job discussing the most obvious limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on 2D floorplan reconstruction from 3D point clouds. The authors represent 2D floorplans as sliced ​​boxes and apply a transformer network to predict box characteristics, as well as post-processing steps to build the final floorplan. The authors conduct experiments on Structure3D and show better performance in both quantitative and qualitative terms compared to state-of-the-art methods.\n\nSTRENGTHS:\n- I like the idea of ​​introducing geometric priors (slicing boxes) into the design of neural network architectures. In my opinion, a combination of learning methods and pure geometric reasoning should be a better way to solve 3D related tasks. I'm happy to see better performance compared to other state-of-the-art method.\n\n- In addition to the evaluation on synthetic Structure3D, the authors also introduce a novel real-world dataset (Gibsonlayout) in the supplement to further demonstrate the effectiveness of the proposed method compared to other state-of-the-art methods.\n\n- The proposed method is technically sound. The authors also conduct ablation studies to verify the effectiveness of the design choices.\n\nWEAKNESSES:\n- I still don't understand why two parallel transformers are used instead of a single transformer for room center and slice box prediction. Perhaps it is better to show quantitatively that joint prediction is not a viable solution. Otherwise I find the proposed method a bit clumsy.\n\n- It also better to conduct evaluation of inference time compared to other methods. I'm wondering how long the pre- and post-processing will take and the cost of running two transformers. If it takes significantly longer than others, it may not be desirable given the amount of improvement compared to other methods on Structure3D.\n\n- Maybe an unnecessary requirement, it's also better to experiment on SceneCAD as other methods too. Although I know the authors propose a new dataset in the supplement, it should stand alone in the main text. Given the many heuristics introduced in pre- and post-processing, I want to make sure this is not tuned specifically for Structure3D and the proposed dataset.\n\n- Minor issue, page 7, line 211, missing citation after Structure3D.\n\nQUESTIONS:\nPlease see *Weaknesses\n\nLIMITATIONS:\nI found the proposed method a bit clumsy, with two parallel transformers instead of a single one. Inference time was not evaluated compared to other methods, so it is not known how the preprocessing and postprocessing steps affect the latency of this method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method for floorplan reconstruction from point clouds. Specifically, the method formulates the problem from predicting polygons for each room to predicting sliced boxes and their corresponding room centers by imposing the Manhattan grid assumption on the floorplans. The authors also use the prior information--cell complex, derived by extending the extracted wall segments from the input point clouds, to improve the results. The qualitative and quantitative results demonstrate that the proposed method outperforms the prior work by a significant margin.\n\nSTRENGTHS:\n1. The proposed method effectively improves the results using the Manhattan assumption and the prior knowledge, cell complex, obtained from the given point clouds. It is also a clever approach to use predicted room centers to cluster the boxes instead of predicting complex, non-rectangular polygons. \n\n2. The figures are clear and helpful in understanding the methods and details.\n\n3. The results outperform prior works by a significant margin, especially in the qualitative results' geometric details, such as wall gaps.\n\nWEAKNESSES:\n1. As the authors mentioned, the main limitation of this method is the Manhattan formulation. \n\n2. Extending the local planes across the whole space might be unnecessary to form the cell complex. The extended planes will intersect at many empty locations. For example, the second line from the bottom in Figure 2(e) creates extra cell units on the bottom right, but it is unrelated to the room on the bottom right. Aren't the extracted wall segments already useful for post-processing? Or have the author tried using local grids (e.g., a plane is extended until it intersects with another plane) instead of global ones for the cell complex?\n\nQUESTIONS:\n1. It is unclear how the cell complex is used for regularization during training. None of the branches have a regularization term, and cell complex seems to be used only in post-processing.\n\n2. The seems to be some redundancy in representing the box losses: why is computing loss on four corners required? Aren't top-left and bottom-right corners enough (i.e., xmin, ymin, xmax, ymax)?\n\n3. It would be interesting to know how important the center transformer is. For example, the room assignment can be done using clustering algorithms on the predicted room centers. \n\n4. How are the extracted wall segments for the cell complex aligned with the ground truth room shapes? Do the misalignments affect the accuracy?\n\n5. The visual information in 3D is potentially helpful. Several works have reconstructed room shapes from panorama views, such as [1]. How are these approaches compared to the proposed method? For example, can one reconstruct the plan room-by-room instead of the whole floorplan?\n\n[1] PSMNet: Position-aware Stereo Merging Network for Room Layout Estimation by Wang et al.\n\nLIMITATIONS:\nYes, the authors have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}