SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jheng-Wei Su $^{1}$ , Kuei-Yu Tung $^{1}$ , Chi-Han Peng $^{2}$ , Peter Wonka $^{3}$ , Hung-Kuo Chu $^{1,\dagger}$ $^{1}$ National Tsing Hua University  $^{2}$ National Yang Ming Chiao Tung University  
 $^{3}$ King Abdullah University of Science and Technology  
 $\dagger$ Correspondence: hkchu@cs.nthu.edu.tw
This paper focuses on improving the reconstruction of 2D floorplans from unstructured 3D point clouds. We identify opportunities for enhancement over the existing methods in three main areas: semantic quality, efficient representation, and local geometric details. To address these, we presents SLIBO-Net, an innovative approach to reconstructing 2D floorplans from unstructured 3D point clouds. We propose a novel transformer-based architecture that employs an efficient floorplan representation, providing improved room shape supervision and allowing for manageable token numbers. By incorporating geometric priors as a regularization mechanism and post-processing step, we enhance the capture of local geometric details. We also propose a scale-independent evaluation metric, correcting the discrepancy in error treatment between varying floorplan sizes. Our approach notably achieves a new state-of-the-art on the Structured3D dataset. The resultant floorplans exhibit enhanced semantic plausibility, substantially improving the overall quality and realism of the reconstructions. Our code and dataset are available online<sup>1</sup>.
Reconstructing 2D floorplans from unstructured 3D point clouds has received significant attention in recent years as it facilitates various applications in scene understanding, robotics, VR/AR, etc. Prior works [24, 8, 9, 22, 14] typically employ a series of operations to address the problem at hand. The raw 3D point cloud is initially projected along the gravity axis (or z-axis) to generate a 2D density map. This density map is then fed into a deep neural network to either extract room masks [8, 22] or regress room corners [9, 14], followed by an optimization process to extract room polygons. In contrast to the two-stage framework, Yue et al. [24] introduced the RoomFormer model, an end-to-end trainable, transformer-type architecture that directly predicts room polygons using two-level queries. Although deep learning approaches have exhibited impressive performance in floorplan reconstruction, we have identified the following opportunities for improvement.
Semantic quality. The plausibility or semantic quality of reconstructed floorplans heavily relies on the accuracy of estimated room corners and masks. Insufficient supervision of the room shape or fragile room representations can result in artificial room shapes, as demonstrated in Figure 1(a), that are not plausible.
Efficient representation. One main challenge is designing a representation that requires only a reasonable number of tokens in the transformer architecture and can handle a variable number of rooms, each with a variable number of corners. For example, the naive approach of predicting the layout room by room, corner by corner does not require too many tokens but is too difficult to
<https://ericusujw.github.io/SLIBO-Net/>
(a) Artificial Corners
(b) Density Map
Figure 1: Limitations for current methods. (a) The lack of supervision on the overall room shape can lead to the generation of artificial corners. (b) The density map of a large floorplan may have insufficient resolution to effectively capture local geometric details, such as wall thickness, which are clearly evident in the (c) ground truth layout.
train. To address this challenge, we employ a novel floorplan representation that slices the floorplan into a set of boxes by scanning it in a top-down manner (Figure 3). This representation offers two key advantages. Firstly, it yields a manageable number of slicing boxes, making it suitable for the transformer architecture as each slicing box can be represented by a query token. Secondly, these slicing boxes provide improved supervision of the room shape by utilizing Intersection over Union (IoU) loss [15]. Furthermore, this representation naturally leads to higher semantic quality, enhancing the overall accuracy and realism of the reconstructed floorplans.
Local details. We observed that deep learning approaches excel in predicting the overall room structure but struggle to capture local geometric details such as wall thickness due to image resolution limitations (Figure 1(b)). We incorporate prior knowledge about geometric details derived from the input 3D point cloud to address this limitation. This knowledge is encoded as cell complexes, which serve multiple purposes. It acts as a regularization mechanism during the training phase and is also utilized as a post-processing step to enhance the accuracy of the reconstructed floorplans.
Evaluation. The metrics employed in previous studies are scale-dependent, resulting in differential treatment of errors between large and small floorplans. To illustrate this, consider starting with a floorplan of a specific size. Adding additional rooms to the floorplan alters its error measurement, leading to a more lenient evaluation of incorrect corner predictions. This behavior is counter-intuitive and primarily masks a drawback of neural networks in scaling to larger inputs. Hence, we advocate for adopting a scale-independent metric that mitigates this issue.
In summary, we make the following contributions: 1) We propose a novel transformer architecture for floorplan reconstruction that only requires a manageable number of tokens. 2) We propose integrating geometric priors as regularization during training and as a post-process to improve local geometric details. 3) We advocate the use of a new metric that is scale independent and that better measures if local details are correctly reconstructed. 4) We produce more semantically meaningful floorplans that do not suffer from implausible geometric details.
**Optimization-based methods.** Early work mostly assumed the task to be based on finding 3D planes from the point clouds using traditional techniques such as Hough transform [16, 2], plane sweeping/plane fitting [4, 23, 17, 12, 18, 21], and leveraging histograms of point cloud positions [7]. Higher-level structured objects such as rooms could then be assembled from the recovered planes. For larger buildings with many rooms, the optimization became increasingly difficult to solve. Researchers then developed graph-based data structures [10, 5, 11, 8, 22] to help organizing the reconstructed sub-models (e.g., corners, walls, objects, and rooms) and guide the optimization to follow certain semantic, topological constraints. In particular, Ikehata et al. [11] used hierarchical tree graphs of reconstructed sub-models, such as objects, walls, and rooms, to guide the 3D reconstructions. Bassier and Vergauwen [3] extracted highly-detailed BIM-compatible objects from point clouds by plane fitting and RANSAC. In the same spirit of extracting more semantic and realistic 3D models, Ochmann et al. [19] proposed methods to extract parametric 3D building models from input point clouds. A key advantage was that their reconstructed floorplans are volumetric - i.e., having wall thickness. Our results shared the same idea. Finally, our cell complex-based constraints was inspired
(a) Input Point Cloud
(b) Segmented Clusters
(c) Wall Clusters
(d) Wall Segments
(e) Cell Complex
Figure 2: Encoding geometric priors into the cell complex. (a) Input 3D point cloud. (b) Segmentation based on normals and proximity. (c) Plane fitting and grouping. Aerial view of wall clusters with orientations. (d) Wall segment formation by collinear line grouping. (e) Cell complex from parametric lines. Non-axis aligned lines removed.
by [13], in which they proposed an optimization-based approach to register multiple 3D scans of multi-story, multi-room buildings.
Deep learning methods. More recent methods began to leverage neural networks to solve the floorplan reconstruction problems. Earlier methods were hybrid methods that utilized both optimization and neural networks. Floor-SP [8] assembled room segments detected by Mask R-CNN [1] to form full floorplans by sequentially solving shortest path problems. MonteFloor [22] later solved the room assembly problem by Monte-Carlo Tree-Search (MCTS). FloorNet [14] took a point cloud and RGB images as inputs, utilized a triple-branch neural network to generate pixel-wise predictions on floorplan geometry and semantics, and then used an integer programming to generate the final floorplans in vector-graphics format.
Later, fully neural network-based methods (which are end-to-end trainable) were being developed and delivered state-of-the-art performances. HEAT [9] was a method to reconstruct planar graphs with underlying geometric structures from 2D raster image inputs. It had two main applications - building segmentation in aerial images and floorplan reconstruction from 2D density maps distilled from point clouds. It utilized an end-to-end trainable edge classification neural network for the task. Recently, RoomFormer [24] proposed a single-stage, end-to-end trainable neural network to directly predict floorplans from a given point cloud. Their main innovation was to encode a floorplan as variable-size set of polygons, which are variable-length sequences of ordered vertices. This design enabled efficient handling by transformers. Our method is an improvement upon it.
3.1 Pre-processing
In the data preprocessing stage, we adopt a similar process as in previous works [8, 9, 14, 22] to project the raw 3D point cloud along the gravity axis (or z-axis), resulting in a 2D density map in the aerial view (or x-y plane). It is important to note that while this density map provides adequate visual information for describing the overall structure (e.g., rooms and walls) of the floorplan, it faces challenges in capturing geometric details like wall thickness due to the limitation of image resolutions.
Encoding geometric priors. We extract wall segments from 3D point cloud data as geometric priors for floorplan reconstruction. We project these wall segments as parametric lines onto the x-y plane, forming a "cell complex" representation. Figure 2 shows the steps of computing a cell complex from a 3D point cloud. First, we segment the point cloud by normals and proximity (Figure 2(b)). Next, we fit planes for each cluster and select those orthogonal to gravity as wall clusters (Figure 2(c)). After that, we project points onto planes and group collinear line segments as wall segments (Figure 2(d)). Finally, we convert wall segments into parametric lines and remove non-axis aligned lines to get the cell complex (Figure 2(e)). After obtaining the cell complex representation, we follow the strategy described in Section 3.4, where we adjust our ground truth by rounding it to the cell complex representation for training.
Figure 3: Floorplan representation. Illustrative examples highlighting floorplans (left) and their respective slicing boxes representation (right).
3.2 Floorplan representation
One key challenge in our work was designing a suitable data structure for floorplans. The representation should work with the transformer architecture and only require a reasonable number of tokens. One possible representation would be to represent each grid cell of the cell complex as a query token, but this would require a large number of tokens, e.g., 2655 for our dataset. Another possible representation is to encode a floorplan as a set of room polygons. However, we observed that predicting rooms as sequences of corners may lead to artificial room shapes as shown in Figure 1(a). We propose to employ a novel floorplan representation that slices the floorplan into a set of boxes by scanning the floorplan along the vertical axis (or y-axis). We denote this box set as  $\mathcal{B} = \{b_i\}_{i=1}^M$ , where  $b_i = \{b_i^j\}_{j=1}^{N_m}$  and  $N_m$  is the number of boxes  $b_i^j$  in the  $m^{th}$  room. Examples of the slicing boxes representation are depicted in Figure 3. The major advantage of this representation lies in its ability to maintain a manageable number of slicing boxes and query tokens ( $\leq 50$  in our dataset). However, each floorplan may consist of a varying number of rooms, with each room containing a different number of slicing boxes. This poses a challenge in clustering the slicing boxes to their respective rooms. The unordered nature of the rooms, combined with the variable number of rooms and slicing boxes, makes it difficult to assign room IDs for the classification task. To tackle this challenge, we transform the original box classification problem into a 2D coordinate regression problem. Specifically, we employ one network to predict the room center for each slicing box, and another network to predict all the room centers from the density map. We compute the room center coordinate as the bounding box center for each room in the floorplan, represented as  $\mathbb{C} = \{c_i\}_{i=1}^M$ , where M denotes the number of rooms and  $c_i \in [-1,1] \times [-1,1]$ . This allows us to cluster the slicing boxes based on the predicted room centers.
3.3 Network architecture
Figure 4 illustrates the SLIBO-Net's architecture, comprising two primary components: the center transformer  $\mathcal{T}_{center}$  and the box transformer  $\mathcal{T}_{box}$ . The center transformer  $\mathcal{T}_{center}$  is responsible for predicting the coordinates of all room centers based on the density map, while the box transformer  $\mathcal{T}_{box}$  predicts both the slicing boxes and their corresponding room center coordinates. Subsequently, a post-processing step is implemented to merge the estimated slicing boxes into a complete floorplan with distinct room layouts, utilizing the predicted room center points. Each component are elaborated in the next subsections.
Center transformer. First, we feed a  $256 \times 256$  density map into a ResNet-50 feature extractor and generate one  $16 \times 16$  feature map. Following the encoder-decoder transformer architecture, we feed the extracted feature maps into the transformer encoder block. The output tokens of the transformer encoder block are then used for cross attention in the transformer decoder block. In order to query all the room centers from the density map, we use  $\bar{M}$  learnable embeddings as the input tokens of the transformer decoder. The output tokens of the transformer decoder are further processed using two auxiliary MLP heads. Specifically, we use: (i) room center MLP head  $\mathcal{F}_{room}^{center}$  that outputs the coordinates of rooms' bounding box center  $\mathbb{C}^{room} = \{c_i^{room}\}_{i=1}^{\bar{M}}$ ; and (ii) room prob MLP head  $\mathcal{F}_{room}^{prob}$  that outputs the probabilities  $\mathbb{P}^{room} = \{p_i^{room}\}_{i=1}^{\bar{M}}$  representing the probability of valid room center.
Box transformer. As for the box transformer  $\mathcal{T}_{\text{box}}$ , we follow the same architecture of center transformer  $\mathcal{T}_{\text{center}}$  except the number of input tokens of the transformer decoder and the output MLP heads. We query all the slicing boxes from the density map via  $\bar{N}$  learnable embeddings. All the output tokens of the box transformer  $\mathcal{T}_{\text{box}}$  are further processed using two auxiliary MLP heads.
Figure 4: Network architecture. SLIBO-Net takes a 2D density map as input, and outputs a realistic floorplan as the final result. (a) The center transformer  $\mathcal{T}_{\text{center}}$  outputs a set of room centers and the probabilities for them to be valid. (b) The box transformer  $\mathcal{T}_{\text{box}}$  predicts a set of slicing boxes, corresponding room centers, and their probabilities of being valid. Room centers and slicing boxes are further post-processed to form the final floorplan.
We use (i) box coord MLP head  $\mathcal{F}_{box}^{coord}$  that outputs six scalars at once. The first two scalars represent the corresponding room's center coordinates  $\mathbb{C}^{box} = \{c_i^{box}\}_{i=1}^{\bar{N}}$  and the remain four scalars represent the upper-left and lower-right corners of slicing boxes  $\mathcal{B} = \{b_i\}_{i=1}^{\bar{N}}$ ; and (ii) box prob MLP head  $\mathcal{F}_{box}^{prob}$  that outputs the probabilities  $\mathbb{P}^{box} = \{p_i^{box}\}_{i=1}^{\bar{N}}$  representing the probability of valid slicing boxes.
Bipartite matching. Note that our neural network would always predict 25 room centers  $(\mathbb{C}^{room})$  and 50 slicing boxes, which generally would be more than the ground-truth room centers  $\overline{\mathbb{C}}^{room}$  and slicing boxes  $\mathcal{B}$ , respectively. Thus, following DETR [6], we conduct a set-based bipartite matching between predictions and targets in the center transformer  $\mathcal{T}_{center}$  and the box transformer  $\mathcal{T}_{box}$ . Since we use the same strategy in both of the transformers, we explain this step using the center transformer  $\mathcal{T}_{center}$  as an example. Our goal is to optimize a bipartite matching objective on a permutation function  $\sigma(\cdot): \mathbb{Z}_+ \to \mathbb{Z}_+$  which maps  $\bar{M}$  prediction indices to  $M$  targets:
$$
\sigma^ {*} = \arg \min  _ {\sigma} \sum_ {i = 1} ^ {\bar {M}} \mathbb {1} _ {\{\sigma (i) \leq M \}} \left[ \lambda_ {1} d _ {1} \left(c _ {i} ^ {\text {r o o m}}, \bar {c} _ {\sigma (i)} ^ {\text {r o o m}}\right) + \lambda_ {2} d _ {2} \left(p _ {i} ^ {\text {r o o m}}\right) \right] \tag {1}
$$
where  $d_{1}(\cdot, \cdot)$  represents  $L_{1}$  distance between predicted and target room center coordinates,  $d_{2}()$  represents the focal loss between predicted and target room center probabilities, and  $\mathbb{1}_{\{\cdot\}}$  is an indicator function. Finally, we use the Hungarian Matching algorithm to calculate the minimal cost  $\sigma^{*}$  that maps  $\bar{M}$  predicted room centers to  $M$  target room centers and  $\bar{N}$  predicted slicing boxes to  $N$  target slicing boxes.
3.4 Post-processing
Here, we describe the post-processing steps in Figure 5 to reconstruct the floorplan using the slicing boxes  $\mathcal{B}$  and room centers  $\mathbb{C}^{room}$  obtained in the previous step by the SLIBO-Net.
The first step in the post-processing phase is to cluster the slicing boxes  $\mathcal{B}$  into rooms. Given the predicted room centers  $\mathbb{C}^{room}$  from the center transformer  $\mathcal{T}_{center}$  and the predicted slicing boxes  $\mathbb{C}^{box}$  corresponded rooms' centers  $\mathbb{C}^{box}$  as the input, we can cluster the slicing boxes  $\mathcal{B}$  into different rooms by assigning the  $\mathbb{C}^{box}$  to the nearest room centers  $\mathbb{C}^{room}$  from the center transformer  $\mathcal{T}_{center}$ .
After clustering the boxes into different rooms, we fill the gaps between the bottom and top of any two adjacent boxes in the same room with a box whose height is the vertical distance between them and whose width is the intersection length of them.
Finally, we union together the boxes for each room to form a room area  $A_{r}$ . We then overlay  $A_{r}$  on the cell complex, which allows us to calculate the intersection ratio for each room per cell. By assigning each cell to the room with the largest intersection ratio, we can obtain a room representation by tracing the boundary formed by the cell groups. For non-connected components within a single room, we choose the component with the largest area. A cell remains empty if no room has more than 50% overlap.
Figure 5: Illustration of the post-processing. We use cell complex and SLIBO-Net's output to get the floorplan. Same-color boxes are one room. (a) Crosses show slicing boxes' rooms' centers, dots show predicted room centers. (b) Box fills gap between boxes, red dashed box shows its size. (c) Colored boundaries are final result by boxes. Cells in green box belong to red room if they intersect more with red room and intersection ratio is over  $50\%$ .
We first elaborate on the loss functions we use for the center transformer  $\mathcal{T}_{center}$  based on the optimal permutation  $\sigma^{*}$  from the bipartite matching. For the center transformer  $\mathcal{T}_{center}$ , we use the index set  $\{i; \sigma^{*}(i) \leq M\}$  representing the best matching prediction indices. For the coordinate regression, we use the smooth  $L_{1}$  loss  $\mathbb{L}_{smooth}(\cdot, \cdot)$  which is defined as follow:
$$
\mathbb {L} _ {\text {s m o o t h}} = \left\{ \begin{array}{l l} \| X - Y \| _ {1} + 0. 5 (X - Y) ^ {2}, & \text {i f} \| X - Y \| _ {1} <   1 \\ \| X - Y \| _ {1} + \| X - Y \| _ {1} - 0. 5, & \text {o t h e r w i s e} \end{array} \right. \tag {2}
$$
where  $X$  is the prediction and  $Y$  is the label.
Room center loss calculates the smooth  $L_{1}$  loss with ground truth room center  $\overline{e}^{room}$ :
$$
\mathcal {L} _ {r c e n t e r} ^ {(i)} = \mathbb {1} _ {\left\{\sigma^ {*} (i) \leq M \right\}} \mathbb {L} _ {s m o o t h} \left(c _ {i} ^ {r o o m}, \bar {c} _ {\sigma (i)} ^ {r o o m}\right) \tag {3}
$$
Room prob loss calculates the focal loss for room probability of being valid:
$$
\mathcal {L} _ {r p r o b} ^ {(i)} = - \mathbb {1} _ {\left\{\sigma^ {*} (i) \leq M \right\}} \left(1 - p _ {i} ^ {r o o m}\right) ^ {\gamma} \log p _ {i} ^ {r o o m} - \mathbb {1} _ {\left\{\sigma^ {*} (i) > M \right\}} \left(p _ {i} ^ {r o o m}\right) ^ {\gamma} \log \left(1 - p _ {i} ^ {r o o m}\right) \tag {4}
$$
Thus, the final loss  $\mathcal{L}_{rtotal}$  of the center transformer  $\mathcal{T}_{center}$  is defined as below:
$$
\mathcal {L} _ {r t o t a l} = \sum_ {i = 1} ^ {M} \lambda_ {1} \mathcal {L} _ {r c e n t e r} ^ {(i)} + \lambda_ {2} \mathcal {L} _ {r p r o b} ^ {(i)} \tag {5}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are the hyperparameters for weighting the loss functions.
Next, we describe the loss functions we compute for the box transformer  $\mathcal{T}_{box}$  based on the optimal permutation  $\sigma^{*}$  from the bipartite matching. For the box transformer  $\mathcal{T}_{box}$ , we use the index set  $\{i; \sigma^{*}(i) \leq N\}$  representing the best matching prediction indices.
Box room center loss calculates the smooth  $L_{1}$  loss for each slicing box room center with ground truth room center  $\overline{c}^{box}$ :
$$
\mathcal {L} _ {b c e n t e r} ^ {(i)} = \mathbb {1} _ {\left\{\sigma^ {*} (i) \leq N \right\}} \mathbb {L} _ {s m o o t h} \left(c _ {i} ^ {b o x}, \bar {c} _ {\sigma (i)} ^ {b o x}\right) \tag {6}
$$
Box prob loss calculates the focal loss for each slicing box probability of being valid:
$$
\mathcal {L} _ {b p r o b} ^ {(i)} = - \mathbb {1} _ {\{\sigma^ {*} (i) \leq N \}} \left(1 - p _ {i} ^ {b o x}\right) ^ {\gamma} \log p _ {i} ^ {b o x} - \mathbb {1} _ {\{\sigma^ {*} (i) > N \}} \left(p _ {i} ^ {b o x}\right) ^ {\gamma} \log \left(1 - p _ {i} ^ {b o x}\right) \tag {7}
$$
Box corner loss calculates the smooth  $L_{1}$  loss for all the corners of each slicing box:
$$
\mathcal {L} _ {b c o r n e r} ^ {(i)} = \mathbb {1} _ {\left\{\sigma^ {*} (i) \leq N \right\}} \mathbb {L} _ {s m o o t h} \left(b _ {i} ^ {c o r n e r}, \bar {b} _ {\sigma (i)} ^ {c o r n e r}\right) \cdot w _ {\sigma (i)} \tag {8}
$$
where  $b_{i}^{\text{corner}}$  represented all 4 corners of a slicing box. We assume a conversion from the original upper-left and lower-right representation of the slicing boxes  $b_{i}$ . We set  $w_{\sigma(i)}$  to 1 if and only if the corner  $\bar{b}_{\sigma(i)}^{\text{corner}}$  equals any corner in the floorplan; otherwise, it is 0.
Table 1: Quantitative comparison of floorplan reconstruction on Structured3D [25] dataset. The best results are in bold font. * means using cell complex to round the floorplan.
Method	Room↑	Corner↑	Angle↑	Delta↓
Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1	Prec.	Rec.	F1
Floor-SP [8]	0.89	0.88	0.885	0.81	0.73	0.768	0.80	0.72	0.758	0.01	0.01	0.01
MonteFloor [22]	0.956	0.944	0.950	0.885	0.772	0.825	0.863	0.754	0.805	0.022	0.018	0.02
HEAT [9]	0.969	0.940	0.954	0.817	0.832	0.824	0.776	0.790	0.783	0.041	0.042	0.041
RoomFormer [24]	0.977	0.965	0.971	0.889	0.851	0.870	0.826	0.791	0.808	0.063	0.06	0.062
RoomFormer* [24]	0.991	0.976	0.983	0.851	0.826	0.838	0.840	0.816	0.828	0.011	0.01	0.01
Ours	0.991	0.978	0.984	0.889	0.821	0.854	0.878	0.812	0.844	0.011	0.009	0.01