Title: PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Structured Reconstruction

Floorplan Reconstruction

HD Map Construction

Diffusion Models
Abstract: This paper presents \textit{PolyDiffuse}, a novel structured reconstruction algorithm that transforms visual sensor data into polygonal shapes with Diffusion Models (DM), an emerging machinery amid exploding generative AI, while formulating reconstruction as a generation process conditioned on sensor data. 
The task of structured reconstruction poses two fundamental challenges to DM: 1) A structured geometry is a ''set'' (e.g., a set of polygons for a floorplan geometry), where a sample of $N$ elements has $N!$ different but equivalent representations, making the denoising highly ambiguous; and 2) A ''reconstruction'' task has a single solution, where an initial noise needs to be chosen carefully, while any initial noise works for a generation task.
Our technical contribution is the introduction of a Guided Set Diffusion Model where 1) the forward diffusion process learns \textit{guidance networks} to control noise injection so that one representation of a sample remains distinct from its other permutation variants, thus resolving denoising ambiguity; and 2) the reverse denoising process reconstructs polygonal shapes, initialized and directed by the guidance networks, as a conditional generation process subject to the sensor data.
We have evaluated our approach for reconstructing two types of polygonal shapes: floorplan as a set of polygons and HD map for autonomous cars as a set of polylines.
Through extensive experiments on standard benchmarks, we demonstrate that PolyDiffuse significantly advances the current state of the art and enables broader practical applications. The code and data are available on our project page: https://poly-diffuse.github.io.

PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models

Abstract

This paper presents PolyDiffuse , a novel structured reconstruction algorithm that transforms visual sensor data into polygonal shapes with Diffusion Models (DM), an emerging machinery amid exploding generative AI, while formulating reconstruction as a generation process conditioned on sensor data. The task of structured reconstruction poses two fundamental challenges to DM: 1) A structured geometry is a 'set' (e.g., a set of polygons for a floorplan geometry), where a sample of N elements has N ! different but equivalent representations, making the denoising highly ambiguous; and 2) A 'reconstruction' task has a single solution, where an initial noise needs to be chosen carefully, while any initial noise works for a generation task. Our technical contribution is the introduction of a Guided Set Diffusion Model where 1) the forward diffusion process learns guidance networks to control noise injection so that one representation of a sample remains distinct from its other permutation variants, thus resolving denoising ambiguity; and 2) the reverse denoising process reconstructs polygonal shapes, initialized and directed by the guidance networks, as a conditional generation process subject to the sensor data. We have evaluated our approach for reconstructing two types of polygonal shapes: floorplan as a set of polygons and HD map for autonomous cars as a set of polylines. Through extensive experiments on standard benchmarks, we demonstrate that PolyDiffuse significantly advances the current state of the art and enables broader practical applications. The code and data are available on our project page: https://poly-diffuse.github.io .

1 Introduction

Reconstruction and generation were once separate research areas with minimal overlaps. While not acknowledged by the broader research communities, state-of-the-art reconstruction and generation techniques have now exhibited increasing similarities. Focusing on structured geometry (e.g., CAD models) in this paper, Auto-regressive Transformer (AR-Transformer) is a family of state-of-the-art generative models that iteratively applies a Transformer network to generate CAD construction sequences [18, 29, 35]. On the reconstruction side, a Transformer network iteratively reconstructs and refines indoor floorplans, outdoor buildings models, and vectorized traffic maps [7, 23, 47], a process resembling the generative AR-Transformer.

In 2023, Generative AI is experiencing significant growth, primarily driven by the emergence of Diffusion Models (DM) [14, 38, 40, 41], where structured geometry generation is not an exception. DM-based approach iteratively denoises coordinates (with associated properties) to generate realistic design layouts [17] or floorplans [36]. A natural question is then 'Is a Diffusion Model also good at structured reconstruction?' However, the answer is not that simple.

Take the high-definition (HD) map reconstruction problem for example [22], which aims to reconstruct a set of polylines given sensor data in a bird's-eye view (see Figure 1). A straightforward DM formulation would assume a maximum number of polylines (with a fixed number of corners per poly-

Ped. crossing

Sensor inputs

Initial proposal

Sensor-conditioned Denoising Process of Guided Set Diffusion Model

⋯

4

4

6

4

22

8

4

6

Divider

Boundary

car

HD map

⋯

Guidance

networks

Guidance

networks

⋯

⋯

Sample

Sample

line [23, 25]), take a particular permutation of the elements to construct a fixed-length vector of corner coordinates, then iteratively denoise the vector subject to sensor data as a condition. The structured reconstruction task poses two fundamental challenges to such standard DM formulations: 1) Structured geometry is often a 'set' of elements (e.g., a set of polylines), where the geometry of N elements has N ! different but equivalent representations, making the denoising factorially ambiguous; 2) The denoising process needs to reach a single solution ( i.e ., one of N ! representations) for a reconstruction task and an initial noise needs to be set carefully, while any initial noise would work for a generation task. Figure 2 shows a toy example of three elements with 6 = 3! permutations, where six permutationequivalent representations of the solution lie in six different sub-spaces. However, they all follow the standard Gaussian and become indistinguishable after the diffusion process, which obfuscates the denoising learning.

(a).

𝑡 = 0

Standard diffusion

GS-DM diffusion

0

1

-1

0

1

0

3

-3

0

3

0

1

-1

0

1

(b).

𝑡 = 𝑇

(standard DM)

(c).

𝑡 = 𝑇

(Ours)

Standard denoising

GS-DM denoising

Our technical contribution is a Guided Set Diffusion Model (GS-DM). The forward diffusion process learns guidance networks controlling noise injection so that one representation of a sample remains separated from its permutation variants, thus resolving denoising ambiguities. Specifically, the guidance networks set an individual Gaussian as the target distribution for each element (instead of a fixed zero-mean unit-variance Gaussian for everything), and are learned before the denoising training by minimizing the permutation ambiguity as a loss function. At test time, the reverse process initializes the element-wise Gaussian noise by the guidance networks and then denoises to reconstruct polygonal shapes conditioned on the sensor data, while the guidance networks take rough initial reconstructions either from an existing method or an annotator (see Figure 1 for examples).

We have conducted extensive experiments on two polygonal structure reconstruction tasks: floorplan reconstruction from a point-cloud density image with Structured3D dataset [48], and HD map construction from onboard camera inputs with nuScenes dataset [2]. PolyDiffuse augments the state-of-the-art approach of the two specific tasks and brings consistent performance improvements, especially on the geometric regularity of the reconstruction results. Our generation process takes 5 to 10 steps to produce decent results and the visual encoder only runs once. We also demonstrate a likelihood-based refinement extension of our system by exploiting the probability flow ODE [41], which enables broader practical applications. To our knowledge, this paper is the first to demonstrate that DM is a powerful machinery for structured reconstruction tasks, in fact, the first in a broader domain of geometry reconstruction as well. We will share all our code and data.

2 Background: Denoising Diffusion Probabilistic Models

Diffusion models or score-based generative models [14, 38, 40, 41] progressively inject noise to data in the forward (diffusion) process and generate data from noise by the reverse (denoising) process. The section provides the key equations on the denoising diffusion probabilistic models (DDPM) [14] that lay the foundation of our method.

DDPM considers a Markovian forward process that turns data x 0 ∼ q ( x 0 ) into Gaussian noise:

t = 1 , . . . , T and x t is the latent at t . The noise schedule { β t } is constant [14, 28] or learned by reparameterization [20]. The above forward process is written in closed form for an arbitrary t :

x t is sampled by x t = √ ¯ α t x 0 + √ (1 -¯ α t ) ϵ for ϵ ∼ N ( 0 , I ) . DDPM parameterizes the reverse process with a noise prediction (or denoising) network ϵ θ ( x t , t ) to make connections with denoising score matching and Langevin dynamics [40, 45], and the sampling step of the reverse process is derived as:

σ 2 t is set to β t or 1 -¯ α t -1 1 -¯ α t β t . The final training objective is a reweighted variational lower bound:

3 Guided Set Diffusion Models (GS-DM)

Consider the task of generating a set of elements x = { x 1 , . . . , x N } under a condition y . A straightforward application of diffusion models would be to take one permutation of the elements to form a vector, then perform diffusion and learn denoising. With an abuse of notation, we let the vector x 0 ( i.e ., the sample or denoised result in DM's notation) represent one particular permutation of x . The challenge is that a set of N elements has N ! different but equivalent representations, and any one of the N ! is a valid x 0 , making the denoising factorially ambiguous 1 . To alleviate the above issue, our idea learns to guide the noise injection in a per-element manner by learning two 'guidance networks', such that a sample x 0 remains separated from its permutation variants throughout the diffusion process. At test time, the guidance networks produce per-element Gaussians and stepwise guidance from an initial reconstruction, from which the denoising process generates the final reconstruction iteratively conditioned on sensor data. We first explain the forward and reverse processes and then the two-stage training for the guidance and denoising networks.

Forward process : With the standard diffusion process, different permutation variants of a sample gradually become indistinguishable through the diffusion process. To keep a particular representation x 0 separated from its permutation variants, we propose to inject noise per element. Let x i t denote the diffused element x i 0 at timestep t , we learn to change the transition distribution q ( x i t | x i t -1 ) by adding µ ϕ ( x 0 , t, i ) to the mean and multiplying σ ϕ ( x 0 , t, i ) to the standard deviation:

µ ϕ and σ ϕ take x 0 and produce outputs for i th element at t . Following the similar derivations as in DDPM [14], we obtain (see Appendix A.1 for derivations):

As q ( x i t | x i 0 ) only explicitly depends on ¯ µ ϕ and ¯ σ ϕ , and x i T ∼ N ( ¯ µ ϕ ( x 0 , T, i ) , ¯ σ 2 ϕ ( x 0 , T, i )) , we consider ¯ µ ϕ and ¯ σ ϕ as the guidance networks. µ ϕ and σ ϕ are defined implicitly by Eq. 8 and Eq. 9. To simplify the formulation, we define ¯ σ ϕ ( x 0 , t, i ) := √ 1 -¯ α t C ( x 0 , i ) where C ( x 0 , i ) is a function independent of the timestep t , thus implicitly defining σ ϕ ( x 0 , t, i ) = ¯ σ ϕ ( x 0 , T, i ) = C ( x 0 , i ) .

Reverse process : Note that x 0 is the ground truth and only available for training. At test time, we introduce a proposal generator to produce an initial reconstruction ˆ x 0 , and run ¯ µ ϕ and ¯ σ ϕ with ˆ x 0 . ˆ x 0 can either be results from an existing method or rough annotations from a human annotator. We first draw the element-wise initial noise with x i T ∼ N ( ¯ µ ϕ (ˆ x 0 , T, i ) , ¯ σ 2 ϕ (ˆ x 0 , T, i )) , and then run denoising steps iteratively. Similar to Eq.4, the sampling step of the reverse process is derived as

The denoising network ϵ θ takes the entire x t ( i.e ., all elements), and ϵ i θ is the output of the i th element. Please see Appendix A.2 for derivations.

Learning the denoising network : Following Eq. 5 of standard DDPM, the denoising objective is

The above formulation from Eq.6 to Eq.11 resembles that of DDPM in §2, where the key difference is the introduction of guidance networks ¯ µ ϕ and ¯ σ ϕ . We then describe the training of the guidance networks, which is the key step of GS-DM.

Learning the guidance network : Before training the denoising network with Eq. 11, we train ¯ µ ϕ and ¯ σ ϕ to ensure that x t in the diffusion process keep separated from other permutation variants of x 0 . We quantify the permutation invariance between x 0 and x t with a triplet loss L Triplet ( · , · , · ) widely used in metric learning [4, 34] (full implementation details are in Appendix B.3):

P ! ( x 0 ) is the set of all permutation variants of x 0 . Using the terminology in metric learning, the three inputs of L Triplet ( · , · , · ) are the anchor, positive, and negative, respectively. Our loss only considers the closest negative permutation, which is similar to the hard negative mining in some variants of the triplet loss [6, 10]. We also add two regularization terms on ¯ µ ϕ and ¯ σ ϕ , so that the means of all elements are not too scattered and the variances do not vanish. The final loss for training the guidance networks is:

Algorithm 1 and Algorithm 2 summarize the two-stage training paradigm of GS-DM.

4 PolyDiffuse: Polygonal shape reconstruction via GS-DM

PolyDiffuse uses the GS-DM to solve polygonal shape reconstruction tasks (see Figure 3), in particular, floorplan and HD map reconstruction. This section explains specific details and specializations.

⋯

𝐱

!

𝐲

𝐱

"

"

|𝐱

!

)

𝐲

𝑝(𝐱

"

|) 𝐱

!

)

𝐱

!

Forward

Reverse

⋯

⋯

Sample

𝐱

"

𝝐

!

⋯

⋯

𝑡

" 𝝁

𝝓

" 𝝈

𝝓

Guidance

networks

" 𝝁

𝝓

" 𝝈

𝝓

Proposal generator

Guidance

networks

) 𝐱

!

?

72

Feature representation : A sample x is a set of elements x i as defined in §3. Each element x i is either a (closed) polygon or a polyline, consisting of an arbitrary (for the floorplan task) or a fixed (for the HD map task) number of vertices as a sequence: x i = [ v i 1 , . . . , v i N i ] . Each vertex contains a 2D coordinate. For the ground-truth data, we determine the first vertex of each element by sorting based on the Y-axis coordinate and then the X-axis coordinate, similarly to PolyGen [27]. The polygons are always in a counter-clockwise orientation.

Guidance network : As derived in §3, the forward (Eq.7) and reverse (Eq.10) processes only depend on ¯ µ ϕ and ¯ σ ϕ , and each element follows x i T ∼ N ( ¯ µ ϕ ( x 0 , T, i ) , ¯ σ 2 ϕ ( x 0 , T, i )) . Therefore, we choose to directly parameterize ¯ µ ϕ ( x 0 , T, i ) = Trans ¯ µ ϕ ( x 0 , i ) and ¯ σ ϕ ( x 0 , T, i ) = Trans ¯ σ ϕ ( x 0 , i ) as two Transformer [44] networks, and define for all timesteps t = 1 , . . . , T -1 as follows:

Note that Eq.14 makes the noise adaptation directly dependent on ¯ α t for simplicity and drops t from the input. Algorithm 1 thus learns the above two Transformers. As for the proposal generator to produce ˆ x 0 at test time, we either employ the state-of-the-art task-specific method ( i.e ., RoomFormer [47] for floorplan and MapTR [23] for HD map) or simulate rough human annotations. Please see §5 for experiments with different proposal generators, and Appendix B.1 for implementation details of the guidance network.

Denoising network : The implementation of the denoising network ϵ θ ( x t , t, y ) borrows the core neural architectures from the state-of-the-art task-specific models RoomFormer [47] and MapTR [23]. While these two models are both adapted from DETR [3] and learn direct mappings from y to x with encoder-decoder Transformers, we need to make a few essential modifications to turn them into valid denoising networks (full implementation details are in Appendix B.2):

Instead of the learnable embeddings or coordinates, x t should be the input nodes to the Transformer decoder. Each vertex is a node, and the node feature is the concatenation of four positional encodings [44]: two for X and Y-axis coordinates and another two for vertex and instance index.

We encode the timestep t (or noise level) with a positional encoding [9, 14], and add this feature to each block of the Transformer decoder.

Our overall formulation with GS-DM is independent of the network architectures, and more advanced networks can be easily integrated into the framework in the future to boost performance.

Sampling acceleration : PolyDiffuse follows Eq.10 to reconstruct polygonal shapes. Song et al.[41] shows that DDPM can be viewed as a discretization of a stochastic differential equation (SDE) with an associated probability flow ODE, and advanced ODE solvers [19, 26, 39] can significantly accelerate the sampling process. At test time, we employ a first-order solver ( e.g ., DDIM [39]) and use 10 sampling steps for both tasks (ablation study is in §5.3). During the generation process, PolyDiffuse only runs the image encoder once as the visual features are shared across all steps.

Likelihood evaluation : With the probability flow ODE [41], PolyDiffuse can estimate its own reconstruction quality via likelihood evaluation, thus enabling potential extensions such as searchbased or human-in-the-loop refinement as a wrapper around our system. This is an exciting property not possessed by previous reconstruction approaches. We show examples in Figure 4.

5 Experiments

We have implemented the system with PyTorch and used a machine with 4 NVIDIA RTX A5000 GPUs. We have borrowed the official codebase of Karras et al.[19] to implement the diffusion model framework. The guidance networks are implemented as set-to-set Transformer decoders, and the loss weights for the guidance training are λ 1 = 1 , λ 2 = 0 . 05 , λ 3 = 0 . 1 . The implementation of the denoising network refers to the competing state-of-the-art, RoomFormer [47] and MapTR [23] for floorplan and HD map reconstruction, respectively. Concretely, we employ the same ResNet [12] image encoder, DETR-style Transformer architectures (discussed in §4), learning rate settings, and optimizer settings as RoomFormer or MapTR, while increasing the number of training iterations ( × 2 . 5 for floorplan and × 1 . 5 for HD map), as the model converges slower under a denoising formulation than a detection/regression formulation. Note that we tried the same training schedule for the competing methods but their performance dropped due to overfitting (see §5.3 for ablation study). For each task, we use the same trained model for different types of proposals ( i.e ., existing methods or rough annotations). Complete implementation details are in Appendix B, and supplementary experiments are in Appendix D

5.1 Floorplan reconstruction

Dataset and metrics : Structured3D dataset [48] contains 3500 indoor scenes (3000/250/250 for training/validation/test) with diverse house floorplans. The average/maximum numbers of polygons and vertex per polygon across the dataset are 6.29/17 and 5.72/38, respectively. Point clouds are converted to 256 × 256 top-view point-density images as inputs. We use the same evaluation metrics as previous works [7, 42, 47], which consists of three levels of metrics with increasing difficulty: room, corner, and angle. The precision, recall, and F1 score are reported at each metric level.

Competing approaches : We compare PolyDiffuse with five approaches from the literature: FloorSP [5], MonteFloor [42], LETR [46], HEAT [7], and RoomFormer [47]. The first two approaches design sophisticated optimization algorithms to solve for the optimal floorplan with learned metric functions, while the latter three use end-to-end Transformer-based neural networks, thus being much faster. For our proposal generator RoomFormer, we simplify the implementation by removing the Dice loss. As the dataset is very small, we add a random rotation data augmentation and train the modified RoomFormer for 4 × the original training iterations, which improves the overall performance.

Table 1: Quantitative evaluation on Structured3D test set [48]. PolyDiffuse outperforms the state-of-the-art methods by clear margins, and works well with rough annotations. Results of previous works are copied from [47]. ∗ indicates our modified implementation. † : The running time is measured on a single Nvidia RTX A5000 GPU, and we only report those run by ourselves. When using RoomFormer to produce the initial proposals, the running time counts both the RoomFormer proposal generator and the GS-DM.

Quantitative & qualitative evaluation : Table 1 presents the quantitative evaluation results. We tried two proposal generators: 1) the improved RoomFormer and 2) rough annotations. The rough annotations are prepared by turning the ground-truth data into fixed-radius small circles centered

RoomFormer

+PolyDiffuse

Input image

G.T.

Rough annot.

+PolyDiffuse

Likelihood-based refinement

NLL = -9.02

NLL = -9.49

NLL = -9.02

1 room

2 rooms

NLL = -9.81

1 room

NLL = -8.17

NLL = -8.42

at the instance centroid (see Figure 4), which simulates what a human annotator can provide easily ( i.e ., indicating instance center and the number of vertices by looking at images). When using RoomFormer as the proposal generator, PolyDiffuse outperforms all competing methods by clear margins across all the evaluation metrics. The performance gap increases with the difficulty level of the metric, indicating that PolyDiffuse is superior in high-level geometry reasoning. With the rough annotations as the initial proposal, PolyDiffuse also produces reasonable reconstructions. The recalls are especially high because of the correct number of vertices. Note that our training is not dependent on any specific proposal generator, but PolyDiffuse works well with different generators at test time. Figure 4 presents concrete qualitative examples of how PolyDiffuse improves the initial reconstruction and enforces better geometry correctness. We also provide more qualitative examples in Appendix D.3.

Running speed analysis : Table 1 also presents the speed-performance tradeoff of PolyDiffuse on the floorplan reconstruction task. During the denoising process, the image encoding parts of the denoising network only run once, while the transformer decoder part runs for multiple rounds. In our computation environment, the time of image encoding vs. the time of transformer decoder is 2:1. Since speed is not a crucial factor for floorplan reconstruction, we can just use more denoising steps in real applications for better reconstruction quality.

Likelihood-based refinement : Figure 4(right) demonstrates how we leverage the likelihood evaluation to refine reconstruction results with minimal user interaction. We manually indicate the imperfect polygons (inside dashed boxes at the left), then run our system with different numbers of vertices for these polygons while evaluating the reconstruction likelihood. The result with the best negative log-likelihood (NLL) per dimension is the refined reconstruction.

5.2 HD map reconstruction

Dataset and metrics : The nuScenes dataset [2] provides a standard benchmark for HD map reconstruction. The dataset contains 1000 ego-centric sequences collected by autonomous vehicles with rich annotations. The data is annotated at 2Hz. Each sample has 6 RGB images and LiDAR sweeps captured by onboard sensors, covering the 360 ◦ horizontal FOV. In the top-down Bird-eye-view (BEV) space, the perception range is [ -15 m, 15 m ] for X-axis and [ -30 m, 30 m ] for Y-axis. The map includes three categories of elements: pedestrian crossing, divider, and road boundary. For fair comparisons, we use the same dataset split and pre-processing setups as previous works [22, 23, 25], and also represent each polyline with 20 uniformly-interpolated vertices. Similar to MapTR [23], we only use the RGB images as the sensor inputs in our experiments.

We follow the common evaluation protocol from the literature [23, 25], which employs the average precision (AP) with the Chamfer distance as a matching criterion. The AP is averaged over three

distance thresholds: { 0 . 5 m, 1 . 0 m, 1 . 5 m } , and the mean average precision (mAP) is obtained by averaging across the three classes of map elements. However, the Chamfer distance does not consider the vertex order and vertex matching between prediction and G.T., thus failing to measure the structured correctness. As directional information is critical for autonomous vehicles, we propose to augment the matching criterion with an extra order-aware angle distance. Concretely, we find the optimal vertex matching between a prediction and a G.T. with the smallest coordinate distance, then trace along the two polylines/polygons to compute the average angle distance (in degrees). A prediction is considered a true positive only when satisfying both the Chamfer and angle distance thresholds. We augment the three-level thresholds to { (0 . 5 m, 5 ◦ ) , (1 . 0 m, 10 ◦ ) , (1 . 5 m, 15 ◦ ) } . Results are reported in both the original and augmented metrics. We provide complete implementation details and visualizations about the augmented metric in the Appendix C.

Competing approaches : We compare our method with VectorMapNet [25] and MapTR [23]. VectorMapNet [25] employs an auto-regressive transformer decoder to generate the vertex of polygon/polyline one by one, conditioned on the image inputs. MapTR [23] is our proposal generator and is a DETR-style hierarchical detection method that directly estimates the polygon/polyline as a set of vertex sequences. MapTR and RoomFormer are technically very similar but focus on different tasks.

Quantitative & qualitative evaluation : Table 2 presents the quantitative evaluation results. PolyDiffuse does not significantly improve the Chamfer-distance mAP of MapTR, as it does not discover new instances. However, it shows clear advantages in the order-aware angle distance, a metric that is indicative of high-level structural regularities. Figure 5 qualitatively shows that PolyDiffuse accurately reconstructs curves with greater resemblance to the ground truth. We also experimented with the same rough annotations as in the floorplan task, and the quantitative results show that the mAP becomes lower compared to using MapTR as the proposal generator. By analyzing the results with rough annotations more carefully, we found the AP with the lowest threshold becomes much worse, while the one with the highest threshold improves. We believe this is because the guidance networks are trained with 'G.T. proposals', where the rough annotations ( i.e ., fixed radius circles) have a very different curve style when serving as the proposals, thus producing inaccurate initial Gaussians and stepwise guidance. This is a potential limitation of the current method. We provide additional qualitative examples in Appendix D.3.

Running speed analysis : Table 2 also presents our speed-performance tradeoff on the HD map reconstruction task. In our computation environment, the time of image encoding vs. the time of transformer decoder is 4:1 when using MapTR's model architecture. As FPS is an important consideration for online applications, we need to pick the number of denoising steps carefully. Furthermore, since PolyDiffuse is not restricted to a specific task-specific method for the model

Surrounding views

G.T.

MapTR

+PolyDiffuse

Rough annot.

+PolyDiffuse

Ped. crossing

Divider

Boundary

MNNN

architecture and proposal generator, our performance/efficiency can keep improving when better task-specific base methods appear in the future.

5.3 Ablation studies

We provide key ablation studies with the floorplan reconstruction benchmark. Please also find additional ablation studies in Appendix D.1.

Standard DM vs. GS-DM : To better understand the effectiveness of our GS-DM, we define a standard DM using a vanilla DDPM with images as the condition. The standard DM does not have the guidance network and proposal generator. The reverse process starts with a noise sampled from the standard Gaussian distribution and gradually denoises it into the final reconstruction using the same sampler as GS-DM. It also has the same denoising network architecture as the GS-DM of PolyDiffuse. As shown in Table 3, although standard DM produces reasonable results with 10 sampling steps, it does not outperform the baseline method RoomFormer. Also, the performance of standard DM drops significantly with fewer sampling steps, as the denoising ambiguity hurts the performance more seriously with a larger sampling step size. We also demonstrate in Appendix D.2 that the standard DMcan fail in a toy experiment due to denoising ambiguity.

Sampling steps : Unlike the standard DM, Table 1, Table 2, and Table 3 all suggest that PolyDiffuse is robust to fewer sampling steps and can still slightly improve RoomFormer or MapTR with a very small number of sampling steps.

Training epochs : We observed that the model converges slower under the denoising formulation than the detection/regression formulation of previous approaches. In Table 4, PolyDiffuse keeps improving when training with longer schedules, while RoomFormer suffers from overfitting. However, even with the default schedule, PolyDiffuse significantly outperforms the baseline.

6 Related Work

Structured reconstruction of polygonal shapes : Structured geometry reconstruction is an active research area in computer vision, focusing on converting raster sensor data into vectorized geometries such as room layouts [8, 13], floorplans [1, 24], planes [11, 37], wireframes [16], etc . Taking advantage of the advancements in deep learning, methods with end-to-end neural architecture [7, 23, 25, 47, 49, 51] gradually dominate the area with superior accuracy and efficiency. In this paper, we formulate the reconstruction of polygonal shapes as a conditioned generation process with diffusion models, one of the essential machinery of the recent exploding generative AI. Our

Table 4: Ablation study on the training schedule of PolyDiffuse and RoomFormer. Results are reported in F1 scores.

method, PolyDiffuse, outperforms the current state of the arts on floorplan [47] and HD map [23] reconstruction while enabling broader practical use cases.

Diffusion-based generative models : Diffusion Models (DM) or score-based generative models [28, 38, 40, 41] have made tremendous progress in the last few years and demonstrated promising performance in content generations [9, 31, 32, 33, 43] and likelihood estimations [19, 20, 28]. In this paper, we explore the potential of DM in the context of structured reconstruction and propose a Guided Set Diffusion Model (GS-DM) by extending the DDPM [14] formulation. GS-DM reconstructs complex polygonal shapes with a guided denoising (reverse) process subject to sensor data and can evaluate its reconstruction likelihood by leveraging the underlying probability flow ODE [41]. Our high-level formulation is relevant to PriorGrad [21], a diffusion model for acoustic data, but has essential differences - GS-DM learns the guidance networks to control the diffusion and direct the denoising in a per-element and stepwise manner, while PriorGrad pre-computes the mean and variance of the target Gaussian from the condition and directly shifts the diffusion/denoising trajectory.

7 Conclusion

This paper introduces PolyDiffuse, a system designed to reconstruct high-quality polygonal shapes from sensor data via a conditional generation procedure. At the heart of PolyDiffuse is a novel Guided Set Diffusion Model, that controls the noise injection in the diffusion process to avoid permutation ambiguity for denoising. PolyDiffuse achieves state-of-the-art performance on two challenging tasks: floorplan reconstruction from a point density image and HD map construction from onboard RGB images. To our knowledge, this paper is the first to demonstrate that Diffusion Models, generally regarded as a generation technique, are also powerful for reconstruction, potentially encouraging the community to further investigate the effectiveness of Diffusion Models in broader domains.

Limitations : PolyDiffuse suffers from two major limitations. First, it does not recover geometry instances (e.g., rooms for floorplan) missing in the initial reconstruction, while the likelihood-based refinement provides a possible solution by incorporating search or external inputs. Second, as shown in the HD mapping results with rough annotations, when the style of initial reconstructions ( e.g ., fixed radius small circles) is different from the curve style of ground truth, the guidance networks could produce bad initial Gaussians and guidance, leading to inaccurate locations of the final results. Training specific guidance and denoising networks for each type of initial reconstruction is a solution but induces more computation.

Broader impact : The paper benefits applications in architecture, construction, autonomous driving, and potentially human-in-the-loop annotations systems to reduce human workload. However, potential malicious or unintended applications might include military scenarios, for example, reconstructing digital models of important buildings or structures for targeted military operations.

Acknowledgements : This research is partially supported by NSERC Discovery Grants with Accelerator Supplements and DND/NSERC Discovery Grant Supplement, NSERC Alliance Grants, and John R. Evans Leaders Fund (JELF). We thank the Digital Research Alliance of Canada and BC DRI Group for providing computational resources.

References

Ricardo Cabral and Yasutaka Furukawa. Piecewise planar and compact floorplan reconstruction from images. In 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2014.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In 16th European Conference of Computer Vision (ECCV) , 2020.

Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research , 11(3), 2010.

Jiacheng Chen, Chen Liu, Jiaye Wu, and Yasutaka Furukawa. Floor-sp: Inverse cad for floorplans by sequential room-wise shortest path. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019.

Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021.

Jiacheng Chen, Yiming Qian, and Yasutaka Furukawa. Heat: Holistic edge attention transformer for structured reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022.

Erick Delage, Honglak Lee, and Andrew Y Ng. A dynamic bayesian network model for autonomous 3d reconstruction from a single indoor image. In IEEE computer society conference on computer vision and pattern recognition (CVPR) , 2016.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-semantic embeddings with hard negatives. BMVC , 2018.

Yasutaka Furukawa, Brian Curless, Steven M Seitz, and Richard Szeliski. Manhattan-world stereo. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 1422-1429. IEEE, 2009.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Varsha Hedau, Derek Hoiem, and David Forsyth. Recovering the spatial layout of cluttered rooms. In IEEE 12th international conference on computer vision (ICCV) , 2009.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Similarity-Based Pattern Recognition: Third International Workshop, SIMBAD 2015, Copenhagen, Denmark, October 12-14, 2015. Proceedings 3 . Springer, 2015.

Kun Huang, Yifan Wang, Zihan Zhou, Tianjiao Ding, Shenghua Gao, and Yi Ma. Learning to parse wireframes in images of man-made environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 626-635, 2018.

Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. CVPR , 2023.

Pradeep Kumar Jayaraman, Joseph G Lambourne, Nishkrit Desai, Karl DD Willis, Aditya Sanghi, and Nigel JW Morris. Solidgen: An autoregressive model for direct b-rep synthesis. TMLR , 2023.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems (NeurIPS) , 2021.

Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie-Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior. ICLR , 2021.

Qi Li, Yue Wang, Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In 2022 International Conference on Robotics and Automation (ICRA) , pages 4628-4634. IEEE, 2022.

Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construction. ICLR , 2023.

Chen Liu, Jiajun Wu, Pushmeet Kohli, and Yasutaka Furukawa. Raster-to-vector: Revisiting floorplan transformation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , 2017.

Yicheng Liu, Yue Wang, Yilun Wang, and Hang Zhao. Vectormapnet: End-to-end vectorized hd map learning. ArXiv , abs/2206.08920, 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927 , 2022.

Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International Conference on Machine Learning (ICML) , 2020.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML) , 2021.

Wamiq Para, Shariq Bhat, Paul Guerrero, Tom Kelly, Niloy Mitra, Leonidas J Guibas, and Peter Wonka. Sketchgen: Generating constrained cad sketches. Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark , 7(15):510, 2008.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-toimage diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Matthew Schultz and Thorsten Joachims. Learning a distance metric from relative comparisons. Advances in neural information processing systems (NeurIPS) , 2003.

Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P Adams. Vitruvion: A generative model of parametric cad sketches. ICLR , 2022.

Mohammad Amin Shabani, Sepidehsadat Hosseini, and Yasutaka Furukawa. Housediffusion: Vector floorplan generation via a diffusion model with discrete and continuous denoising. CVPR , 2023.

Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. ECCV , 2012.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML) , 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ArXiv , abs/2010.02502, 2020.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems (NeurIPS) , 2019.

Yang Song, Jascha Narain Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR , 2021.

Sinisa Stekovic, Mahdi Rad, Friedrich Fraundorfer, and Vincent Lepetit. Montefloor: Extending mcts for reconstructing accurate large-scale floor plans. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021.

Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-aware diffusion. ArXiv , abs/2307.01097, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems (NeurIPS) , 2017.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 2011.

Yifan Xu, Weijian Xu, David Cheung, and Zhuowen Tu. Line segment detection using transformers without edges. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021.

Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Connecting the dots: Floorplan reconstruction using two-level queries. CVPR , 2023.

Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In ECCV , 2020.

Yichao Zhou, Haozhi Qi, Yuexiang Zhai, Qi Sun, Zhili Chen, Li-Yi Wei, and Yi Ma. Learning to reconstruct 3d manhattan wireframes from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 2020.

Chuhang Zou, Alex Colburn, Qi Shan, and Derek Hoiem. Layoutnet: Reconstructing the 3d room layout from a single rgb image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2051-2059, 2018.

Appendix:

PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models

This appendix is organized as follows:

§A: The derivation of the Guided Set Diffusion Models based on the DDPM [14] formulation.

§B: Additional implementation details, including:

-Architectures and implementation details of the guidance networks.

-Architectures and implementation details of the denoising networks.

-Implementation details of the permutation loss L perm ( ϕ ) for the guidance training.

-Implementation details of the diffusion models framework.

§C: More details and analyses on the augmented matching criterion for the mean AP metric to evaluate HD map reconstruction.

§D: Extra experimental results including:

-Additional ablation studies on the design choices of the denoising networks.

-Additional qualitative results for floorplan and HD map reconstruction.

A Derivation of Guided Set Diffusion Models (GS-DM)

In this section, we present more details of the derivation of the Guided Set Diffusion Models (GS-DM) proposed in §3 of the main paper.

A.1 Forward process

We derive Eq. 7, 8, and 9 in Sec. 3 of the main paper by induction. The trivial case of t = 1 can be easily verified. Let x i t -1 = √ ¯ α t -1 x i 0 + ¯ µ ϕ ( x 0 , t -1 , i ) + ¯ σ ϕ ( x 0 , t -1 , i ) ϵ i t -1 for t > 1 and ϵ i t -1 ∼ N ( 0 , I ) . By definition, we have

where ϵ i t is a standard Gaussian variable independent of ϵ i t -1 . Since ϵ i t -1 and ϵ i t are independent, √ α t ¯ σ ϕ ( x 0 , t -1 , i ) ϵ i t -1 + √ 1 -α t σ ϕ ( x 0 , t, i ) ϵ i t is a Gaussian random variable of zero mean and a variance of α t ¯ σ 2 ϕ ( x 0 , t -1 , i ) + (1 -α t ) σ 2 ϕ ( x 0 , t, i ) . Thus we have

which correspond to Eq.8 and Eq.9 of the main paper, and we further obtain

which is the Eq.7 of the main paper.

A.2 Reverse process

Assuming we are given the guidance networks µ ϕ and σ ϕ and the noise scales σ t s of the reverse process, we follow a style similar to DDPM [14] to derive a sampling step in the reverse process. At each sampling step in the reverse process, p θ ( x i t -1 | x i t ) , the Gaussian distribution to sample x i t -1 from is supposed to have the same mean as q ( x i t -1 | x i t , x 0 ) defined by the forward process, which is also Gaussian, to minimize their KL divergence. As the joint distribution of x i t -1 and x i t conditioned on

x 0 is Gaussian, the mean of q ( x i t -1 | x i t , x 0 ) can be easily derived with the following closed form [30, Chapter 8.1.3]:

Since i √ i i i

As our formulation directly parameterizes ¯ σ ϕ , we further simplify Eq. 20 by defining ¯ σ ϕ ( x 0 , t, i ) := √ 1 -¯ α t C ( x 0 , i ) where C ( x 0 , i ) is independent of the timestep t , thus implicitly defining σ ϕ ( x 0 , t, i ) = ¯ σ ϕ ( x 0 , T, i ) = C ( x 0 , i ) . Such a parameterization makes { ¯ σ ϕ ( x 0 , t, i ) } t simply an interpolation between 0 and ¯ σ ϕ ( x 0 , T, i ) and further simplifies Eq. 20 as

During inference, we replace ¯ µ ϕ ( x 0 , t, i ) and ¯ σ ϕ ( x 0 , t, i ) with ¯ µ ϕ (ˆ x 0 , t, i ) and ¯ σ ϕ (ˆ x 0 , t, i ) respectively and replace ϵ i with the denoising network ϵ i θ to define the mean of sampling distribution p θ ( x i t -1 | x i t ) and derive a sampling step in the reverse process as

which is the Eq.10 of the main paper.

B Additional implementation details

In this section, we complement §4 of the main paper by providing the complete implementation details of PolyDiffuse for the two different tasks.

B.1 Guidance networks

The implementation details of the guidance networks are shared across the two tasks.

Model architectures : In §4 of the main paper, we parameterize ¯ µ ϕ ( x 0 , T, i ) and ¯ σ ϕ ( x 0 , T, i ) with two Transformers Trans ¯ µ ϕ ( x 0 , i ) and Trans ¯ σ ϕ ( x 0 , i ) , and define ¯ µ ϕ ( x 0 , t, i ) and ¯ σ ϕ ( x 0 , t, i ) with Eq.14. The two Transformers are implemented with a DETR-style [3] Transformer decoder 2 : The Transformer decoder contains two shared attention-based decoder layers and two separate linear projection heads for ¯ µ ϕ and ¯ σ ϕ , respectively. Each decoder layer consists of an intra-element selfattention layer and a global self-attention layer, whose outputs are fused by addition. Each vertex in the input x 0 becomes an input node of the Transformer decoder, and the node feature consists of three positional encodings [44]: 1) positional encoding of the X-axis coordinate, 2) positional

encoding of the Y-axis coordinate, 3) positional encoding of the vertex index inside the element. The sequence of each element starts with a special dummy node (similar to the SOS token in language modeling), whose final encodings are used to compute the ¯ µ ϕ and ¯ σ ϕ of the element. The dimension of the positional encoding is 128, and the hidden dimension of the Transformer decoder is 256.

Training details : The guidance training is summarized by Algorithm 1 of the main paper. We train the Transformer decoder for 3125 iterations with batch size 32. Adam optimizer is employed with a learning rate of 2e-4 and a weight decay rate of 1e-4.

B.2 Denoising networks

Since the implementations of the denoising networks are based on the corresponding state-of-the-art task-specific models, we describe them separately.

Floorplan reconstruction : The denoising network for the floorplan reconstruction task refers to RoomFormer [47] and its official implementation 3 . RoomFormer is a DETR-based [3] model consisting of a ResNet50 [12] image backbone, a Transformer encoder to process and aggregate image information, and a Transformer decoder with two-level learnable embeddings/cooridnates to predict a set of polygon vertices from the image information. The Transformer has 6 encoder layers and 6 decoder layers with an embedding dimension of 256, and deformable-attention [50] is employed for all cross-attention layers and the self-attention layers in the Transformer encoder.

We follow the same architecture as RoomFormer, and have discussed the key modifications to turn the model into a denoising function (§4 of the main paper) and our improvements to lift up its overall performance (§5 of the main paper). A minor modification omitted in the main paper is that we add an intra-element attention layer to each Transformer decoder layer to increase the model capacity, as we found the model converges obviously slower under the denoising formulation than the regression/detection formulation of RoomFormer. In each decoder layer, the outputs of the intra-element attention are added to the outputs of the original global attention. The intra-element attention increases the number of overall parameters by ~5%, and the corresponding ablation study is in §D.1 of this appendix, showing minor but recognizable improvements.

For training the denoising network, we keep the same setups as RoomFormer except that we increase the number of training iterations (ablation study is in §5 of the main paper). Adam optimizer is employed with a base learning rate of 2e-4 for all parameters, and the learning rate decays by a factor of 0.1 for the last 20% iterations.

HD map reconstruction : The denoising network for the HD map reconstruction task refers to MapTR [23] and its official implementation 4 . The overall design of MapTR is very similar to RoomFormer, where the keys are the 'hierarchical' or 'two-level' query embeddings for DETR-style Transformer and a loss based on 'hierarchical bipartite matching'. We use the same model config file provided in MapTR's official codebase and convert the model into a denoising function as described in §4 of the main paper. Similar to what we did to RoomFormer above, we add an intra-element attention layer to each Transformer decoder layer to facilitate convergence.

We employ an Adam optimizer with a base learning rate of 6e-4 and a weight decay factor of 1e-4. A cosine learning rate scheduler is used. To further facilitate convergence under the denoising formulation, we load the ResNet image backbone and Transformer encoder from the pre-trained MapTR, and set the initial learning rate of the image backbone to be 0.1 of the base learning rate.

When using MapTR as the proposal generator to produce the initial reconstruction for PolyDiffuse, we only consider predicted instances with a confidence score higher than 0 . 5 as MapTR's positive predictions. PolyDiffuse takes and updates the positive predictions of MapTR and keeps the remaining low-confidence predictions unchanged. Only the positive predictions are visualized for the qualitative results of MapTR and PolyDiffuse.

B.3 Permutation loss

Directly computing the L perm ( ϕ ) as defined in Eq.14 of the main paper requires finding x ∗ 0 = arg max x ′ 0 ∈P ! ( x 0 ) \{ x 0 } L Triplet ( x t , x 0 , x ′ 0 ) and x ∗ t = arg max x ′ t ∈P ! ( x t ) \{ x t } L Triplet ( x 0 , x t , x ′ t ) , where the size of P ! ( x 0 ) and P ! ( x t ) is N ! . This enumeration over all N ! permutations of x 0 and x t immediately becomes computationally prohibitive when N gets large.

To reduce the computational cost and make Eq.14 practically feasible, we propose to approximate L perm ( ϕ ) with element-level triplet losses. Concretely, we first enumerate all pairs of elements x i 0 and x j t for i, j = 1 , . . . , N to compute a N × N distance matrix D . The entry D ( i, j ) is the minimum distance between two elements x i 0 and x j t , considering all possible vertex-level permutations ( i.e ., two variants for a polyline, and 2( N i -1) variants for a polygon with N i vertices, similar to the 'point-level matching' in MapTR [23]). We then replace the sample-level triplet loss in L perm ( ϕ ) with N element-level triplet losses, and define the computationally feasible proxy loss ˆ L perm ( ϕ ) as:

̸

̸

α is the soft margin hyperparameter of the hinge-style triplet loss [15, 34] and we set α = 0 . 1 . All coordinate values are re-scaled into [ -1 , 1] . In this way, we reduce the computational cost from O ( N !) to O ( N 2 M ) , where M = max N i =1 N i is the maximum number of vertices of an element of x 0 . In practical implementation, the guidance training (Algorithm 1 of the main paper) uses ˆ L perm ( ϕ ) rather than L perm ( ϕ ) .

B.4 Diffusion models framework

Karras et al.[19] (EDM) presents a general diffusion model framework, where DDPM [14] and SDEbased DM formulations from Song et al.[41] can all be viewed as specializations of the proposed framework. We borrow its official codebase 5 to implement our GS-DM as it provides a general and clean base implementation suitable for all DM-based formulations. We then describe how we adapt the GS-DM into the EDM-based framework and list the relevant hyperparameter settings. This subsection follows the notations in Karras et al., where y is the data sample, n is the sampled Gaussian noise, x is the noisy sample, σ is the noise level (equivalent to the timestep), while c skip ( σ ) , c in ( σ ) , c out ( σ ) , and c noise ( σ ) are the preconditioning factors [19, Section 5]. Similar to the notations of our main paper, we let y i and x i denote the i th element of y and x , respectively. The sensor condition is omitted for notation simplicity.

To adapt our GS-DM into the EDM framework, we first set σ data = 1 . 0 for EDM [19, Table 1], and then adapt the preconditioning equation [19, Section 5, Eq.7] based on §3 of our main paper:

where the per-element noise injection is defined as x i = y i + n ¯ σ ϕ ( y , σ, i ) , and noise n ∼ N ( 0 , σ 2 I ) . D θ ( x i ; σ, y ) is the reconstructed y i . F θ is the denoising network taking all elements of a noisy sample x , and F i θ is the output of the i th element. With the above modifications, we implement our GS-DM with the general EDM framework. We then describe the concrete hyperparameter settings [19, Table 1] for our two tasks. Please refer to Karras et al. for detailed explanations of each hyperparameter.

Floorplan reconstruction : For the guidance training, we set the P mean = 1 . 0 and P std = 4 . 0 to ensure sufficient coverage of the forward process. For the denoising training, we set the P mean = -0 . 5 and P std = 1 . 5 . For inference (sampling), we set σ max = 5 . 0 and σ min = 0 . 01 . Instead of the 2 nd order Heun ODE solver adopted by the original EDM, we simply employ the 1 st order Euler solver for sampling, which is equivalent to DDIM [39]. All other hyperparameters are kept unchanged.

HD map reconstruction : The settings of P mean and P std for guidance and denoising training are the same as the floorplan reconstruction task. We set c skip ( σ ) = 0 and c out ( σ ) = 1 so that the denoising

G.T.

CD matching

+OAD matching

G.T.

CD matching

+OAD matching

G.T.

CD matching

+OAD matching

G.T.

CD matching

+OAD matching

G.T.

CD matching

+OAD matching

G.T.

CD matching

+OAD matching

network directly estimates the sample y , as we found this choice stabilizes the denoising training on the HD map construction task. For inference (sampling), we set σ max = 5 . 0 and σ min = 0 . 1 , and use the Euler solver. All other hyperparameters are kept unchanged.

C The augmented matching criterion for HD map reconstruction

In §5.2 of the main paper, we augment the Chamfer-distance (CD) matching criterion for the mean AP (mAP) metric used in previous works [23, 25] with an order-aware angle distance (OAD) to better evaluate the structural regularity and directional correctness of the results. This section first discusses the limitations of the original Chamfer-distance matching criterion and then provides complete implementation details of the order-aware angle distance.

C.1 The limitations of the Chamfer-distance matching criterion

Figure 6 shows examples of the Chamfer-distance (CD) matching (threshold is 1 . 0 m ). Since CD considers each vertex separately, it mostly evaluates the global location of the instance and ignores the structural/directional information of the prediction. In the figure, some predictions with obviously wrong shapes are considered true positives under the CD matching criterion, while the augmented OADmatching criterion can effectively reject these bad shapes and align better with human judgment.

C.2 Implementation details

The computation of the order-aware angle distance is based on an optimal vertex-level matching between a predicted element (instance) and a ground truth (G.T.) element, similar to the 'point-level matching' in the loss computation of MapTR [23]. Concretely, we enumerate through all equivalent representations of a G.T. map element and compute the average vertex-level L 1 distance between the G.T. vertices and the predicted vertices. The variant with the smallest average vertex-level L 1 distance forms the optimal matching with the predicted element. There are two equivalent variants for a polyline and 2( N i -1) equivalent variants for a polygon with N i vertices. After obtaining the optimal one-to-one vertex-level matching, we trace along the vertices of the G.T. and predicted elements to compute their average angle distance.

With the augmented OAD matching criterion, we change the three-level thresholds of the original CDbased AP from { 0 . 5 m, 1 . 0 m, 1 . 5 m } into { (0 . 5 m, 5 ◦ ) , (1 . 0 m, 10 ◦ ) , (1 . 5 m, 15 ◦ ) } . However, compared to polylines ( i.e ., road dividers and boundaries), we noticed that the angle direction of polygons (pedestrian crossings) is much more challenging to recover. With an OAD threshold of 5 ◦ , MapTR's average precision for the pedestrian crossing class (AP p ) is zero. Therefore, we loosen the OAD thresholds for the pedestrian crossing class by a factor of 2 ( i.e ., { (0 . 5 m, 10 ◦ ) , (1 . 0 m, 20 ◦ ) , (1 . 5 m, 30 ◦ ) } ) while not changing the thresholds for the other two classes.

D Additional experimental results

This section provides additional experimental results, including extra ablation studies and qualitative comparisons.

D.1 Additional ablation studies

Table 5 presents the ablation study of the intra-element attention with the floorplan reconstruction task. Comparing the first and second rows, adding intra-element attention to RoomFormer slightly improves the room and corner results but deteriorates the angle-level performance. Table 6 provides a similar comparison for the HD map reconstruction task. Augmenting MapTR with intra-element attention worsens the mAP with the CD matching criterion while slightly improving the mAP with the OAD-augmented matching criterion. These comparisons provide a potential empirical explanation for the architecture choice of RoomFormer and MapTR - they only employ global attention layers without extra attention layers at the element or instance level.

On the contrary, as indicated by the third and fourth rows of both Table 5 and Table 6, applying intraelement attention consistently boosts PolyDiffuse across all metrics, although the improvements are not huge. A potential explanation here is that the denoising task of PolyDiffuse is more challenging than the regression/detection task of RoomFormer and MapTR, thus benefiting from extra modeling capacities.

D.2 A toy experiment with standard DM

In Figure 7, we provide a toy experiment to support what we motivated in §1 of the main paper, which demonstrates how standard DM easily fails even with a single data sample. Note that we have clarified the definition of standard DM in §5.3 of the main paper. In this experiment, the data contains a single toy sample with 6 rectangular shapes, so there are permutation-equivalent representations. After sufficient training, we draw four samples using the image-conditioned denoising process. The DDIM sampler is used with 10 sampling steps, so the randomness only comes from the initial noise. As the figure shows, only the third sample gets the correct reconstruction result. With the challenges of set ambiguity, a standard conditional DM has trouble overfitting a single data sample and easily gets wrong outputs when the initial noise is inappropriate.

Toy input image

There are 6 elements, and the G.T. has 6! = 720 permutation-equivalent representations

Sampling results of Standard DM (DDIM sampler, 10 steps)

Sample 1

Sample 2

Sample 3

Sample 4

t = 0

t = 10

⋯

⋯

t = 0

t = 10

⋯

⋯

t = 0

t = 10

⋯

⋯

t = 0

t = 10

⋯

⋯

8

8

D.3 More qualitative examples

We provide additional qualitative results to compare PolyDiffuse against the state-of-the-art floorplan and HD map reconstruction methods, respectively. Note that the state-of-the-art method ( i.e ., RoomFormer or MapTR) produces the initial reconstruction for PolyDiffuse. Figure 8 to Figure 10 are for the floorplan reconstruction task, while Figure 11 to Figure 13 are for the HD map reconstruction task.

Since the likelihood-based refinement is not employed in the qualitative examples of this subsection, PolyDiffuse cannot discover missing instances that are not covered by the proposal generator ( i.e ., RoomFormer and MapTR in the qualitative results). However, the visual comparisons clearly demonstrate that PolyDiffuse significantly improves the structural regularity of the reconstructed polygonal shapes.

RoomFormer

+PolyDiffuse

Input image

G.T.

RoomFormer

+PolyDiffuse

Input image

G.T.

口

+PolyDiffuse

Input image

G.T.

Boundary

Surrounding views

G.T.

MapTR

+PolyDiffuse

Ped. crossing

Divider

Boundary

Surrounding views

G.T.

MapTR

+PolyDiffuse

Ped. crossing

Divider

Boundary

Surrounding views

G.T.

MapTR

+PolyDiffuse

Ped. crossing

Divider

TWO

sonopolisT

INT

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a diffusion-based approach for reconstructing polygonal shapes from floorplan images and auto-drive sensor images. The idea is to refine an initial rough reconstruction through iterative denoising steps which revert a forward process that diffuses regular reconstructions into random noises through per-element noise injection (though the notions of elements are not clearly given for different tasks).  The challenge with learning such a denoising diffusion model, as identified by the paper, is the robustness to permutations of orders of elements that are introduced by linearization (though it's not convincing why a set-based transformer model would introduce such ordering problem inevitably). The paper proposes to address this challenge by learning a guided forward diffusion process, such that the diffusion paths for different permutations are encouraged to be separate. In test time, the rough initial reconstruction is first converted to corresponding target noise distributions by the learned guided diffusion networks, and then reverted back to an accurate reconstruction through iterative denoising. \n\nThe approach is tested on two polygonal shape reconstruction tasks, including floorplans and autonomous driving maps, and shows improvements over the baselines that produce initial reconstructions. Ablation studies are conducted on diffusion guidance and choices of several hyper parameters.\n\nSTRENGTHS:\nThe paper proposes to improve polygonal reconstruction works that produce results in a single network pass by iterative denoising of the reconstruction. It is thus expected that the results can have better quality than the initial estimates, which is confirmed by tests on two tasks.\n\nThe paper formulates the diffusion model for structured elements like polygons and polylines, and identifies the problem of ordering permutation in representing the structures, which may cause severe ambiguity for a network to learn the reverse process. \n\nThe paper designs a learned guided forward diffusion process to distinguish the different permutations causing representational ambiguity, and derives the updated diffusion models biased by the learned diffusions. Such adaptive and learned diffusion can be inspiring to other situations where hand-crafting a diffusion process can be problematic.\n\nWEAKNESSES:\nThe computational cost of the iterative process can be better illuminated, to help readers appreciate the comparative advantage obtained over the initial estimates produced by other methods. \n\nThe paper identifies another challenge with diffusion-based approach for reconstruction: a reconstruction task has a single solution, so the initial noise needs to be chosen carefully. I do not fully understand this statement, and cannot find any validation of this challenge in the experiments and discussions. On the other hand, the proposed guided diffusion process does not produce a single noise for an input polygonal shape either, but samples from the predicted Gaussian distribution. \n\nThe guided diffusion process is trained through contrastive learning on the initial reconstructions of training data by a different method. It is not clear how well the guided diffusion generalizes for data samples outside the training set. In contrast, a fixed diffusion procedure as adopted by DDPM and others does not have generalization issue, as the final noise distributions are known a prior. Indeed, the paper acknowledges that when the initial reconstructions come from hand annotations that are largely different from training reconstructions for HD Maps, the guided diffusion and denoiser do not work robustly. \n\nIt might be difficult to regard this work as the first one using diffusion models for reconstruction. In fact, diffusion models have been widely used for conditioned generation that resemble reconstruction, e.g. ControlNet. Also, people have explored using diffusion models for segmentation [1], which is a kind of reconstruction. \n\n[1] SegDiff: Image Segmentation with Diffusion Probabilistic Models.\n\nQUESTIONS:\n1. Computational costs of the iterative denoisers should be reported, and compared with the base cost of running the initial reconstructions.\n\n2. Please provide clarifications and experimental evidence that perturbations of the optimal noise for a particular reconstruction can cause failure of denoise-based reconstruction. \n\n3. In the footnote of page 3, it's stated that even a permutation-invariant model like Transformer has to pick a permutation as the representation of a data sample. This statement seems self-contradictory and needs clarification. For example, in DETR like works (including RoomFormer and MapTR) there is no ordering of the elements, and they rely on matching to compute the loss functions between two sets. Why cannot this approach to extended to the diffusion setting?\n\n4. When defining the permutation contrastive loss (12), why not compare the x_t and x'_t, but compare x_0 and x_t instead? Is it essential to constrain the similarity between x_0 the uncorrupted data and x_t the noised version? This is not common for standard diffusion models.\n\n5. Sec.4, feature representation, it's not clear how polylines/polygons of different vertex numbers are encoded and represented. This is an important detail as its encoding directly impacts how likelihood-based refinement (Fig4) can be done. \n\n6. The line matching metrics augmented with angle threshold is not defined clearly. There is no formal description of its computation in either text or supplemental. \n\n7. Define the standard DM model of Sec.5.3 clearly. Otherwise it's hard to justify any proposed component. \n\n8. What is training schedule? It looks like training epochs but why would one bother with the training iterations so much. \n\n9. Since it is acknowledged that the guided diffusion network may not generalize to hand-annotated input, I hope the authors can thoroughly discuss the limitations of the learned diffusion network, including how to decide if it's not general enough for a particular test dataset or input pattern, and what are the practical ways to enhance its generalization scope.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

comments: SUMMARY:
This paper proposes a Guided Set Diffusion Model for reconstruction, addressing the challenges of ambiguous denoising and selecting appropriate initial noise. By learning guidance networks, the model ensures distinct representations for samples with multiple permutations in structured geometry. During testing, the model uses the guidance networks to initialize Gaussian noise and denoises to reconstruct polygonal shapes conditioned on sensor data. The approach is evaluated on floorplans and HD maps, demonstrating significant advancements over existing methods and enabling practical applications.

STRENGTHS:
The paper takes a step towards formulating reconstruction as a generation process conditioned on sensor data, providing insights into advancing Diffusion Models for shape reconstruction. The approach of guiding noise injection per element to ensure separation from permutation variants during the diffusion process effectively addresses the ambiguous denoising issue. The PolyDiffuse method is well-presented and achieves promising results for layout reconstruction. Overall, this paper makes valuable contributions and is commendable.

WEAKNESSES:
While floorplan and HD map reconstruction are important problems, it appears that the GS-DM formulation is more suitable for layout or other semantic reconstruction tasks rather than real 3D shape reconstruction. It would be beneficial if the authors could provide a discussion on this aspect and given some suggestion of addressing the limitations of PolyDiffuse in extending to 3D shape reconstruction tasks.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

comments: SUMMARY:
This paper proposes a novel method for reconstructing multiple polygon shapes using a conditioned diffusion model. The method first learns a score-matching-based prior diffusion model from data. This model is then used to denoise the sensor data, resulting in the reconstruction of the polygon shapes. To handle the permutation order of the polygonal elements, the method separates the target Gaussian distribution of a sample from its permutation variants.

STRENGTHS:
1. A nice solution to handle the permutation order in the reconstruction of multiple geometric elements in conditioned diffusion model.
2. Comprehensive experiments to evaluate the performance of the proposed method.
3.This paper is well-written and easy to follow. Figure 2 provides a clear and concise illustration of the basic idea.

WEAKNESSES:
1. The maximum number of polygons and their vertices used in the training of the diffusion model should be clarified. Are these numbers consistent with the ones listed in lines 183-184?

QUESTIONS:
Overall, I like the idea of integrating a diffusion model as a prior into the reconstruction pipeline. However, I wonder if it is possible to handle the permutation order in a different way. For example, we could first train a network to predict the structure of the floorplan, such as the number of rooms and their basic bounding boxes. Then, we could denoise each shape individually. This approach might be easier to train and faster in inference.

LIMITATIONS:
The authors have discussed the limitations of the proposed method in Sec. 7.  I also want to know whether the denoising speed is fast enough for a real-time HD map reconstruction.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
- The paper introduces PolyDiffuse, a novel structured reconstruction algorithm that incorporates Guided Set Diffusion Models (GS-DM).
- The core concept involves splitting the reconstruction pipeline into two distinct stages.
   - The forward diffusion process focuses on learning guidance networks to address denoising ambiguity effectively.
   - The guidance networks establish individual Gaussians as target distributions for each polygon and are learned prior to the denoising training phase.
   - The reverse denoising process utilizes the sensor data as a condition to reconstruct polygonal shapes.
- The authors conduct extensive experiments involving two polygonal structure reconstruction tasks.
- Through comprehensive quantitative and qualitative evaluations, the results demonstrate that GS-DM outperforms existing state-of-the-art methods in terms of performance and quality of the reconstructions.

STRENGTHS:
- The paper is well-written and easy to follow.
- The derivation of the GS-DM is appropriately detailed.
- By employing a multi-stage approach and introducing guidance, the proposed method effectively resolves ambiguity in polygonal shape scenarios.
- The proposed method can evaluate reconstruction quality through likelihood evaluation.
- The proposed method performs well even when using only rough annotations as input.

WEAKNESSES:
The GS-DM may be sensitive to the specific guidance, as mentioned in Section 7 (Limitations).

QUESTIONS:
- It is worth exploring whether incorporating the condition $y$ as an input to the guidance network leads to performance improvements.
- What are the failure cases when applying PolyDiffuse to off-the-shelf methods? Is there further analysis regarding these cases?

LIMITATIONS:
The authors addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript introduces a adaptation of the DDPM paradigm to enable denoising sets (instead of single data points like images). The method called GS-DM does this by introducing by adding noise per set element via learned guidance networks.\nThis approach requires the addition of a proposal generator that initializes the number of sets and their parameters for the denoising process.\nThe manuscript applies this new set-based diffusion model to two tasks: generating floorplan polygons from floorplan images and maps from autonomous car images. The proposed method performs well in comparison to related work.\n\nSTRENGTHS:\nThe authors identify a core limitation in DDPMs when it comes to set-based data and propose a sound way that empirically works well to solve it. The paper is well written, figures are high quality and support the writing well.\nThe evaluation is thorough and clearly demonstrates that (1) the set-based diffusion is superior to the default DDPM (table 3) and (2) that the denoising approach helps improve state of the art results from MapTR and RoomFormer when applied on top of their final outputs. It is also very interesting to see how simple the inputs for denoising can be (as simple as the polygon centers) and still perform at almost state of the art levels.\n\nSince the underlying concept of set diffusion models is very general and could apply in various other domains, this work is significant to a larger audience than just the 3d reconstruction or perception community.\n\nWEAKNESSES:\nWhile the paper is well written it could use some more clarity around how the guidance network is learned(line 122ff) since this is one of the key novelties. Just from reading the text I would not be able to reproduce the method at this point.\n- How is the neares negative permutation found? It seems like this could be an intractable problem very quickly?\n- is the triplet loss computed based simply on the polygon corners (for example?)\n\nThe key limitation of the current approach is the fact that the number of elements in the set has to be initialized correctly by the proposal generator during inference. This is acknowledged by the authors and a reasonable limitation that can be addressed in follow on work imo.\n\nQUESTIONS:\n- In Fig3: is it mu/sigma _phi or _psi?\n\nLIMITATIONS:\nThe limitations are adequately addressed by the authors at the end of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}