PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models
Authors: Jiacheng Chen
Ruizhi Deng
Yasutaka Furukawa
Venue: NeurIPS 2023 poster (2023)
Keywords: Structured Reconstruction
Floorplan Reconstruction
HD Map Construction
Diffusion Models
TL;DR: This paper presents PolyDiffuse, a novel structured reconstruction algorithm that transforms visual sensor data into polygonal shapes with Diffusion Models (DM), an emerging machinery amid exploding generative AI.

Abstract: This paper presents \textit{PolyDiffuse}, a novel structured reconstruction algorithm that transforms visual sensor data into polygonal shapes with Diffusion Models (DM), an emerging machinery amid exploding generative AI, while formulating reconstruction as a generation process conditioned on sensor data. 
The task of structured reconstruction poses two fundamental challenges to DM: 1) A structured geometry is a ''set'' (e.g., a set of polygons for a floorplan geometry), where a sample of $N$ elements has $N!$ different but equivalent representations, making the denoising highly ambiguous; and 2) A ''reconstruction'' task has a single solution, where an initial noise needs to be chosen carefully, while any initial noise works for a generation task.
Our technical contribution is the introduction of a Guided Set Diffusion Model where 1) the forward diffusion process learns \textit{guidance networks} to control noise injection so that one representation of a sample remains distinct from its other permutation variants, thus resolving denoising ambiguity; and 2) the reverse denoising process reconstructs polygonal shapes, initialized and directed by the guidance networks, as a conditional generation process subject to the sensor data.
We have evaluated our approach for reconstructing two types of polygonal shapes: floorplan as a set of polygons and HD map for autonomous cars as a set of polylines.
Through extensive experiments on standard benchmarks, we demonstrate that PolyDiffuse significantly advances the current state of the art and enables broader practical applications. The code and data are available on our project page: https://poly-diffuse.github.io.

PolyDiffuse: Polygonal Shape Reconstruction via Guided Set Diffusion Models
Jiacheng Chen
Ruizhi Deng Simon Fraser University
Yasutaka Furukawa
Abstract
This paper presents PolyDiffuse, novel structured reconstruction algorithm that transforms visual sensor data into polygonal shapes with Diffusion Models (DM), an emerging machinery amid exploding generative AI, while formulating recon- struction as a generation process conditioned on sensor data. The task of structured reconstruction poses two fundamental challenges to DM: 1) A structured geometry is "set" (e-g., set of polygons for a floorplan geometry), where a sample of N elements has N! different but equivalent representations, making the denoising highly ambiguous; and 2) "reconstruction" task has a single solution, where initial noise needs to be chosen carefully, while any initial noise works for generation task. Our technical contribution the introduction of a Guided Set Diffusion Model where 1) the forward diffusion process learns guidance networks to control noise injection so that one representation a sample remains distinct from its other permutation variants, thus resolving denoising ambiguity; and 2) the reverse denoising process reconstructs polygonal shapes, initialized and directed by the guidance networks, data. We have evaluated our approach for reconstructing two types of polygonal shapes: floorplan set of polygons and HD) map for autonomous of polylines. Through extensive experiments on standard benchmarks, we demon- strate that PolyDiffuse significantly advances the current state of the art and enables broader practical applications. The code and data are available on our project page: httts://poly-diffit sealffuueegithueee tit.b...hhu..
conditional generation process subject to the sensor
set
1 Introduction
Reconstruction and generation were once separate research areas with minimal overlaps. While not acknowledged by the broader research communities, state-of-the-az reconstruction and generation techniques have now exhibited increasing similarities. Focusing on structured geometry (c.g., CAD models) in this paper, Auto-regressive Transformer (AR-Transformer) a family of state-of-the-art generative models that iteratively applies Transformer network to generate CAD construction the reconstruction side, Transformer network iteratively reconstructs and refines indoor floorplans, outdoor buildings models, and vectorized traffic maps 7 process resembling the generative Transfforrer..
sequences [8 29 35).
In 2023, Generative AI is experiencing significant growth, primarily driven by the emergence of
41 where structured geometry generation is not an exception. DM-based approach iteratively denoises coordinates (with associated properties) to generate realistic
Diffusion Models (DM) 14 38
40
design layouts [17 or floorplans (36]. natural question then "Is a Diffusion Model also good at structured reconstruction?" However, the answer is not that simple.
Take the high-defnnition (HD) map reconstruction problem for example (22] which aims recon- struct a set of polylines given sensor data in a bird's-eye view (see Figure 1). A straightforward DM formulation would assume maximum number of polylines (with a fixed number of corners per poly-
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Sensor inputs
Initial proposal
Sensor-conditioned Denoising Process of Guided Set Diffusion Model
Floorplan
Guidance networks
Sample
HD map
Guidance
networks
Sample
Ped. crossing
Divider
Boundary
CCO
Figure 1: PolyDiffuse for floorplan and HD map reconstruction. Starting from an initial proposal (e.g., from human annotator existing method), the censor-conditioned denoising process of our Guided Set Diffusion Model (GS-DM) generates shape reconstructions in a few sampling steps, initialized and directed by the guidance networks. The initial proposal above mimics simple human inputs that indicate rough locations and specify the number of vertices for the polygonal shapes.
line 23 25 ), take a particular permutation of the elements to construct a fixed-length vector corner coordinates, then iteratively denoise the vector subject to sensor data condition. The structured reconstruction task poses two fundamental chal- lenges to such standard DM formulations: 1) Structured geometry is often a "set" of elements (e.g., set of polylines), where the geometry of N elements has N! different but equivalent rep- resentations, making the denoising factorially ambiguous: 2) The denoising process needs to reach single solution (i.e., of N! represen- tations) for reconstruction task and an initial noise needs be set carefully, while any initial noise would work for generation task. /Figure 2 shows toy example of three elements with 6 = 3! permutations, where six permutation- equivalent representations the solution lie in six different sub-spaces. However, they all fol- low the standard Gaussian and become indis- tinguishable after the diffusion process, which obfuscates the denoising learning.
(a). t 0
(b). z T (standard DM)
Standard diffusion
Standard denoising
GS-DM diffusioo
GS-DM denoising
(c), ==7 (Ouns)
Figure 2: (a) A toy visualization for a sample with three elements, thus 6 different but equiva- lent points in the data space; (b) Standard DM at t t= T; (c) Our GS-DM
Our technical contribution is a Guided Set Diffusion Model (GS-DM), The forward diffusion process learns guidance networks controlling noise injection so that one representation of sample remains separated from its permutation variants, thus resolving denoising ambiguities. Specifically, the guidance networks set an individual Gaussian as the target distribution for each element (instead of fixed zero-meat unit-variance Gaussian for everything), and are learned before the denoising training by minimizing the permutation ambiguity as loss function. At test time, the reverse process initializes the element-wise Gaussian noise by the guidance networks and then denoises to reconstruct polygonal shapes conditioned on the sensor data, while the guidance networks take rough initial reconstructions either from an existing method or annotator (see Figure T for examples).
We have conducted extensive experiments on two polygonal structure reconstruction tasks: floorplan reconstruction from a point-cloud density image with Structured3D dataset [48), and HD map construction from onboard camera inputs with nuScenes dataset (2). PolyDiffuse augments the state-of-ther -oe--e--rttttttt approach of the two specific tasks and brings consistent performance improvements, especially on the geometric regularity of the reconstruction results. Our generation process takes to 10 steps to produce decent results and the visual encoder only runs once. We also demonstrate refinement extension of our system by exploiting the probability flow ODE [41), which enables broader practical applications. To our knowledge, this paper is the first to demonstrate that DM a powerful machinery for structured reconstruction tasks, in fact, the first in a broader domain of geometry reconstruction as well. We will share all our code and data.
likelihood-baal

---

2 Background: Denoising Diffusion Probabilistic Models
progressively inject noise to data in the forward (diffusion) process and generate data from noise by the reverse (denoising) process. The section provides the key equations on the denoising diffusion probabilistic models (DDPM) [4 that lay the foundation of our method.
Diffusion models score-based generative models 14
DDPM considers Markovian forward process that turns data Xo ((xo) into Gaussian noise:
((x/(x<-1) == ((xx v - 11 (.I)
t= ........ T and x; is the latent The noise schedule {..} is constant 14
28 or learned by
reparameterization [20]. The above forward process is written in closed form for an arbitrary t:
g(xe|xx) = N (x+i v axx (1 -ā+)
0g = 1, Oz = ar 10t, 04 == - Bt.
x, sampled by x, = X0,
a.)E for e (N I). DDPM parameterizes the reverse process with a noise prediction (or denoising) network Ea (X,, t) to make connections with denoising score matching and Langevin dynamics [40] (5), and the sampling step of the reverse process is derived as:
1 at āz
X
X
Eo (x1, 1) VI
+ o12, z N).
Org
of is set to 3r
1- 1-a--1 T-
Br. The final training objective is reweighted variational lower bound:
Issmpiee (0) = Exo,t. [[|e= E0 (v + v1 - oze, 1)]
(5)
3 Guided Set Diffusion Models (GS-DM)
Consider the task of generating a set of elements straightforward application of diffusion models would be to take one permutation of the elements form vector, then perform diffusion and learn denoising. With an abuse of notation, we let the vector X (i.e., the sample or denoised result in DM's notation) represent one particular permutation of x. The challenge is that set of N elements has N! different but equivalent representations, and any one of the N! is a valid Xu. making the denoising factorially ambiguous [ the above issue, our idea learns to guide the noise injection per-element manner by learning two "guidance networks", such that sample Xo remains separated from its permutation variants throughout the diffusion process. At test time, the guidance networks produce per-element Gaussians and stepwise guidance from an initial reconstruction, from which the denoising process generates the final reconstruction iteratively conditioned on sensor data. We first explain the forward and reverse processes and then the two-stage training for the guidance and denoising networks.
{xt,..... ,x.,..,,x') under condition y. A
To alleviate
Forward process: With the standard diffusion process, different permutation variants of sample gradually become indistinguishable through the diffusion process. To keep a particular representation Xy separated from its permutation variants, we propose to inject noise per element. Let x; denote the diffused element xj timestep t, we learn change the transition distribution (x)) xx{_)) by adding Ho (Xo, f, i) to the mean and multiplying a (Xo. t, i) to the standard deviation:
xo) = Ax(;; V 11-3,x(-- + 4,,, 3,0 (x0 1))
Ng and a take Xy and produce outputs for ith element at t. Following the similar derivations as in DDPM [14, we obtain (see Appendix for derivations):
((x/1xx) -N(x/; vārzh + (x t, i), (xo, i))
n. (xo. 0, O, oo t,) Varlts (xo, 1, + ps (xo, t, i), 2 (xo, 0,1) O. oz t) a. (xo. t 1, + (1 oz ) (x,,t,
(8)
(9)
'This permutation ambiguity issue persists with permutation-invaatie models (e.g.. Transformers), as it is inevitable pick particular permutation of x the conerete data representation Xo.
3

---

Algorithm Guidance training (stage 1)
Algorithm 2 Denoising training (stage 2)
1: d: trainable, 0: frozen
1: Φi frozen, 9: trainable
2: repeat
2: repeat
3: Xo ~V q(xo)
Xo g(xo)
4: >> Uniform((1,.....
t Uniform([,..... 7})
5: E ((0, I), then compute X
5: E 6 N(, I), then compute X
6: Take gradient descent step V guidd (Eg 13
6: Take gradient descent step on V Loimlle (Eg 11
7:
until converged
7:
until converged
As q(x] x)) only explicitly depends Ps and Ss and x .V (N i, o (xo, T, i)) we consider P& and a as the guidance networks. Ms and a are defined implicitly by Eq. [nd Eq. p To simplify the formulation, we define 5g (X0. t, := vI ā, (xo, i) where C(xo, i) is function independent of the timestep t, thus implicitly defining a (xo,,tt t, ox( I.
ix).
Note that X is the ground truth and only available for training. At test time, introduce a proposal generator to produce an initial reconstruction Ny, and run μ and o with xo- No can either be results from existing method rough annotations from human annotator. We first draw the element-wise initial noise with xi ~ .V (Vps (Ro, T, i), o (Xo, T, i)). and then run denoising steps iteratively. Similar to Eq
Reverse process:
the sampling step of the reverse process is derived as
1 x
Po (Xo, i, - δ (x 1 att aq
c ex, t.)) + ao(So, 1, + to,z' (10)
The denoising network eg takes the entire x, (i.e., all elements), and ep the output of the i element. Please see Appendix A.2 for derivations.
Learning the denoising network: Following Eq. of standard DDPM, the denoising objective is
N
Lsimple (f) = Exx,
Σ
||
((Vaxs + (xo, t) + ox(xx,
(11)
The above formulation from Eq to Eq 11 [1 resembles that of DDPM in 2 where the key difference
the introduction of guidance networks μ& and σ We then describe the training of the guidance networks, which is the key step of GS-DM.
we train js and ds ensure that x; in the diffusion process keep separated from other permutation variants of Xg. We quantify the permutation invariance between Xo and x, with triplet loss (t;;;) widely used in metric learning 4 34] (full implementation details are in Appendix
Learning the guidance network: Before training the denoising network with Eq. I1
Liperm (Ø)
X (x.) (xa] Triiplet (Xe, Xo, x(() +
max
max EX. (x.)
Lrriplet
(Xo, (Xo, X1, x()
(12)
(xxx)) is the set of all permutation variants of Xp- Using the terminology in metric learning, the three inputs of Trriptet (:,'.) are the anchor, positive, and negative, respectively. Our loss only considers the closest negative permutation, which is similar the hard negative mining in some variants of the triplet loss [6] 10). We also add two regularization terms on fe and o so that the means of all elements not too scattered and the variances do not vanish. The final loss for training the guidance networks is:
N
N
I Leguide(0) (Ø) = Er Ex.... I (0) + A2
Σ 2x0,
+ As
Σ (xx, t,0
(13)
Algorithm
and Algorithm summarize the two-stage training paradigm of GS-DM.
PolyDiffuse: Polygonal shape reconstruction via GS-DM
PolyDiffuse uses the GS-DM to solve polygonal shape reconstruction tasks (see Figure 3). in particu- lar, floorplan and HD map reconstruction. This section explains specific details and specializations.

---

Forward
F 0 +
Guidance
q (xr1xx)
Xg
Reverse
Proposal generator
&
8
F O
+
Sample
Guidunce networks
p(xr(80)
X
X
Figure 3: Illustration of the forward and reverse processes of PolyDiffuse with floorplan data.
Feature representation: A sample x is set of elements x' as defined in Each element x' is either a (closed) polygon or polyline, consisting of an arbitrary (for the floorplan task) or a fixed (for the HD map task) number of vertices as sequence: x' [u] ..i..... vN.. Each vertex contains 2D coordinate. For the ground-trus truth data, we determine the first vertex of each element by sorting based
on the Y-axis coordinate and then the X-axis coordinate, similarly PolyGen 27 The polygons always in a counter-clockwise orientation.
Guidance network: As derived in } the forward (Eq and reverse (Eq processes only depend on p and Φg. and each element follows x N (M (Xo, T) o (xo. I)) Therefore, we choose to directly parameterize a (x,, T, i) Transp, (xo. i) and o (Xo, T, i) Transe, (Xo, 2) two Transformer [44] networks, and define for all timesteps f I....,T
as follows:
1 , t, i) = (1 - a.,) Transp, (Xo, i), (x.. t, == vI - ā. Transo, i). (14) Note that Eq 14 makes the noise adaptation directly dependent on ag for simplicity and drops from the input. Algorithm thhus learns the above two Transformers. As for the proposal generator to produce &n at test time, we either employ the tate-of-theear task-specific method (iee., Room- Former [47] for floorplan and MapTI 23 for HD map) simulate rough human annotations. Please see $5 for experiments with different proposal generators, and Appendix B.] for implementation details of the guidance network.
The implementation of the denoising network (x,,,, y, borrows the core neural architectures from the state-of-the-ar task-specific models RoomFormer [47 and MapTR (23). While these two models are both adapted from DETR (3] and learn direct mappings from y to x with encoder-decoder Transformers, we need to make few essential modifications to turn them into valid denoising networks (full implementation details are in Appendix B.2)
Denoising network:
Instead of the learnable embeddings or coordinates, x, should be the input nodes the Transformer decoder. Each vertex is node, and the node feature is the concatenation of four positional encodings : two for and Y-axis coordinates and another two for vertex and instance index. We encode the timestep (or noise level) with a positional encoding [9 14], and add this feature to each block of the Transformer decoder.
Our overall formulation with GS-DM is independent of the network architectures, and more advanced networks can be easily integrated into the framework in the future boost performance.
Sampling acceleration: PolyDiffuse follows Eq reconstruct polygonal shapes. Song et al. 41 shows that DDPM can be viewed as discretization of a stochastic differential equation (SDE) with an
associated probability flow ODE, and advanced ODE solvers [99 26 39] can significantly accelerate
the sampling process. At test time, we employ a first-order solver (e.g., DDIM (39) and use 10
sampling steps for both tasks (ablation study is in (5.3). During the generation process, PolyDiffuse only runs the image encoder once as the visual features are shared across all steps.
Likelihood evaluation: With the probability flow ODE 41,, PolyDiffuse can estimate its own reconstruction quality via likelihood evaluation, thus enabling potential extensions such as search- based or -uunan-i--the-loyy refinement wrapper around our system. This is an exciting property not possessed by previous reconstruction approaches. We show examples in Figure 4
5

---

Experiments
We have implemented the system with PyTorch and used a machine with 4 NVIDIA RTX A5000
GPUs. We have borrowed the official codebase Karras et 19] to implement the diffusion model framework. The guidance networks are implemented as set-to-set set Transformer decoders, and the
loss weights for the guidance training are λι 1, 22 = 0.05, λ 0.1. The implementation of the denoising network refers to the competing state-of-the-art, RoomFormer [47] and MapTR [23] for floorplan and HD map reconstruction, respectively. Concretely, we employ the same ResNet [12] image encoder, DETR- Transformer architectures (discussed in $ learning rate settings, and optimizer settings RoomFormer MapTR, while increasing the number of training iterations (x2.5 for floorplan and x 1.5 for HD map), as the model converges slower under a denoising formulation than a detectiio/rregression formulation. Note that tried the same training schedule for the competing methods but their performance dropped due to overfitting (see $5.3 for ablation study). For each task, we use the same trained model for different types of proposals (i.e., existing methods or rough annotations). Complete implementation details are in Appendix B and supplementary experiments are in Appendix D
5.1 Floorplan reconstruction
Dataset and metrics: Structured3D dataset 48 contains 3500 indoor scenes (3000/250/250 for saining/validatiionness)) with diverse house floorplans. The average/maxximm numbers of polygons and vertex per polygon across the dataset are 6.29/17 and 5.72/38, respectively. Point clouds are converted to 256 x 256 top-view point-density images as inputs. We use the same evaluation metrics as previous works 7 42 47 which consists of three levels of metrics with increasing difficulty: room, corner, and angle. The precision, recall, and F1 score are reported each metric level.
Competing approaches: We compare PolyDiffuse with five approaches from the literature: Floor- SP [ MonteFloor [42), LETR [46), HEAT 77. and RoomFormer (47). The first two approaches design sophisticated optimization algorithms to solve for the optimal floorplan with learned metric functions, while the latter three use end-to-end Transformer-basee faster. For our proposal generator RoomFormer, we simplify the implementation by removing the Dice loss. As the dataset is very small, we add a random rotation data augmentation and train the modified RoomFormer for 4x the original training iterations, which improves the overall performance.
neural networks, thus being much
Table 1: Quantitative evaluation on Structured3D test set [48]. PolyDiffuse outperforms the state-of-the-art methods by clear margins, and works well with rough annotations. Results of previous works are copied from 447). indicates our modified implementation. f; The running time is measured on a single Nvidia RTX A5000 GPU, and we only report those run by ourselves. When using RoomFormer to produce the initial proposals, the running time counts both the RoomFormer proposal generator and the GS-DM.
Evaluation Level
Room
Corner
Angle
Method
Stages
Steps
FPS
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
F1
Floor-S2 SP
5
2
89.
88.
88.
81.
73.
76.
80.
72.
75.
MonteFloor
42
2
500
95.6
94.4
95.0
88.5
77.2
82.5
86.3
75.4
80.5
LETR
46
1
94.5
90.0
92.2
79.7
78.2
78.9
72.5
71.3
71.9
HEAT
2
96.9
94.0
95.4
81.7
83.2
82.5
77.6
79.0
78.3
RoomFormer
47
1
97.9
96.7
97.3
89.1
85.3
87.2
83.0
79.5
81.2
RoomFormer"
1
29.9
96.3
96.2
96.2
89.7
86.7
88.2
85.4
82.5
83.9
+PolyDiffuse
2
11.7
96.9
96.4
96.6
90.3
87.1
88.7
85.8
82.8
84.3
+PolyDiffuse
2
5
7.1
98.5
97.9
98.2
92.5
89.0
90.7
90.3
86.9
88.6
(Ofuuss))))))OUus)
2
10
4.4
98.7
98.1
98.4
92.8
89.3
91.0
90.8
87.4
89.1
Rough annotations
17.9
18.2
18.0
1.3
1.4
1.3
0.1
0.1
0.1
(Offusssfuss(Ours))
2 10
19.2
97.4
98.2
97.8
91.7
92.2
91.9
89.2
89.7
89.4
presents the quantitative evaluation results. We tried two proposal generators: 1) the improved RoomFormer and 2) rough annotations. The rough annotations prepared by turning the ground-truth data into fixed-radius small circles centered
Quantitative & qualitative evaluation: Table

---

denoising steps Input image
G.T.
RoomFormer
+PolyDiffux
Rough annot.
+PolyDiffuse
Likelihood-bawee refinement
Refine moom
.9.99
moots
room
Figure Qualitative results for floorplan reconstruction. Left: PolyDiffuse improves the geometry of RoomFormer and turns rough annotations into decent reconstructions. Right: PolyDiffuse enables search-based self-refinement by enumerating different vertex numbers of polygons and evaluating its own reconstruction likelihood (i.e., NLL per dimension).
at the instance centroid (see Figure 4), which simulates what human annotator can provide easily (i.e., indicating instance center and the number of vertices by looking at images). When using RoomFormer the proposal generator, PolyDiffuse outperforms all competing methods by clear margins across all the evaluation metrics. The performance gap increases with the difficulty level of the metric, indicating that PolyDiffuse is superior in high-level geometry reasoning. With the rough annotations as the initial proposal, PolyDiffuse also produces reasonable reconstructions. The recalls are especially high because of the correct number of vertices. Note that our training is dependent on any specific proposal generator, but PolyDiffuse works well with different generators at test time. Figure 4 Apresents concrete qualitative examples of how PolyDiffuse improves the initial reconstruction and enforces better geometry correctness, We also provide more qualitative examples in Appendix D.3
Running speed analysis: Table I |lso presents the speed- seeedd-effrmmance tradeoff of PolyDiffuse the floorplan reconstruction task. During the denoising process, the image encoding parts of the denoising network only once, while the transformer decoder part runs for multiple rounds. In our computation environment, the time of image encoding vs. the time transformer decoder is 2:1. Since speed is crucial factor for floorplan reconstruction, we can just in real applications for better reconstruction quality.
Likelihood- based refinement: Figure 4[right) demonstrates how we leverage the likelihood evalua- tion to refine reconstruction results with minimal user interaction. We manually indicate the imperfect polygons (inside dashed boxes at the left), then run our system with different numbers of vertices for these polygons while evaluating the reconstruction likelihood. The result with the best negative log-likelihood (NLL) per dimension is the refined reconstruction.
5.2
HD map reconstruction
The nuScenes dataset provides standard benchmark for HD map recon- struction. The dataset contains 1000 ego-centric sequences collected by autonomous vehicles with rich annotations. The data is annotated at 2Hz. Each sample has 6 RGB images and LiDAR sweeps captured by onboard sensors, covering the 360 horizontal FOV. In the top-down Bird-eye-view (BEV) space, the perception range is |-5m, 15ml for X-axis and [-30m, 30m] for Y-axis. The map includes three categories of elements: pedestrian crossing, divider, and road boundary. For fair comparisons,
Dataset and metrics:
the same dataset split and processicn setups previous works [22 23 25). and also represent each polyline with 20 uniformly-intepollaee vertices. Similar to MapTR 23], we only use the RGB images the sensor inputs in experiments.
We follow the common evaluation protocol from the literature 23 25
which employs the average
precision (AP) with the Chamfer distance matching criterion. The AP is averaged over three

---

Table 2: Quantitative evaluation reusits of HD map construction on nuScenes All methods in the table use RGB inputs only and employ ResNet50 as the image backbone. + Results are obtained by running the official code with the released model checkpoint. i The running time here is measured on a single Nvidia RTX A5000 GPU, and we only report for the methods run by ourselves. When using TR to produce the proposals, the running time counts both the MapTR proposal generator and the GS-DM.
Matching Criterion
Chamfer distance
Ordered angle distance
Method
Stages
Steps
FPSt
APP
APa
APp
mAP
APp
APa
APb
mAP
VeptorMapNet
25
20
36.1
47.3
39.3
40.9
VectorMapNet
25
20
42.5
51.4
44.1
46.0
VectorMapNNet MapTR
20
3.9
40.0
47.6
39.0
42.2
33.0
44.5
27.3
34.9
23
56.2
59.8
60.1
58.7
MapTR* +
1
14.3
55.8
60.9
61.1
59.3
46.1
43.4
41.9
43.8
+PolyDiffuse(Oum
6.3
56.8
59.8
60.9
59.2
50.3
48.2
44.3
47.6
(Oflusssfus(Ourr)))
2
5
4.8
58.1
59.7
61.2
59.6
51.8
49.5
45,4
48.9
+PolyDiffuse((urs))
10
3.4
58.2
59.7
61.3
59.7
52.0
49.5
45.4
49.0
Rough annotations +PolyDiffuse(Ours)
18.5
2.2
0.7
7.1
8.7
0.0
0.0
2.9
10
11.3
55.3
60.4
55.3
57.0
48.3
49.5
38.1
45.3
distance thresholds: {0.5m, 1.0m, 1.5m), and the mean average precision (mAP) is obtained by aver- aging across the three classes of map elements. However, the Chamfer distance does not consider the vertex order and vertex matching between prediction and G.T., thus failing measure the structured correctness. As directional information is critical for autonomous vehicles, we propose to augment the matching criterion with an extra order-aware angle distance. Concretely, matching between a prediction and a G.T. with the smallest coordinate distance, then trace along the two polylines/polygons compute the average angle distance (in degrees). A prediction is considered a true positive only when satisfying both the Chamfer and angle distance thresholds. We augment the three-level thresholds to ((0.5m, 59), (1.0m, 109), (1.5m, 15")). Results reported in both the original and augmented metrics. We provide complete implementation details and visualizations about the augmented metric in the Appendix C
find the optimal vertex
Competing approaches: We compare method with VectorMapNet [25] and MapTR [23). VectorMapNet (25) employs an auto-regressive transformer decoder to generate the vertex of poly- gon/polyline one by one, conditioned on the image inputs. MapTR [23) is proposal generator and is a DETR-style hierarchical detection method that directly estimates the polygon/polyline as a set of vertex sequences. MapTR and RoomFormer are technically very similar but focus different tasks.
Quantitative & qualitative evaluation: Table 2 presents the quantitative evaluation results. PolyDif- fuse does not significantly improve the Chamfer-distance mAP of MapTR, as it does discover new instances. However, it shows clear advantages in the order-aware angle distance, metric that is indicative of high-level structural regularities, |Figure 5) Jlqualitatively shows that PolyDiffuse accurately reconstructs curves with greater resemblance to the ground truth. We also experimented with the same rough annotations as the floorplan task, and the quantitative results show that the mAp becomes lower compared to using MapTR as the proposal generator. By analyzing the results with rough annotations more carefully, we found the AP with the lowest threshold becomes much worse, while the one with the highest threshold improves. We believe this is because the guidance networks are trained with "G.T. proposals", where the rough annotations (i.e., fixed radius circles) have very different curve style when serving as the proposals, thus producing inaceurate initial Gaussians and stepwise guidance. This is a potential limitation of the current method. We provide additional qualitative examples in Appendix D33
Running speed analysis: Tablle 2 also presents our speed-performanee tradeoff on the HD map reconstruction task. In our computation environment, the time of image encoding vs. the time of transformer decoder is 4:1 when using MapTR's model architecture. As FPS is an important consideration for online applications, we need to pick the number of denoising steps carefully. Furthermore, since PolyDiffuse is not restricted to specific task-specific method for the model

---

Surrounding views
G.T.
MapTR +PolyDiftuse Rough annot.
-PolyDiffuse
8
Ped. crossing
Divider
Boundary
Figure Qualitative results for HD map reconstruction. PolyDiffuse improves the geometry correctness of MapTRR (marked in red) and can turn rough annotations into reasonable reconstructions.
architecture and proposal generator, our performance/efficiennyy can keep improving when better task-specific base methods appear in the future.
5.3
Ablation studies
We provide key ablation studies with the floorplan reconstruction benchmark. Please also find additional ablation studies in Appendix D.]
Standard DM vs. GS-DM: To better understand the effectiveness of our GS-DM, we define standard DM using a vanilla DDPM with images as the condition. The standard DM does not have the guidance network and proposal generator. The reverse process starts with noise sampled from the standard Gaussian distribution and gradually denoises into the final reconstruction using the same sampler GS-DM. It also has the same denoising network architecture the GS-DM of PolyDiffuse. As shown in Table 3 although standard DM produces reasonable results with 10 sampling steps, it does not outperform the baseline method RoomFormer. Also, the performance of standard DM drops significantly with fewer sampling steps, as the denoising ambiguity hurts the performance more seriously with larger sampling step size. We also demonstrate in Appendix D.2 that the standard DM can fail in a toy experiment due to denoising ambiguity.
Sampling steps: Unlike the standard DM, Table Table 2, and Table all suggest that PolyDiffuse is robust to fewer sampling steps and can still slightly improve Rooml or MapTR with very small number of sampling steps.
Training epochs: We observed that the model converges slower under the denoising formulation than the detection/regressionn formulation of previous approaches. In Table 4] PolyDiffuse keeps improving when training with longer schedules, while RoomFormer suffers from overfitting. However, even with the default schedule, PolyDiffuse significantly outperforms the baseline.
Related Work
Structured reconstruction of polygonal shapes: Structured geometry reconstruction is an active
research area in computer vision, focusing on converting raster sensor data vectorized geometries
such as room layouts 88 [3], floorplans [ 24), planes [1]] 337, wireframes [16), etc. Taking advantage of the advancements in deep learning, methods with end-to-end neural architecture [
23 25
47 49 51 gradually dominate the area with superior accuracy and efficiency. In this paper, we formulate the reconstruction of polygonal shapes as conditioned generation process with diffusion models, of the essential machinery of the recent exploding generative AL. Our

---

Table 3: Comparison between standard DM and GS-DM with different sampling steps. Results are reported in Fl scores.
Table 4: Ablation study on the training schedule of PolyDiffuse and RoomFormer. Results are reported in Fl scores.
Method
Steps
Room
Corner
Edge
Method
Epochs
Room
Corner
Edge
RoomFormer
96.2
88.2
83,9
1x
96.2
88.2
83.9
RoomFormer
2
12.5
6.5
5.4
2x
96.0
87.9
83.7
Standard DM
86.7
79.6
77.0
1x
98.0
90.6
88.6
10
90.4
83.6
81.1
1.5x
97.7
90.6
88.7
PolyDiffuse
2
96.6
88.7
84.3
2x
97.8
90.7
88.9
PolyDiffuse
98.1
90.7
88.6
2.5x
98.4
91.0
89.1
(w/ GS-DM)
10
98.4
91.0
89.1
method, PolyDiffuse, outperforms the current state of the arts on floorplan and HD map reconstruction while enabling broader practical use cases.
Diffusion-based generative models: Diffusion Models (DM) or score-based generative models 28
38
40 4T
have made tremendous progress in the last few years and demonstrated promising
performance in content generations 9 31
32
33 43 and likelihood estimations [19 20 28). In this
paper, we explore the potential of DM the context structured reconstruction and propose Guided Set Diffusion Model (GS-DM) by extending the DDPM [4] formulation. GS-DM reconstructs complex polygonal shapes with a guided denoising (reverse) process subject to sensor data and can evaluate its reconstruction likelihood by leveraging the underlying probability flow ODE [1]. Our high-level formulation is relevant to PriorGrad [21] diffusion model for acoustic data, but has essential differences GS-DM learns the guidance networks to control the diffusion and direct the denoising a per-element and stepwise manner, while PriorGrad pre-computes the mean and variance of the target Gaussian from the condition and directly shifts the diffusion/denoisiig trajectory.
7 Conclusion
This paper introduces PolyDiffuse, system designed to reconstruct high-quality polygonal shapes from sensor data via a conditional generation procedure. At the heart of PolyDiffuse a novel Guided Set Diffusion Model, that controls the noise injection in the diffusion process to avoid permutation ambiguity for denoising. PolyDiffuse achieves state-of-the-ar performance on two challenging tasks: floorplan reconstruction from point density image and HD map construction from onboard RGB images. To our knowledge, this paper is the first to demonstrate that Diffusion Models, generally regarded generation technique, are also powerful for reconstruction, potentially encouraging the community to further investigate the effectiveness of Diffussinn Models in broader domains.
PolyDiffuse suffers from two major limitations. First, it does not recover geometry instances (e.g., rooms for floorplan) missing in the initial reconstruction, while the likelihood- based refinement provides possible solution by incorporating search or external inputs. Second, as shown in the HD mapping results with rough annotations, when the style of initial reconstructions (e.g., fixed radius small circles) is different from the curve style of ground truth, the guidance networks could produce bad initial Gaussians and guidance, leading to inaccurate locations of the final results. Training specific guidance and denoising networks for each type of initial reconstruction is a solution but induces more computation.
Limitations:
The paper benefits applications in architecture, construction, autonomous driving, and potentially -amaan-n--the-lop annotations systems to reduce human workload. However, potential malicious or unintended applications might include military scenarios, for example, reconstructing digital models of important buildings or structures for targeted military operations.
Broader impact:
This research is partially supported by NSERC Discovery Grants with Accel- erator Supplements and DND/NSERC Discovery Grant Supplement, NSERC Alliance Grants, and John R. Evans Leaders Fund (JELF). We thank the Digital Research Alliance of Canada and BC DRI Group for providing computational resources.
Acknowledgements:
10

---

References
[1] Ricardo Cabral and Yasutaka Furukawa. Piecewise planar and compact floorplan reconstruction from images. In 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2014.
[2] Holger Caesar, Varun Bankiti, Alex H Lang. Sourahh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In 16th European Conference of Computer Vision (ECCV), 2020.
[4] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11(3), 2010.
[5] Jiacheng Chen, Chen Liu, Jiaye Wu, and Yasutaka Furukawa. Floor-sp: Inverse cad for floorplans by sequential room-wise shortest path. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[6] Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Leaminng the best pooling strategy for visual semantic embedding. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021.
Jiacheng Chen, Yiming Qian, and Yasutaka Furukawa. Heat: Holistic edge attention transformer for structured reconstruction. In Proceedings of the IEEFICVE Conference on Computer Vision and Pattern Recognition, 2022.
[8] Erick Delage, Honglak Lec, and Andrew Y Ng. A dynamic bayesian network model for autonomous 3d reconstruction from a single indoor image. In IEEE computer society conference on computer vision and pattern recognition (CVPR), 2016.
[91 Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 1188780-8744 2021.
[10] Fartash Faghri. David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving disual-semante embeddings with hard negatives. BMVC, 2018.
[11] Yasutaka Furukawa, Brian Curless, Steven M Seitz, and Richard Szeliski. Manhattan-woff stereo. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1422-1429. IEEE, 2009.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[13] Varsha Hedau, Derek Hoiem, and David Forsyth. Recovering the spatial layout of cluttered rooms. In IEEE 12th international conference on computer vision (ICCV), 2009.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[15] Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Similarity-Bai Based Pattenn Recognition: Third International Workshop. SIMBAD 2015, Copenhagen, Denmark, October 12-14, 2015. Proceedings 3. Springer, 2015.
[16] Kun Huang, Yifan Wang, Zihan Zhou, Tianjiao Ding, Shenghua Gao, and Ma. Learning to parse wireframes in images of man-made environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 626-66 635, 2018
[17] Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete diffusion model for controllable layout generation. CVPR, 2023.
[18] Pradeep Kumar Jayaraman, Joseph G Lambourne, Nishkrit Desai, Karl DD Willis, Aditya Sanghi, and Nigel JW Morris. Solidgen: An autoregressive model for direct b-rep synthesis. TMLR, 2023.
[19] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[20] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems (NeurIPS), 2021.

---

[21] Sang-gil Lee, Heeseung Kim, Chachun Shin, Xu Tan, Chang Liu, Qi Meng. Tao Qin. Wei Chen, Sungroh Yoon, and Tie-Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior. ICLR, 2021.
[22] Qi Li. Yue Wang. Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In 2022 International Conference on Robotics and Automation (ICRA), pages 4628-4634.
IEEE, 2022.
[23] Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang. Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construction. ICLR, 2023.
[24] Chen Liu, Jiajun Wu, Pushmeet Kohli. and Yasutaka Furukawa. Raster-to-vector: Revisiting floorplan transformation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
[25] Yicheng Liu, Yue Wang, Yilun Wang. and Hang Zhao. Vectormapnet: End-to-end vectorized hd map learning. ArXiv, abs/2206.08920, 2022.
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022.
[27] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International Conference on Muchine Learning (ICML), 2020.
[28] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning (ICML), 2021.
[29] Wamiq Para, Shariq Bhat, Paul Guerrero, Tom Kelly, Niloy Mitra, Lconidas J Guibas, and Peter Wonka. Sketchgen: Generating constrained cad sketches. Advances in Neural Information Processing Systems (NeurlPS), 2021.
[30] Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 715):110, 2008
[31] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 22Xiv2204.06122) 2022.
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36444 -644944 2022.
[34] Matthew Schultz and Thorsten Joachims. Learning a distance metric from relative comparisons. Advances in neural information processing systems (NeurIPS), 2003.
[35] Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P Adams. Vitruvion: A generative model of parametric cad sketches. ICLR, 2022.
[36] Mohammad Amin Shabani, Sepidehsadat Hosseini, and Yasutaka Furukawa. Housediffusion: Vector floorplan generation via a diffusion model with discrete and continuous denoising. CVPR, 2023.
[37] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbdl images. ECCV, 2012.
[38] Jascha Sohl-Dickstein, Fric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International Conference on Machine Learning (ICMI), 2015.
[39] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ArXiv, abs/2010.02502, 2020.
[40] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems (NeuriPS), 2019.
[41] Yang Song, Jascha Narain Sohl-Dicksee Dickstein, Diederik P. Kingma, Abhishek Kumar. Stefano Ermon, and Ben Poole. Score-basee generative modeling through stochastic differential equations. ICLR, 2021.
12

---

[42] Sinisa Stekovic, Mahdi Rad, Friedrich Fraundorfer. and Vincent Lepetit. Montefloor: Extending mets for reconstructing accurate large-scale floor plans. In Proceedings the IEEE/CVF International Conference on Computer Vixion (ICCV), 2021.
[43] Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang. and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-avee diffusion. ArXiv, abs/2307.01097 2023.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin. Attention all you need, Advances in neural information processing systems (NeurIPS), 2017.
[45] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation. 2011.
[46] Yifan Xu, Weijian Xu, David Cheung. and Zhuowen Line segment detection using transformers without edges. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[47] Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann. Connecting the dots: Floorplan reconstruction using wovelevel queries. CVPR, 2023.
[48] Jia Zheng, Junfei Zhang. Jing Li, Rui Tang. Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In ECCV, 2020.
[49] Yichao Zhou, Haozhi Qi, Yuexiang Zhai, Qi Sun, Zhili Chen, Li-Yi Wei, and Yi Ma. Learning to reconstruct 3d manhattan wireframes from single image. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2019.
[50] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to end object detection. arXiv preprint arXiv:2010.0415) 2020.
[51] Chuhang Zou, Alex Colburn, Qi Shan, and Derek Hoiem. Layoutnet: Reconstructing the 3d room layout from single rgb image. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 20512005220 2018
13

---

Appendix:
PolyDiffuse:
Polygonal Shape Reconstruction via Guided Set Diffusion Models
This appendix organized as follows:
The derivation of the Guided Set Diffusion Models based on the DDPM 14 formulation.
B
Additional implementation details, including:
Architectures and implementation details the guidance networks.
Architectures and implementation details of the denoising networks.
Implementation details of the permutation loss [perm(() for the guidance training. Implementation details of the diffusion models framework.
C More details and analyses on the augmented matching criterion for the mean AP metric to evaluate HD map reconstruction.
DD
Extra experimental results including:
Additional ablation studies on the design choices of the denoising networks.
Additional qualitative results for floorplan and HD map reconstruction.
A Derivation of Guided Set Diffusion Models (GS-DM)
this section, we present more details of the derivation of the Guided Set Diffusion Models (GS-DM) proposed in $3 of the main paper.
A.1 Forward process
We derive Eq. 7, 8, and in Sec. 3 of the main paper by induction. The trivial case of t 1 can be easily verified. Let x/-1 = v + Ps (Xo, 1, + ox.x, 1, 1,1) for > 1 and
Ei_1 ~ (O, I). By definition, we have
x = + + + (x,,
-VailVān- + (X0, 1, + ox 1 - 1, 1))
Mo(xo.t, t, + / a, (xo.t,))
(15)
Varxi + va (Xx, 1, + Hal (x t, i)+
Vano | 1) -I-aa Xo: L, i)e;
where of is standard Gaussian variable independent of é Since ei_ and ef are independent, Varo,(X t 1, + vI a t, i)e is Gaussian random variable of zero mean and variance of a (xu, - 1, (1 a,)o (xo, i). Thus have
(x0. (, aa Xo. / 1, + pa Xo, 1, and (x.,
(16)
o (x,, t, i) a (xo. t 1, (1 as) 3 t,
(17)
which correspond 8q.8 and Eq.9 of the main paper, and we further obtain q({ xx)) = ~x(x: Vasaá + Po (Xo, o (x0 f, I).,
(18)
which is the 7 of the main paper.
A.2 Reverse process
Assuming we are given the guidance networks μs and 0 and the noise scales dys of the reverse process, we follow a style similar to DDPM [14 derive a sampling step in the reverse process. At each sampling step in the reverse process, Po (x) ix)), the Gaussian distribution to sample x;_, from is supposed to have the same mean as q(x 1 xo) defined by the forward process, which is also Gaussian, to minimize their KL divergence. As the joint distribution of x and x; conditioned on
14

---

Xo is Gaussian, the mean of (xx--1xx),x0 can be easily derived with the following closed form 30
Chapter 8.1.3]:
V + (x: 1, + Varo 1,
(x(x0,t)) t, (x) varxó (x0 4, 4,)).
(19)
Since x] v 1 (xo
(xo. i) for E' ~ (0),,), Equation 19 can be rewritten as
0 (Xo, 10 (xo, L 1,i) 11,))
v Va + (x
(x0, (,
a (x. t 1,
(Vaxx) + tx((x.tt1
c²) + pe (xg 1,
Vai
(x),t,)
(x0, b, (1 11-04)00 D (x0: 1,
(Vaxx) + (x0 (,)
6¹) + (x 1,
Vat
02 (,
(20)
(1 an) (xo,t,
(Vorxb ox(xx, ()) (,) - 1,
Vou
T x (xn, 1,
"x(xx t 1,
t, (1 - (( 1, E²) + po 1, Txo, t,
Va (x)
As our formulation directly parameterizes oZ. we further simplify Eq. 20 by defining 0x(Xx, t,
vi vi=á,C(xn,.) where C(xo, i) is independent of the timestep 1, thus implicitly defining Ts (x,,x),, = (xo: i) C(xo: i). Such parameterization makes (x,, simply
an interpolation between 0 and o (xo. i) and further simplifies Eq. 20
(x) (x(xx t, i) ox(x0, t, Va
1
at e²) (x. t- 1,
(21)
During inference, we replace Ps (Xo, i) and 0 (X,,,,i) with (x( (, and o (80. i) respec- tively and replace et with the denoising network e to define the mean of sampling distribution po(x (x{) and derive a sampling step in the reverse process as
1
1 - at (,(xx,,,)) + t 1, tarzi, (22) t,
x
x
( t, i) (x(o,,
Vou
ai
which is the Eq.10 10 of the main paper.
B Additional implementation details
In this section, we complement $4 of the main paper by providing the complete implementation details of PolyDiffuse for the two different tasks.
B.1 Guidance networks
The implementation details of the guidance networks are shared across the two tasks.
Model architectures: In $4 of the main paper, we parameterize P& (xo, T, i) and o (xo, T, i) with two Transformers Transp, (xo, and Transo, (xo, i), and define P& (Xo, t, and o (xo, &, i) with Eq. 14. The two Transformers are implemented with a DETR-style 3 Transformer decode The Transformer decoder contains two shared attention- projection heads for jág and as respectively. Each decoder layer consists of intra-element self- attention layer and global self-attention layer, whose outputs are fused by addition. Each vertex in the input No becomes an input node of the Transformer decoder, and the node feature consists of three positional encodings 44]: 1) positional encoding of the X-axis coordinate, 2) positional
decoder layers and two separate linear
15

---

encoding of the Y-axis coordinate, 3) positional encoding of the vertex index inside the element. The sequence of each element starts with a special dummy node (similar the SOS token in language modeling), whose final encodings are used to compute the P and T of the element. The dimension of the positional encoding is 128, and the hidden dimension of the Transformer decoder is 256.
Training details:
The guidance training is summarized by Algorithm of the main paper. We train the Transformer decoder for 3125 iterations with batch size 32. Adam optimizer is employed with a learning rate of 2e-4 and weight decay rate of le-4.
B.2 Denoising networks
Since the implementations of the denoising networks are based on the corresponding state-of-the-art task-specific models, we describe them separately.
Floorplan reconstruction: The denoising network for the floorplan reconstruction task refers to RoomFormer [47] and its official implementation RoomFormer is DETR-based [3] model consisting of a ResNet50 [12] image backbone, a Transformer encoder to process and aggregate image information, and a Transformer decoder with two-level learnable embeedingssccorrdnaatss to predict a set of polygon vertices from the image information. The Transformer has 6 encoder layers and 6 decoder layers with an embedding dimension of 256, and deformable-attentin (50) is
employed for all cross-attention layers and the self-attention layers the Transformer encoder.
We follow the same architecture as RoomFormer, and have discussed the key modifications to turn the model into a denoising function (54 of the main paper) and our improvements lift up its overall performance ($5 of the main paper). A minor modification omitted in the main paper is that we add an intra-element attention layer each Transformer decoder layer to increase the model capacity, as we found the model converges obviously slower under the denoising formulation than the regression/detection formulation of RoomFormer. In each decoder layer, the outputs of the intra-elemen cerra-element attention are added to the outputs of the original global attention. The intra-element attention increases the number of overall parameters by -5% and the corresponding ablation study in D of this appendix, showing minor but recognizable improvements.
For training the denoising network, we keep the same setups as RoomFormer except that increase the number of training iterations (ablation study is in $5 of the main paper). Adam optimizer is employed with a base learning rate of 2e-4 for all parameters, and the learning rate decays by factor of 1 for the last 20% iterations.
The denoising network for the HD map reconstruction task refers to Map (23) and its official implementationf] The overall design of MapTR is very similar to RoomFormer, where the keys are the "hicrarchical" or "two-level" query embeddings for -tyle Transformer and loss based "hierarchical bipartite matching". We use the same model config file provided in MapTR's s official codebase and convert the model into a denoising function as described in $4 of the main paper. Similar what we did to RoomFormer above, add intra-element attention layer to each Transformer decoder layer to facilitate convergence.
HD map reconstruction:
We employ Adam optimizer with base learning rate of 6e-4 and a weight decay factor of le-4. cosine learning rate scheduler is used. To further facilitate convergence under the denoising formulation, we load the ResNet image backbone and Transformer encoder from the pre-trained MRpTR, and set the initial learning rate of the image backbone to be 0.1 of the base learning rate.
When using MapTR as the proposal generator to produce the initial reconstruction for PolyDiffuse, we only consider predicted instances with a confidence score higher than 0.5 as MapTR's positive predictions. PolyDiffuse takes and updates the positive predictions of MapTRR and keeps the remaining low-confidence predictions unchanged. Only the positive predictions are visualized for the qualitative results of MapTR and PolyDiffuse.
httpee//ggihal.ccc//yyyy//eaaaar http:s//ggicabb..co/uuuttt/////
16

---

B.3 Permutation loss
Directly computing the Lperm (6) as defined in Eq.14 of the main paper requires finding xi arg maxx,eP"(xx) (xca Lriplltt (X,, Xu. x6) and x; = arg e
Iriplee (Xo, X2, x²).
where the size of PPl (xxx) and Pl (x(; is N!. This enumeration over all N! permutations of Xu and X; )
immediately becomes computationally prohibitive when N gets large.
To reduce the computational cost and make Eq. 14 practically feasible, we propose to approximate Lperm(( (() with element-level triplet losses. Concretely, we first enumerate all pairs of elements x
and x for i.j ,,..., N to compute N N distance matrix D. The entry Dii,j) is the minimum distance between two elements x and x/. considering all possible vertex-level permutations (i.e., two variants for a polyfine, and 2 N;
variants for a polygon with N; vertices, similar to the "point-level matching" in MapTR 23 We then replace the sample-level triplet loss in I (Ø)
with N element-level triplet losses, and define the computationally feasible proxy loss [pper(()) as:
Eperm (o)
Σ max Tripler (x), X x()) + i1..... ISISN
max Triples (xiipletxxh:100 x²,
(23)
ISiSNo
Liriplet (x), x x() = max (0, a + D(i, - D(j,)))
(24)
Irippltt (x). x() max (0, + D(i, (ii.
(25)
is the soft margin hyperparameter of the hinge-style triplet loss 115] 34 and we a 0.1. All
coordinate values are re-scaled into [-1,1] 1]. In this way, we reduce the computational cost from
O(V!) M), where M max N₁ is the maximum number of vertices of an element of
Xp. In practical implementation, the guidance training (Algorithm of the main paper) uses Eperrn (Ø)
rather than [pem(())
B.4 Diffusion models framework
Karras et al. 19 (EDM) presents a general diffusion model framework, where DDPM [4] and SDE- based DM Tormulations from Song et al. 4T can all be viewed as specializations of the proposed framework. We borrow its official codebase" to implement our GS-DM as it provides a general and clean base implementation suitable for all DM-ddsed formulations. We then describe how we adapt the GS-DM into the EDM-based framework and list the relevant hyperparameter settings. This subsection follows the notations in Karras et al.] where y is the data sample, n is the sampled Gaussian noise, æ is the noisy sample, a is the noise level (equivalent to the timestep), while (o) Gn(a), Cou (a)) and Croise (a))se(a) are the preconditioning factors [19] Section 5]. Similar to the notations of our main paper, we let y and x² denote the :t element of y and æ, respectively. The sensor condition is omitted for notation simplicity.
To adapt our GS-DM into the EDM framework, we first dataa = 1.0 for EDM 19 Table 1], and then adapt the preconditioning equation [19] Section 5, Eq. 7] based on $3 of our main paper:
Do ; y) Cakip (a) x² + Comn (a) F' (())(() x' + (1 (in(m)) ( σ, i)}; Cuoise (a)) (26)
where the per-element noise injection is defined as x² = y'+ a, i), and noise n ~ NOO IT)..
Do(x"; o, y) is the reconstructed y'. Fg is the denoising network taking all elements of a noisy sample x. and Fj is the output of the hh element. With the above modifications, our GS-DM with the general EDM framework. We then describe the concrete hyperparameter settings 19 Table 1] for our two tasks. Please refer Karras et al. for detailed explanations of each hyperparameter.
implement
Floorplan reconstruction: For the guidance training, we set the Pmean 1.0 and Pas = 4.0 to ensure sufficient coverage of the forward process. For the denoising training, we set the Pmean = -0.5 and Pas 1.5. For inference (sampling), we set Fmax = 5.0 and amin = 0.01. Instead of the 2nd order Heun ODE solver adopted by the original EDM, we simply employ the 1st order Euler solver for sampling, which is equivalent to DDIM [3). All other hyperparameters are kept unchanged.
HD map reconstruction: The settings of P Preaa and Pas for guidance and denoising training are the same as the floorplan reconstruction task. We set Cssip (o) = O and Cour (o) = 1 so that the denoising
17

---

G.T.
CD matching
+OAD matching
G.T.
CD matching
+OAD matching
G.T.
CD matching
+OAD matching
G.T.
CD matching
matching
G.T.
CD matching
+OAD matching
G.T.
CD matching
+OAD matching
Figure 6: Illustration of the matching results with MapTR 2 predictions. matching: the original Chamfer distance (CD) matching criterion with threshold of 1.0m. +OAD matching: the original CD criterion with threshold 1.0m, augmented with the order-aware angle distance (OAD) criterion with threshold of 10 The matched instances (i.e., true positives) are marked in red except for the two endpoints.
network directly estimates the sample y, found this choice stabilizes the denoising training the HD map construction task. For inference (sampling), we set Tmax = 5.0 and a = 0.1, and use the Euler solver. All other hyperparameters are kept unchanged.
C The augmented matching criterion for HD map reconstruction
In $5.2 of the main paper, augment the Chamfer-distance (CD) matching criterion for the mean AP (mAP) metric used in previous works [23] 25) with order-aware angle distance (OAD) to better evaluate the structural regularity and directional correctness of the results. This section first discusses the limitations of the original Slamfer-distanee matching criterion and then provides complete implementation details of the order-aware angle distance.
C.1 The limitations the Chamfer-distance matching criterion
Figure 6 shows examples of the Chamfer-distame (CD) matching (threshold is 1.0m). Since CD considers each vertex separately, mostly evaluates the global location of the instance and ignores the structural/direettional information of the prediction. In the figure, some predictions with obviously wrong shapes are considered true positives under the CD matching criterion, while the augmented OAD matching criterion can effectively reject these bad shapes and align better with human judgment.
18

---

C.2 Implementation details
The computation of the order-aware angle distance is based on an optimal vertex-level matching between predicted element (instance) and ground truth (G.T.) element, similar to the "point-level matching" in the loss computation MapTR [23]. Concretely, we enumerate through all equivalent representations of G.T. map element and compute the average vertex-level L1 distance between the G.T. vertices and the predicted vertices. The variant with the smallest average vertex-level L1 distance forms the optimal matching with the predicted element. There are two equivalent variants for a polyline and 2 1) equivalent variants for a polygon with N; vertices. After obtaining the optimal one-to-one vertex-level matching, trace along the vertices of the G.T. and predicted elements to compute their average angle distance.
With the augmented OAD matching criterion, we change the three-level thresholds of the original CD- based AP from [0.5m, 1.0m,, 1.5m] into {(0.5m, 5"), ) (1.0m, 10"), ) (1.5m, 15") However, com- pared to polylines (iee, road dividers and boundaries), we noticed that the angle direction of polygons (pedestrian crossings) is much more challenging to recover. With an OAD threshold 5", MapTR's average precision for the pedestrian crossing class (AP,) is zero. Therefore, we loosen the OAD thresh- olds for the pedestrian crossing class by a factor of (i.e., {0..mm, 10%), (1.0m, 20%), (1.5m, 30") while not changing the thresholds for the other two classes.
D Additional experimental results
This section provides additional experimental results, including extra ablation studies and qualitative comparisons.
Table 5: Ablation study for the intra-element attention layer on the floorplan reconstruction task. Note that PolyDiffuse uses the RoomFormer from the first row to produce the initial reconstruction.
Evaluation Level >
Room
Corner
Angle
Method
Intra-element-attn
Prec.
Rec.
Fl
Prec.
Rec.
Prec.
Rec.
RoomFormer
96.3
96.2
96.2
89.7
86.7
88.2
85,4
82.5
83.9
RoomFormer
96.9
96.4
96.6
90.3
87.0
88.6
84.9
81.9
83.4
(olyDiffuseOūrr))
X
98.4
97.8
98.1
92.4
89.0
90.7
90.2
87.0
88.6
PolyDiffuse(Ours))
98.7
98.1
98.4
92.8
89.3
91.0
90.8
87.4
89.1
Table 6: Ablation study for the intra-element attention layer on the HD map reconstruction task. Note that PolyDiffuse uses the MapTR from the first row to produce the initial reconstruction.
Matching Criterion
+
Chamfer distance
+ Ordered angle distance
Method
Intra-elem eltraeeeeenn--atnn
AP,
APa
APp
mAP
APp
APa
AP,
mAP
MapTR MapTR
55.8
60.9
61.1
59.3
46.1
43.4
41.9
43.8
56.5
59.9
60.2
58.8
48.6
44.8
41.6
45.0
Poly (olyDiifuue(Ours)
56.9
59.2
60.66
58.9
50.8
48.3
44.9
48.0
PolyDiffuse(Ours)
58.2
59.7
61.3
59.7
52.0
49.5
45.4
49.0
D.1
Additional ablation studies
Table presents the ablation study of the intra-element attention with the floorplan reconstruction task. Comparing the first and second rows, adding intra-element attention to RoomFormer slightly improves the room and corner results but deteriorates the angle-Ievel performance, [Talle 6jrovides similar comparison for the HD map reconstruction task. Augmenting MapTR with intra-element attention worsens the mAP with the CD matching criterion while slightly improving mAP with the OAD-au -uggmeeedd matching criterion. These comparisons provide potential empirical explanation for the architecture choice of RoomFormer and MapTR they only employ global attention layers without extra attention layers at the element or instance level.
19

---

On the contrary, as indicated by the third and fourth rows of both Table 5 and Table 6 applying intra- element attention consistently boosts PolyDiffuse across all metrics, although the improvements are not huge. A potential explanation here is that the denoising task of PolyDiffuse is more challenging than the regression/deeection task RoomFormer and MapTR, thus benefiting from extra modeling capacities.
D.2 A toy experiment with standard DM
In Figure 7, we provide toy experiment support what we motivated in $1 of the main paper, which demonstrates how standard DM easily fails even with single data sample. Note that we have clarified the definition of standard DM in $5.3 of the main paper. In this experiment, the data contains single toy sample with 6 rectangular shapes, so there are permutation -euitaiiioneuuvveentt representations. After sufficient training, we draw four samples using the image-conditioned denoising process. The DDIM sampler is used with 10 sampling steps, so the randomness only comes from the initial noise. As the figure shows, only the third sample gets the correct reconstruction result. With the challenges of set ambiguity, standard conditional DM has trouble overfitting a single data sample and easily gets wrong outputs when initial noise is inappropriate.
Toy input image
Sampling results Standard (DDIM sampler, steps)
t= 10
0
10
Sample
Sample 2
There are elements, and the G.T. has 6! 720 permutation-qquivalent representations
10
0
10
Sample 3
Sample 4
Figure 7: A simple toy experiment of using standard DM to fit single data sample with 6 elements. Four sampling results with different initial noises are shown. The DDIM sampler used with 10 denoising steps. Only one of the four samples (i.e., Sample 3) gets the correct final result due to the challenges induced by the ambiguity, as explained in the main paper.
D.3 More qualitative examples
We provide additional qualitative results to compare PolyDiffuse against the state-of-the-art floorplan and HD map reconstruction methods, respectively. Note that the state-of-the-ar method (i.e., Room- Former or MapTRR produces the initial reconstruction for PolyDiffuse, Figure 8 to Figure 10 are for the floorplan reconstruction task, while Figure 11 Figure 13 are for the HD map reconstruction
task.
Since the likelihood-basee refinement is employed in the qualitative examples of this subsection, PolyDiffuse cannot discover missing instances that are not covered by the proposal generator (i.e., RoomFormer and MapTR in the qualitative results). However, the visual comparisons clearly demonstrate that PolyDiffuse significantly improves the structural regularity of the reconstructed polygonal shapes.
20

---

Input image
G.T.
RoomFormer
+PolyDiffuse
Figure 8: Additional qualitative results for the floorplan reconstruction task.
21

---

Input image
G.T.
RoomFormer
+PolyDiffuse
Figure 9: Additional qualitative results for the floorplan reconstruction task.
22

---

Input image
G.T.
RoomFormer
+PolyDiffuse
Figure 10: Additional qualitative results for the floorplan reconstruction task.
23

---

Surrounding views
G.T.
MapTR
+PolyDiffuse
AL
Ped. crossing
Divider
Boundary
Figure 11:
Additional qualitative results of the HD map reconstruction task.
24

---

Surrounding views
G.T.
MapTR
+PolyDiffuse
Ped. crossing
Divider
Boundary
Figure 12: Additional qualitative results of the HD map reconstruction task.
25

---

Surrounding views
G.T.
MapTR
+PolyDiffuse
Two
onopolss T
Ped. crossing
Divider
Boundary
Figure 13: Additional qualitative results of the HD map reconstruction task.
26


Reviews:
[{"review_id": "qwDL06nlHg", "reviewer": "Reviewer_HnVz", "summary": "The manuscript introduces a adaptation of the DDPM paradigm to enable denoising sets (instead of single data points like images). The method called GS-DM does this by introducing by adding noise per set element via learned guidance networks.\nThis approach requires the addition of a proposal generator that initializes the number of sets and their parameters for the denoising process.\nThe manuscript applies this new set-based diffusion model to two tasks: generating floorplan polygons from floorplan images and maps from autonomous car images. The proposed method performs well in comparison to related work.\n\n", "questions": "- In Fig3: is it mu/sigma _phi or _psi?\n", "limitations": "The limitations are adequately addressed by the authors at the end of the paper.\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "\nThe authors identify a core limitation in DDPMs when it comes to set-based data and propose a sound way that empirically works well to solve it. The paper is well written, figures are high quality and support the writing well.\nThe evaluation is thorough and clearly demonstrates that (1) the set-based diffusion is superior to the default DDPM (table 3) and (2) that the denoising approach helps improve state of the art results from MapTR and RoomFormer when applied on top of their final outputs. It is also very interesting to see how simple the inputs for denoising can be (as simple as the polygon centers) and still perform at almost state of the art levels.\n\nSince the underlying concept of set diffusion models is very general and could apply in various other domains, this work is significant to a larger audience than just the 3d reconstruction or perception community.\n", "weaknesses": "\nWhile the paper is well written it could use some more clarity around how the guidance network is learned(line 122ff) since this is one of the key novelties. Just from reading the text I would not be able to reproduce the method at this point.\n- How is the neares negative permutation found? It seems like this could be an intractable problem very quickly?\n- is the triplet loss computed based simply on the polygon corners (for example?)\n\nThe key limitation of the current approach is the fact that the number of elements in the set has to be initialized correctly by the proposal generator during inference. This is acknowledged by the authors and a reasonable limitation that can be addressed in follow on work imo.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the valuable comments and appreciate the overall positive feedback on the writing, the experiments, and the potential extension of our approach to broader domains. We will address your questions/comments as follows.\n\n---\n\n**Weaknesses: Clarifications of the guidance network learning.**\n \nWe are sorry for the confusion. Details are presented in the supplementary (in particular, the permutation loss Eq.12 in *Sec.3.3 of the supplementary document*). We briefly answer your two sub-questions below and will add these clarifications to the main paper.\n\n(1). **nearest negative permutation in the permutation loss**: Yes, finding the nearest negative permutation immediately becomes intractable as the number of elements ($N$) grows. As described in Sec.3.3 of the supplementary document, our solution is to approximate the permutation loss (Eq.12 of the main paper) with $N$ element-level triplet losses (Eq.9-12 of the supp), which reduces the computational cost from $O(N!)$ to $O(N^2M)$, where $M$ is the maximum number of vertices of an element (i.e., a polygon or polyline) in the data sample $\\mathbf{x}_0$. Empirically, our experimental results show that this approximation works well and helps learn reasonable guidance under feasible computations.\n\n(2). **Triplet loss computation:** The triplet loss is computed based on the L1 distance between the corners' coordinates of the two polygons or polylines (the $D(i, j)$ in Sec3.3 of the supp). We will clarify this detail.\n\n---\n\n**Questions: Symbols in Fig.3.** \n\nIt should be phi for both bar_mu and bar_sigma in Fig.3. We thank you for the catch and will fix the mistake. "}, {"review_id": "G8lcjwwsoi", "reviewer": "Reviewer_pHyL", "summary": "- The paper introduces PolyDiffuse, a novel structured reconstruction algorithm that incorporates Guided Set Diffusion Models (GS-DM).\n- The core concept involves splitting the reconstruction pipeline into two distinct stages.\n   - The forward diffusion process focuses on learning guidance networks to address denoising ambiguity effectively.\n   - The guidance networks establish individual Gaussians as target distributions for each polygon and are learned prior to the denoising training phase.\n   - The reverse denoising process utilizes the sensor data as a condition to reconstruct polygonal shapes.\n- The authors conduct extensive experiments involving two polygonal structure reconstruction tasks.\n- Through comprehensive quantitative and qualitative evaluations, the results demonstrate that GS-DM outperforms existing state-of-the-art methods in terms of performance and quality of the reconstructions.", "questions": "- It is worth exploring whether incorporating the condition $y$ as an input to the guidance network leads to performance improvements.\n- What are the failure cases when applying PolyDiffuse to off-the-shelf methods? Is there further analysis regarding these cases?", "limitations": "The authors addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The derivation of the GS-DM is appropriately detailed.\n- By employing a multi-stage approach and introducing guidance, the proposed method effectively resolves ambiguity in polygonal shape scenarios.\n- The proposed method can evaluate reconstruction quality through likelihood evaluation.\n- The proposed method performs well even when using only rough annotations as input.\n\n", "weaknesses": "The GS-DM may be sensitive to the specific guidance, as mentioned in Section 7 (Limitations).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the time and effort in providing insightful feedback and appreciate the overall positive comments. We answer the questions and concerns as follows.\n\n---\n**Q1. It is worth exploring whether incorporating the condition as an input to the guidance network leads to performance improvements.**\n\nYes, this is a potential strategy to improve the quality of the guidance. However, we chose not to do this in the current formulation due to three considerations: \n\n(1). The proposal generator (i.e., either an existing method like RoomFormer or MapTR, or a human annotator to provide rough annotations) already takes $\\mathbf{y}$ as the input to produce the initial proposal $\\hat{\\mathbf{x}}_0$, so having both $\\hat{\\mathbf{x}}_0$ and $\\mathbf{y}$ as inputs to the guidance network might not be necessary. Furthermore, the outputs of the guidance network are combined with Gaussian noise to produce noisy data during denoising training and are adjusted by the denoising network to get the final reconstruction during sampling, so they do not have to be precise.\n\n(2). Incorporating the sensor input $\\mathbf{y}$ could bring significant computation costs since it requires image encoding and cross-attention layers (for extracting image features). Our current guidance network is simply a lightweight Transformer with two self-attention layers; \n\n(3). The guidance training loss is designed to preserve the permutation of the input data in the forward process, and there is no regression or classification-style supervision as in single-shot methods (e.g., RoomFormer and MapTR). With this loss design, the guidance network might not effectively use the information of $\\mathbf{y}$.\n\n---\n**Q2. What are the failure cases when applying PolyDiffuse to off-the-shelf methods? Is there further analysis regarding these cases?**\n\nThree types of failure modes are observed in our experiments, and they are mentioned and roughly explained in the main paper (Sections 5.1, 5.2, and 7). We will provide more detailed discussions in the following and will also add them to the paper. Please also see extended discussions in *GR2 of the global response*, where we provide a figure to show concrete examples of the failure cases on the HD mapping task.\n\n(1). **Wrong number of vertices.** This type of failure happens when the proposal generator predicts wrong vertex numbers for the polygons/polylines, which has been shown in Fig.4 of the main paper. We have provided a feasible solution to address this failure mode by leveraging the likelihood evaluation property of our DM-based method, where we can locally search different vertex numbers for the polygons and get the one with the highest likelihood as the final reconstruction result.\n\n(2). **Wrong number of polygonal instances (i.e., elements of the set).** As mentioned in the Limitations in Sec.7, this type of failure case happens when the proposal generator misses entire polygons/polylines, or predicts redundant elements (e.g., the ground truth is a large polygon, but the proposal generator predicts two separate small polygons). \n\nA potential solution for the former case is still based on the likelihood evaluation property: i) Run the proposal generator with a lower confidence threshold to get an initial result with more instances, which has a better recall but lower precision; ii) Extract subsets of the initial result as different initial proposals for PolyDiffuse, run PolyDiffuse and evaluate the likelihood of the final reconstruction; iii) Pick the one with the highest likelihood as the final result. This strategy could potentially recover missing instances but needs much more running time.\n\nThe latter case is more challenging, and PolyDiffuse cannot rectify these redundant elements in the initial proposals. We hope there can be future works to handle the number of instances in an elegant way.\n\n(3). **Inaccurate shape/location caused by limited generalization ability of the networks to unseen styles of initial proposals.** This type of failure case is also mentioned in the Limitations subsection of Sec.7. Concretely, in our current training settings, the guidance network is trained only with \"ground-truth proposals\" during the guidance training stage (Algorithm1 in the paper), and it also only takes the \"ground-truth proposals\" to produce the guidance for training the denoising network during the denoising training stage (Algorithm2 in the paper). As a result, the networks might not perfectly generalize to other styles of the initial proposals, like the circle-shaped rough annotations used in our experiments, which can lead to *inaccurate location or shape* of the final reconstruction. \n\nOne potential solution to alleviate this generalization issue is to train the guidance network and denoising network with different types of initial proposals at training time by augmenting the ground-truth data. Possible augmentation strategies include: adding noise to the vertex coordinates, or converting the G.T. elements into some canonical shapes (e.g., circles as in our experiments) to mimic the style of rough annotations, etc.\n\nAnother potential solution is to train separate guidance and denoising networks for each type of initial proposal and applies the corresponding models at test time with respect to the exact type of initial reconstruction. But this solution will induce more computational cost for training.\n\n(Note: we use \"initial proposal\" and \"initial reconstruction\" interchangeably in the above descriptions)"}, {"review_id": "XKk8VPZLKq", "reviewer": "Reviewer_t57P", "summary": "This paper proposes a novel method for reconstructing multiple polygon shapes using a conditioned diffusion model. The method first learns a score-matching-based prior diffusion model from data. This model is then used to denoise the sensor data, resulting in the reconstruction of the polygon shapes. To handle the permutation order of the polygonal elements, the method separates the target Gaussian distribution of a sample from its permutation variants.", "questions": "Overall, I like the idea of integrating a diffusion model as a prior into the reconstruction pipeline. However, I wonder if it is possible to handle the permutation order in a different way. For example, we could first train a network to predict the structure of the floorplan, such as the number of rooms and their basic bounding boxes. Then, we could denoise each shape individually. This approach might be easier to train and faster in inference.", "limitations": "The authors have discussed the limitations of the proposed method in Sec. 7.  I also want to know whether the denoising speed is fast enough for a real-time HD map reconstruction. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A nice solution to handle the permutation order in the reconstruction of multiple geometric elements in conditioned diffusion model.\n2. Comprehensive experiments to evaluate the performance of the proposed method.\n3.This paper is well-written and easy to follow. Figure 2 provides a clear and concise illustration of the basic idea.\n", "weaknesses": "1. The maximum number of polygons and their vertices used in the training of the diffusion model should be clarified. Are these numbers consistent with the ones listed in lines 183-184?\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the constructive questions and appreciate the overall positive comments on our presentation, idea, and experiments. We answer your questions/concerns as follows.\n\n---\n\n**Q1. The maximum number of polygons and their vertices used in the training of the diffusion model should be clarified.**\n\nThank you for the catch. We provide the details and will also clarify them in the paper. For the floorplan reconstruction task, the maximum number of polygons and maximum vertices per polygon used during training are 20 and 40, respectively. These choices are inherited from the base method RoomFormer and are actually larger than the maximum possible numbers of the Structured3D dataset (reported in L183-184).  For the HD map reconstruction task, the maximum number of polygons/polylines and maximum vertices per instance used during training are 30 and 20, respectively. Note that we employ the same map representation as the base method MapTR -- all the map instances have 20 uniformly interpolated vertices.\n\nAt test time, PolyDiffuse takes whatever the proposal generator produces and does not limit the maximum number of polygons and vertices per polygon.\n\n---\n\n**Q2. If it is possible to handle the permutation order in a different way. For example, we could first train a network to predict the structure of the floorplan, such as the number of rooms and their basic bounding boxes. Then, we could denoise each shape individually. This approach might be easier to train and faster in inference.**\n\nThis is an interesting idea -- first training a simple network to predict the \"meta-information\" of the structures (i.e., number of instances, bounding box of each instance, and the rough number of vertices), and then employing a diffusion model to denoise the meta-information of each instance individually (should be in-parallel denoising for acceleration). Since the shapes are denoised separately, there is no bothering with the set ambiguity issue. However, we have two concerns below, which might potentially limit the performance:\n\n(1). Some pre-processing steps might be necessary for employing a diffusion model to denoise each polygonal instance separately. For example, we might need to normalize the coordinate space and crop image features based on the bounding box of each instance, such that the denoised polygons are bounded by the boxes and do not interfere with each other. The ground-truth bounding boxes can be used at training time. But at test time, recovering from an inaccurate initial bounding box (e.g., a too-small box) could be hard. \n\n(2). The inter-instance relation is a crucial type of pattern in structured polygonal data. For example, neighboring polygons effectively share corners and edges (as in floorplan), and polylines are usually parallel, orthogonal, or connected to each other based on certain regularities (as in HD maps). The individual design makes it impossible for the denoising network to model the inter-instance interactions, which could largely impair the performance. To alleviate this issue, we might need additional clever designs to allow message passing between the instances during the in-parallel denoising. On the contrary, our PolyDiffuse can directly borrow the network architecture of state-of-the-art one-shot methods (e.g., RoomFormer, MapTR) to implement the denoising network.\n\n(3). Using bounding boxes as the intermediate representation is a bit task-specific and might not support arbitrary initial inputs (e.g., human scribbles as rough annotations). On the contrary, the guidance network in our GS-DM formulation is very general: 1) the training simply relies on a distance-based loss without specifying a typical type of input; 2) it can take different initial results at test time (i.e., either the results from existing methods or arbitrary human inputs).\n\nOverall speaking, we believe this idea is feasible and can be faster than the current formulation of PolyDiffuse w/ GS-DM. But given the above concerns, more complicated designs will be needed to get a satisfactory performance, and the method will be a bit task-specific.\n\n---\n**Q3. Whether the denoising speed is fast enough for a real-time HD map reconstruction.**\n\nWe thank you for the question and have measured the running time of PolyDiffuse based on MapTR-tiny (using ResNet-50 image backbone as in Table.2 of the main paper). We provide the details in this rebuttal and will include related information in the paper.\n\nThe running time is measured with *a single Nvidia RTX A5000 GPU* on our machine. The time used by the image encoding (i.e., processing six perspective images with the ResNet and aggregating all features into the BEV space by a Transformer) is **roughly four times** the time used by the Transformer decoder to produce the final denoising outputs. Since all the denoising steps share the same BEV features, the image encoding will only run once at test time. \n\n**FPS stats:** MapTR-tiny has 14.3 FPS in our computation environment. With the same computing resources, a 5-step PolyDiffuse has 7.2 FPS, and a 10-step PolyDiffuse has 4.4 FPS. If we count both the running time of the MapTR proposal generator and PolyDiffuse, the results are 4.8 FPS for 5-step and 3.4 FPS for 10-step.\n\nWe also provide an updated table in *GR1 of the global response* to better demonstrate the \"running time vs. performance\" relation for both tasks. Please consider taking a look there.\n"}, {"review_id": "eJiF5yZynD", "reviewer": "Reviewer_Fcug", "summary": "This paper proposes a Guided Set Diffusion Model for reconstruction, addressing the challenges of ambiguous denoising and selecting appropriate initial noise. By learning guidance networks, the model ensures distinct representations for samples with multiple permutations in structured geometry. During testing, the model uses the guidance networks to initialize Gaussian noise and denoises to reconstruct polygonal shapes conditioned on sensor data. The approach is evaluated on floorplans and HD maps, demonstrating significant advancements over existing methods and enabling practical applications.", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper takes a step towards formulating reconstruction as a generation process conditioned on sensor data, providing insights into advancing Diffusion Models for shape reconstruction. The approach of guiding noise injection per element to ensure separation from permutation variants during the diffusion process effectively addresses the ambiguous denoising issue. The PolyDiffuse method is well-presented and achieves promising results for layout reconstruction. Overall, this paper makes valuable contributions and is commendable.\n\n", "weaknesses": "While floorplan and HD map reconstruction are important problems, it appears that the GS-DM formulation is more suitable for layout or other semantic reconstruction tasks rather than real 3D shape reconstruction. It would be beneficial if the authors could provide a discussion on this aspect and given some suggestion of addressing the limitations of PolyDiffuse in extending to 3D shape reconstruction tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the valuable comments and appreciate the overall positive feedback. \n\nThe question of extending PolyDiffuse to 3D reconstruction is quite open-ended and worth a detailed discussion. We provide our thoughts in the following, and the answer is organized into two parts: 1) describe the settings of 3D shape reconstruction tasks where extensions of PolyDiffuse are possible; 2) explain the key challenges of applying PolyDiffuse to these tasks in the current stage. We will also add discussions for future works to the paper.\n\n---\n\n**1. Task settings:** Since PolyDiffuse reconstructs the data as a set of polygonal structures, its potential extensions to 3D shape reconstruction will be reconstructing 3D CAD models or compact low-poly meshes. We briefly describe two related lines of previous works for indoor scenes and objects, respectively:  \n\n(1). *Indoor CAD reconstruction:* This line of work aims to reconstruct CAD-quality models from RGB-D captures of a large indoor scene, facilitating valuable applications in the construction industry. Typical works include Structured Indoor Modeling[R1] and some of its follow-ups[R2, R3].  The entire scene usually contains many rooms, and the rooms consist of a set of planar polygonal structures for walls, floors, and ceilings.\n\n(2). *Object shape reconstruction:* Another line of work studies generative models of objects (usually from a single category) as compact low-poly meshes. As these generative models can naturally be paired with encoders (for point clouds or images) to support conditional generation, they are also able to reconstruct object shapes from sensor inputs. From our knowledge, typical works include PolyGen[R4] and BSP-Net[R5]. The compact meshes can be regarded as a set of variable-length polygons, while the polygons efficiently share the vertices.\n\nGiven the similarities in the data representation, the above two lines of work are potential directions to extend PolyDiffuse.\n\n---\n**2. Key challenges:** (1). the lack of large datasets is an immediate challenge, as paired data with sensor inputs and CAD-quality annotations is very rare, even considering synthetic ones. PolyGen[R4] already observed overfitting issues when training on pre-processed ShapeNet data. Previous works on indoor CAD reconstruction [R1-R3] rely on heuristics to produce the final compact CAD-quality models, and there are no end-to-end deep learning methods due to the lack of training data; (2). While PolyDiffuse is good at reconstructing the geometry of the polygonal shapes (i.e., the accurate coordinates of vertices), it needs a reasonable proposal generator to provide the \"meta-information\" -- the number of polygons and the number of vertices per polygon. While getting reliable meta-information is not hard in floorplan and HD map reconstructions, it becomes more challenging for large-scale indoor scenes or complicated objects (e.g., there might be curved surfaces).\n\n---\n\nIn summary, extending GS-DM and PolyDiffuse to more challenging 3D reconstruction tasks is an interesting yet challenging future direction. We need large datasets as well as some reliable approaches to estimate the meta-information. \n\n---\n\n### References\n\n[R1]. Ikehata, Satoshi, Hang Yang, and Yasutaka Furukawa. \"Structured indoor modeling.\" In Proceedings of the IEEE international conference on computer vision, 2015.\n\n[R2]. Macher, Hélène, Tania Landes, and Pierre Grussenmeyer. \"From point clouds to building information models: 3D semi-automatic reconstruction of indoors of existing buildings.\" Applied Sciences 7, no. 10 (2017): 1030.\n\n[R3]. Tang, Shengjun, Xiaoming Li, Xianwei Zheng, Bo Wu, Weixi Wang, and Yunjie Zhang. \"BIM generation from 3D point clouds by combining 3D deep learning and improved morphological approach.\" Automation in Construction 141 (2022): 104422.\n\n[R4]. Nash, Charlie, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. \"Polygen: An autoregressive generative model of 3d meshes.\" In International conference on machine learning, pp. 7220-7229. PMLR, 2020.\n\n[R5]. Chen, Zhiqin, Andrea Tagliasacchi, and Hao Zhang. \"Bsp-net: Generating compact meshes via binary space partitioning.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 45-54. 2020.\n\n\n"}, {"review_id": "oeqgfPxSVn", "reviewer": "Reviewer_uTWe", "summary": "The paper presents a diffusion-based approach for reconstructing polygonal shapes from floorplan images and auto-drive sensor images. The idea is to refine an initial rough reconstruction through iterative denoising steps which revert a forward process that diffuses regular reconstructions into random noises through per-element noise injection (though the notions of elements are not clearly given for different tasks).  The challenge with learning such a denoising diffusion model, as identified by the paper, is the robustness to permutations of orders of elements that are introduced by linearization (though it's not convincing why a set-based transformer model would introduce such ordering problem inevitably). The paper proposes to address this challenge by learning a guided forward diffusion process, such that the diffusion paths for different permutations are encouraged to be separate. In test time, the rough initial reconstruction is first converted to corresponding target noise distributions by the learned guided diffusion networks, and then reverted back to an accurate reconstruction through iterative denoising. \n\nThe approach is tested on two polygonal shape reconstruction tasks, including floorplans and autonomous driving maps, and shows improvements over the baselines that produce initial reconstructions. Ablation studies are conducted on diffusion guidance and choices of several hyper parameters.", "questions": "1. Computational costs of the iterative denoisers should be reported, and compared with the base cost of running the initial reconstructions.\n\n2. Please provide clarifications and experimental evidence that perturbations of the optimal noise for a particular reconstruction can cause failure of denoise-based reconstruction. \n\n3. In the footnote of page 3, it's stated that even a permutation-invariant model like Transformer has to pick a permutation as the representation of a data sample. This statement seems self-contradictory and needs clarification. For example, in DETR like works (including RoomFormer and MapTR) there is no ordering of the elements, and they rely on matching to compute the loss functions between two sets. Why cannot this approach to extended to the diffusion setting?\n\n4. When defining the permutation contrastive loss (12), why not compare the x_t and x'_t, but compare x_0 and x_t instead? Is it essential to constrain the similarity between x_0 the uncorrupted data and x_t the noised version? This is not common for standard diffusion models.\n\n5. Sec.4, feature representation, it's not clear how polylines/polygons of different vertex numbers are encoded and represented. This is an important detail as its encoding directly impacts how likelihood-based refinement (Fig4) can be done. \n\n6. The line matching metrics augmented with angle threshold is not defined clearly. There is no formal description of its computation in either text or supplemental. \n\n7. Define the standard DM model of Sec.5.3 clearly. Otherwise it's hard to justify any proposed component. \n\n8. What is training schedule? It looks like training epochs but why would one bother with the training iterations so much. \n\n9. Since it is acknowledged that the guided diffusion network may not generalize to hand-annotated input, I hope the authors can thoroughly discuss the limitations of the learned diffusion network, including how to decide if it's not general enough for a particular test dataset or input pattern, and what are the practical ways to enhance its generalization scope.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper proposes to improve polygonal reconstruction works that produce results in a single network pass by iterative denoising of the reconstruction. It is thus expected that the results can have better quality than the initial estimates, which is confirmed by tests on two tasks.\n\nThe paper formulates the diffusion model for structured elements like polygons and polylines, and identifies the problem of ordering permutation in representing the structures, which may cause severe ambiguity for a network to learn the reverse process. \n\nThe paper designs a learned guided forward diffusion process to distinguish the different permutations causing representational ambiguity, and derives the updated diffusion models biased by the learned diffusions. Such adaptive and learned diffusion can be inspiring to other situations where hand-crafting a diffusion process can be problematic. \n\n", "weaknesses": "The computational cost of the iterative process can be better illuminated, to help readers appreciate the comparative advantage obtained over the initial estimates produced by other methods. \n\nThe paper identifies another challenge with diffusion-based approach for reconstruction: a reconstruction task has a single solution, so the initial noise needs to be chosen carefully. I do not fully understand this statement, and cannot find any validation of this challenge in the experiments and discussions. On the other hand, the proposed guided diffusion process does not produce a single noise for an input polygonal shape either, but samples from the predicted Gaussian distribution. \n\nThe guided diffusion process is trained through contrastive learning on the initial reconstructions of training data by a different method. It is not clear how well the guided diffusion generalizes for data samples outside the training set. In contrast, a fixed diffusion procedure as adopted by DDPM and others does not have generalization issue, as the final noise distributions are known a prior. Indeed, the paper acknowledges that when the initial reconstructions come from hand annotations that are largely different from training reconstructions for HD Maps, the guided diffusion and denoiser do not work robustly. \n\nIt might be difficult to regard this work as the first one using diffusion models for reconstruction. In fact, diffusion models have been widely used for conditioned generation that resemble reconstruction, e.g. ControlNet. Also, people have explored using diffusion models for segmentation [1], which is a kind of reconstruction. \n\n[1] SegDiff: Image Segmentation with Diffusion Probabilistic Models. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the valuable questions and thoughtful feedback. We answer your questions and comments as follows.\n\n(A clarification: \"elements\" are the polygons/polylines in our tasks)\n\n---\n**W1&Q1. Computational costs of the iterative denoisers.**\n\nDue to the space limit, we refer to *Q3 of Reviewer t57P*  and *GR1 of the global response* for discussions and tables. \n\n---\n**W2&Q2. Clarifications and experimental evidence that perturbations of the optimal noise for a particular reconstruction can cause failure of denoise-based reconstruction.**\n\nTable 3 of the main paper has shown quantitatively that GS-DM is better than a standard DM model that uses initial noise drawn from standard Gaussian (i.e., non-optimal noise distribution compared to the per-element Gaussian distribution estimated by our guidance network). We also included an illustrative toy experiment in *GR3 of the global response*, showing standard DM model easily fails with bad initial noises.\n\n---\n**Q3. The footnote on page-3, and why cannot DETR's approach be extended to the diffusion setting?**\n\nWe thank for a great question. Here is the clarification, which we will also be added to the paper: The use of a matching loss (e.g., the Hungarian matching loss in DETR) could make DMs permutation-invariant indeed. However, we cannot pick an arbitrary loss in the formulation. The L2 denoising loss of DMs is strictly derived from either the variational lower bound (DDPM) or denoising score matching (NCSN) and is order-aware by definition (i.e., based on one specific serialization of the data). Specifically, in DDPM's derivations, the loss consists of a sequence of KL divergences between the forward and the reverse Gaussians, which cannot be easily extended to a permutation-invariant version -- We are unaware of any DM formulation that is inherently permutation-invariant. \n\n---\n**Q4. Eq.12: why not compare the x_t and x'_t, but compare x_0 and x_t instead?**\n\nThe x_t and x'_t are permutation variants of the same noisy sample. They should not interfere (i.e., get close) with each other, which would make denoising ambiguous. We provide more explanations here and will clarify in the paper. \n\nThe core motivation of GS-DM is \"The guidance network learns to guide the noise injection in a per-element manner, such that a sample x_0 remains separated from its permutation variants throughout the diffusion process\". The second triplet loss term in Eq.12 encourages x_t to be closer to x_0 than any other permutation-equivalent variants x'_t. x_0 is the *anchor* using the terminology of triplet loss, and x_t and x'_t are the *positive* and *negative*, respectively. In triplet loss, the positive (x_t) and negative (x'_t) are not directly compared to each other, but compared against the anchor (x_0) respectively. Please see Sec.3.3 of the supp for full details of the permutation loss.\n\n---\n**Q5. Sec.4, feature representation.**\n\nWe thank for the question and will clarify in the paper. A data sample is represented by a $N\\times M\\times 2$ tensor. $N$ is the number of polygonal instances and $M=\\max_{i=1}^N N_i$ is the maximum number of vertices in an instance. The last dimension is each vertex's (x,y) coordinates. Instances with less than $M$ vertices are padded, and a mask is generated to handle the padding in the model. The order of instances is based on an arbitrary permutation. In the model, the vertex coordinates are transformed into sinusoidal positional encodings and augmented with two additional position encodings for the vertex index and instance index. All these positional encodings are concatenated and processed by an MLP to produce the feature with shape $(N*M)\\times D$, where $D$ is the hidden dimension of the denoising network (we use the same hyperparameter as the base method).\n\n---\n**Q6. Clear descriptions of the line matching metrics augmented with angle threshold.**\n\nThe full details of the angle-aware matching criterion are in *Sec.4.2 of the supplementary document*. The original average precision(AP) metric considers a predicted instance as true positive once the Chamfer-distance criterion is met. Our augmented AP considers a predicted instance as true positive only when both the Chamfer-distance and angle-distance criteria are met. We will add more explanations.\n\n---\n**Q7. Define the standard DM model of Sec.5.3 clearly.**\n\nWe thank for the question. The following is the definition, and we will clarify in the paper. The standard DM model is the vanilla DDPM with images as the condition. There is *no guidance network* and *no proposal generator*. The reverse process starts with a noise sampled from the standard Gaussian distribution and gradually denoises it into the final reconstruction using the same sampler as GS-DM. It also uses the same denoising network architecture as GS-DM. This model helps to ablate the key designs of our GS-DM.\n\n---\n**Q8. What is the training schedule, and why bother with it so much.**\n\n\"Training schedule\" means the training iterations and the corresponding learning rate decay strategy. We mentioned this because diffusion models need more iterations to converge well than the one-shot counterparts (i.e., RoomFormer and MapTR). To make our comparisons as fair as possible, we increase and match the training iterations of the base methods with ours.\n\n---\n**W3&Q9. Discussions about the limitations of the learned diffusion network, including how to decide if it's not general enough for a particular test dataset or input pattern, and what are the practical ways to enhance its generalization scope.**\n\nDue to the space limit, we refer to our answer in *Q2 of Reviewer pHyL* for thorough discussions and analyses of failure modes. Please also see extended discussions with concrete examples on the HD mapping task in *GR2 of the global response*. \n\n---\n**W4. The first one using diffusion models for reconstruction.**\n\nWe thank for the comment. We will revise the claim and limit the scope to geometry reconstruction."}]

Author rebuttal:
We thank all reviewers for your time and efforts in providing valuable comments and constructive feedback. We are glad that all the initial reviews are on the positive side (two accept, two weak accept, and one borderline accept). 

---
We use this global response as **a complement to the individual responses**, which will answer reviewers' questions that require additional figures or tables. The content of this global response is organized as follows, and we will also incorporate them into the paper.

**GR1:** Tables and discussions for the computational costs of PolyDiffuse on the two tasks. **(for Reviewer-t57P and Reviewer-uTWe)**

**GR2:** A figure of PolyDiffuse's example failure cases on the HD mapping task, as well as corresponding discussions. *This is an extension of our response to Q2 of Reviewer-pHyL*. **(for Reviewer-pHyL and Reviewer-uTWe)**

**GR3:** An illustrative toy experiment to show how standard DM fails, with a figure and discussions. **(for Reviewer-uTWe)**

---
**GR1. (Reviewer-t57P, Reviewer-uTWe) The computational cost of the denoising process, and the comparison to the methods for generating initial proposals.**

Table 1 and Table 2 in the rebuttal PDF present the speed-performance tradeoff of PolyDiffuse on the HD map and floorplan reconstruction tasks, respectively. Note that during the denoising process, the image encoding parts of the denoising network only run once, while the transformer decoder part runs for multiple rounds. In our computation environment, *the time of image encoding vs. the time of transformer decoder* is 4:1 for HD map reconstruction (w/ MapTR's network architecture) and 2:1 for floorplan reconstruction (w/ RoomFormer's network architecture). 

Speed is not a crucial factor for floorplan reconstruction, so we can just use more denoising steps in real applications for better reconstruction quality. For HD map reconstruction, since FPS is an important consideration for online applications, we might have to pick the number of denoising steps more carefully. In general, our method provides a reasonable speed-performance tradeoff. Furthermore, since PolyDiffuse is not restricted to a fixed task-specific model for the proposal generator, our performance/efficiency can keep improving when better task-specific base methods appear in the future.

---
**GR2. (Reviewer-pHyL, Reviewer-uTWe) Extended discussions/analyses on the failure modes of PolyDiffuse.**

*(This response is an extension of our response to Q2 of Reviewer-pHyL, please read that response first)*

We have discussed three types of failure modes in *Q2 of Reviewer-pHyL*. The first type (i.e., wrong number of vertices) has been covered in *Figure 4 of the main paper*. We now use Figure 1 of the rebuttal PDF to show the other two types of failure cases with the HD mapping task and provide some discussions below.

**Wrong number of instances.** The examples (1) (2) (3) in Figure 1 of the rebuttal PDF are typical failure modes of having a wrong number of instances. When MapTR serves as the proposal generator, the mistakes of missing instances or predicting duplicate/redundant instances are hard to be recovered by PolyDiffuse. Our *response to Q2 of Reviewer-pHyL* only provides an alleviating strategy based on search and likelihood evaluation, and more elegant approaches are needed to better handle this challenge. Also, if the proposal generator predicts wrong semantic labels, PolyDiffuse is not able to recover.

**Inaccurate shape/location caused by limited generalization ability of the networks to unseen styles of initial proposals.** When using rough human annotations as the initial proposals in the paper, we assume that the correct number of polygonal instances and semantics labels are given, and PolyDiffuse is responsible for generating accurate coordinates for all the vertices. However, as demonstrated by (4) (5) (6) of Figure 1, although the results are visually reasonable, many predictions are not considered as true positive by the matching criteria. 

As analyzed in our *response to Q2 of Reviewer-pHyL*, the networks don't perfectly generalize to the circle-shaped initial proposals, leading to slight shifts in instance location or errors in shape. Note that the matching criteria of the HD mapping task are very strict (autonomous driving must guarantee safety). This inaccurate shape/location issue also appears when using MapTR initial proposals due to some inaccurate initial predictions, but the overall precision of using MapTR proposals is higher than using rough annotations, as shown in Table 2 of the main paper. 

Another cause of inaccurate location/shape is the noisy image inputs caused by occlusions or bad weather conditions, but this is a common challenge for all HD mapping methods (e.g., MapTR, VectorMapNet, etc.) and is out of the scope of this paper. 

---

**GR3. (Reviewer-uTWe)  Clarifications and experimental evidence that perturbations of the optimal noise for a particular reconstruction can cause failure of denoise-based reconstruction.**

In Figure 2 of the rebuttal PDF, we provide a toy experiment to demonstrate how standard DM models easily fail and why a good initial noise is important. Note that we have clarified the definition of standard DM in the *response to Q7 of Reviewer-uTWe*. In this example, the data contains a single toy example with 6 rectangular shapes, so there are $6! = 720$ permutation-equivalent representations. After sufficient training, we draw four samples using the image-conditioned denoising process. The DDIM sampler is used with 10 sampling steps, so the randomness only comes from the initial Gaussian noise. As the figure shows, only Sample 3 gets the correct reconstruction result. With the challenges of set ambiguity, a standard conditional DM even has trouble overfitting a single data sample, and easily gets wrong outputs when the initial noise is inappropriate.



Meta-review: The paper has achieved a consensus for acceptance. Initially, reviewers were positive about both the technical novelty and the results presented. The rebuttal resolved most minor concerns and issues related to clarity. The AC agrees that the technical contribution of this paper, particularly the diffusion model on polygonal shapes with special care on handling permutation order, is clearly articulated. The results effectively support the paper's claims. Therefore, acceptance should be granted. Congratulations!
Decision: Accept (poster)