SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiacheng Chen Ruizhi Deng Yasutaka Furukawa
This paper presents PolyDiffuse, a novel structured reconstruction algorithm that transforms visual sensor data into polygonal shapes with Diffusion Models (DM), an emerging machinery amid exploding generative AI, while formulating reconstruction as a generation process conditioned on sensor data. The task of structured reconstruction poses two fundamental challenges to DM: 1) A structured geometry is a "set" (e.g., a set of polygons for a floorplan geometry), where a sample of  $N$  elements has  $N!$  different but equivalent representations, making the denoising highly ambiguous; and 2) A "reconstruction" task has a single solution, where an initial noise needs to be chosen carefully, while any initial noise works for a generation task. Our technical contribution is the introduction of a Guided Set Diffusion Model where 1) the forward diffusion process learns guidance networks to control noise injection so that one representation of a sample remains distinct from its other permutation variants, thus resolving denoising ambiguity; and 2) the reverse denoising process reconstructs polygonal shapes, initialized and directed by the guidance networks, as a conditional generation process subject to the sensor data. We have evaluated our approach for reconstructing two types of polygonal shapes: floorplan as a set of polygons and HD map for autonomous cars as a set of polylines. Through extensive experiments on standard benchmarks, we demonstrate that PolyDiffuse significantly advances the current state of the art and enables broader practical applications. The code and data are available on our project page: https://poly-diffuse.github.io.
Reconstruction and generation were once separate research areas with minimal overlaps. While not acknowledged by the broader research communities, state-of-the-art reconstruction and generation techniques have now exhibited increasing similarities. Focusing on structured geometry (e.g., CAD models) in this paper, Auto-regressive Transformer (AR-Transformer) is a family of state-of-the-art generative models that iteratively applies a Transformer network to generate CAD construction sequences [18, 29, 35]. On the reconstruction side, a Transformer network iteratively reconstructs and refines indoor floorplans, outdoor buildings models, and vectorized traffic maps [7, 23, 47], a process resembling the generative AR-Transformer.
In 2023, Generative AI is experiencing significant growth, primarily driven by the emergence of Diffusion Models (DM) [14, 38, 40, 41], where structured geometry generation is not an exception. DM-based approach iteratively denoises coordinates (with associated properties) to generate realistic design layouts [17] or floorplans [36]. A natural question is then "Is a Diffusion Model also good at structured reconstruction?" However, the answer is not that simple.
Take the high-definition (HD) map reconstruction problem for example [22], which aims to reconstruct a set of polylines given sensor data in a bird's-eye view (see Figure 1). A straightforward DM formulation would assume a maximum number of polylines (with a fixed number of corners per poly
Figure 1: PolyDiffuse for floorplan and HD map reconstruction. Starting from an initial proposal (e.g., from a human annotator or an existing method), the sensor-conditioned denoising process of our Guided Set Diffusion Model (GS-DM) generates shape reconstructions in a few sampling steps, initialized and directed by the guidance networks. The initial proposal above mimics simple human inputs that indicate rough locations and specify the number of vertices for the polygonal shapes.
line [23, 25]), take a particular permutation of the elements to construct a fixed-length vector of corner coordinates, then iteratively denoise the vector subject to sensor data as a condition. The structured
reconstruction task poses two fundamental challenges to such standard DM formulations: 1) Structured geometry is often a "set" of elements (e.g., a set of polylines), where the geometry of  $N$  elements has  $N!$  different but equivalent representations, making the denoising factorially ambiguous; 2) The denoising process needs to reach a single solution (i.e., one of  $N!$  representations) for a reconstruction task and an initial noise needs to be set carefully, while any initial noise would work for a generation task. Figure 2 shows a toy example of three elements with  $6 = 3!$  permutations, where six permutation-equivalent representations of the solution lie in six different sub-spaces. However, they all follow the standard Gaussian and become indistinguishable after the diffusion process, which obfuscates the denoising learning.
Figure 2: (a) A toy visualization for a sample with three elements, thus 6 different but equivalent points in the data space; (b) Standard DM at  $t = T$ ; (c) Our GS-DM at  $t = T$ .
Our technical contribution is a Guided Set Diffusion Model (GS-DM). The forward diffusion process learns guidance networks controlling noise injection so that one representation of a sample remains separated from its permutation variants, thus resolving denoising ambiguities. Specifically, the guidance networks set an individual Gaussian as the target distribution for each element (instead of a fixed zero-mean unit-variance Gaussian for everything), and are learned before the denoising training by minimizing the permutation ambiguity as a loss function. At test time, the reverse process initializes the element-wise Gaussian noise by the guidance networks and then denoises to reconstruct polygonal shapes conditioned on the sensor data, while the guidance networks take rough initial reconstructions either from an existing method or an annotator (see Figure 1 for examples).
We have conducted extensive experiments on two polygonal structure reconstruction tasks: floorplan reconstruction from a point-cloud density image with Structured3D dataset [48], and HD map construction from onboard camera inputs with nuScenes dataset [2]. PolyDiffuse augments the state-of-the-art approach of the two specific tasks and brings consistent performance improvements, especially on the geometric regularity of the reconstruction results. Our generation process takes 5 to 10 steps to produce decent results and the visual encoder only runs once. We also demonstrate a likelihood-based refinement extension of our system by exploiting the probability flow ODE [41], which enables broader practical applications. To our knowledge, this paper is the first to demonstrate that DM is a powerful machinery for structured reconstruction tasks, in fact, the first in a broader domain of geometry reconstruction as well. We will share all our code and data.
2 Background: Denoising Diffusion Probabilistic Models
Diffusion models or score-based generative models [14, 38, 40, 41] progressively inject noise to data in the forward (diffusion) process and generate data from noise by the reverse (denoising) process. The section provides the key equations on the denoising diffusion probabilistic models (DDPM) [14] that lay the foundation of our method.
DDPM considers a Markovian forward process that turns data  $\mathbf{x}_0\sim q(\mathbf{x}_0)$  into Gaussian noise:
$$
q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t - 1}\right) := \mathcal {N} \left(\mathbf {x} _ {t}; \sqrt {1 - \beta_ {t}} \mathbf {x} _ {t - 1}, \beta_ {t} \mathbf {I}\right) \tag {1}
$$
$t = 1,\dots ,T$  and  $\mathbf{x}_t$  is the latent at  $t$ . The noise schedule  $\{\beta_t\}$  is constant [14, 28] or learned by reparameterization [20]. The above forward process is written in closed form for an arbitrary  $t$ :
$$
q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {0}\right) = \mathcal {N} \left(\mathbf {x} _ {t}; \sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right), \tag {2}
$$
$$
\bar {\alpha} _ {0} = 1, \quad \bar {\alpha} _ {t} := \bar {\alpha} _ {t - 1} \alpha_ {t}, \quad \alpha_ {t} := 1 - \beta_ {t}. \tag {3}
$$
$\mathbf{x}_t$  is sampled by  $\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{(1 - \bar{\alpha}_t)}\epsilon$  for  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ . DDPM parameterizes the reverse process with a noise prediction (or denoising) network  $\epsilon_{\theta}(\mathbf{x}_t,t)$  to make connections with denoising score matching and Langevin dynamics [40, 45], and the sampling step of the reverse process is derived as:
$$
\mathbf {x} _ {t - 1} = \frac {1}{\sqrt {\alpha_ {t}}} \left[ \mathbf {x} _ {t} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\theta} (\mathbf {x} _ {t}, t) \right] + \sigma_ {t} \mathbf {z}, \quad \mathbf {z} \sim \mathcal {N} (\mathbf {0}, \mathbf {I}). \tag {4}
$$
$\sigma_t^2$  is set to  $\beta_{t}$  or  $\frac{1 - \bar{\alpha}_{t - 1}}{1 - \bar{\alpha}_t}\beta_t$ . The final training objective is a reweighted variational lower bound:
$$
L _ {\text {s i m p l e}} (\theta) := \mathbb {E} _ {\mathbf {x} _ {0}, t, \epsilon} \left[ | | \epsilon - \epsilon_ {\theta} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \epsilon , t\right) | | ^ {2} \right] \tag {5}
$$
3 Guided Set Diffusion Models (GS-DM)
Consider the task of generating a set of elements  $\mathbf{x} = \{\mathbf{x}^1, \ldots, \mathbf{x}^N\}$  under a condition  $\mathbf{y}$ . A straightforward application of diffusion models would be to take one permutation of the elements to form a vector, then perform diffusion and learn denoising. With an abuse of notation, we let the vector  $\mathbf{x}_0$  (i.e., the sample or denoised result in DM's notation) represent one particular permutation of  $\mathbf{x}$ . The challenge is that a set of  $N$  elements has  $N!$  different but equivalent representations, and any one of the  $N!$  is a valid  $\mathbf{x}_0$ , making the denoising factorially ambiguous<sup>1</sup>. To alleviate the above issue, our idea learns to guide the noise injection in a per-element manner by learning two "guidance networks", such that a sample  $\mathbf{x}_0$  remains separated from its permutation variants throughout the diffusion process. At test time, the guidance networks produce per-element Gaussians and stepwise guidance from an initial reconstruction, from which the denoising process generates the final reconstruction iteratively conditioned on sensor data. We first explain the forward and reverse processes and then the two-stage training for the guidance and denoising networks.
Forward process: With the standard diffusion process, different permutation variants of a sample gradually become indistinguishable through the diffusion process. To keep a particular representation  $\mathbf{x}_0$  separated from its permutation variants, we propose to inject noise per element. Let  $\mathbf{x}_t^i$  denote the diffused element  $\mathbf{x}_0^i$  at timestep  $t$ , we learn to change the transition distribution  $q(\mathbf{x}_t^i | \mathbf{x}_{t-1}^i)$  by adding  $\pmb{\mu}_{\phi}(\mathbf{x}_0, t, i)$  to the mean and multiplying  $\pmb{\sigma}_{\phi}(\mathbf{x}_0, t, i)$  to the standard deviation:
$$
q \left(\mathbf {x} _ {t} ^ {i} \mid \mathbf {x} _ {t - 1} ^ {i}, \mathbf {x} _ {0}\right) := \mathcal {N} \left(\mathbf {x} _ {t} ^ {i}; \sqrt {1 - \beta_ {t}} \mathbf {x} _ {t - 1} ^ {i} + \mu_ {\phi} \left(\mathbf {x} _ {0}, t, i\right), \beta_ {t} \sigma_ {\phi} ^ {2} \left(\mathbf {x} _ {0}, t, i\right) \mathbf {I}\right) \tag {6}
$$
$\mu_{\phi}$  and  $\sigma_{\phi}$  take  $\mathbf{x}_0$  and produce outputs for  $\mathrm{i}^{\mathrm{th}}$  element at  $t$ . Following the similar derivations as in DDPM [14], we obtain (see Appendix A.1 for derivations):
$$
q \left(\mathbf {x} _ {t} ^ {i} \mid \mathbf {x} _ {0} ^ {i}\right) = \mathcal {N} \left(\mathbf {x} _ {t} ^ {i}; \sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} ^ {i} + \bar {\mu} _ {\phi} \left(\mathbf {x} _ {0}, t, i\right), \bar {\sigma} _ {\phi} ^ {2} \left(\mathbf {x} _ {0}, t, i\right) \mathbf {I}\right), \tag {7}
$$
$$
\bar {\boldsymbol {\mu}} _ {\phi} (\mathbf {x} _ {0}, 0, i) = 0, \quad \bar {\boldsymbol {\mu}} _ {\phi} (\mathbf {x} _ {0}, t, i) := \sqrt {\alpha_ {t}} \bar {\boldsymbol {\mu}} _ {\phi} (\mathbf {x} _ {0}, t - 1, i) + \boldsymbol {\mu} _ {\phi} (\mathbf {x} _ {0}, t, i), \tag {8}
$$
$$
\bar {\sigma} _ {\phi} ^ {2} \left(\mathbf {x} _ {0}, 0, i\right) = 0, \quad \bar {\sigma} _ {\phi} ^ {2} \left(\mathbf {x} _ {0}, t, i\right) := \alpha_ {t} \bar {\sigma} _ {\phi} ^ {2} \left(\mathbf {x} _ {0}, t - 1, i\right) + \left(1 - \alpha_ {t}\right) \sigma_ {\phi} ^ {2} \left(\mathbf {x} _ {0}, t, i\right). \tag {9}
$$
This permutation ambiguity issue persists with permutation-invariant models (e.g., Transformers), as it is inevitable to pick one particular permutation of  $\mathbf{x}$  as the concrete data representation  $\mathbf{x}_0$ .
Algorithm 1 Guidance training (stage 1)
1:  $\phi$ : trainable,  $\theta$ : frozen
2: repeat
3:  $\mathbf{x}_0\sim q(\mathbf{x}_0)$
4:  $t\sim \mathrm{Uniform}(\{1,\ldots ,T\})$
5:  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  , then compute  $\mathbf{x}_t$
6: Take gradient descent step on  $\nabla_{\phi}L_{\mathrm{guide}}$  (Eq.13)
7: until converged
Algorithm 2 Denoising training (stage 2)
1:  $\phi$ : frozen,  $\theta$ : trainable
2: repeat
3:  $\mathbf{x}_0\sim q(\mathbf{x}_0)$
4:  $t\sim \mathrm{Uniform}(\{1,\ldots ,T\})$
5:  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  , then compute  $\mathbf{x}_t$
6: Take gradient descent step on  $\nabla_{\theta}L_{\mathrm{simple}}$  (Eq.11)
7: until converged
As  $q(\mathbf{x}_t^i|\mathbf{x}_0^i)$  only explicitly depends on  $\bar{\mu}_{\phi}$  and  $\bar{\sigma}_{\phi}$ , and  $\mathbf{x}_T^i \sim \mathcal{N}(\bar{\mu}_{\phi}(\mathbf{x}_0, T, i), \bar{\sigma}_{\phi}^2(\mathbf{x}_0, T, i))$ , we consider  $\bar{\mu}_{\phi}$  and  $\bar{\sigma}_{\phi}$  as the guidance networks.  $\pmb{\mu}_{\phi}$  and  $\pmb{\sigma}_{\phi}$  are defined implicitly by Eq. 8 and Eq. 9. To simplify the formulation, we define  $\bar{\sigma}_{\phi}(\mathbf{x}_0, t, i) \coloneqq \sqrt{1 - \bar{\alpha}_t} C(\mathbf{x}_0, i)$  where  $C(\mathbf{x}_0, i)$  is a function independent of the timestep  $t$ , thus implicitly defining  $\pmb{\sigma}_{\phi}(\mathbf{x}_0, t, i) = \bar{\sigma}_{\phi}(\mathbf{x}_0, T, i) = C(\mathbf{x}_0, i)$ .
Reverse process: Note that  $\mathbf{x}_0$  is the ground truth and only available for training. At test time, we introduce a proposal generator to produce an initial reconstruction  $\hat{\mathbf{x}}_0$ , and run  $\bar{\mu}_{\phi}$  and  $\bar{\sigma}_{\phi}$  with  $\hat{\mathbf{x}}_0$ .  $\hat{\mathbf{x}}_0$  can either be results from an existing method or rough annotations from a human annotator. We first draw the element-wise initial noise with  $\mathbf{x}_T^i \sim \mathcal{N}(\bar{\mu}_{\phi}(\hat{\mathbf{x}}_0,T,i),\bar{\sigma}_{\phi}^2 (\hat{\mathbf{x}}_0,T,i))$ , and then run denoising steps iteratively. Similar to Eq.4, the sampling step of the reverse process is derived as
$$
\mathbf {x} _ {t - 1} ^ {i} = \frac {1}{\sqrt {\alpha_ {t}}} \left[ \mathbf {x} _ {t} ^ {i} - \bar {\mu} _ {\phi} (\hat {\mathbf {x}} _ {0}, t, i) - \bar {\sigma} _ {\phi} (\hat {\mathbf {x}} _ {0}, t, i) \frac {1 - \alpha_ {t}}{1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon} _ {\theta} ^ {i} (\mathbf {x} _ {t}, t, \mathbf {y}) \right] + \bar {\mu} _ {\phi} (\hat {\mathbf {x}} _ {0}, t - 1, i) + \boldsymbol {\sigma} _ {t} \mathbf {z} ^ {i} (1 0)
$$
The denoising network  $\epsilon_{\theta}$  takes the entire  $\mathbf{x}_t$  (i.e., all elements), and  $\epsilon_{\theta}^{i}$  is the output of the  $i^{\text{th}}$  element. Please see Appendix A.2 for derivations.
Learning the denoising network: Following Eq. 5 of standard DDPM, the denoising objective is
$$
L _ {\text {s i m p l e}} (\theta) := \mathbb {E} _ {\mathbf {x} _ {0}, t, \boldsymbol {\epsilon}} \left[ \sum_ {i = 1} ^ {N} | | \boldsymbol {\epsilon} ^ {i} - \boldsymbol {\epsilon} _ {\theta} ^ {i} \left(\left\{\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} ^ {i} + \bar {\mu} _ {\phi} \left(\mathbf {x} _ {0}, i, t\right) + \bar {\sigma} _ {\phi} \left(\mathbf {x} _ {0}, i, t\right) \boldsymbol {\epsilon} ^ {i} \right\}, t, \mathbf {y}\right) | | ^ {2} \right] \tag {11}
$$
The above formulation from Eq.6 to Eq.11 resembles that of DDPM in §2, where the key difference is the introduction of guidance networks  $\bar{\mu}_{\phi}$  and  $\bar{\sigma}_{\phi}$ . We then describe the training of the guidance networks, which is the key step of GS-DM.
Learning the guidance network: Before training the denoising network with Eq. 11, we train  $\bar{\mu}_{\phi}$  and  $\bar{\sigma}_{\phi}$  to ensure that  $\mathbf{x}_t$  in the diffusion process keep separated from other permutation variants of  $\mathbf{x}_0$ . We quantify the permutation invariance between  $\mathbf{x}_0$  and  $\mathbf{x}_t$  with a triplet loss  $L_{\mathrm{Triplet}}(\cdot ,\cdot ,\cdot)$  widely used in metric learning [4, 34] (full implementation details are in Appendix B.3):
$$
L _ {\text {p e r m}} (\phi) := \max  _ {\mathbf {x} _ {0} ^ {\prime} \in \mathcal {P} ^ {!} (\mathbf {x} _ {0}) \backslash \{\mathbf {x} _ {0} \}} L _ {\text {T r i p l e t}} \left(\mathbf {x} _ {t}, \mathbf {x} _ {0}, \mathbf {x} _ {0} ^ {\prime}\right) + \max  _ {\mathbf {x} _ {t} ^ {\prime} \in \mathcal {P} ^ {!} \left(\mathbf {x} _ {t}\right) \backslash \{\mathbf {x} _ {t} \}} L _ {\text {T r i p l e t}} \left(\mathbf {x} _ {0}, \mathbf {x} _ {t}, \mathbf {x} _ {t} ^ {\prime}\right) \tag {12}
$$
$\mathcal{P}^! (\mathbf{x}_0)$  is the set of all permutation variants of  $\mathbf{x}_0$ . Using the terminology in metric learning, the three inputs of  $L_{\mathrm{Triplet}}(\cdot ,\cdot ,\cdot)$  are the anchor, positive, and negative, respectively. Our loss only considers the closest negative permutation, which is similar to the hard negative mining in some variants of the triplet loss [6, 10]. We also add two regularization terms on  $\bar{\mu}_{\phi}$  and  $\bar{\sigma}_{\phi}$ , so that the means of all elements are not too scattered and the variances do not vanish. The final loss for training the guidance networks is:
$$
L _ {\text {g u i d e}} (\phi) := \mathbb {E} _ {t, \mathbf {x} _ {0}, \boldsymbol {\epsilon}} \left[ \lambda_ {1} L _ {\text {p e r m}} (\phi) + \lambda_ {2} \sum_ {i = 1} ^ {N} \| 1 / \bar {\sigma} _ {\phi} (\mathbf {x} _ {0}, t, i) \| ^ {2} + \lambda_ {3} \sum_ {i = 1} ^ {N} \| \bar {\mu} _ {\phi} (\mathbf {x} _ {0}, t, i) \| ^ {2} \right] \tag {13}
$$
Algorithm 1 and Algorithm 2 summarize the two-stage training paradigm of GS-DM.
4 PolyDiffuse: Polygonal shape reconstruction via GS-DM
PolyDiffuse uses the GS-DM to solve polygonal shape reconstruction tasks (see Figure 3), in particular, floorplan and HD map reconstruction. This section explains specific details and specializations.
Figure 3: Illustration of the forward and reverse processes of PolyDiffuse with floorplan data.
Feature representation: A sample  $\mathbf{x}$  is a set of elements  $\mathbf{x}^i$  as defined in §3. Each element  $\mathbf{x}^i$  is either a (closed) polygon or a polyline, consisting of an arbitrary (for the floorplan task) or a fixed (for the HD map task) number of vertices as a sequence:  $\mathbf{x}^i = [v_1^i,\dots,v_{N_i}^i]$ . Each vertex contains a 2D coordinate. For the ground-truth data, we determine the first vertex of each element by sorting based on the Y-axis coordinate and then the X-axis coordinate, similarly to PolyGen [27]. The polygons are always in a counter-clockwise orientation.
Guidance network: As derived in §3, the forward (Eq.7) and reverse (Eq.10) processes only depend on  $\bar{\mu}_{\phi}$  and  $\bar{\sigma}_{\phi}$ , and each element follows  $\mathbf{x}_T^i \sim \mathcal{N}(\bar{\mu}_{\phi}(\mathbf{x}_0,T,i),\bar{\sigma}_{\phi}^2 (\mathbf{x}_0,T,i))$ . Therefore, we choose to directly parameterize  $\bar{\mu}_{\phi}(\mathbf{x}_0,T,i) = \mathrm{Trans}_{\bar{\mu}_{\phi}}(\mathbf{x}_0,i)$  and  $\bar{\sigma}_{\phi}(\mathbf{x}_0,T,i) = \mathrm{Trans}_{\bar{\sigma}_{\phi}}(\mathbf{x}_0,i)$  as two Transformer [44] networks, and define for all timesteps  $t = 1,\dots ,T - 1$  as follows:
$$
\bar {\boldsymbol {\mu}} _ {\phi} (\mathbf {x} _ {0}, t, i) := \left(1 - \sqrt {\bar {\alpha} _ {t}}\right) \operatorname {T r a n s} _ {\bar {\boldsymbol {\mu}} _ {\phi}} (\mathbf {x} _ {0}, i), \quad \bar {\boldsymbol {\sigma}} _ {\phi} (\mathbf {x} _ {0}, t, i) := \sqrt {1 - \bar {\alpha} _ {t}} \operatorname {T r a n s} _ {\bar {\boldsymbol {\sigma}} _ {\phi}} (\mathbf {x} _ {0}, i). \tag {14}
$$
Note that Eq.14 makes the noise adaptation directly dependent on  $\bar{\alpha}_t$  for simplicity and drops  $t$  from the input. Algorithm 1 thus learns the above two Transformers. As for the proposal generator to produce  $\hat{\mathbf{x}}_0$  at test time, we either employ the state-of-the-art task-specific method (i.e., Room-Former [47] for floorplan and MapTR [23] for HD map) or simulate rough human annotations. Please see §5 for experiments with different proposal generators, and Appendix B.1 for implementation details of the guidance network.
Denoising network: The implementation of the denoising network  $\epsilon_{\theta}(\mathbf{x}_t,t,\mathbf{y})$  borrows the core neural architectures from the state-of-the-art task-specific models RoomFormer [47] and MapTR [23]. While these two models are both adapted from DETR [3] and learn direct mappings from  $\mathbf{y}$  to  $\mathbf{x}$  with encoder-decoder Transformers, we need to make a few essential modifications to turn them into valid denoising networks (full implementation details are in Appendix B.2):
- Instead of the learnable embeddings or coordinates,  $\mathbf{x}_t$  should be the input nodes to the Transformer decoder. Each vertex is a node, and the node feature is the concatenation of four positional encodings [44]: two for X and Y-axis coordinates and another two for vertex and instance index.
- We encode the timestep  $t$  (or noise level) with a positional encoding [9, 14], and add this feature to each block of the Transformer decoder.
Our overall formulation with GS-DM is independent of the network architectures, and more advanced networks can be easily integrated into the framework in the future to boost performance.
Sampling acceleration: PolyDiffuse follows Eq.10 to reconstruct polygonal shapes. Song et al.[41] shows that DDPM can be viewed as a discretization of a stochastic differential equation (SDE) with an associated probability flow ODE, and advanced ODE solvers [19, 26, 39] can significantly accelerate the sampling process. At test time, we employ a first-order solver (e.g., DDIM [39]) and use 10 sampling steps for both tasks (ablation study is in §5.3). During the generation process, PolyDiffuse only runs the image encoder once as the visual features are shared across all steps.
Likelihood evaluation: With the probability flow ODE [41], PolyDiffuse can estimate its own reconstruction quality via likelihood evaluation, thus enabling potential extensions such as search-based or human-in-the-loop refinement as a wrapper around our system. This is an exciting property not possessed by previous reconstruction approaches. We show examples in Figure 4.