LayoutDM: Discrete Diffusion Model for Controllable Layout Generation
Naoto Inoue1

Kotaro Kikuchi1
Edgar Simo-Serra2
Mayu Otani1
1
2
CyberAgent, Japan
Waseda University, Japan

Kota Yamaguchi1{inoue naoto, kikuchi kotaro xa}@cyberagent.co.jp ess@waseda.jp{otani mayu, yamaguchi kota}@cyberagent.co.jp

Abstract

Initial layout

Final layout

corrupt...

Controllable layout generation aims at synthesizing
plausible arrangement of element bounding boxes with optional constraints, such as type or position of a specific element. In this work, we try to solve a broad range of layout generation tasks in a single model that is based on discrete state-space diffusion models. Our model, named LayoutDM, naturally handles the structured layout data in thediscrete representation and learns to progressively infer anoiseless layout from the initial input, where we model the
layout corruption process by modality-wise discrete diffusion. For conditional generation, we propose to inject layout constraints in the form of masking or logit adjustment
during inference. We show in the experiments that our LayoutDM successfully generates high-quality layouts and outperforms both task-specific and task-agnostic baselines on
several layout tasks.1...

denoisecondition

M

M

...

M

M

M

1

...

3

M

(optional)

1

4

...

3

2

Generated layout(s)LayoutDM

Completion

Category

Category +size

.

..

+

Refinement

Category+relationship

Figure 1. Overview of LayoutDM. Top: LayoutDM is trained togradually generate a complete layout from a blank state in discrete
state space. Bottom: During sampling, we can steer LayoutDM
to perform various conditional generation tasks without additional
training or external models.

1. Introduction
Graphic layouts play a critical role in visual communication. Automatically creating a visually pleasing layout hastremendous application benefits that range from authoringof printed media [50] to designing application user interface [5], and there has been a growing research interest in
the community. The task of layout generation considers thearrangement of elements, where each element has a tupleof attributes, such as category, position, or size, and depending on the task setup, there could be optional control
inputs that specify part of the elements or attributes. Dueto the structured nature of layout data, it is crucial to consider relationships between elements in a generation. For
this reason, current generation approaches either build anautoregressive model [2, 12] or develop a dedicated inference strategy to explicitly consider relationships [21–23].
In this paper, we propose to utilize discrete state-spacediffusion models [3, 10, 16] for layout generation tasks.
Diffusion models have shown promising performance forvarious generation tasks, including images and texts [14].
We formulate the diffusion process for layout structure bymodality-wise discrete diffusion, and train a denoising backbone network to progressively infer the complete layout
with or without conditional inputs. To support variablelength layout data, we extend the discrete state-space witha special PAD token instead of the typical end-of-sequence
token used in autoregressive models. Our model can incorporate complex layout constraints via logit adjustment, sothat we can refine an existing layout or impose relative size
constraints between elements without additional training.
We discuss two key advantages of LayoutDM over existing models for conditional layout generation. Our model
avoids the immutable dependency chain issue [22] that hap-

1 Please find the code and models at:https://cyberagentailab.github.io/layout-dm.

1pens in autoregressive models [12]. Autoregressive models fail to perform conditional generation when the condition disagrees with the pre-defined generation order of
elements and attributes. Unlike non-autoregressive models [22], our model can generate variable-length elements.
We empirically show in Sec. 4.5 that naively extending
non-autoregressive models by padding results in suboptimalvariable length generation while padding combined with
our diffusion formulation leads to significant improvement.
We evaluate LayoutDM on various layout generation
tasks tackled by previous works [22, 23, 37, 40] using twolarge-scale datasets, Rico [5] and PubLayNet [50]. LayoutDM outperforms task-agnostic baselines in the majority of cases and shows promising performance comparedwith task-specific baselines. We further conduct an ablation
study to prove the significant impact of our design choices
in LayoutDM, including quantization of continuous variables and positional embedding.
We summarize our contributions as follows:• We formulate the discrete diffusion process for layout
generation and propose a modality-wise diffusion and apadding approach to model highly structured layout data.
• We propose to inject complex layout constraints via
masking and logit adjustment during the inference, so that
our model can solve diverse tasks in a single model.
• We empirically show solid performance for various conditional layout generation tasks on public datasets.

a single model.
Another layout-specific consideration is the complex
user-specified constraints, such as the positional requirements between two boxes (e.g., a header box should be on
top of a paragraph box). Earlier approaches [34, 36, 48]
propose hand-crafted cost functions representing the violation degree of aesthetic constraints so that those constraints guide the optimization process of layout inference.
CLG-LO [21] proposes an aesthetically constrained optimization framework for pre-trained GANs. Our LayoutDM
solves such constrained generation tasks on top of the taskagnostic iterative prediction via logit adjustment.

2.2. Discrete Diffusion Models
Diffusion models [42] are generative models characterized by a forward and reverse Markov process. The forward process corrupts the data into a sequence of increasingly noisy variables. The reverse process gradually denoises the variables toward the actual data distribution. Diffusion models are stable to train and achieve faster sampling
than autoregressive models by parallel iterative refinement.
Recently, many approaches have learned the reverse process by a neural network and show strong empirical performance [7,14,44] in continuous state spaces, such as images.
Discrete state spaces are a natural representation of discrete variables, such as text. D3PM [3] extends the pioneering work of Hoogeboom et al. [16] to structured categoricalcorruption processes for diffusion models in discrete state
spaces, while maintaining the advantages of diffusion models for continuous state spaces. VQDiffusion [10] develops a corruption approach called mask-and-replace, so as
to avoid accumulated prediction errors that are common inmodels based on iterative prediction. Following the corruption model of VQDiffusion, we carefully design a modalitywise corruption process for layout tasks that involve tokens
from disjoint sets of vocabulary per modality.
Several studies consider a conditional input to the inference process of diffusion models. Some approaches alter
the reverse diffusion iteration to carefully inject given conditions for free-form image inpainting [31] or image editing by strokes or composition [33]. We extend the discrete
state-space diffusion models via hard masking or logit adjustment to support the conditional generation of layouts.

2. Related Work2.1. Layout Generation
Studies on automatic layout generation have appeared
several times in literature [1, 29, 36, 47]. Layout tasks arecommonly observed in design applications, including magazine covers, posters, presentation slides, application userinterface, or banner advertising [5, 9, 11, 20, 39, 46, 47, 49].
Recent approaches to layout generation consider both unconditional generation [2, 12, 17, 18] and conditional generation in various setups, such as conditional inputs of category or size [21–23, 25], relational constraints [21, 23], element completion [12], and refinement [40]. Some attempt
at solving multiple tasks in a single model [22, 37].
BLT [22] points out that the recent autoregressive decoders [2, 12] are not fully capable of considering partial
inputs, i.e. known elements or attributes, during generation
because they have a fixed generation order. BLT addresses
the conditional generation by fill-in-the-blank task formulation using a bidirectional Transformer encoder similar tomasked language models [6]. However, BLT cannot solve
layout completion demonstrated in the decoder-based models because of the requirement of the known number of elements. Our LayoutDM enjoys the best of both worlds andsupports a broader range of conditional generation tasks in

3. LayoutDM
Our LayoutDM builds on discrete-state space diffusion
models [3, 10]. We first briefly review the fundamental of
discrete diffusion models in Sec. 3.1. Sec. 3.2 explains ourapproach to layout generation within the diffusion framework while discussing features inherent in layout comparedwith text. Sec. 3.3 discusses how we extend denoising steps
to perform various conditional layout generation by imposing conditions in each step of the reverse process.
23.1. Preliminary: Discrete Diffusion Models
Diffusion models [42] are generative models characterized by a forward and reverse Markov process. While
many diffusion models are defined on continuous spacewith Gaussian corruption, D3PM [3] introduces a general
diffusion framework for categorical variables designed primarily for texts. Let T ∈ N be a total timestep of the diffusion model, we first explain the forward diffusion process. For a scalar discrete variable with K categories
zt ∈ {1, 2, . . . , K} at timestep t ∈ N, probabilities thatzt−1 transits to zt are defined by using a transition matrix
Qt ∈ [0, 1]K×K , with [Qt ]mn = q(zt = m|zt−1 = n),  q(z_{t}|z_{t-1}) = \bm {v}(z_{t})^{\!\top } \mathbf {Q}_{t} \bm {v}(z_{t-1}), 

...

...

Figure 2. Overview of the corruption and denoising processes in
LayoutDM. For simplicity, we use a toy layout consisting of twoelements and the model generates three elements at maximum.

token [MASK] and three probabilities γt of replacing thecurrent token with the [MASK] token, βt of replacing thetoken with other tokens, and αt of not changing the token.
The [MASK] token never transitions to other states. The
transition matrix Qt ∈ [0, 1](K+1)×(K+1) is defined by:(1)

where v(zt ) ∈ {0, 1}K is a column one-hot vector of zt .
The categorical distribution over zt given zt−1 is computedby a column vector Qt v(zt−1 ) ∈ [0, 1]K . Assuming theMarkov property, we can derive q(zt |z0 ) = v(zt )⊤ Qt v(z0 )where Qt = Qt Qt−1 · · · Q1 and:  &q(z_{t-1}|z_{t}, z_{0}) = \frac { q(z_{t}|z_{t-1}, z_{0})\,q(z_{t-1}|z_{0}) }{ q(z_{t}|z_{0}) } \nonumber \\ &= \frac { \left (\bm {v}\!\left (z_{t}\right )^{\!\top }\!\mathbf {Q}_{t}\bm {v}\!\left (z_{t-1}\right )\right ) \left ( \bm {v}\!\left (z_{t-1}\right )^{\!\top }\overline {\mathbf {Q}}_{t-1}\bm {v}\!\left (z_{0}\right ) \right ) }{ \bm {v}\!\left (z_{t}\right )^{\!\top }\overline {\mathbf {Q}}_{t}\bm {v}\!\left (z_{0}\right ) }. \label {eq:q_posterior}

  \mathbf {Q}_{t} = \begin {bmatrix} \alpha _{t}+\beta _{t} & \beta _{t} & \cdots & \beta _{t} & 0 \\ \beta _{t} & \alpha _{t}+\beta _{t} & \cdots & \beta _{t} & 0 \\ \vdots & \vdots & \ddots & \beta _{t} & 0 \\ \beta _{t} & \beta _{t} & \beta _{t} & \alpha _{t}+\beta _{t} & 0 \\ \gamma _{t} & \gamma _{t} & \gamma _{t} & \gamma _{t} & 1 \\ \end {bmatrix}. \label {eq:Q_mask_and_replace} 

(5)

(2)
(αt , βt , γt ) is carefully designed so that zt converges to the[MASK] token for sufficiently large t. During testing, westart from zT filled with [MASK] tokens and iteratively
sample new set of tokens zt−1 from pθ (zt−1 |zt ).

Note that due to the Markov property, q(zt |zt−1 , z0 ) =q(zt |zt−1 ). When we consider N -dimensional variables
zt ∈ {1, 2, . . . , K}N , the corruption is applied to each variable zt independently. In the following, we explain with
N -dimensional variables zt .
In contrast to the forward process, the reverse denoising process considers a conditional distribution of zt−1 overzt by a neural network pθ (zt−1 |zt ) ∈ [0, 1]N ×K . zt−1 issampled according to this distribution. Note that the typical
implementation is to predict unnormalized log probabilities
log pθ (zt−1 |zt ) by a stack of bidirectional Transformer encoder blocks. D3PM uses a neural network p̃θ (z̃0 |zt ), combines it with the posterior q(zt−1 |zt , z0 ), and sums overpossible z˜0 to obtain the following parameterization:  p_{\theta }(\bm {z}_{t-1}|\bm {z}_{t}) \propto \sum _{\tilde {\bm {z}}_{0}}q(\bm {z}_{t-1}|\bm {z}_{t},\tilde {\bm {z}}_{0})~\tilde {p}_{\theta }(\tilde {\bm {z}}_{0}|\bm {z}_{t}). \label {eq:single_step_in_inference} 3.2. Unconditional Layout Generation
A layout l is a set of elements represented by l ={(c1 , b1 ) , . . . , (cE , bE )}. E ∈ N is the number of elements in the layout. ci ∈ {1, . . . , C} is categorical information of the i-th element in the layout. bi ∈ [0, 1]4 isthe bounding box of the i-th element in normalized coordinates, where the first two values indicate the center location,and the last two indicate the width and height. Following
previous works [2, 12, 22] that regard layout generation as
generating a sequence of tokens, we quantize each value inbi and obtain [xi , yi , wi , hi ]⊤ ∈ {1, . . . , B}4 , where B isa number of the bins. The layout l is now represented byl = {(c1 , x1 , y1 , w1 , h1 ) , . . .}.
In this work, we corrupt a layout in a modality-wisemanner in the forward process, and we denoise the corrupted layout while considering all elements and modalities in the reverse process, as we illustrate in Fig. 2.
Similarly to D3PM [3], we parameterize pθ by a Transformer encoder [45], which processes an ordered 1D sequence. To process l by pθ while avoiding the order dependency issue [22], we randomly shuffle l inelement-wise manner and then flatten it to produce lflat =(c1 , x1 , y1 , w1 , h1 , c2 , x2 , . . .).

(3)

In addition to the commonly used variational lower
bound objective Lvb , D3PM introduces an auxiliary denoising objective. The overall objective is as follows:  \mathcal {L}_{\lambda } = \mathcal {L}_\mathrm {vb} + \lambda \underset {\substack { \bm {z}_{t} \sim q(\bm {z}_{t}|\bm {z}_{0}) \\ \bm {z}_{0} \sim q(\bm {z}_{0})}}{\mathbb {E}} \left [ -\log \tilde {p}_{\theta }\left (\bm {z}_{0}|\bm {z}_{t}\right ) \right ], \label {eq:total_loss} 

(4)where λ is a hyper-parameter to balance the two loss terms.
Although D3PM proposes many variants of Qt , VQDiffusion [10] offers an improved version of Qt called maskand-replace strategy. They introduce an additional special
3

Variable length generation Existing diffusion models
generate fixed-dimensional data and are not directly applicable to the layout generation because the number of elements in each layout varies. To handle this, we introduce a[PAD] token and define a maximum number of elementsin the layout as M ∈ N. Each layout is fixed-dimensional
data composed of 5M tokens by appending 5(M − E)
[PAD] tokens. [PAD] is treated similarly to the ordinary token in VQDiffusion and Qt ’s dimension becomes(K + 2) × (K + 2).

Strong Constraints The most typical condition is partially known layout fields. Let z known ∈ ZN contain the
known fields and m ∈ {0, 1}N be a mask vector denoting the known and unknown field as 1 and 0, respectively.
In each timestep t, we sample ẑt−1 from pθ (zt−1 |zt ) in
Eq. (3) and then inject the condition by zt−1 = m ⊙z known +(1−m)⊙ ẑt−1 , where 1 denotes a N -dimensional
all-ones vector and ⊙ denotes element-wise product.
Weak Constraints We may impose a weaker constraint
during generation, such as an element in the center. We
offer a way to impose such constraints in a unified framework without additional training or external neural networkmodels. We propose to adjust the logits to inject weak constraints in log probability space byModality-wise diffusion Discrete state-space models assume that all the standard tokens are switchable by corruption. However, layout tokens comprise a disjoint set of token groups for each attribute in the element. For example,
applying the transition rule Eq. (5) may change a token representing an element’s category to another token representing the width. To avoid such invalid switching, we proposehto apply disjoint corruption matrices Qct , Qxt , Qyt , Qw
t , Qtfor tokens representing different attributes c, x, y, w, h, as
we show in Fig. 2. The size of each matrix is (C + 2) ×
(C + 2) for Qct and otherwise (B + 2) × (B + 2), where+2 is for [PAD] and [MASK].

  \log \hat {p}_{\theta }(\bm {z}_{t-1}|\bm {z}_{t}) \propto \log p_{\theta }(\bm {z}_{t-1}|\bm {z}_{t}) + \lambda _{\pi } \bm {\pi }, \\ \label {eq:prior_addition} 

(6)

N ×K

where π ∈ R
is a prior term that weights the desired outputs, and λπ ∈ R is a hyper-parameter. The
prior term can be defined either hard-coded (Refinement in
Sec. 4.5) or through differentiable loss functions (Relationship in Sec. 4.5). Let {Li }L
i=1 be a set of differentiable loss
functions given the prediction, the later prior definition canbe written by:

Adaptive Quantization The distribution of the positionand size information in layouts is highly imbalanced; e.g.,elements tend to be aligned to either left, center, or right.
Applying uniform quantization to those quantities as in existing layout generation models [2,12,22] results in the lossof information. As a pre-processing, we propose to apply
a classical clustering algorithm, such as KMeans [32] on x,y, w, and h independently to obtain balanced position andsize tokens for each dataset. We show in Sec. 4.7 how quantization strategy affects the resulting quality.

  \bm {\pi } = -\nabla _{p_{\theta }(\bm {z}_{t-1}|\bm {z}_{t})} \sum _{i=1}^{L} \mathcal {L}_{i}\left (p_{\theta }\left (\bm {z}_{t-1}|\bm {z}_{t}\right )\right ). \\ \label {eq:prior_addition_by_gradient} 

(7)

Although the formulation of Eq. (7) resembles steering diffusion models by gradients from external models [7, 28],
our primal focus is incorporating classical hand-crafted energies for aesthetics principles of layout [36] that do not depend on an external model. In practice, we tune the hyperparameters for imposing weak constraints, such as λπ . Notethat these hyper-parameters are only for inference and areeasier to tune than the other training hyper-parameters.

Decoupled Positional Encoding Previous works apply
standard positional encoding to a flattened sequence of layout tokens lflat [2, 12, 22]. We argue that this flattening approach could lose the structure information of the layout andlead to inferior generation performance. In layout, each token has two types of indices: i-th element and j-th attribute.
We empirically find that independently applying positional
encoding to those indices improves final generation performance, which we study in Sec. 4.7.

4. Experiment4.1. Datasets
We use two large-scale datasets for comparison, Rico [5]and PubLayNet [50]. As we mention in Sec. 3.2, an element in a layout for each dataset is described by the five
attributes. For preprocessing, we set the maximum numberof elements per layout M to 25. If a layout contains moreelements, we discard the whole layout.
We provide an overview of each dataset. Rico is a datasetof user interface designs for mobile applications containing
25 element categories such as text button, toolbar, and icon.
We divide the dataset into 35,851 / 2,109 / 4,218 samples fortrain, validation, and test splits. PubLayNet is a dataset ofresearch papers containing five element categories, such astable, image, and text. We divide the dataset into 315,757 /
16,619 / 11,142 samples for train, validation, and test splits.

3.3. Conditional Generation
We elaborate on solving various conditional layout generation tasks using pre-trained frozen LayoutDM. We injectconditional information in both the initial state zT and sam−1
pled states {zt }Tt=0
during inference but do not modify thedenoising network pθ . The actual implementation of the injection differs by the type of conditions.
44.2. Evaluation Metrics

number of [MASK] tokens from a uniform distribution between one and the sequence length, and perform masking
based on the number.
VQDiffusion∗ is a diffusion-based model originally for
text-to-image generation [10]. We adapt the model for layout using K = C + 4B + 2 tokens, including [PAD].

We employ two primary metrics: FID and Maximum
IoU (Max.). These metrics take into account both fidelity
and diversity [13], which are two mutually complementary properties widely used in evaluating generative models. FID [13] captures the similarity of generated data toreal ones in feature space. We employ an improved feature
extraction model for layouts [21] instead of a conventional
method [23] to compute FID. Maximum IoU [21] measuresthe conditional similarity between generated and real layouts. The similarity is measured by computing optimal
matching that maximizes average IoU between generated
and real layouts that have an identical set of categories. For
reference, we show the FID and Maximum IoU computed
between the validation and test data as Real data.

4.4. Implementation Details
We re-implement most of the models since there are few
official implementations publicly available except [12, 21,22]2 . We train all the models on the two datasets with three
independent trials and report the average of the results.
LayoutDM follows VQDiffusion for hyper-parameters
unless specified, such as configurations for pθ and the transition matrix parameters i.e. αt and γt . We set the loss
weight λ = 0.1 (in Eq. (4)) and the diffusion timesteps
T = 100. For optimization, we use AdamW [30] withlearning rate of 5.0 × 10−4 , β1 = 0.9, and β2 = 0.98.
Many models, including LayoutDM, use Transformer [45] encoder backbone. We define a sharedconfiguration as follows: 4 layers, 8 attention heads, 512
embedding dimensions, 2048 hidden dimensions, and 0.1
dropout rate. For other models with extra modules, we adjust the number of hidden dimensions to roughly match thenumber of parameters for a fair comparison. We randomly
shuffle elements in the layout to avoid fixed-order generation during training. We search best hyper-parameters to
obtain the best FID using the validation set.

4.3. Tasks and Baselines
We test LayoutDM on six tasks for evaluation.
Unconditional generates layouts without any conditional
input or constraint.
Category→size+position (C→S+P) is a generation task
conditioned on the category of each element [22].
Category+size→ position (C+S→P) is conditioned on the
category and size of each element.
Completion is conditioned on a small number of elements
whose attributes are all known. Given a complete layout,
we randomly sample from 0% to 20% of elements.
Refinement is conditioned on a noisy layout in which
only geometric information is perturbed [40]. Following
RUITE [40], we synthesize the input layout by adding random noise to the size and position of each element. We sample noise from a standard normal distribution with a meanof 0 and a variance of 0.01.
Relationship is conditioned on the category of each element and some relationship constraints between the elements [23]. Following CLG-LO [21], we employ the sizeand location relationships and randomly sample 10% relationships between elements for the experiment.
The first four tasks handle basic layout fields. We include
a few task-agnostic models for comparison using existing
controllable layout generation methods or simple adaptationof generative models in the following:
LayoutTrans is a simple autoregressive model [12] trainedon a element-level shuffled layout, following [37]. We set avariable generation order to c→w→h→x→y.
MaskGIT∗ is originally a non-autoregressive model for unconditional fixed-length data generation [4]. We use [PAD]to enable variable-length generation.
BLT is a non-autoregressive model with layout-specific decoding strategy [22].
BART is a denoising autoencoder that can solve bothcomprehension and generation tasks based on Transformer
encoder-decoder backbone [24]. We randomly generate a

4.5. Quantitative Evaluation
C→S+P, C+S→P, Completion In these tasks, we inject conditions by masking. We summarize comparisonsin Tab. 1. As task-specific models, we include LayoutVAE [18], NDN-none [23], and LayoutGAN++ [21] for
C→S+P. We also adapt these models for C+S→P. LayoutDM outperforms other models except LayoutTrans [12]in completion. The significant performance gap between
LayoutDM and VQDiffusion* suggests the contribution of
our proposals to go beyond the simple discrete diffusion
models discussed in Sec. 3.2. Results in the completionsuggest that a combination of padding and diffusion models is the primal key to the generation quality. We find that
FID and Maximum IoU are not highly correlated only in thecompletion task. We conjecture that Maximum IoU may
become unstable when categories are also predicted, unlike
the C→S+P and C+S→P tasks where categories are given.
Fig. 3 shows the qualitative results of some models, including LayoutDM. We can see that LayoutDM generates
2 Unfortunately, most datasets have no official train-val-test splits, andprevious approaches work on different splits and pre-processing strategies.
Furthermore, models for FID computation also vary. Thus, we cannot directly compare our results with the reported figure in the literature.

5

Table 1. Quantitative comparison in conditional generation given partially known fields. Top two results are highlighted in bold andunderline, respectively. † indicates the results of BLT trained with [PAD] as an additional vocabulary since the original model cannot
perform unordered completion in practice.

Task
DatasetModel

Category → Size+Position

Category+Size → Position

Completion

Rico
PubLayNet
Rico
PubLayNet
Rico
PubLayNet
FID ↓ Max. ↑ FID ↓ Max. ↑ FID ↓ Max. ↑ FID ↓ Max. ↑ FID ↓ Max. ↑ FID ↓ Max. ↑Task-specific models

LayoutVAE [18]NDN-none [23]LayoutGAN++ [21]

33.3
28.4
6.84

0.249
0.158
0.267

26.0
61.1
24.0

0.316
0.162
0.263

30.6
62.8
6.22

0.283
0.219
0.348

27.5
69.4
9.94

0.315
0.222
0.342

-

-

-

-

LayoutTrans [12]MaskGIT∗ [4]
BLT [22]
BART [24]
VQDiffusion∗ [10]LayoutDM

5.57
26.1
17.4
3.97
4.34
3.55

0.223
0.262
0.202
0.253
0.252
0.277

14.1
17.2
72.1
9.36
10.3
7.95

0.272
0.319
0.215
0.320
0.319
0.310

3.73
8.05
4.48
3.18
3.21
2.22

0.323
0.320
0.340
0.334
0.331
0.392

16.9
5.86
5.10
5.88
7.13
4.25

0.320
0.380
0.387
0.375
0.374
0.381

3.71
33.5
117†
8.87
11.0
9.00

0.537
0.533
0.471†
0.527
0.541
0.576

8.36
19.7
131†
9.58
11.1
7.65

0.451
0.484
0.345†
0.446
0.373
0.377

Real data

1.85

0.691

6.25

0.438

1.85

0.691

6.25

0.438

1.85

0.691

6.25

0.438

Task-agnostic models

Rico

PubLayNet
LayoutCondition
BLT [22] BART [24] LayoutDM
Trans [12]

Real

Real

Completion

C+S→P

C→S+P

LayoutCondition
BLT [22] BART [24] LayoutDM
Trans [12]Figure 3. Comparison in conditional generation given partially known fields.

high-quality layouts with fewer layout aesthetics violations,such as misalignment and overlap, given diverse conditions.

generation, LayoutDM achieves the best FID under random
element order setting. We conjecture that BLT’s poor performance is due to train-test mask distribution inconsistency
caused by their hierarchical masking strategy for training.
BLT masks a randomly sampled number of fields from asingle semantic group i.e. category, position, or size. However, decoding starts with all masked tokens in inference.
The alignment metric of Real data stays at 0.109 in Rico.
Too small alignment values of LayoutTrans and MaskGIT
can be a signal of producing trivial outputs in Rico.

Unconditional Generation Tab. 2 summarizes the resultsof unconditional generation. Unconditional layout generation methods often assume fixed order for element generation e.g. top-to-bottom rather than random order for better
generation quality by constraining the prediction. For reference, we additionally report the results of LayoutTrans [12]trained on the fixed element order (LayoutTrans-fixed). Although we design LayoutDM’s primarily for conditional
6

Table 2. Quantitative comparison in unconditional generation. Top
two results are highlighted in bold and underline, respectively.

LayoutTrans-fixed [12]LayoutTrans [12]MaskGIT∗ [4]
BLT [22]
BART [24]
VQDiffusion∗ [10]LayoutDM

6.47
7.63
52.1
88.2
11.9
7.46
6.65

0.133
0.068
0.015
1.030
0.090
0.178
0.162

17.1
13.9
27.1
116
16.6
15.4
13.9

0.084
0.127
0.101
0.153
0.116
0.193
0.195

Real data

1.85

0.109

6.25

0.0214

40

40

30

30

20
10

20

101
FID
LayoutGAN++
w/ CLG-LO

0
102 100

101
FID

LayoutDM
w/o logit adjustment

102
NDN-partial

Figure 5. Quality-violation trade-off in the relationship task.
Lower scores indicate better performance for both metrics.
Relationship

Rico
PubLayNet
FID ↓ Max. ↑ Sim ↑ FID ↓ Max. ↑ Sim ↑w/o rel.

w/ rel.

Real

Rico

DatasetPubLayNet

10

0 0
10

Table 3. Quantitative comparison in the refinement task. Top two
results are highlighted in bold and underline, respectively.

Model

50

Violation

Model

Rico
PubLayNet
FID ↓ Align. ↓ FID ↓ Align. ↓Violation

DatasetRico

50

Task-specific models

RUITE [40]3.23 0.421 0.221 6.39 0.415 0.174

Noisy input
LayoutDM
w/o logit adj.

134 0.213 0.177 130 0.242 0.147
2.77 0.370 0.205 6.75 0.352 0.149
3.55 0.277 0.168 7.95 0.310 0.127

Real data

1.85 0.691 0.260 6.25 0.438 0.216

PubLayNet

Task-agnostic models

Figure 6. Qualitative comparison in the relationship task.
Input

RUITE [40] LayoutDM

Realand define by:
Rico  \left [ \bm {\pi }_x^i \right ]_j = \begin {cases} 1 & \text {if}\ |\mathrm {loc}(j)-\hat {x}_{i}| < m~\text {and}~j \in X \\ 0 & \text {otherwise}, \end {cases} 

(8)

PubLayNet

where m is a hyper-parameter indicating a margin, X is aset of indices denoting tokens for x in the vocabularies, andloc(j) is a function that returns the centroid value of j-th
token in the vocabularies. We define similar constraints forthe other geometric variables and elements.
We summarize the performance in Tab. 3. We additionally report DocSim [38] (Sim) to measure the similarity of apredicted and its corresponding ground truth layout. Imposing noisy geometric fields as a weak prior significantly improves the masking-only model and makes the performancemuch closer to RUITE [40], which is a denoising model notapplicable to other layout tasks. We compare some resultsin Fig. 4. Both LayoutDM and RUITE successfully recovercomplete layouts from non-trivially noisy layouts.

Figure 4. Qualitative comparison in the refinement task.

Refinement Our LayoutDM performs this task with a
combination of the strong constraints of element categories,
i.e., setting z known = {(c1 , [MASK], . . . , [MASK]), . . .},and the weak constraints that geometric outputs appear near
noisy inputs. As an example of the weak constraint, we
describe a constraint that imposes the x-coordinate estimateof i-th element close to the noisy continuous observation x̂i .
We denote a sliced vector of the prior term π in Eq. (6) that
corresponds to the x-coordinate of i-th element as πxi ∈ RK

Relationship We use Eq. (7) to incorporate the relational
constraints during the sampling step of LayoutDM. We follow [21] to employ the loss functions penalizing size and7

PubLayNet

FID

102

Runtime is also essential for a controllable generation.
We show a speed-quality trade-off curve for C+S→P asshown in Fig. 7. The Transformer encoder-only models,such as LayoutDM and BLT, can achieve fast generation at
the sacrifice of quality. We employ fast-sampling strategy
employed in discreteP
diffusion models [3] for LayoutDM
by pθ (zt−∆ |zt ) ∝z̃0 q(zt−∆ , zt |z̃0 )p̃θ (z̃0 |zt ), whereT
steps. De∆ ∈ N indicates a step size for generation in ∆spite being a task-agnostic model, LayoutDM achieves thebest quality-speed trade-off except for task-specific LayoutGAN++ [21] that runs under 10ms.

101

100
LayoutVAE
NDN-none
LayoutGAN++

4.6. Speed-Quality Trade-off

LayoutTrans.
BART
MaskGIT*

0

50
100
150
Time per sample [ms]BLT
VQDiffusion*

LayoutDM
Real Data

Figure 7. Speed-quality trade-off of different models for C+S→P.

4.7. Ablation Study
We investigate whether techniques in Sec. 3.2 improve
the performance. First, we evaluate a choice of quantization
methods for the geometric fields of elements. Instead of
KMeans, we compute centroids for the quantization by:• Uniform: This is dataset-agnostic quantization, which is
popular in previous works [2, 12, 22]. Following [12], we1B−1choose {0.0, B1 , . . . , B−1
B } and { B , . . . , B , 1.0} forthe position and size, respectively.
• Percentile: we sort the data into equally sized groups andobtain average values for each group as the centroids.
This is dataset-specific quantization similar to KMeans.
We show the result at the bottom of Tab. 4. We additionallyreport the Alignment metric (Align.) used in [21] since thechoice of the quantization affects the alignment between elements. Compared to Linear and Percentile, KMeans quantization significantly improves both FID and Alignment.
We confirm that our modality-wise diffusion and decoupled positional encoding both moderately improve the performance, as we show at the top half of Tab. 4.

Table 4. Ablation study results on layout-specific modification in
unconditional generation of Rico [5] dataset.

FID ↓ Align. ↓LayoutDM
w/o modality-wise diff.
w/o decoupled pos. enc.
w/ uniform-quantization
w/ percentile-quantization

6.65
7.32
6.78
7.58
9.79

0.162
0.156
0.227
0.256
0.232

Real Data

1.85

0.109

location relationships between elements that do not matchuser specifications. We define the loss functions for continuous bounding boxes, and we have to convert the predicteddiscrete bounding boxes to continuous ones in a differentiable manner. Given estimated probabilities of discrete xcoordinates p(x), for example,
P we compute the continuous
x-coordinate x̄ by x̄ =n∈X p(x = n) loc(n). Similar
conversion applies to the other attributes. Empirically, we
find that applying the logit adjustment multiple times (threetimes in our experiments) to each diffusion step moderately
improves performance.

5. Discussion
LayoutDM is based on diffusion models for discrete
state-space. Using continuous state-space as in latent diffusion models [41] would be interesting. Extension of
LayoutDM to handle various layout properties such ascolor [19] and image/text [46] is also appealing.
We believe our proposed logit adjustment can incorporate more attributes. Attribute-conditional LayoutGAN [26]considers area, aspect ratio, and reading order of elements
for fine-grained control. Since these attributes can be easily
converted to the size and location relationship constraints,
incorporating them with our LayoutDM is not very difficult.

We compare LayoutDM with two task-specific approaches: NDN-partial [23] and CLG-LO based on LayoutGAN++ [21]. We show the results in Fig. 5. We additionally report constraint violation error rates [21]. LayoutDM can control the strength of the logit adjustment asin Eq. (6) and produces an FID-violation trade-off curve.
LayoutDM is comparable to NDN-partial in Rico and outperforms NDN-partial by a large margin in PubLayNet. Although LayoutDM is inferior to CLG-LO in both datasets,
note that the average runtime of CLG-LO is 4.0s, which ismuch slower than 0.5s in LayoutDM. We show some resultsof LayoutDM in Fig. 6.

Potential negative impact Our model might be used toautomatically generate the basic structure of fake websites
or mobile applications, which could lead to scams or the
spreading of misinformation.
8

References[17] Zhaoyun Jiang, Shizhao Sun, Jihua Zhu, Jian-Guang Lou,and Dongmei Zhang. Coarse-to-fine generative modeling forgraphic layouts. In AAAI, 2022. 2[18] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. LayoutVAE: Stochastic scene layout
generation from a label set. In CVPR, 2019. 2, 5, 6, 12[19] Kotaro Kikuchi, Naoto Inoue, Mayu Otani, Edgar SimoSerra, and Kota Yamaguchi. Generative colorization of structured mobile web pages. In WACV, 2023. 8[20] Kotaro Kikuchi, Mayu Otani, Kota Yamaguchi, and Edgar
Simo-Serra. Modeling visual containment for web page layout optimization. Computer Graphics Forum, 40(7), 2021.
2[21] Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota
Yamaguchi. Constrained graphic layout generation via latent
optimization. In ACM MM, 2021. 1, 2, 5, 6, 7, 8, 11, 12, 13[22] Xiang Kong, Lu Jiang, Huiwen Chang, Han Zhang, Yuan
Hao, Haifeng Gong, and Irfan Essa. BLT: Bidirectional layout transformer for controllable layout generation. In ECCV,
2022. 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 16, 17, 18, 19, 20, 21[23] Hsin-Ying Lee, Weilong Yang, Lu Jiang, Madison Le, Irfan Essa, Haifeng Gong, and Ming-Hsuan Yang. Neural design network: Graphic layout generation with constraints. In
ECCV, 2020. 1, 2, 5, 6, 8, 12, 13[24] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, andLuke Zettlemoyer. BART: Denoising sequence-to-sequence
pre-training for natural language generation, translation, andcomprehension. In ACL, 2020. 5, 6, 7, 12, 13, 16, 17, 18, 19,
20, 21[25] Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang,
and Tingfa Xu. LayoutGAN: Generating graphic layouts
with wireframe discriminators. In ICLR, 2019. 2, 13[26] Jianan Li, Jimei Yang, Jianming Zhang, Chang Liu,
Christina Wang, and Tingfa Xu. Attribute-conditioned layout gan for automatic graphic design. IEEE TVCG, 27(10),2020. 8, 11, 13[27] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy
Liang, and Tatsunori B Hashimoto. Diffusion-LM improves
controllable text generation. In NeurIPS, 2022. 11, 12[28] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, andJoshua B Tenenbaum. Compositional visual generation withcomposable diffusion models. In ECCV, 2022. 4[29] Simon Lok and Steven Feiner. A survey of automated layout
techniques for information presentations. In SmartGraphics,
2001. 2[30] Ilya Loshchilov and Frank Hutter. Decoupled weight decay
regularization. In ICLR, 2019. 5[31] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher
Yu, Radu Timofte, and Luc Van Gool. RePaint: Inpainting
using denoising diffusion probabilistic models. In CVPR,
2022. 2[32] J MacQueen. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability,
1967. 4[33] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided[1] Maneesh Agrawala, Wilmot Li, and Floraine Berthouzoz.
Design principles for visual communication. Communications of the ACM, 54(4), 2011. 2[2] Diego Martin Arroyo, Janis Postels, and Federico Tombari.
Variational transformer networks for layout generation. In
CVPR, 2021. 1, 2, 3, 4, 8
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In NeurIPS, 2021. 1, 2,
3, 8[4] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T
Freeman. MaskGIT: Masked generative image transformer.
In CVPR, 2022. 5, 6, 7, 11, 12, 13, 16, 17, 18, 19, 20, 21[5] Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building datadriven design applications. In UIST, 2017. 1, 2, 4, 8, 12[6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. 2[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models
beat gans on image synthesis. In NeurIPS, 2021. 2, 4
[8] Patrick Esser, Robin Rombach, and Bjorn Ommer. Tamingtransformers for high-resolution image synthesis. In CVPR,
2021. 11
[9] Tsu-Jui Fu, William Yang Wang, Daniel McDuff, and Yale
Song. DOC2PPT: Automatic presentation slides generationfrom scientific documents. In AAAI, 2022. 2[10] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo
Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In
CVPR, 2022. 1, 2, 3, 5, 6, 7, 11, 12, 13, 16, 17, 18, 19, 20,
21[11] Shunan Guo, Zhuochen Jin, Fuling Sun, Jingwen Li, Zhaorui
Li, Yang Shi, and Nan Cao. Vinci: an intelligent graphic
design system for generating advertising posters. In CHI,
2021. 2[12] Kamal Gupta, Alessandro Achille, Justin Lazarow, Larry
Davis, Vijay Mahadevan, and Abhinav Shrivastava. LayoutTransformer: Layout generation and completion with selfattention. In ICCV, 2021. 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 16, 17,
18, 19, 20, 21[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner,
Bernhard Nessler, and Sepp Hochreiter. Gans trained by a
two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017. 5[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 1, 2, 12[15] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin
Choi. The curious case of neural text degeneration. In ICLR,
2019. 11[16] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick
Forré, and Max Welling. Argmax flows and multinomial
diffusion: Towards non-autoregressive language models. In
NeurIPS, 2021. 1, 2

9

image synthesis and editing with stochastic differential equations. In ICLR, 2022. 2[34] Paul Merrell, Eric Schkufza, Zeyang Li, Maneesh Agrawala,and Vladlen Koltun. Interactive furniture layout using interior design guidelines. ACM TOG, 30(4), 2011. 2[35] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh,
Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity
metrics for generative models. In ICML, 2020. 13[36] Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann.
Learning layouts for single-pagegraphic designs. IEEE
TVCG, 20(8), 2014. 2, 4[37] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten
Kreis, Andreas Geiger, and Sanja Fidler. ATISS: Autoregressive transformers for indoor scene synthesis. In NeurIPS,
2021. 2, 5[38] Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar
Averbuch-Elor. READ: Recursive autoencoders for document layout generation. In CVPRW, 2020. 7[39] Chunyao Qian, Shizhao Sun, Weiwei Cui, Jian-Guang Lou,
Haidong Zhang, and Dongmei Zhang. Retrieve-then-adapt:
Example-based automatic generation for proportion-related
infographics. IEEE TVCG, 27(2), 2020. 2[40] Soliha Rahman, Vinoth Pandian Sermuga Pandian, andMatthias Jarke. RUITE: Refining ui layout aesthetics usingtransformer encoder. In 26th International Conference onIntelligent User Interfaces-Companion, 2021. 2, 5, 7, 12, 22[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz,
Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 8[42] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan,and Surya Ganguli. Deep unsupervised learning using
nonequilibrium thermodynamics. In ICML, 2015. 2, 3
[43] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 12[44] Yang Song and Stefano Ermon. Improved techniques fortraining score-based generative models. In NeurIPS, 2020.
2[45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In NeurIPS, 2017. 3,
5[46] Kota Yamaguchi. CanvasVAE: Learning to generate vector
graphics documents. In ICCV, 2021. 2, 8
[47] Xuyong Yang, Tao Mei, Ying-Qing Xu, Yong Rui, andShipeng Li. Automatic generation of visual-textual presentation layout. TOMM, 12(2), 2016. 2[48] Lap Fai Yu, Sai Kit Yeung, Chi Keung Tang, Demetri Terzopoulos, Tony F Chan, and Stanley J Osher. Make it home:
automatic optimization of furniture arrangement. ACM TOG,
30(4), 2011. 2[49] Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH
Lau. Content-aware generative modeling of graphic design
layouts. ACM TOG, 38(4), 2019. 2[50] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. PubLayNet: largest dataset ever for document layout analysis. In
ICDAR, 2019. 1, 2, 4

10

A. Implementation Details

where γ is a tolerance parameter, which is empirically set
to 0.1. If we want to make the j-th element above the i-th
element, the loss is defined by:

A.1. Baseline
We explain more details on task-agnostic layout generation baselines using masking, especially when the originalmodel is not designed for layout generation. We mostly
describe unconditional generation cases, but partial layout
fields can be easily injected by hard masking.
BART: BART is a denoising autoencoder and was originally designed for learning a sequence-to-sequence modelfor text generation. Text is usually represented as a 1D sequence of discrete tokens. Since we also handle the shuffledlayout as a 1D sequence of discrete tokens during training,a BART-like model may be another solid baseline. To build
a task-agnostic layout generation model, we apply randommasking similar to the noise pattern of MaskGIT [4], instead of text-specific noises, such as span-level masking.
MaskGIT∗ : MaskGIT [4] is originally built for unconditional image generation. Following recent two-stage
approaches for efficient image modeling, such as VQGAN [8], MaskGIT first generates a small number of discrete tokens and subsequently decodes those tokens into a
continuous high-dimensional image by a pre-trained neural
decoder. We consider the first generation part of MaskGIT
to be another baseline. We use [PAD] to enable variablelength generation. For a masking schedule during decoding,i.e. fraction of the tokens masked in each iteration, we employ a cosine schedule as in MaskGIT.
VQDiffusion∗ : VQDiffusion [10] is a discrete diffusionbased model designed for text-to-image generation. To
adapt VQDiffusion for conditional layout generation withminimal modification, we (i) remove the text conditioning branch in the reverse process, (ii) replace the image
tokens with layout tokens, and (iii) add [PAD] token to
enable variable-length generation. As described in themain manuscript, there are three major differences between
VQDiffusion∗ and our proposed LayoutDM: modality-wise
diffusion, decoupled positional encoding, and adaptive
quantization.
We adjust the number of parameters for each model tohave about 12M parameters for a fair comparison. We show
the exact numbers in Tab. 5.

  g_{ab}(i, j) = \max \left (\left (\hat {y}_{j} + \frac {\hat {h}_{j}}{2}\right ) - \left (\hat {y}_{i} - \frac {\hat {h}_{i}}{2}\right ), 0\right ),  (10)which compares the bottom of the j-th element and the topof the i-th element. Please refer to the code for losses forthe rest of the relationships.
Although it is not experimentally demonstrated, we believe that it is also possible to incorporate area, aspect ratio,
and reading order constraints used in Attribute-conditioned
GAN [26].
• Area: given a target area of the element ai ∈ R, we use|ai − ĥi ŵi | as a loss.
• Aspect ratio: Given a target aspect ratio ri ∈ R, we useĥi| as a loss.
|ri − ŵi
• Reading order: we follow [22] and define that the reading
order solely depends on the distance between the left-topof the canvas and eachq element. We first compute theˆdistance by di =(x̂i − ŵi )2 + (ŷi − ĥi )2 . We can2

A.3. Hyper-parameters
We search for the best hyper-parameters using a validation set. During sampling from pθ (zt−1 |zt ) for all the tasks,
we search for p used in nucleus (or top-p) sampling [15] outof {0.90, 0.95, 0.99, 1.0}. We train the models for 50 and
20 epochs in Rico and PubLayNet, respectively.
We attempt a grid search for additional hyper-parametersin the refinement task. The ranges of possible values are
the following: the distance margin m in {0.1, 0.2} and theweighting term λπ in {1.0, 2.0, 3.0, 4.0, 5.0}.

A.4. Evaluation
In unconditional generation, the model generates 1,000
samples from the random seed. In conditional generation,
the test set of each dataset is used to make a partial input forconditional generation and the model generates one sample
per each data in the test set.

A.2. Relationship Guidance

B. Additional Results

Similarly to the main manuscript, let us denote the predicted coordinates of an i-th element (x̂i , ŷi , ŵi , ĥi ) ∈[0, 1]4 . We follow [21] to define the loss for penalizing
size and location relationships between elements that do notmatch user specifications. For example, if we want to makethe j-th element larger than the i-th element, the loss is defined by:  g_{lg}(i, j) = \max \left (\left (1+\gamma \right )\hat {w}_{i}\hat {h}_{i}-\hat {w}_{j}\hat {h}_{j}, 0\right ), 2

use max(dˆi − dˆj , 0) as a loss to make the i-th element
come before the j-th element in the reading order.

B.1. Ablation Study
State space Continuous state space diffusion models havegained much attention compared to discrete state spacemodels. Recently, Li et al. [27] propose DiffusionLM that
adapts the continuous models to handle discrete text generation. DiffusionLM introduces an embedding and rounding
step to bridge the continuous and discrete state spaces. We(9)11

LayoutVAE [18] (C→S+P)
NDN-none [23] (C→S+P)
LayoutGAN++ [21] (C→S+P)
LayoutVAE [18] (C+S→P)
NDN-none [23] (C+S→P)
LayoutGAN++ [21] (C+S→P)
LayoutTrans [12]LayoutTrans-fixed [12]MaskGIT* [4]BLT [22]
RUITE [40]BART [24]
VQDiffusion* [10]LayoutDM

Rico

PubLayNet

13.2
21.8
12.9
14.7
14.8
13.0
12.7
12.7
12.7
12.7
12.7
12.8
12.4
12.4

13.0
21.8
12.9
14.5
14.8
13.0
12.7
12.7
12.7
12.7
12.7
12.8
12.4
12.4

Default
Gaussian
Negation

DiffusionLM [27]Max. ↑

Sim ↑

2.77
5.82
3.78

0.370
0.330
0.276

0.205
0.188
0.169

Table 7. Ablation study results on the choice of logit adjustment
methods in the refinement task. Top two results are highlighted inbold and underline, respectively.

loc(j) is closer to x̂i . The prior is defined by:
  \begin {aligned} & \left [ \bm {\pi }_x^i \right ]_j = \\ & \begin {cases} \left (\mathrm {loc}\left (j\right )-\hat {x}_{i}\right )^{2} & \text {if}\ \left |\mathrm {loc}(j)-\hat {x}_{i}\right | < m~\text {and}~j \in X \\ 0 & \text {otherwise}, \end {cases} \end {aligned} (11)
The ranges of possible values are similar to our method
used in the main manuscript (Default).
• Negation: j-th token is never sampled when loc(j) is far
away from x̂i . The prior is defined by:

Table 5. The number of parameters [M] used for each model.

LayoutDM
VQDiffusion* [10]

FID ↓

State

#stepsSampler

FID ↓

dis.
dis.
con.
con.
con.
con.

100
100
100
100
1000
1000

DDIM
DDPM
DDIM
DDPM

6.65
7.46
34.5
24.8
33.8
22.8

  \begin {aligned} \left [ \bm {\pi }_x^i \right ]_j = \begin {cases} 0 & \text {if}\ \left |\mathrm {loc}(j)-\hat {x}_{i}\right | < m~\text {and}~j \in X \\ -\infty & \text {otherwise}. \end {cases} \end {aligned} 
(12)
The ranges of possible values are the following: the distance margin m in {0.2, 0.4, 0.6, 0.7, 0.8, 0.9}.
We show the quantitative evaluation results in Tab. 7. We
can see that Default outperforms other possible choices bya large margin.

Table 6. Ablation study results on the choice of state spaces: discrete (dis.) and continuous (con.), in the unconditional generation
task of Rico [5] dataset. Top two results are highlighted in bold
and underline, respectively.

B.2. Speed-Quality Trade-off
We show more speed-quality trade-off curves in Fig. 8.
We perform generation with a batch size of 64 and report the
average runtime to generate a single layout for all the models. Lightly colored regions around the line plots, such as
the one in BLT for C→S+P in Rico represent the standard
deviation of three trials for each model, though the deviations are too small to see in most cases.

train DiffusionLM (with 12.6M parameters) and show theresults in Tab. 6. We show the results of DiffusionLM withembedding dimensions d = 16 because it works best outof {16, 64, 128} in Rico [5] dataset. Although We tried different samplers (DDPM [14] and DDIM [43]) and training
timesteps, DiffusionLM is still far behind the discrete state
space models in layout generation as shown in Tab. 6.

B.3. More Results
We show more results compared with task-specific baselines in C→S+P (Fig. 9), C+S→P (Fig. 11), unconditional
generation (Fig. 13), the refinement task (Fig. 15) for PubLayNet. Typical failure cases are frequent overlap between
elements (often in BLT), unnecessarily broad blank space(often in LayoutTrans.), and lack of diversity. We show
more results in C→S+P (Fig. 10), C+S→P (Fig. 12), unconditional generation (Fig. 14), the refinement task (Fig. 16)
for Rico. Rico is more difficult to generate since the number of categories is large and elements are less aligned compared to PubLayNet.

Refinement The logit adjustment proposed in the main
manuscript has some choices for injecting positional prior.
Without loss of generality, we describe a constraint that imposes the x-coordinate estimate of i-th element close to thenoisy continuous observation x̂i . We denote a sliced vector of the prior term π(zt−1 ) that corresponds to the xcoordinate of i-th element as πxi ∈ RK .
• Gaussian: j-th token is more likely to be sampled when12

B.4. Diversity-Fidelity Trade-off
We introduce density and coverage metrics by [35] to analyze the results from a different viewpoint. Density measures fidelity; i.e., how closely generated samples resemble real ones. Coverage measures diversity; i.e., whether
generated samples cover the full variability of the real
samples. We plot the diversity and fidelity of iterativerefinement-based models in Fig. 17 as we increase thenumber of timesteps for the iterative prediction. Discrete
diffusion-based models usually have higher coverage scoresand lower density scores. We conjecture that the coverage difference comes from the inference decoding strategy.
BLT [22] and MaskGIT∗ [4] fix high-confident predictions
and re-initialize lower-confident fields by [MASK] for the
next step that leads to higher fidelity. In contrast, discrete
diffusion-based models randomly corrupt the predictions
and result in higher diversity.

B.5. Alignment and Overlap
We additionally show the metrics reported in many previous works: Alignment and Overlap. Note that these metrics only capture the fidelity of generated layouts. There
are a few variants for both Alignment [21, 23, 25, 26] andOverlap [21, 25, 26]. We employ the definition in [21]. We
scale the values of Alignment by 100× for visibility. For
reference, we show Alignment and Overlap computed ina validation set as Real data. The lowest score in Alignment or Overlap does not always mean the best performancefor a model, but a model closest to Real data is the bestmodel. We show the result in Fig. 18. In the fixed-length
generation i.e. C→S+P and C+S→P, LayoutDM performs
almost comparably to VQDiffusion* [10] and BART [24],and better than the other models. In the variable-length generation i.e. the completion task and unconditional generation, autoregressive models, such as BART [24] and LayoutTrans. [12], are moderately better than LayoutDM. This
result is reasonable since these models predict the fields one
by one. Diffusion-based models, such as LayoutDM and
VQDiffusion∗ , are better than BLT and MaskGIT∗ . We believe this is because diffusion models avoid the error accumulation in iterative prediction according to [10].

13C→S+P

Rico

101

100

0

25

PubLayNet

102

FID

FID

102

100

50 75 100 125 150
Time per sample [ms]LayoutVAE
NDN-none
LayoutGAN++

101

LayoutTrans.
BART
MaskGIT*

0

25

BLT
VQDiffusion*

50 75 100 125 150
Time per sample [ms]LayoutDM
Real Data

C+S→P

Rico

101

100

0

25

PubLayNet

102

FID

FID

102

100

50 75 100 125 150
Time per sample [ms]LayoutVAE
NDN-none
LayoutGAN++

101

LayoutTrans.
BART
MaskGIT*

0

25

BLT
VQDiffusion*

Figure 8. Speed-quality trade-off of different models.

14

50 75 100 125 150
Time per sample [ms]LayoutDM
Real Data

Partial

Rico

101

100

0

25

101

100

50 75 100 125 150
Time per sample [ms]LayoutTrans.
BART

PubLayNet

102

FID

FID

102

MaskGIT*
BLT*

0

25

50 75 100 125 150
Time per sample [ms]VQDiffusion*
LayoutDM

Real Data

Unconditional

Rico

101

100

0

25

101

100

50 75 100 125 150
Time per sample [ms]LayoutTrans.
BART

PubLayNet

102

FID

FID

102

MaskGIT*BLT

VQDiffusion*
LayoutDM

025

50 75 100 125 150
Time per sample [ms]LayoutTrans.-OrderedReal Data

Figure 8. (cont.) Speed-quality trade-off of different models.

15

LayoutTrans. [12]MaskGIT∗ [4]

BLT [22]

BART [24]

VQDiffusion∗ [10]LayoutDM

Input

Real Data

Input

Real Data

Figure 9. Comparison of conditional generation in C→S+P for PubLayNet. We obtain three samples from each model to demonstrate thediversity.

16

LayoutTrans. [12]MaskGIT∗ [4]

BLT [22]

BART [24]

VQDiffusion∗ [10]LayoutDM

Input

Real Data

Input

Real Data

Figure 10. Comparison of conditional generation in C→S+P for Rico. We obtain three samples from each model to demonstrate thediversity.

17

LayoutTrans. [12]MaskGIT∗ [4]

BLT [22]

BART [24]

VQDiffusion∗ [10]LayoutDM

Input

Real Data

Input

Real Data

Figure 11. Comparison of conditional generation in C+S→P for PubLayNet. We obtain three samples from each model to demonstrate thediversity. Note that the size condition of each element is not shown for limited space. Please refer to Real Data for the size.

18

LayoutTrans. [12]MaskGIT∗ [4]

BLT [22]

BART [24]

VQDiffusion∗ [10]LayoutDM

Input

Real Data

Input

Real Data

Figure 12. Comparison of conditional generation in C+S→P for Rico. We obtain three samples from each model to demonstrate thediversity. Note that the size condition of each element is not shown for limited space. Please refer to Real Data for the size.

19Real Data

LayoutTrans. [12]MaskGIT∗ [4]

BLT [22]

BART [24]

VQDiffusion∗ [10]LayoutDM

Figure 13. Comparison of unconditional generation for PubLayNet. We obtain five samples from each model to demonstrate the diversity.

20Real Data

LayoutTrans. [12]MaskGIT∗ [4]

BLT [22]

BART [24]

VQDiffusion∗ [10]LayoutDM

Figure 14. Comparison of unconditional generation for Rico. We obtain five samples from each model to demonstrate the diversity.

21

InputRUITE [40]LayoutDM

Ground Truth

Figure 15. Comparison of the refinement task for PubLayNet. We obtain three samples from LayouytDM to demonstrate the diversity.

Input

RUITE [40]LayoutDM

Ground Truth

Figure 16. Comparison of the refinement task for Rico. We obtain three samples from LayouytDM to demonstrate the diversity.

22

C→S+P

Rico

0.8

0.8

0.6

0.6

0.4
0.2
0.0
0.0

PubLayNet

1.0

Coverage

Coverage

1.0

0.4
0.2

0.5

1.0
Density

1.5

MaskGIT*

0.0
0.0

2.0

LayoutDM

0.5

VQDiffusion*

1.0
Density

1.5

2.0

1.5

2.0

BLT

C+S→P

Rico

0.8

0.8

0.6

0.6

0.4
0.2
0.0
0.0

PubLayNet

1.0

Coverage

Coverage

1.0

0.4
0.2

0.5

1.0
Density

1.5

MaskGIT*

0.0
0.0

2.0

LayoutDM

0.5

VQDiffusion*

Figure 17. Density-coverage trade-off of different models.

23

1.0
Density
BLT

Partial

Rico

0.8

0.8

0.6

0.6

0.4
0.2
0.0
0.0

PubLayNet

1.0

Coverage

Coverage

1.0

0.4
0.2

0.5

1.0
Density

1.5

MaskGIT*

0.0
0.0

2.0

LayoutDM

0.5

1.0
Density

VQDiffusion*

1.5

2.0

1.5

2.0

BLT*

Unconditional

Rico

0.8

0.8

0.6

0.6

0.4
0.2
0.0
0.0

PubLayNet

1.0

Coverage

Coverage

1.0

0.4
0.2

0.5

1.0
Density

1.5

LayoutDM

0.0
0.0

2.0

MaskGIT*

0.5

VQDiffusion*

Figure 17. (cont.) Density-coverage trade-off of different models.

24

1.0
DensityBLTC→S+P

Rico

0.7

PubLayNet
0.5

0.6
0.4
Alignment

Alignment

0.5
0.4
0.3
0.2

0.3
0.2
0.1

0.1
0.6

0.7

0.8

0.9 1.0
Overlap

1.1

LayoutVAE
NDN-none
LayoutGAN++

0.0

1.2
LayoutTrans.
MaskGIT*
BLT

0.0

0.2

BART
VQDiffusion*

0.4 0.6
Overlap

0.8

1.0

LayoutDM
Real Data

C+S→P

Rico
0.7

0.6
0.5

0.5

Alignment

Alignment

0.6

0.4

0.4
0.3

0.3

0.2

0.2

0.1

0.1

PubLayNet

0.7

0.7

0.8

0.9
Overlap

1.0

LayoutVAE
NDN-none
LayoutGAN++

0.0

1.1
LayoutTrans.
MaskGIT*
BLT

0.0

0.2

BART
VQDiffusion*

Figure 18. Alignment and overlap of different models.

25

0.4
0.6
Overlap
LayoutDM
Real Data

0.8

Partial

Rico

0.11

PubLayNet

0.10

8

0.08

Alignment

Alignment

0.09
0.07
0.06
0.05

6
4
2

0.04
0.03

0
1.0

1.5
Overlap

2.0

2.5

LayoutTrans.
MaskGIT*

0.00
BLT
BART

0.05

VQDiffusion*
LayoutDM

0.10
Overlap

0.15

Real Data

Unconditional

Rico
1.0

0.175

0.8

0.150

0.6

0.125

Alignment

Alignment

PubLayNet

0.200

0.4

0.100
0.075

0.2

0.050

0.0

0.025
0.8

1.0
Overlap

1.2

BART
LayoutTrans.

1.4

0.0

0.2

VQDiffusion*
LayoutDM

LayoutTrans.-Ordered
MaskGIT*

Figure 18. (cont.) Alignment and overlap of different models.

26

0.4
0.6
Overlap
BLT
Real Data

0.8