Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints
Authors: Jian Chen
Ruiyi Zhang
Yufan Zhou
Changyou Chen
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion model
Layout generation
Constrained Optimization
Primary area: generative models
TL;DR: Unified model for layout generation using constrained diffusion.

Abstract: Controllable layout generation refers to the process of creating a plausible visual arrangement of elements within a graphic design (*e.g.*, document and web designs) with constraints representing design intentions. Although recent diffusion-based models have achieved state-of-the-art FID scores, they tend to exhibit more pronounced misalignment compared to earlier transformer-based models. In this work, we propose the **LA**yout **C**onstraint diffusion mod**E**l (LACE), a unified model to handle a broad range of layout generation tasks, such as arranging elements with specified attributes and refining or completing a coarse layout design. The model is based on continuous diffusion models. Compared with existing methods that use discrete diffusion models, continuous state-space design can enable the incorporation of continuous aesthetic constraint functions in training more naturally. For conditional generation, we propose injecting layout conditions in the form of masks or gradient guidance during inference. Empirical results show that LACE produces high-quality layouts and outperforms existing state-of-the-art baselines. We will release our source code and model checkpoints.

Published as a conference paper at ICLR 2024
TOWARDS ALIGNED LAYOUT GENERATION VIA DIFFUSION MODEL WITH AESTHETIC CONSTRAINTS
Jian Chen', Ruiyi Zhang', Yufan Zhou², Rajiv Jain', Zhiqiang Xu Ryan Rossi², Changyou Chen'
University at Buffalo', Adobe Research?, MBZUAI @iche338, changyou] ( @buffalo. edu, rui wwizzhnng@adbbeccom
ABSTRACT
Controllable layout generation refers to the process of creating a plausible visual arrangement of elements within a graphic design (e.g.. document and web de- signs) with constraints representing design intentions. Although recent diffusion- based models have achieved sate-ooftheearr FID scores, they tend exhibit more pronounced misalignment compared to earlier transformer-based models. In this work, we propose the LAyout Constraint diffusion modEl [ a unified model to handle broad range of layout generation tasks, such arranging ele- ments with specified attributes and refining or completing a coarse layout design. The model is based on continuous diffusion models. Compared with existing methods that use discrete diffusion models, continuous state-space design can en- able the incorporation of differentiable aesthetic constraint functions in training. For conditional generation, we introduce conditions via masked input. Extensive experiment results show that LACE produces high-quality layouts and outper- forms existing state-of-ther baselines.
INTRODUCTION
Leveraging advanced algorithms and artificial intelligence, automated layout generation serves as a cost-effective and scalable tool that facilitates a diverse range of applications including website development (Pang al., 2016). UI design (Deka et al., 2017), urban planning (Yang et al., 2013), and editing in printed media Zhong et al. 2019). Layout generation tasks fall into either uncon- ditional conditional categories. Unconditional generation refers to the process where layouts generated from scratch without predefined conditions or constraints. Conditional generation, on the other hand, is guided by user specification, such as element types, positions and sizes, or unfinished layouts, allowing for more controlled and targeted results.
Previous research utilized generative models such as GANs (Goodfellow al. 2014 Kikuchi al,
2021 VAEs (Kingma & Welling 2013), and transformer-based ( (Vaswani et al. [017] Kong et al.,
2022 models. Recent studies, however, have shifted the focus towards the application of diffusion
models for better generative quality and versatility in conditional generation (Hui et al., 2023).
Diffusion-basee models handle layout attributes as discrete or continuous variables and corrupt the data using categorical and Gaussian noise based on the discrete (Austin et al. (021) and continuous (Ho et al. 2020) diffusion frameworks. These different corruption mechanisms result distinct patterns during the generation process, illustrated in Figure [ Discrete diffusion, starting from a blank canvas, generates elements incrementally. Emerged elements tend to remain static, thus limiting the model's ability make global adjustments. Continuous diffusion, by contrast, starts with a random arrangement and refines it to an organized one over time, which is arguably more flexible in modeling.
Diffusion models, while achieving state-of-the-art FID scores, usually underperform in terms of alignment and MaxIoU scores compared to earlier transformer-based models (Gupta et al.) 2021), especially in unconditional generation (Inoue et al. 2023). These metrics can be leveraged for con- straint optimization in continuous diffusion models enhance layout aesthetics. However, due to
Cooee available at http://ghhhhi...ooommuuuuuuuuuug

---

Published as conference paper at ICLR 2024
Continuous
Diffusion
Discrete Diffusion
Figure Comparisons of latent states in continuous and discrete diffusion for layout generation. Discrete diffusion adds elements to blank canvas incrementally, and the added elements remain fixed. Continuous diffusion gradually refines a random layout into an organized one over time.
the non-differentiability of quantized geometric attributes, discrete models miss out on this opti- mization potential. On the other hand, continuous diffusion models face challenges task unifica- tion because the sample spaces of Gaussian distribution and the data distribution (canvas range and probability simplex) are different. In contrast to discrete diffusion models (Inoue et al. 2023 Hui et al. 2023), which can direct unconditional models to meet conditional goals by masking specific attributes, this strategy proves inefficient for continuous models. This limitation could explain why earlier research continuous diffusion models (Chai al. 2023 He al. 2023) focused single layout generation task.
In this study, introduce the LAyout Constriant Diffusion modEl (LACE). A unified model de- signed to generate both geometric and categorical attributes for various tasks in a continuous space. While the neural network is trained predict noise following the classic DDPM framework (He et al. 2023), we employ simple reparameterization technique to compute a layout prediction (Song al. 2020). This prediction then serves target to apply differentiable aesthetic constraint func- tions, thereby enhancing the model performance. In addition, we designed a global alignment loss and pairwise overlap loss that serve as constraint functions during the training and post- pest-processing stages. Global alignment loss promotes the learning of alignment patterns from real data during training and is used to refine global alignment in the post-processing stage. train models with masked input to unify unconditional and conditional generation, following approach similar to masked autoencoders, as seen in DiftMAE (Wei et al.) 2023). To avoid convergence to the local minimum introduced by the constraints, propose a time- deeendep weight to deactivate the con- straints for noisier time steps.
Our contribution is summarized as:
Built upon a diffusion model, we formulate various controllable layout generation tasks as conditional generation processes in continuous space, enabling constraint optimization for enhanced quality.
We propose two aesthetic constraint loss functions that promote global alignment and min- imize overlap in the layout. These functions serve as constraints during both the training and prosesrccssing phases.
We conducted extensive experiments and achieved state-of-the-art results on public bench- marks across various layout generation tasks.
2 METHODOLOGY
2.1 PRELIMINARY: CONTINUOUS DIFFUSION MODELS
Diffusion models (Ho al. 2020 are generative models characterized by forward and reverse Markov process. In the forward process, continuous valued data mata-point X is gradually corrupted to intermediate noisy latents X1-7, which converges random variable of the standard variant Gaussian distribution N (/00. I) after T steps. The transition steps between adjacent intermediate predictions is modeled as Gaussian distributions, (xx+(xx-) N vT Bex- 1, I),, with
2

---

Published as a conference paper at ICLR 2024
variance schedule 31,-. .3r- The forward process admits closed-form sampling distribution, (xx, (x0) = (xxi Vārxo, (1- )I at arbitrary timestep t, where at II.= ag and a; (1-B,).
In the reverse (generative) process, the data x is reconstructed gradually by sampling from a series of estimated transition distributions po (xxx-//x)) (xx) start from standard Gaussian random variable Xr ~ (0 I). The transition distributions are learned by optimizing the evidence lower bound consists of series of KL-divergence:
Bg DKL (q(xr (x)ll)(())) +
Σ E>1
DEL (q(xx-1/xx x (x)_lpo - log pe (xu xx.)
(1)
Since the posterior distributions ( xx, xo) are also Gaussian for an infinitesimal variance St (Feller 2015) and can be approximated by -0- parameterizztion (Song et al. 2020) as:
(xx,-11x(,x0) = -N(x; (x(e, t), 3,1)
(2)
where i)(x,,t) 1
Var VT at :)
3e
((,,) and Be 1 ār-1 B2.
(3)
āt the transition distributions could be estimated as a Gaussian with mean:
1
Bg
no t) where Eg is function approximator and is usually trained using simplified objective:
co (xx,
(4)
var
VT 0z
Simmle εo(Vái t1,,
(5)
where E NOO, I).
2.2 CONTINUOUS LAYOUT GENERATION
Similar to previous work (Inoue et al. 2023 Chat et al. 2023 He et al. 2023 we define a layout with elements as {(c1. b1),... .... (ct, bi) where e E {0.
. 1} and b, E [0, 1] the label (spanning N classes) and the bounding box for the i-th element. The bounding box is defined by its center coordinates (x, y) and sizes ratio (width, height). order to allow variable length generation, we extend layouts to uniform length L with padding elements, which has an extra class label N and bounding box initialized zero, denoted as b = (0, O, 0). Consequently, vector sequence with N + 5 dimensions
any layout within dataset of N classes is represented and length of L.
In contrast to previous discrete diffusion (Austin et al. 2021 based methods that quantize bounding box attributes to discrete bins, our approach considers them as continuous size and position ratio ranging from 0 to 1. This shift towards continuous variables enables the integration of continuous constraint functions to optimize aesthetic qualities. Moreover, following previous work (Han et al, 2022), employ continuous label vectors of (N 1) dimension to represent noisy classification logits. In other words, label vectors are not confined in the probability simplex when / >> O. How- ever, the clean data at = 0 still has one-hot label vectors. This design allows direct application of the classic diffusion model using Gaussian noise (Ho et al., 2020) without modality-wist corruption
tricks (Hui et al. 2023 Inoue al. 2023). An example is shown in Figure 2
Conditional Generation Instead of training separate models for unconditional and various condi- tional generation, we train a single neural network to handle multiple generation tasks. In training, we employ three types of binary condition masks as data augmentation that fix the label, size at- tributes of all elements or all attributes of partial elements. Specifically, given a binary condition mask m, the noisy latent is augmented as X, m + (1 m) xt, where 1 denotes all-ones matrix.
2.3 RECONSTRUCTION AND AESTHETIC CONSTRAINTS
Following recent work, introduce reconstruction loss to encourage plausible predictions of x at each time step (Austin et al., 202T Hui et al.) 2023). Thus, the total loss is defined as L = fiximle + feee-
3

---

Published as a conference paper at ICLR 2024
XT
X
X1
X
(x X;-1)
one-hot
labels
bounding
insonditional generation
boxcs
standard Ganssian
masking
padding elements
Pa 1 (N1, Xy. m)
coustraint loes
class
Diffusion Model co(Xe, f, Xo. Xo. m)
muak
given condition
class, size
conditional generation
bounding boxes
pososa
sesppe
partial element
in Ka(3.)
The figure shows layout generation of up to 15 elements from 5 classes. The layout is padded for consistency using padding elements, represented by extra class in the 6-dimensional one-hot vector. The bounding box uses 4-dimensional continuous vector. Dashed lines represent multi-step processes and solid lines represent a single step. The forward process corrupt data with Gaussian noise, while the reverse process trains neural network to denoise the noisy latent x, and its three augmentations using condition masks. Predicted bounding boxes is used compute constraint loss against real ones.
Figure 2: Overview of the layout generation model:
Specifically, by rewriting the closed-form sampling distribution for the forward process, we have x; V0 v1 046, E ~ (O, I). This equation yield way predict X each time step using the predicted noise Eg as:
xo(x;) = (x:-v1 /1 - Es (x,, Va,
The reconstruction loss is typically defined as the Mean Squared Error (MSE) between the prediction Ng and data Xg. However, when we reframe the Layout generation as continuous diffusion process, the results often exhibit poor global alignment and undesirable overlaps. This phenomenon occurs because the diffusion loss and the FID metric are not sensitive to minor variations in coordinates, yet they can cause tangible differences in visual quality. Since the contiuous search space for geometric attributes is much larger than in a discrete diffusion model, the diffusion loss holds many local minimums with similar FIDs and varying alignment scores. To address these issues, we propose to combine two additional aesthetic constraints in the reconstruction loss:
Lee = MSE($N)) ( Xy) + We (Catg (x:) X²) + (x)x))))
where wi a time-dependent weight prevent convergence to local minimum. Colp is the overlap constraint function, while Calg can refer to either the local or global alignment constraint (represented as Cl-alg: Cgaolg), which will be defined below.
2020) is common metric to evaluate the Alignment constraint The alignment loss Li et al.
alignment between layout elements for aesthetics assessment. While calculating this metric, bound- ing boxes denoted by its six coordinates b, (b4, LXC, bR, bJ. bYC. by corresponds to six possi- ble alignment types: (L.), X-center (XC), right (R), top (T), Y-center (YC), and bottom (B) alignment. Since this loss function promotes an alignment pattern where each element is aligned with just one other element based on one type, we term it local alignment loss. For set of ( bounding boxes bi.... by layout x, the local alignment loss is defined as:
CLalg(x) (x) Σ min (g(A))) ((ANN))
s(ARR) (g(A))) 9(A6))) g(Ab))
where g(x) = -log(1 - 2). Ab; = miney 16; - b3 * A= (L. XC, R, T, YC, B), and the minimum of the matrix above is defined as the smallest value among all its elements.
However, local alignment pattern is inconsistent with alignment observed in real-world graphic de- signs. For instance, in the context of website layouts, there exist elements that do not align with any others, while publication layouts, certain elements might align with multiple others in multiple ways.
4

---

Published as a conference paper at ICLR 2024
In order to encourage global alignment of elements, we propose novel alignment loss function that encourages the model to learn global aligmment pattern in the real data using an alignment mask. Specifically, for each of the six aligmment types, define the coordinate difference matrix for layout x as A A*(x),, = b6] bj6. E A. Then the alignment loss function between a given prediction & at time and real layout x is defined
- Σ
||*( 1A-())=011 11
al - log
[A-(x)=ol1;
KEA
where 1 (x)= the binary ground truth alignment mask matrix selecting the zero entries of the coordinate difference matrix of the true layout, which represents the true alignment pattern in x, and is the Hadamard product.
Overlap constraint To prevent overlapping elements in the generated layout, x, we apply mean pairwise intersection over union loss function, characterized by the pairwise IoU matrix, (xx); (b; n by)/(b; U bj). This function monotonically identifies similarities between sets but it has a continuous plane of stationary points, corresponding to the cases when a bounding box is fully contained within another. In such situations, infinitesimal adjustments to the bounding boxes do not cause the enclosed box to exceed the boundary of the larger box. To further penalize aforementioned instances, define another matrix using the distance between centers of boxes, D(x),, = e V02e + -69)), which has value range from 0 to 1. Then we combine it with a mask matrix selecting overlapping pairs using the pairwise IoU matrix. The overlap constraint function is given as:
(xp = mean (O(x) + D(x) o 10())=0)
(10)
Minimizing the D(x) term push elements away from each other when one is entirely encompassed by another and for disjoint element pairs, the mask 1 1x70 deactivates the divisive affect.
Time-dependent constraint weight In observations, directly applying the alignment and overlap loss to noisy layout hinders the faithful reconstruction. This is due to the constraint function introducing numerous local minima within the parameter space as demonstrated in Figure [8.2] To mitigate this, we employ a series of time-dependent constraint weight to enforce the constraint only for smaller time Ê to finetune the misaligned coordinates in less noisy prediction Xo. We empir- ically choose WN = (1 - āz) of a constant 33 schedule as the constraint weight series. A detailed explanation and illustration provided in Appendix B.I
post-processing The model exhibited enhanced performance across FID, alignment, and Max- IoU metrics when trained with constraints. However. LACE may generate layouts with minor misalignment as the coordinate searching space is much larger than that of the discrete diffusion model. Fortunately, further refinement of its visual quality is straightforward and efficient. Guided by the strategies outlined in (Kikuchi et al.] 202II. processing phase, focusing on global alignment and, when suitable, the overlap constraint, targeting the geometric attributes directly. The application of global alignment demands alignment mask, training, this mask is calculated using training data, ensuring the accurate learning of human -deigned alignment patterns, Since there are no ground truth in the inference post-processing stage, we propose to use a threshold 5 to identify nearly aligned entries in the coordinate difference matrix of the produced layout. Thus, the post-process forces elements to align with each other based on the pattern inferred from the raw output. should be noted that excessively large threshold will force more alignment, erasing the subtle structure of layout and leading to coarse or even collapsing generation. A small threshold, on the other hand, might be ineffective in inducing modifications. Thus, the raw outputs must have a low coordinate difference in misaligned entries. In our experiment we set equal to 1/64 of the scaled canvas range, as explained Appendix B.2
employ constraint optimization in the post-
selecting coordinate pairs for minimization.
introduced in Eq. (
RELATED WORK
Layout Generation Various automatic layout generation methods have emerged to aid the effi- cient creation of visually attractive and organized content across digital and print mediums, includ- ing websites, magazines, and mobile apps. Early works OO'Donvvan al. 2014 O'Donovan al.,
5

---

Published as a conference paper at ICLR 2024
2013
create layouts by optimizing energy- objectives derived from design principles. With the
advent of deep generative models, data-driven models LayoutGAN (Li al. 2020) and LayoutVAE
Jyothi et al. 2019) are developed and enables more flexible conditional layout generation. Layout- GAN++ (Kikuchi et al. 2021) improves the performance of LayoutGAN for unconditional layout generation and proposes CLG-LLO framework optimize pre-trained LayoutGAN++ model to fulfill aesthetic constraints. LayoutTransforne (Gupta et al., 2021) and VTN (Arroyo et al.) (021)) are autoregressive models based on transformer architectures, they improve the generation diversity and quality. However, the autoregressive mechanism hinders their application on conditional gen- eration. BLT (Kong et al.] 2022) introduces a bidirectional transformer encoder similar to masked language models to facilitate conditional generation with transformer models. However, the BLT model requires specifying the number of elements in advance. Thus, can not solve completion tasks based on partial layout. LayoutFormer-+ (Jiang et al., (023) introduces a decoding strategy in its transformer encoder-decoder architecture to enable high-quality, conditional layout generation in autoregressive manner. However, its backtracking mechanism may necessitate multiple rever- sions in the decoding process to rectify any invalid generations. Additionally, retraining is necessary for adapting to different conditional tasks. Zheng et al.] (2019) incorporated image and text features into their layout generation model to achieve content-awarenee
Diflusion-based layout generation Recently, diffusion models have been adopted to develop uni- fied models that adapt to various conditions and improve generation quality. PLay (Cheng al. 2023) uses a latent diffusion model to generate layout conditioned on guidelines lines align el- ements. LayoutDM (Inoue al. 2023) and LDGM (Hui et al.) 2023) build unified model for various generation conditions without guidelines using the discrete diffusion (Austin et al., 2021) framework. They independently developed the same attribute-specifie corruption strategy restrict variables of different attributes to their respective sample spaces in the mask-and-repless process (Gu et al., 2022). In addition, LDGM employs discretized Gaussian noise facilitate grad- ual coordinate changes. LayoutDiffusion (Zhang et al.) 2023), adopting a similar design to LDGM, enhances visual quality of three generation tasks significantly by using a larger transformer back- bone, particularly improving alignment and overlap metrics. In contrast, our method aims to en- hance visual quality by applying constraint functions without scaling up the network architecture. Motivated by the superior performance of DDPM (Ho et al. 2020) in image generation, two recent studies have aimed to generate layouts in a continuous space. A eeffusiio-basse model for uncon- ditional generation was proposed by (He et al.) (023). Although it generate embedding vectors in continuous space, geometric attributes are quantized to discrete tokens, thus, is not differentiable. Another work (Chai al., 2023) adopt DDPM for conditional generation that directly generate continuous geometric attributes (size and position) based on given categorical attributes. Because the noisy attributes share the same continuous sample space, DDPM-based models do not need the sttribute-specifiy corruption strategy. Our model further innovates by directly generating geometric and categorical attributes in continuous space and utilizing differentiable aesthetic constraint func- tions to enhance generation quality. In addition, we train diffusion models using both masked and unmasked layouts similar to DiffMAE (Wei et al. 2023) to unify various tasks in a single model.
forward
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets We use two large-scale datasets for comparison. PubLayNet (Zhong et al., 2019) consists of 330K document layouts with five class element anotation retrived from articles from PubMed CM TM Rico (Deka et al. 2017) consists of 72k user interfaces designs from 9.7k Android apps spanning 27 classes. Following LayoutDM (Inoue al. 2023), for both datasets, card instance that has more than 25 elements, and pad the remaining layouts to a unified maximum length of 25. For Rico dataset, we use layouts that has only the most frequent 25 classes. The re- fined Publ Lat and Rico datasets were then split into training, validation, and test sets containing 315, 757/1571666 619/11, 142 and 35, 851 /2, 109 4, 218 samples respectively.
dis-
Evaluation Metrics To assess the quality of generation, we employ four computational metrics. (1) Fréchet Inception Distance (FID) (Heusel et al.) (017) compute the distance between the feature distributions of generated and real layout in the feature space of pre-trained feature extraction

---

Published as a conference paper at ICLR 2024
model. It evaluate both fidelity and diversity of generative models. We employ the same feature extraction model used in LayoutDM to compute FID. (2) Maximum Intersection-over-UUU Union (Max.) is metric for conditional generation. It compute the average IoU between optimally matched pairs of elements from two layouts that contain elements with identical category sets. (3) Alignment (Align.) and (4) Overlap is used for aesthetics assessment which compute the normalized alignment score. We adopt the implementation of all metrics as proposed in LayoutGAN++ (Kikuchi et al. 202T
Generation Tasks We train a unified model for five generation tasks defined in previous works Guppa et al., 2021 Jiang et al.) 2022, Rahman et al. 2021) including unconditional generation (U-Cond), conditional generation based solely on class (C+P+ conditional generation based on both class and size (C+S of each element, completion with attributes given for subset noisy layout. For completion task, following LayoutDM (Inoue
of elements, and refinement et al. 2023). randomly sample 0% to 20% of elements from real samples the binary condition mask. In the completion task, we follow LayoutDM (IInoue et al. (023)), where we create a binary condition mask by randomly sampling between 0% and 20% of elements from real samples. In the refinement task, following RUTTE ((ahmman et al.) (021)), we introduce minor Gaussian noise to the size and position of each element, characterized by mean of and variance of 0.01, to simulate a perturbed layout. For more implementation details, please see Appendix B
4.2 QUANTITATIVE RESULTS
We benchmark our method, LACE, against several state-of-the-arr (SOTA) models on five generation tasks using the PubLayNet and Rico datasets. These models are categorized into traditional and diffusion- basod models. Traditional models are further split into task task-specific models, which include
PubLay Net
C-+S+P
C+S
Completion
U-Cond
Model
Metric
FID]
Max.
FID
Max.1
FIDI
Max.1
FIDJ
Align.
Takopectic
LayoutVAE
26.0
0,316
27.5
0,315
NDN-none
61.1
0.162
69.4
0.222
LayouGGAN++
24.0
0,263
9.94
0,342
Takoganie modch
LayoutTrans
14.1
0.272
0.320
8.36
0.451
13.9
BLT
72.
0.215
5.10
0.387
131
0.345
116
0.153
BART
9.36
0.320
5.88
0.375
9.58
0.446
16.6
0.116
MaskGIT
17.2
0.319
5.86
0.380
19.7
0.484
27.1
0.101
VQDiffusion
10.3
0.319
7.13
0.374
11.1
0.373
15.4
0.193
LayouDM
7.95
0.310
4,25
0.381
7.65
0.377
13.9
0,195
LACE (local)
4.88
0.35T
2.00
U.437
3.86
0.401
8.45
U.T44
LACE (global)
0.383
3.07
6.03
0.396
8.35
0.185
LACE (local) w/ post
4.63
0.390
2.69
0.462
5.90
0.399
8.47
0.032
LACE (global) post
0.388
7.43
Validation data
6.25
01438
6.35
0.4.8
6.25
0.438
6.25
0.021
Rico
Task
C-
C+$ P
Completion
U-Cond
Model
Metric
FID1
Max.1
FID1
Max.1
FIDL
Max.7
FID
Align.
LayoulVAE
33.3
0.249
30.6
0.283
NDN-aone
28.4
0.158
62.8
0,219
LayouGGAN+
6.84
0.267
0,348
LayoulTrans
5.57
0.223
3.73
0.323
3.71
0.537
7.63
0.068
BLT
17.4
0,202
4,48
0,340
117
0.471
88.2
1.030
BART
0.253
0.334
8.87
0.527
11.9
0.090
MaskGIT
26.1
0.262
8.05
0.320
33.5
0.533
$2.1
0.015
Bitham fuae modds
VQDifTusion
4.34
0.252
3.21
0,331
11.0
0,541
7.46
0,178
LayonDMM
0.277
0.392
9.00
0,576
6.65
0.162
LACE (local)
3.31
0.336
2.66
0.418
5.09
0.518
4.71
0,107
LACE (global)
3.24
0.340
2.87
0.418
4.45
0.527
4.63
0.117
LACE (local) post
2.88
01.347
2.17
01..30
3.82
0.539
3,59
0.031
LACE (global) w/ post
3.24
0.344
2.16
0.428
4.30
0,540
4.51
0.035
Validation data
1.85
0.691
Table I: Quantitative results on PubLayNet and Rico for four generation tasks. The top two results are highlighted with deep and light red shades, respectively.
7

---

Published as conference paper at ICLR 2024
PubLayNet
Rico
Condition LMyou:DM
LACE
LACE (post)
Real
Condition
LM
LACE
LACE (post)
Real
C-8 p
Completion
Refmement
Figure 3: Qualitative comparison between LACE and LayoutDM in conditional generation tasks.
LayoutVAE (Jyothi et al. 2019), NDN-none (Lee et al. 2020), LayoutGAN++ (Kiuuchi et al,
2021. and RUTTE (Rahman al. 2021), and task-agnostic models like LayoutTrans (Gupta al. BLT (Kong et all 2022). and MaskGIT (Chang et al. 2022). On the other hand, the diffusion- based models considered are VODiffusion (Gu et (002) and LayouDDM (Inoue
2021
2023).
For evaluating unconditional generation, we utilize FID and Alignment metrics. Mcanwhile, for refinement and conditional generation based on partial attributes, we apply FID and MaxloU. Since the overlap patten is prevalent in the Rico dataset, we only apply the overlap constraint during training on the PubLayNet dataset. Therefore the overlap metric only used in experiments on the PubLayNet dataset. Furthermore, results from models using both local and global alignment constraints are reported the benchmark experiments. the post-processing stage, we optimize the global alignment using the d-alignment mask introduced in section 2.3
Conditional and Unconditional Generation We demonstrate that LACE, even without post- processing, achieves state-of-the-art (SOTA) performance four tasks including unconditional and conditional generation given partial attributes. LACE surpasses other models that utilize discrete diffusion in benchmark experiments conducted on Rico and PubLayNet datasets. The results for LACE. along with nine baseline methods, are presented in Table [ and Appendix A We observed Both LACE versions, using local and global aligmment constraints, show similar performance. LACE (global) exhibited superior performance prevalence alignment patterns in PubLayNet. The results of LACE with post-processing show significant improvement in alignment, while either maintaining slightly enhancing the FID. This is attributed to the adjustments that fine-tune the alignment without substantially altering the global organization of elements. Figure shows the qualitative comparison results between LACE and LayoutDM for conditional generation. Additional qualitative results are in the Appendix C
the PubLayNet dataset, possibly due to the greater
Refinement Continuous diffusion using Gaussian noise is naturally an expert at refining noisy layouts because noisy layouts can be viewed noisy states during the forward process. And the continuous diffusion model is trained to reverse the corruption. Since it is only beneficial to refine layout that is clear enough to show the relative position elements, we assume the noise level the input matches the noise states time step where the time-dependent constraint weight starts to encourage the importance of the constraint functions (w; = 0.1). Thus, we use fixed LACE model to refine noisy layout, treat the input noisy state at time and start partial reverse process to LACE outperforms competing models, achieving
refine it. The results shown in Table FID and MaxIoU scores comparable that of LACE with post-processing.
8

---

Published as a conference paper at ICLR 2024
Dataset
Rico
PubLayNet
Model
FID4
Maxt
FIDL
Maxt
RUITE RRahman et al. 2021)
263.23
0.421
6.39
0.415
Taek agsonii modes
Noisy input
0.213
0.242
LayoutDM
2.77
0.370
4.25
0.381
LACE (local)
1.79
0.485
1.65
0.491
Peo
LACE (local)
1.76
0.484
1.80
0.500
Validation data
6.25
0.438
6.25
0,438
Table 2: Results in the refinement task on PubLayNet and Rico datasets.
4.3 ABLATION STUDY
We conducted ablation study the PubLayNet dataset to evaluate the role of aesthetic straints and masked input during training. We trained a LACE model without aesthetic constraints and three diffusion models with constraints for unconditional generation and two conditional tasks (C-S+P, C+S-P). Using the unconditional model, we also applied the in-painting technique (Lug- mayr et al. 2022 for completion. We only include LACE using local alignment constraint (noted as LACE w/ C). in the ablation study, as LACE with both constraints exhibit similar performance. Table displlas the results. The result of task-specific models is comparable LACE, which proves the effectiveness of our task unification method. LACE achieves a much lower alignment score and slightly reduced FID compared to LACE without aesthetic constraints, suggesting the effectiveness of constraint optimization. In addition, the post-process notably improves the alignment in uncon- ditional generation without sacrificing the FID score. However, the post-process is not as effective for LACE without constraints. A possible explanation is the coordinate difference in the raw output is too large for the threshold to detect, resulting in a smaller number of coordinate differences being minimized, We include detailed alignment and overlap results in Appendix A
Task
Metric
Tpec-ip
LACE
LACE
LACE w/o w/ post
LACE
w/o
w/ post
FID.
6.76
8.70
8.45
10.31
8.47
U-Cond
Align,
0,128
0.238
0,141
0.215
0.032
FID.
6.12
5.08
4.88
4.76
4.63
C-+S+P
Muxt
0,386
0.383
0,332
0,389
0,390
FID.
2.25
3.21
2.80
2.57
2.69
C+S-*P
Muxt
0.478
0.460
0.437
0.461
0.462
Completion
FID.
7.73*
6.42
5.86
6.01
5.90
Muxt
0.399
0.403
0.401
0.385
0.399
Table 3: Results of the ablation study conducted the PubLayNet dataset. The asterisk (*) denotes in-painting results using the unconditional model. Symbol C represents 'constraints', while 'post' refers to post-processing. The top two results highlighted in deep and light red shades.
5 CONCLUSIONS
We introduced LACE, diffusion a unified model for controllable layout generation in a continuous state space. Using the diffusion framework with alignment and overlap loss constraints, LACE out- performs previous state-of-the-art baselines in both FID and visual quality. Efficient post-processing can further improve generation quality regarding alignment and overlap, without sacrificing FID. While LACE demonstrates advancements in layout generation. it has several limitations. First, it restricts layout elements to box shapes, limiting representation flexibility. Additionally, lacks background and content awareness, which could be crucial for more context-driven designs. The model also handles only limited number elements and relies on a closed label set. These con- straints may restrict its applicability in complex, varied design scenarios. For future work, adopting arbitrary shapes can better mirror real-world graphic design scenarios as most existing work relies on rectangular boxes for element representation. In addition, using images text conditions to generate more relevant layout for the content should improve the performance downstream tasks, such syyout-guided image generation and automated web design.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is partially supported by NSF AT institute-2229877 NSP RI-2223292, an Amazon re- search award, and an Adobe gift fund. Any opinions, findings and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, the Institute of Education Sciences, the U.S. Department of Educa- tion.
REFERENCES
Diego Martin Arroyo, Janis Postels, and Federico Tombari. Variational transformer networks for layout gen- cration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13642-13652, 2021.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981- 17993, 2021.
Shang Chai, Liansheng Zhuang. and Fengying Yan. Layoutdm: Transformer-basee diffusion model for layout generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp-
18349-18358, 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
11315-11325, 2022.
Chin-Yi Cheng, Forrest Huang, Gang Li, and Yang Li. Play: Parametrically conditioned layout generation using latent diffusion. arXiv preprint arXiv:2301.115222 2023.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. Proceedings of the 30th annual ACM symposium on user interface software and technology, pp. 845-854, 2017.
Vincent Dumoulin, Jonathon Shiens, and Manjunath Kudlur. leamedd representation for artistic style. In International Conference on Learning Representations, 2017.
William Feller. On the theory stochastic processes, with particular reference to applications. Selected Papers I, pp. 769-798, Springer, 2015.
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing xystems, 27, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16696-17766 2022.
Kamal Gupta, Justin Lazarow, Alessandro Achille, Larry Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layouttransforerr: Layout generation and completion with self offfaattetion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1004- 1014, 2021.
Xizewen Han, Huangjie Zheng, and Mingyuan Zhou. Card: Classification and regression diffusion models. Advances in Neural Information Processing Systems, 15:18100018811 2022.
Liu He, Yijuan Lu, John Corring, Dinei Florencio, and Cha Zhang. Diffusion-hased document layout genera- tion. arXiv preprint XrXiv:2303 10787, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising difffusion probabilistic models. Advances in Neurul Information Processing Systems, 33:6840-666 (001, 2020.
Mude Hui, Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, Yuwang Wang. and Yan Lu. Unifying layout generation with decoupled diffusion model. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1942-1951, 2023.
10

---

Published as a conference paper at ICLR 2024
Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Layoutdm: Discrete dif- fusion model for controllable layout generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10167-10176, June 2023.
Zhaoyun Jiang, Huayu Deng, Zhongkai Wu, Jiaqi Guo, Shizhao Sun, Vuksan Mijovic, Zijiang Yang, Jian- Guang Lou, and Dongmei Zhang. Unilayout: Taming unified sequence-t--squenne transformers for graphic layout generation. arXiv preprint 2xXf:22088000::1 2022.
Zhaoyun Jiang, Jiaqi Guo, Shizhao Sun, Huayu Deng. Zhongkai Wu, Vuksan Mijovic, Zijiang James Yang. Jian- Guang Lou, Dongmei Zhang. Layoutformer++: Conditional graphic layout generation con- straint scrialization and decoding space restriction. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 18403-18412, 2023.
Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from label set. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 9895-9904, 2019.
Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, and Kota Yamaguchi. Constrained graphic layout generation via latent optimization. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 88-96,
2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Xiang Kong, Lu Jiang, Huiwen Chang. Han Zhang, Yuan Hao, Haifeng Gong, and Irfan Essa. Blt: bidirectional layout transformer for controllable layout generation. In European Conference Computer Vision, pp. 474- 490. Springer, 2022.
Hsin- Ying Lee, Lu Jiang, Irian Essa, Phuong Le, Haifeng Gong. Ming- Hsuan Yang. and Weilong Yang. Neural design network: Graphic layout generation with constraints. In Computer VisionEECCT 2020: 16th European Conference, Glasgow, UK. August 28, 2020, Proceedings, Part III 16, pp. 491-506. Springer, 2020.
Jianan Li, Jimei Yang, Jiamming Zhang, Chang Liu, Christina Wang, and Tingfa Xu. Attribuueecomditionee layout gan automatic graphic design. IEEE Transactions on Visualization and Computer Graphics, 27 (10)4039 4048, 2020.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11461-11471, 2022.
Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. Designscape: Design with interactive layout suggestions. Proceedings of the 33rd annual ACM conference on human factors in computing systems, pp. 1121-12244 2015.
Peter O' O'Donvann, Aseem Agarwala, and Aaron Hertzmann. Leamning layouts for kinle-pagegraphar designs. IEEE Irunsactions on visualization and computer gruphics, 2008):1200-1211 1213, 2014.
Xufang Pang, Ying Cao, Rynson WH Lau, and Antoni B Chan. Directing user attention via visual flow web designs. ACM Transactions on Graphics (TOG), 35(6) 2016.
Soliha Rahman, Vinoth Pandian Sermuga Pandian, and Matthias Jarke. Ruite: Refining ui layout aestheties
using transformer encoder. In 26th International Conference on Intelligent User Interfaces Cetefface--Coopaiion pp- 81 81-83, 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arX arX1v:2010.02502, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
Chen Wei, Karttikeya Mangalam, Yo-Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. arXiv preprint arXiv:2304.0322 2023.
Yong-Liang Yang, Jun Wang, Etienne Vouga, and Peter Wonka. Urban pattern: Layout design by hierarchical domain splitting ACM Transactions on Graphics (TOG), 32(6):1-12, 2013.
11

---

Published as a conference paper at ICLR 2024
Junyi Zhang, Jiaqi Guo, Shizhao Sun, Jian-Guang Lou, and Dongmei Zhang. Layoutdiffusion: Improving graphic layout generation by discrete diffusion probabilistic models, arXiv preprint arXiv:2303.//589 2023.
Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson WH Lau. Content-aware generative modeling of graphic design layouts. ACM Transactions on Graphics (TOG), 38(4):1-13, 2019.
Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset for document layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1015 1022. IEEE, 2019.
12

---

Published as a conference paper at ICLR 2024
A ADDITIONAL ABLATION RESULTS
We compared LACE with LayoutDM and LayoutGAN++ in terms of overlap and alignment. Lay- outGAN++ is a task-specific model that also uses constraints training and post-processing. We directly adopt their results in the original paper. In addition, apply post-processing to LayoutDM to demonstrate the advantage in using constraint during training for diffusion model. Results are shown in Table A.I and Table A.2 Both LayoutDM and LayoutGAN++ show notable FID in- crease after post-pus processio..... In contrast, LACE maintains a stable FID and achieves lower overlap and alignment scores before post-process proccssioc. which are further improved afterward.
Task
C-+S+P
Model
Metric
FID4
Alignl
Overlap
Task-paciii madch
NDN-none
61.1
0.350
16.5
LayouiGAN++
24.0
0.190
22.80
LayoutGAN++ w/ C
22.3
0.160
14.27
LayoutGAN+ w/ & post
26.2
0.160
1.18
Diffaiin-basod
LayoutDM
7.95
0.106
LayoutDM w/ w/post post
15.2
0.083
6.076
LACE w/o C
6.12
0.054
T.636
LACE (local)
4.88
0.043
1.638
LACE (global)
5.14
0.046
1.791
LACE (local) w/ post
4.63
0.010
1,211
LACE (global) w/ post
4.56
0.009
0.906
Validation data
6.25
0.021
0.117
Table A.1: FID, overlap and alignment results in the C-+S+P task on the PubLayNet dataset.
Additionally, LACE, even without post-processee outperforms LayoutDM with post-prose post-processing in alignment and overlap scores. This serves as proof the effectiveness of constraint loss in our approach.
Task
Completion
Model
Metric
Aligne
Overlap.
Align]
Overlap.4
Align./
Overlap.
LayoutDM
0.119
18.91
0.107
15.04
0.195
13.43
LayourDM w/ post
0.117
6.506
0.073
5.220
0.200
4.641
TACH w/o C
0.065
3.162
0.0154
3.223
0.238
7.533
LACE (locul)
0.061
3.309
0.040
2.772
0.141
3.615
LACE (global)
0.061
3.439
0.042
3.056
0.185
4.140
(locul) w/ post
0.016
1.400
0.014
1.723
LACE (global) w/ post
0.017
1.363
0.017
1.573
0.074
0.768
Validation data
0.021
0.117
0.021
0.117
0.021
0.117
Table A.2: Overlap and alignment results on PubLayNet for three generation tasks.
B IMPLEMENTATION SPECIFICATIONS
B. TIME-DEPENDENT CONSTRAINT WEIGHT
The time denendeenndnnt constraint weight is critical for effective model convergence and output quality. Without this weight, the model struggles to converge, leading to high Fréchet Inception Distance (FID) score, typically remaining above 100, which indicates poor layout quality. We choose wi (1- (1-6 āz) of a constant 3 schedule as the constraint weight series. The 3 schedule is set empirically such that the weight activates the constraint only when is small when the corruption process has not introduced too much overlap, as demonstrated in Figure B.T Thus, in the reverse process, the coarse structure of the layout has emerged. Figure B.2 demonstrates the local minimum induced by the constraint functions at noisy steps, hindering convergence.
13

---

Published as a conference paper at ICLR 2024
1.0
1.0
Timedeppendetl constraint weight
Time-dependent constraint weight
Mean painwise loU
0.8
0.8
Mean painwise loU
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0
200
400
600
800
1000
time
Figure B.I: Time-dependent constraint weight and Mean Pairwise IoU in the forward process.
cases converging to local minitmmm
Optnnise
clean layout
noisy layout &
MSE CR.aix
MSE Colo
MSE Cg.alx + Colp
Figure B.2: Examples of convergence to local minimum with alignment and overlap constraints
B.2 POST-PROCESSING THRESHOLD
We use the global alignment and overlap (only on PubLay constraints to optimize the raw out- put of LACE. Since there is target layout compute the ground truth alignment mask matrix in Eq. (9). we use a threshold value to compute an alignment mask matrix using the coordinate difference matrix of the generated layout. To determine the threshold for enhanced visual quality post- post-processing we first scaled the normalized canvas according to its width/height ratio. We then tested various threshold values, including 1/16, 1/32, 1/64, 1/128, and 1/256. The optimal threshold was empirically determined to be 1/64 of the scaled normalized canvas size. This setting aligns with real dataset observations, where only 0.5% unaligned coordinate pairs have a smaller difference.
B.3 MODEL ARCHITECTURE
As illustrated in Figure B.3 we adopt transformer architecture that is implemented in the source code of LayoutDM Inoue et al. (202) to predict the noise term in Eq. (5). We also choose similar hyper-paranee apprr-pareettr settings for fair comparison: layers, 16 attention heads, 2048 hidden dimension in the FNN (feed-forward networks), and the embedding dimension is 1024 for PubLayNet and 512 for Rico. addition, we add two FNNs to encode and decode element vectors. Time embeddings
are injected by an modified adaptive layer normalization Dumoulin et al. (2017). Specifically, the layer normalization is: y (1 fv(.))
x- 1
+ fz(v.).
(B.1)
where y the imput and output of the normalization function. fz. fo are FFN that encode the demendependent scale and shift, a are x's mean and standard deviation, v, is the time embedding, is vector of all represents residue connection. is the Hadamard product.
14

---

Published as a conference paper at ICLR 2024
element sequence
Input
Time Embedling
Inbel
box
LayerNocm
FNN
FNN
FNN
Position Eubedding
Norm
Multi-Head Self Arteucion
Embedding
Transformer Block x 4
LagerNorm
FNN
FNN
Transformer Block
(o(X..t)
Hadamard product
Summation
Outpul
Figure B.3: Neural network architecture for the layout generation diffusion models. The the network takes sequence of layout elements the time variables as input and output the predicted noise. The pink blocks in the figure represent the trainable network components, gray blocks represent input tensors.
B.4 TRAINING DETAILS
We train the model using the Adam optimizer. The batch size is 256. We used a learning rate schedule that included a warmup phase followed by half-cycle cosine decay. The initial learning rate is set to 0.001. Training is divided into two phases: initially, the model is trained without constraints (w; 0) until convergence observed in the FID score. In the second phase, constraints are added to the total loss, and training continues until convergence is achieved in both alignment and FID scores. For the Rico dataset, the overlap constraint is excluded due to prevalent overlap patterns in real data. However, in the PubLayNet dataset training, the overlap constraint is applied to prevent undesirable overlaps publication layouts. The diffusion model employs total of 1000 forward steps. For efficient generation, use DDIM sampling with 100 steps.
15

---

Published as a conference paper at ICLR 2024
C
QUALITATIVE COMPARISON
LACE
Real
LayoutDM
LACE
Real
LayoutDM
Figure C.I: Qualitative comparison between LACE w/ post-processing (left), real (middle), and LayoutDM (right) in conditional generation tasks (C+S - P) the PublayNet dataset.
16

---

Published as a conference paper at ICLR 2024
LACE
Real
LayoutDM
LACE
Real
LayoutDM
Figure C.2: Qualitative comparison between LACE w/ post-processigg (left), real (middle), and LayoutDM (right) in conditional generation tasks (C+S - P) the Rico dataset.
17

---

Published as a conference paper at ICLR 2024
Figure C.3: Qualitative comparison between LACE w/ post-processing (left) and LayoutDM (right)
in unconditional generation tasks on the PublayNet dataset.
Figure C.4: Qualitative comparison between LACE w/ post-processing (left) and LayoutDM (right) unconditional generation tasks on the Rico dataset.
18


Reviews:
[{"review_id": "LKXHHcPcrE", "reviewer": "Reviewer_DJP7", "summary": "The paper proposes a model for unconditional and conditional layout generation, and layout de-noising, where layout is defined as a set of rectangles of particular clas placed on the larger canvas, and conditioning can take form of number of rectangles of each class.The main contribution of the paper is modelling this process as continuous diffusion process, as opposed to the previous work which did it as a discrete diffusion process, after bining the rectangular coordinates. The method outperforms other layout generation models, as well as generic models such as MaskGIT, as measured by FID score and the IoU-related metric. The additional contribution is the introduction of a new aesthetic constraint, which further improves the results, used both as a loss during training, and during post-processing, where the rectangles are adjusted to reduce the object overlap.", "questions": "Do you envision any practical applications of the proposed approach? Do you believe it could be extended in a way that would be useful for applications? In which way?\n\nSec. 2.2, \"Conditional generation\": Typo \"We\" --> \"we\"\nPage 4, section \"Overlap constraint\": Typo \"is monotonically identify\" --> \"monotonically identifies\"\nPage 6, Sec. 4.1, \"Datasets\": Typo \"for both dataset\" --> \"for each dataset\" or \"for both datasets\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: While neither the method nor the application domain is new, it is novel to apply the continuous diffusion to the layout generation problem.\nClarity: The writing is sufficiently clear.\nQuality: The paper features an ablation study highlighting the most important design decisions, such as post-processing, and the use of aesthetic constraint to minimize the loss. The authors promise to make the code and model checkpoints available, but do not provide them together with the submission.\nSignificance: I believe the work has significance in exploring the use of continuous diffusion to layout generation task, but given the somewhat \"toy\" setup of the experiments, the practical applicability and the wider significance of layout generation when specified in this formulation (given only types of objects or initial layout, without any information about the textual content of text fields, or image contents of the image fields, with evaluation done through FID and not human studies) may be limited.", "weaknesses": "There aren't significant drawbacks in the manuscript itself, with main potential weakness for me being the question of practicality of the proposed approach or its extensions to any practical application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3czvhtdYy", "reviewer": "Reviewer_6Y7v", "summary": "This paper introduces a novel diffusion model and training process for various conditional and unconditional layout generation tasks. Utilizing a continuous state-space design directly on the target attributes of elements in the layout, the proposed method incorporates both the standard diffusion training objective and novel aesthetic-based objectives (i.e., overlap, alignment) to improve the quality of generated layouts. These improvements lead to a new SoTA model across the majority of layout generation tasks in both the Rico and PubLayNet dataset.", "questions": "- Can the author(s) provide more details of model training (e.g., loss schedule, model architecture), given that the introduced method applied diffusion in a novel and complex state-space that is non-trivial?\n\n- Will the code be open-sourced if this paper is accepted?\n\n- Can you further discuss the importance of the time-dependent constraint weight (preferably empirically)? This appears to be a very important parameter in incorporating aesthetic constraints.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novel formulation and diffusion model for directly using a continuous state-space for geometric and class properties, which could opens up a new avenue of research for layout generation by exploring diffusion in this space\n\n- Unified framework for both unconditional generation and multiple conditional tasks\n\n- Achieved SoTA in the majority of conditional tasks in both Rico and PubLayNet\n\n- Novel formulation of Overlap and Alignment losses, and a time-dependent constraint schedule for effective optimization of aesthetic constraints during the diffusion process, which can independently benefit future research given the importance of these constraints in good layouts.", "weaknesses": "- *Incomplete details for reproduction*: It appears that no training details of the models are reported in the paper and the appendix. It would be extremely helpful for future research for the author(s) the report details such as model architecture, noise schedule, loss schedule, constraint weights, etc, given that this continuous state-space of diffusion for layout generation was previously unexplored.\n\n- *Limited inclusion of qualitative results and human-rater experiments*: It would further improve the paper if the author(s) can provide more qualitative results that demonstrates the differences between the proposed method and existing SoTA, and preferably include a lightweight human-rating study given that the metrics might not fully reflect human preference for layout generation.\n\n- *Missing common metrics*: While FID and MaxIoU is quite representative and established in the layout generation literature, it will be helpful if the author(s) can include alignment and overlap indices, which can further show the effect of the proposed aesthetic constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wNlGtdeyO0", "reviewer": "Reviewer_5Gyc", "summary": "This paper focuses on the controllable layout generation tasks, and formulates them as conditional generation processes in continuous space for aesthetic quality optimization. Two aesthetic constraint losses are proposed for global alignment and minimizing overlap in the layout during the training and post-processing stages. The experimental results show that the proposed method achieves the state-of-the-art performances on several benchmarks.", "questions": "1. How to determine the time-dependent constraint weight and the threshold used in the post-processing stage?\n2. Are there any qualitative results that demonstrate diversity of the generated layouts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Compared to existing works that use discrete diffusion models, the proposed method based on continuous diffusion models can incorporate continuous aesthetic constraints in training. A new alignment loss is proposed to encourage the global alignment of elements. The proposed model can handle multiple generation tasks without retraining.", "weaknesses": "1. The key idea of this paper is to enable constraint optimization by formulating layout generation as conditional generation processes in continuous space. However, a post-processing step is still conducted. I was wondering about the advantage of the proposed method compared to those that directly apply the post-processing algorithm to the discrete diffusion models.\n2. Several layout generation related works [1-3] are neither cited nor discussed in the paper.\n3. For the overlap constraint, the current loss pushes elements away from each other. However, in some cases, designers would intentionally use overlap to make the top element look closer to the viewer. For example, text elements are often located above the image elements in banner ads. The usefulness of such a constraint would be questionable by directly applying it to all elements in graphic designs.\n4. Why the evaluation metrics do not consider overlap, while the aesthetic constraints contain the overlap in layout?\n\n[1] Zheng, Xinru, et al. \"Content-aware generative modeling of graphic design layouts.\" ACM TOG, 2019.\n[2] Zhang, Junyi, et al. \"LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models.\" ICCV 2023.\n[3] Jiang, Zhaoyun, et al. \"LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restriction.\" CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t7pWGSj6qI", "reviewer": "Reviewer_w7ws", "summary": "This paper presents a novel unified continuous diffusion model, namely LAyout Constriant Diffusion modEl (LACE) for layout generation. To address the aesthetic issue of the diffusion model, the authors propose two differential constraints, including alignment constraint and overlap constraint, to address element arrangement and overlap issues. The experiments show that the proposed LACE makes a good performance on PubLayNet and Rico datasets.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper is interesting, well-written and original, and my impression is positive.\n* The proposed framework for various generation tasks is smart and effective.\n* The quantitative experiment results show the superiority of the proposed method over existing approaches.", "weaknesses": "* I would appreciate it if the authors give more details of the proposed framework given different conditions.\n* The ablation study can be extended to illustrate the contribution of two constraints. For example, Figure 3 only shows the visualization result of global alignment and overlap constraints. I would recommend studying the effectiveness of all sub-constraints, e.g., local and global alignment, quantitatively and qualitatively.\n* The limitations of the proposed method should be discussed. This can give good guidance for the application in real-world situations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)