Markup-to-Image Diffusion Models with Scheduled Sampling
Authors: Yuntian Deng
Noriyuki Kojima
Alexander M Rush
Venue: ICLR 2023 poster (2023)

Abstract: Building on recent advances in image generation, we present a fully data-driven approach to rendering markup into images. The approach is based on diffusion models, which parameterize the distribution of data using a sequence of denoising operations on top of a Gaussian noise distribution. We view the diffusion denoising process a sequential decision making process, and show that it exhibits compounding errors similar to exposure bias issues in imitation learning problems. To mitigate these issues, we adapt the scheduled sampling algorithm to diffusion training. We conduct experiments on four markup datasets: formulas (LaTeX), table layouts (HTML), sheet music (LilyPond), and molecular images (SMILES). These experiments each verify the effectiveness of diffusion and the use of scheduled sampling to fix generation issues. These results also show that the markup-to-image task presents a useful controlled compositional setting for diagnosing and analyzing generative image models.

Published as a conference paper at ICLR 2023
MARKUP-TO-IMAGE DIFFUSION MODELS WITH SCHEDULED SAMPLING
Yuntian Deng', Noriyuki Kojima", Alexander M. Rush
harvard. edu 2 Cornell University {nk654, arush) www.sheccorreell.. edu
Harvard University dengyuntiand iansseas www.ssaaa.aaaaa.....mo
ABSTRACT
Building on recent advances in image generation, we present fully data-driven approach to rendering markup into images. The approach is based on diffusion models, which parameterize the distribution of data using a sequence of denois- ing operations on top of Gaussian noise distribution. We view the diffusion denoising process as a sequential decision making process, and show that it ex- hibits compounding errors similar to exposure bias issues in imitation learning problems. To mitigate these issues, we adapt the scheduled sampling algorithm diffusion training. We conduct experiments ematical formulas (LaTeX), table layouts (HTML), sheet music (LilyPond), and molecular images (SMILES). These experiments each verify the effectiveness of the diffusion process and the use of scheduled sampling to fix generation issues. These results also show that the markup-to-image task presents useful controlled compositional setting for diagnosing and analyzing generative image models.
four markup datasets: math-
INTRODUCTION
Recent years have witnessed rapid progress text-to-image generation with the development and deployment of pretrained image/text encoders (Radford et al., 2021 Raffel et al. 22200 and pow- erful generative processes such as denoising diffusion probabilistic models (Sohl-Dicksterl et al.]
Ho al. 2020). Most existing image generation research focuses generating realistic im- ages conditioned on possibly ambiguous natural language (Nichol et al. 2021 [ahariaa et al. [022] Ramesh et al. 2022). In this work, instead study the task of markup-to-imagg generation, where the presentational markup describes exactly one-to-one what the final image should look like.
2015
While the task of markup-to-inen -m-amaae generation can be accomplished with standard renderers, argue that this task has several nice properties for acting as a benchmark for evaluating and analyzing text-to-image generation models. First, the deterministic nature of the problem enables exposing and analyzing generation issues in setting with known ground truth. Second, the compositional nature of markup language nontrivial for neural models to capture, making it challenging benchmark for relational properties. Finally, developing model-based markup renderer enables interesting applications such as markup compilers that are resilient to typos, or even enable mixing natural and structured commands (Glennie, 1960 Teitelman 1972).
We build collection of markup-to-image datasets shown in Figure I mathematical formulas, table layouts, sheet music, and molecules (Nienhuys & Nieuwenhuizen] 2003 Weininger] 1988). These datasets can be used to assess the ability of generation models to produce coherent outputs structured environment. We then experiment with utilizing diffusion models, which represent the current state-of-the-an
conditional generation of realistic images, on these tasks.
The markup-to-image challenge exposes new class of generation issues. For example, when gen- erating formulas, current models generate perfectly formed output, but often generate duplicate or misplaced symbols (see Figure 2). This type of error is similar to the widely studied exposure bias issue in autoregressive text generation (Ranzato et al. 2015). To help the model fix this class of emors during the generation process, we propose to adapt scheduled sampling (Bengio al. 2015)

---

Published as a conference paper at ICLR 2023
Math
Table Layouts
G (-a)m 22n
fj
Yhopf ~ Σ n>0
i \gamma { Anathrm hool Widettlde
<apan style=" Cont wont-weight. :bol;;
G F Vaineq
text align:center; size:1509; </span> </div>
Virac (1-a a 3 3
Sheet Music
Molecules
Wolcee (caiN) (=0)) No2ccanc2
Arelative of Mimm 4/4
12 64
b2 cest bd" 32 E4 ad
d8 ed
g16 g2 E2
| des2 d8 da E8 e4 a16 b1s
e2
d2. sa" g4 s16" e16. F4
62
54.
16 a15 84, 22" 04
62
d4
rB.
Figure Markup-to-Image suite with generated images. Tasks include mathematical formulas (LaTeX), table layouts (HTML), sheet music (LilyPond), and molecular images (SMILES). Each example conditioned on a markup (bottom) and produces a rendered image (top). Evaluation directly compares the rendered image with the ground truth image.
Specifically, we train diffusion models by using the model's generations input such that the model learns to correct its own mistakes.
Experiments all four datasets show that the proposed scheduled sampling approach improves the generation quality compared to baselines, and generates images of surprisingly good quality for these tasks. Models produce clearly recognizable images for all domains, and often do very well at representing the semantics of the task. Still, there is more to be done to ensure faithful and consistent generation in these difficult deterministic settings. All models, data, and code are publicly available at /attps://gii com da03/ 20.033mmarkupimm
MOTIVATION: DIFFUSION MODELS FOR MARKUP-TO-IMACE GENERATION
Task We define the task markup-to-image generation as converting a source in a markup image that target image. The input is sequence of M tokens I1, ... IM E X, and the target an image y E y S R of height H and width W (for simplicity we only consider grayscale images here). The task of rendering is defined
language describing
mapping X V. Our goal is to approximate the rendering function using a model parameterized by {1,2, ,N). To make the task
fe + y trained on supervised examples {(x'.)) (of.y))
tangible, we show several examples of x, pairs in Figure
Challenge The markup-to-image task contains several challenging properties that are not present other image generation benchmarks. While the images are much simpler, they act more dis- cretely than typical natural images. Layout mistakes by the model can lead to propagating emror throughout the image. For example, including an extra mathematical symbol can push everything one line further down. Some datasets also have long-tern ternttermmmm symbolic dependencies, which may be difficult for non con-sequentaa models to handle, analogous to some the challenges observed non- autoregressive machine translation (Guu et al. 2018).
2

---

Published as a conference paper at ICLR 2023
850
750
650 %
550
"
nz 20
1/2
450
% a +
nn 0
7/2 0
0.0.
350
a + a 2n$
1/4 a
m #0
250
% = a + d 7
ā 0.
0
% + 2n
7 m) of 0.
Figure 2: The generation process of diffusion (left) versus diffusion+schedule sampling (right). The numbers on the y-axis are the number of diffusion steps (t). The ground truth LaTeX is \gamma_ (n) \mmu)\aaphha (n) (( (amm)+(tiidd \alpha)_[mn (mmu), "
Generation with Diffusion Models 2020) parameterize probabilistic distribution P(yo|2) xz) as Markov chain P(y 134)) with iniial distribution (yr).. These models conditionally generate an image by sampling iteratively from the following distribution (we omit the dependence for simplicity):
Denoising diffusion probabilistic models (DDPM) (Ho et al.,
P(ur) = 10,,1)
P(y--1l9e) = ((uo()it))
where y1. ".. .TT are latent variables of the same size as yo E V. po(,,t) t) is a neural network parameterizing map y.
Diffusion models have proven to be effective for generating realistic images (Nichol et al. 2021
Saharia et al., 2022, Ramesh et af. (022) and are more stable to train than alternative approaches for image generation such as Generative Adversarial Networks (Goodfellow et al.) [2014). Diffusion models are surprisingly effective on the markup-to-image datasets as well. However, despite gener- ating realistic images, they make major mistakes in the layout and positioning of the symbols. For an example of these mistakes see Figure (left).
We attribute these mistakes to eтгог propagation in the sequential Markov chain. Small mistakes early in the sampling process can lead intermediate yt states that may have diverged significantly far from the model's 's observed distribution during training. This issue has been widely studied in the inverse RL and autoregressive token generation literature, where it is referred to as exposure bias
(Ross al. 201
Ranzato et al. 2015).
SCHEDULED SAMPLING FOR DIFFUSION MODELS
In this work, we adapt scheduled sampling, simple and effective method based on DAgger (Ross al. [211]] Bengio al. 2015) from discrete autoregressive models the training procedure of diffusion models. The core idea is to replace the standard training procedure with a biased sampling approach that mimics the test-time model inference based its own predictions. Before describing this approach, we first give a short background on training diffusion models.
Background: Training Diffusion Models Diffusion models maximize an evidence lower bound (ELBO) the above Markov chain. We introduce auxiliary Markov chain Q(y,," (yo)
3

---

Published as a conference paper at ICLR 2023
IT Q(vlle-1) to compute the ELBO
Pyo, ... ,9T)
log P(yn) 2 By. u log Q ... ,
= Ee log P(yo131) Σ (l(
DKL (RL( (000l/rr0)))pp/r) (1)
Diffusion models fix Q to a predefined Markov chain:
Q(vclvc-)== = WW(1 B, Quvi." .vrlv))) II Qwelv)-1),
1=1
where $1.-.. .rr is sequence of predefined scalars controlling the variance schedule.
Since Q is fixed, the last term only need to optimize
in Equation (1) is a constant, and we
T
Eq log P(volen)
Σ
DKL 1ww.:0/wwwwwwooolllp0llo-
=EQ(n (oo) log P(volv1) - Σ DKL
With large T, sampling from Q (vo) can be made efficient since Queelvn) has analytical form:
Quulvv)) = / - Q(vaalvo) N v1 =ā!),
where a. II'm as and a4 Br.
To simplify the P(yc-1lye) terms. Ho et al. (2020) parameterize this distribution by defining po(ye, t) through an auxiliary neural network co co t):
B2 VI
po(ye,
(ye
Co(Be. t))
Van
a2
With P this form, applying Gaussian identities, reparameterization (Kingma & Welling 2013). and further simplification leads to final MSE training objective,
max
Eyg Q(v
yt VI Cre
Voiya
co colyee t)
where y: the sampled latent, ai a constant derived from the variance schedule, yo is the training image, and eg is neural network predicting the update y, that leads to Wt-1-
Scheduled Sampling Our main observation is that at training time, for each f, the objective func- tion Equation takes the expectation with respect to a Q(y, (yo). test time the model instead uses the learned P(y.) leading to exposure bias issues like Figure
Scheduled sampling (Bengio et al., 2015) suggests alternating sampling training from the standard distribution and the model's 's distribution based schedule that increases model usage through training. Ideally, we would sample from
T
P(y+)
P((T) II P(y--1lys)
s
For detailed derivation, see Appendix
4

---

Published as a conference paper at ICLR 2023
However, sampling from P(y)) is expensive since it requires rolling out the intermediate steps .r... y
We propose an approximation instead. First use Q as approximate posterior of an carlier step t + m, and then roll out a finite number of steps m from yt+m ~ @3))+mlbb)
++m
P(yclyo) -
Q (30)
II
P(yo-1ly6)
8-141
Note that when m (v (b) Quy (o) and we recover normal diffusion training.
Standard Diffusion
When m = T t, P(yoluo) = (()) if
Q (yo) No I). An example of m =
P(sh kn))
is shown in Figure 3 Substituting back, the ob- jective becomes
yO
M2
M1
No
Q(n bm)
Vosyo
E P(ual8o)
co(
Scheduled Sampling
VI at
1=1
(3)
P(u (ta)
To compute its gradients, in theory need to back-propagatt through P since depends oP but practice to save memory we ignore 700 and only consider the term inside expectation.
Ms
y/2
D1
No
Q
Figure 3: Diffusion samples y1 from Q. Sched- uled sampling instead samples an upstream la- tent variable 1/2 and then yı based on the model's Markov chain P(y
MARKUP-TO-IMAGE SETUP
4.1
DATA
We adapt datasets from four domains to the task of markup-to-image. Table provides a summary of dataset statistics.
Math Our first dataset, LaTeX-to-Man is large collection of real-world mathematical expres- sions written in LaTeX markups and their rendered images. We adopt IM2LATEX- 100K introduced in Deng et al. (2016), which is collected from Physics papers on arXiv. IM2LATEX- nally created for the visual markup decompiling task, but we adapt this dataset for the reverse task of torkkuptto-iaage We pad all images size x 320 and remove images larger than that size. For faster evaluation, we form smaller test set by subsampling 1,024 examples from the original test IM2LATEX-100K.
is origi-
Table Layouts The second dataset we use is based on the 100k synthesized HTML snippets and corresponding rendered webpage images from Deng al. (2016). Each HTML snippet contains nested <div>> with a solid border, a random width, and a random float. The maximum depth of nest is limited two. We make change to this dataset, except that subsample 1,024 examples from the original test set to form a new test set.
Sheet Music We generate a third dataset of sheet music. The markup language LilyPond is file format for music engraving (Nienhuys Nieuwenhuizen. 2003). LilyPond is powerful language for writing music scores: it allows specifying notes using letters and note durations using numbers. One challenge in the LilyPond-to-S -he-t music task to deal with the possible "relative" mode, where the determination of each note relies where the previous note is. We generate 35k synthetic LilyPond files and compile them into sheet music. downsample images by factor of two and then filter out images greater than 192 448.
Thhre is analytical solution since the transition probabilities in this Markow chain are parameterized by neural network uio.
5

---

Published as a conference paper at ICLR 2023
Dataset
Input Format
Input Length
# Train
Val
# Test
Image Size
Grayscale
Math
LaTeX Math
113
55,033
6,072
1,024
64 x 320
Table Layouts
HTML Snippet
481
80,000
10,000
1,024
64 64
Sheet Music
LilyPond File
240
30,902
989
988
192 x 448
Molecules
SMILES String
30
17,925
1,000
1,000
128 x 128
N
Table I: Markup-to-image datasets. Inputs each dataset are described in Section 4.1 in detail. Input length is measured as the median number of characters in the validation set.
Molecules The last dataset we use is from the chemistry domain. The input is string of Simpli- fied Molecular Input Line Entry System (SMILES) which specifies atoms and bonds of a molecule (Weiiiner)] 1988). The output is a scheme of the input molecule. We use a solubility dataset by Wilkinson et al. (2022). , containing 19,925 SMILES strings. The dataset is originally proposed to improve the accessibility of chemical structures for deep learning research. 2D molecules images rendered from SMILES strings using the Python package RDKIT (Landrum al., (016). We partition the data into training, validation, and test sets. We downsample images by a factor of two.
4.2 EVALUATION
Popular metrics for conditional image generation such as Inception Score (Salimans et al.) 2016) or Fréchet Inception Distance (Heusel et al. 2017) evaluate the fidelity and high-level semantics of generated images. markup-to-image tasks, we instead emphasize the pixel-level similarity between generated and ground truth images because input markups describe exactly what the image should look like.
Pixel Metries Our primary evaluation metric Dynamic Time Warping (DTW) Müller 2007
which calculates the pixel-level similarities of images by treating them as column time-series. We preprocess images by binarizing them. We treat binarized images as time-series by viewing each image sequence of column feature vectors. We evaluate the similarity of generated and ground truth images by calculating the cost of alignment between the two time-series using DTW We use Euclidean distance feature matching metric. We allow minor perturbations of generated images by allowing up to 10% of upward/downward movement during feature matching.
Our secondary evaluation metric is the root squared mean (RMSE) of pixels between generated and ground truth images. We convert all images to grayscale before calculating RMSE. While RMSE compares two images the pixel level, one drawback that RMSE heavily penalizes the score of symbolically equivalent images with minor perturbations.
Complimentary Metries Complementary to the above two main metrics, we report one learned and six classical image similarity metrics. We use CLIP score (Radford et al. [202]) as learned metric to calculate the similarity between the CLIP embeddings of generated and ground truth im- ages. While CLIP score is robust minor perturbations of images, it is unclear if CLIP embeddings capture the symbolic meanings of the images in the domains of rendered markups. For classical im- age similarity metric we report SSIM (Wang et al. 2004). PSNR (Wang et al.
| 2004), UQI (Wang & Bovik 2002), ERGAS [Wald] 2000), SCC (Zhou et al. 1998), and RASE (Godzalez-Audiaaaa
al. 2004).
4.3
EXPERIMENTAL SETUP
Model For Math, Table Layouts, and Sheet Music datasets, use GPT-Neo- Per-Neo-175M. 175M Black al.
2021 Gao et al., 2020) as the input encoder, which incorporates source code in its pre-training For the Molecules dataset. we use ThemBERTa-77MMMMM
from DeepChem (Ramsundar et al. 2019 Chithrananda et al. 2020 to encode the input. To parameterize the diffusion decoder, we experiment
with three variants of U-Net (Ronneberger al. 2015): 1) standard U-Net conditioned
use the DTW
guide/ /e/ ttw clmml
Wee use the similarity metric implementation httppp/s//gtttub..cc////aaeeeeaa

---

Published as a conference paper at ICLR 2023
Pixel
Complimentary
Approach
D RMSE, CLIPt SSIM PSNR1 UQIT ERGAS) SCCT RASE]
Math
Stable Diffusion
115.77
144.65
0.71
0.15
5.12
0.54
35366.43
0.00
8721.85
Base-Attn,-Pos
27.73
44.72
0.95
0.70
15.35
0.97
2916.76
0.02
729.19
Base+Atttn,PPs
20.81
39.53
0.96
0.76
16.62
0.98
2448.35
0.06
612.09
PPss
19.45
37.81
0.97
0.78
17.12
0.98
2314.31
0.07
578.58
Scheduled Sampling
18.81
37.19
0.97
0.79
17.25
0.98
2247.41
0.07
561,85
Table Layouts
Buse+Atttn-
6.09
22.89
0.95
0.92
38.55
0.98
2497.51
0.44
624.38
Atsn+Attn,
5.91
22.17
0.95
0.93
38.91
0.98
2409.28
0.44
602.32
Scheduled Sampling
5.64
21.11
0.95
0.93
40.20
0.98
2285.83
0.45
571.46
Sheet Music
Base+Atttn
81.21
45.23
0.97
0.67
15.10
0.97
3056.72
0.02
764.18
Base+Attn,+Pos
80.63
45.16
0.97
0.68
15.11
0.97
3032.40
0.02
758.10
Scheduled Sampling
79.76
44.70
0.97
0.68
15.20
0.97
2978.36
0.02
744.59
Molecules
Ause+Attn,-Pos
24.87
38.12
0.97
0.61
16.66
0.98
2482.08
0.00
620.52
Atte+Attn,+Pos
24.95
38.15
0.96
0.61
16.64
0.98
2455.18
0.00
613.79
Scheduled Sampling
24.80
37.92
0.96
0.61
16.69
0.98
2467.16
0.00
616.79
Table Evaluation results of markup-to-imaaee generation across four datasets. (t/-)Attn indicates
model with or without attention, and (+/-)Pos is model with without positional embeddings. Scheduled Sampling is applied to training of models with attention and positional embeddings. Best results are in bold (based on full precision although the numbers are rounded to two decimal places).
average-pooled encoder embedding (denoted "-AAtn,-Pos"), 2) a U-Net alternating with cross- attention layers over the full resolution the encoder embeddings (denoted as "+Attn,-Pos") and 3) U-Net with both cross- coos-attention and additional position embeddings the query marking row ids and column ids (denoted as *+Attt,+Pos"" (Vaswani et al. 2017).
Hyperparameters We train all models for 100 epochs using the Adam optimizer (Kingma & Ba 2014 Loshchilov & Hutter 2018). The learning rate is le- with cosine decay schedule over 100 epochs and 500 warmup steps. We use a batch size of 16 for all models. For scheduled sampling, we use m 1. We linearly increase the rate of applying scheduled sampling from 0% to 50% from the beginning of the training to the end.
Implementation Details Our code is built top of the HuggingFace diffusers library We use a single Nvidia A100 GPU train on the Math, Table Layouts, and Molecules datasets; We use four A100s to train the Sheet Music dataset. Training takes approximately 25 minutes per epoch for Math and Table Layouts, 30 minutes for Sheet Music, and 15 minutes for Molecules. Although one potential concern is that the scheduled sampling approach needs more compute due to the extra computation to get P for > 0, in practice, find that the training speed is not much affected: on the Math dataset, scheduled sampling takes minutes 59 seconds per training epoch, whereas without scheduled sampling it takes 24 minutes 13 seconds per epoch.
RESULTS
Table 2 summarizes the results of markup-to-image tasks across four domains. We use DTW and RMSE our primary evaluation metrics make our experimental conclusions. First, train and evaluate the variations of diffusion models on the Math dataset. Comparing the model with attention ('-Attn,-Pos") to without attention ("+Attn,-Pos"" using attention in the model results in
fuueess

---

Published as a conference paper at ICLR 2023
Math
Tabbles
Molecules
25
70
23
DTW
20
60
22
50
21
20
40
Perturbed
Symbols Removed
Diffusion
Figure 4:
Perturbation results. For each dataset, we compare the DTW score of images generated by compiler, removing an increasing number of symbols from markups, and best model.
4Q
2201)
111 + P(I + co;)3
02(1+
4Q
1
+ +(1+ P(I + +,)²
Figure 5: Qualitative results showing typical mistakes. (Top row) Model-generated images across datasets. (Bottom row) Ground truth images.
significant improvement by reducing DTW (25% reduction) and RMSE (12% reduction). Therefore, always use attention for experiments on other datasets. We observe that additionally, using positional embeddings ("+Attn,+Pos")) is helpful for the Math dataset. The proposed scheduled sampling approach improves the model's performance using attention and positional embeddings. We also evaluate pre-trained Stable Diffusion (Rombach et al.) 2022). but our evaluation results suggest that pre-trained Stable Diffusion does not have zero-shot transferability to the markup domain. We provide qualitative results for Stable Diffusion in Figure 14
We observe a similar trend in the other three datasets-Table Layouts, Sheet Music, and Molecules. Using positional embeddings improves the performance measured by DTW and RMSE (except for the Molecules dataset). Training models with the proposed scheduled sampling achieves the best results consistently across all the datasets. As noted in Figure 2, we qualitatively observe that schedule sampling, which exposes the model to its own generations during training time, comes with the benefits of the model being capable of correcting its own mistakes at inference time.
Absolute Evaluation Our evaluation metrics enable relative comparisons between models in the markup-to-imaag task. However, it remains unclear how capable the models are in an absolute sense-if the models are generating near- sear-peerfcct images or even the best model is missing lot of symbols. We investigate this question by removing an increasing number of symbols from the ground truth markups and evaluating the perturbed images against the ground truth images. Our results in Figure 4 highlight that our best model performs roughly equivalent to the ground truth images with three symbols removed on the Math dataset. On the other hand, our best model performs better than ground truth images with only a single symbol removed on the Table Layouts dataset and two symbols removed on the Molecules dataset, indicating that our best model adapts to these datasets well. Results for music are less strong.
Qualitative Analysis We perform qualitative analysis on the results of our best models, and we observe that diffusion models show different level of adaptation to four datasets. First, we observe that diffusion models fully learn the Table Layouts dataset, where the majority of generated images equivalent to the ground truth images for human eyes. Second, diffusion models perform mod- erately well on the Math and Molecules datasets: diffusion models generate images similar to the the Molecules
ground truth images most of the time the Math dataset, but less frequently dataset. The common failure modes such as dropping few symbols, adding extra symbols, and repeating symbols illustrated in Figure 5
8

---

Published as a conference paper at ICLR 2023
On the Sheet Music dataset, diffusion models struggle by generating images that deviate significantly from the ground truth images. Despite this, we observe that diffusion models manage to generate the first few symbols correctly in most cases. The intrinsic difficulty of the Sheet Music dataset is long chain of dependency of symbols from left to right, and the limited number of denoising steps might be bottleneck generating images containing this long chain. We provide additional qualitative results for all four datasets in Appendix A
6 RELATED WORK
Text-to-Image Generation Text-to-image generation has been broadly studied in the machine learning literature, and several model families have been adopted approach the task. Generative Adversarial Networks (Goodfellow et al. 2014) is one of the popular choices to generate realistic images from text prompts. Initiating from the pioneering work by Reed al. (2016a), numerous approaches are developed to improve the quality of eoxt-to-image generation (Reed et al., 2016b)
Zhang et al.) 2017 2018 Zhu et al. 2019; Tao al., 2020 Koh al.) [021] inter alia). Another common method is based on VQ-VAE (Van Den Oord et al. 2017]. treating text-to-image genera-
tion as a sequence-to-seene toaeeeetttt-equueee tosseunnceeeeeeeeeee task of predicting discretized image tokens autoregressively from text prompts (Ramesh et al. 2022 Ding et al.] 2021, Gafni et al. 2022 Gu et al. 2022) Aghajanyan et al.) 2022
Yu et al. (0222) Diffusion models SSohl-Dickstenn et al. (2015) are the most recent progress in text-to-image generation. The simplicity of training diffusion models introduces significant utility, which often reduces to the minimization of mean-square error for estimating noises added to im- ages (Ho et al. (2020). Diffusion models are freee from training instability or model collapses (Brock)
let al. 2018 Dhariwal & Nichol 2021), and yet manage to outperform Generative Adversarial Net-
works text-to-image generation in the MSCOCO domain (Dhariwal & Nichol 2021). Diffusion
models trained on large-scale image-text pairs demonstrate impressive performance in generating
creative natural or artistic images (Nichol et al. 2021 Ramesh et al. 2022 Saharia al. 2022).
So far, the demonstration of successful text-to-image generation models is centered around the sce- nario with flexible interpretations of text prompts (e.g.. artistic image generation). When there is an exact interpretation of the given text prompt (e.g., markup-to-imagg generation). text-to-image generation models are understudied (with a few exceptions such as Liu et al. (202T) which stud- ied controlled text-to-image generation in CLEVR (Johnson et al. (017)) and iGibson (Shen et al.) 202T) domains). Prior work reports that state-of-the-arr diffusion models face challenges in the ex- act interpretation scenario. For example, Ramesh et al. (022) report unCLIP struggles to generate coherent texts based on images. In this work, we propose a controlled compositional testbed for the exact interpretation scenario across four domains. Our study brings potential opportunities for evaluating the ability of generation models to produce coherent outputs in a structured environment, and highlights open challenges of deploying diffusion models in the exact interpretation scenario.
Scheduled Sampling In sequential prediction tasks, the mismatch between teacher forcing train- ing and inference known as an exposure bias problem (Ranzato al. 2015 Spencer all 202 During teacher forcing training, a model's next-step prediction is based on previous steps from the ground truth sequence. During inference, the model performs the next step based on its own previous predictions. Training algorithms such as DAgger (Ross et al., (211)) or scheduled sampling (Bengio jet al., 2015) are developed to mitigate this mismatch problem, primarily by forcing the model to use its own previous predictions during training with some probability. In this work, we observe a problem similar to exposure bias in diffusion models, and we demonstrate that training diffusion models using scheduled sampling improves their performance markup-to-image generation.
7 CONCLUSION
We propose the task of markup-to-image generation which differs from natural image generation in that there are ground truth images and deterministic compositionality. We adapt four instances of this task to analyze state-of-the-art diffusion-basee image generation models. Motivated by the observation that a diffusion model cannot correct its own mistakes at inference time, we propose to use scheduled sampling to expose it to its own generations during training. Experiments confirm our approach's 's effectiveness, although perfect rendering is not yet achieved. We believe that rendering markup is interesting benchmark and potential application of pretrained models with diffusion.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
YD supported by an Nvidia Fellowship. NK is supported by Masason Fellowship. AR is supported by NSF CAREER 2037519, NSF 1704834, and a Sloan Fellowship. Thanks to Bing Yan for preparing molecule data and Ge Gao for editing drafts of this paper. We would also like to thank Harvard University FAS Research Computing for providing computational resources.
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi. Gargi Ghosh, Mike Lewis, al. Cm3: A causal masked multi- modal model of the internet. arXiv preprint arXiv:2201.0777 0702200000.0...... 2022.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Teenorrllow, March 2021. URL https://doi.
org/10. 52811005221110 /ewoodooo.oooooooooooooooooooo 5297715
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: large-scale self- supervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.00 09885, 2020.
Yuntian Deng, Anssi Kanervisto, and Alexander Rush. What you get what you see: visual markup decompiler. arXiv preprint arXiv: /60,:06609 10:32-37, 2016.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis, Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Prucessing Systems, 34: 19822-9985. 2021.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. arXiv preprint arXiv:2203.13131, 2022.
Lco Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima. et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
AE Glennie. On the syntax machine and the construction a universal compiler. Technical report, CARNEGIE INST OF TECH PITTSBURGH PA COMPUTATION CENTER, 1960.
María Gonzılez-Audícana. José Luis Saleta, Raquel Gareía Catalán, and Rafael García. Fusion of multispectral and panchromatic images using improved ihs and pea mergers based on wavelet decomposition. IEEE Transactions on Geoscience and Remote sensing, 42(6):1291-1299 2004.
lan J Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. In International Conference Learning Representations, 2018
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to -mage synthesis. In Proceedings of the (EEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706. 2022.
10

---

Published as a conference paper at ICLR 2023
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint WrXiv:14/2.6990 2014.
arXiv preprint Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv:1312.61 2013.
Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Text-to-image image generation grounded by fine- one-grained user attention. Proceedings of the IEEE/CVF Winter Conference on Applica- lions of Computer Vision, pp. 237-246, 2021.
Greg Landrum al. Rdkit: Open-source cheminformaties software. 2016. github. com/ rdkit/rdkit//
URL https:/
Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning compose visual relations. Advances in Neural Information Processing Systems, 11-23166-231777 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pp. 69-84, 2007.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12 2021.
Han-Wen Nienhuys and Jan Nieuwenhuizen. Lilypond, system for automated music engraving. In Proceedings of the XIV Colloquium on Musical Informatics (XIV CIM 2003), volume 1, pp- 167-171. Citeseer, 2003.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-878 8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21((14):1-667 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-imaae generation. In International Conference Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O' O'Reilly Media, 2019.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint / 2015.

---

Published as a conference paper at ICLR 2023
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016a.
Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. Learning what and where to draw. Advances in neural informatiin processing systems, 29, 2016b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241 Springer, 2015.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11453 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Bokui Shen, Fei Xia, Chengshu Li, Roberto Martin-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez-D' Arpino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, et al. igibson 1.0: a simu- lation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7520-7527. IEEE, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, pp. 2256-2265. PMLR, 2015.
Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and Andrew Bag- nell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872, 2021.
Ming Tao, Hao Tang, Songsong Wu, Nieu Sebe, Xiao- Jing, Fei Wu, and Bingkun Bao. Df- gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arXiv:2008.0 05865, 2020.
Warren Teitelman. Automated programmering: the programmer's assistant. In Proceedings of the December 5-7, 1972, fall joint computer conference, part II, pp. 917-921, 1972.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Lucien Wald. Quality of high resolution synthesised images: Is there simple criterion? In Third conference" Fusion of Earth data: merging point measurements, raster maps and remotely sensed images". ", pp. 99-103. SEEUURISCA 2000.
Zhou Wang and Alan C Bovik. A universal image quality index. IEEE signal processing letters, (3):81-84, 2002.
12

---

Published as a conference paper at ICLR 2023
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from enror visibility to structural similarity. IEEE transactions on image processing, 13(4))600- 612, 2004.
David Weininger. Smiles, chemical language and information system. introduction method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Matthew R Wilkinson, Uriel Martinez-Hemandee Chick C Wilson, and Bernardo Castro- Dominguez. Images of chemical structures molecular representations for deep learning. Jour- nal of Muterials Research, 37(14):2933-303, 2022.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to choto-realistic image synthesis with stacked generative ad- versarial networks. In Proceedings of the IEEE international conference on computer vision, pp- 5907-5915, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang. Xiaolei Huang, and Dim- itris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial net- works. IEEE transactions on pattern analysis and machine intelligence, 1147)19477 2018.
Jie Zhou, Daniel L Civco, and JA Silander. A wavelet transform method to merge landsat tm and spot panchromatic data. International journal of remote sensing, 19(4):743-757, 1998.
Minfeng Zhu, Pingho Pan, Wei Chen, and Yang. Dm-gan: Dynamic memory generative ad- versarial networks for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5802-5810, 2019.
13

---

Published as a conference paper at ICLR 2023
A QUALITATIVE RESULTS
We provide additional qualitative results from models trained with or without scheduled sampling on four datasets in Figure 6
Figure
Figure 8 and Figure
B
B a
B
4T
1
/2
4T
+ //22/44
a
4T
1 + B/ 4m
8ºF
F
F
Ciyk
Ôt θt 0t
Cijk
Ot 8 8h
Ot 0t
(f) = / DE f [0)
(f) = f P
(f) / DE f P [5,
A(C) = f du a(u)e
A(C) / du alz) ettss,
A(C) / du a(u)e
Figure 6: Qualitative results in the Math domain. Left column: ground truth images. Middle column: generations from +Ats,, Right column: generations from Scheduled Sampling. The top two random selections, and the bottom two rows examples of good generations.
a
a
h
h
h
p
p
p
b
b
q
a q
m
m v
m
V
Figure 7: Qualitative results in the Table Layouts domain. Left column: ground truth images. Mid- dle column: generations from +ots., Right column: generations from Scheduled Sampling. The top two rows are random selections, and the bottom two rows are examples good generations.
14

---

Published as a conference paper at ICLR 2023
Figure 8: Qualitative results the Sheet Music domain. Left column: ground truth images. Middle column: generations from +Attn++Pon +Pos. Right column: generations from Scheduled Sampling. The top two rows are random selections, and the bottom two rows are examples of good generations.
Figure 9: Qualitative results in the Molecules domain. Left column: ground truth images. Middle column: generations from +Ps.. Right column: generations from Scheduled Sampling. The top two rows are random selections, and the bottom two rows are examples of good generations.
15

---

Published as a conference paper at ICLR 2023
B DETAILED DERIVATION OF THE DIFFUSION OBJECTIVE
In this section, we provide a more detailed derivation of the objective function of the diffusion process.
Generative Process Denoising diffusion probabilistic models (DDPM) (Ho et al. 2020 parame- terize the probabilistic distribution P(yol=) by parameterizing Markov chain P (1 with an initial distribution P(yr) (we omit the dependence on æ for simplicity):
P(yr) = (0,
P(y:-1lye) (V(ou t): oî1)
P(yo.T) = P(VT) II P(yn-1lbe)
where y1. '.. ,rr latent variables of the same size yo V. po(;,t) t) is neural network parameterizing a map y y.
Evidence Lower Bound To learn the model,
variational inference and maximize the evi- dence lower bound (ELBO) (Bishop & Nasrabadi] 2006). We introduce an auxiliary Markov chain
Q ,VT ())
II ( 1) as the approximate posterior distribution to compute the
ELBO:
log P(yn) 2 By @ log P(yo, yrr
Q ... yr
P(UT) 11 P(yt-11 19b)
EEq log
Q(vi (vo) 11 (1ve Vt-1)
P(yz) IT,- P(yc-1196)
E.q log
Q( lvo) I (Q(
P(yr) P M/1) [[, P l11e)
EEq log
Q(yrlv)) (o) II,2 Q Me y0)
T
P
P(TT)
EEq
log P(yolun) + log
II
vo)) + log
Q Vt, vo)) Qurluyn
t=
EEQ
log P(yo 91)
Σ DKL(QQQ P() DKL
(KL(Q(uTToo))) (10 -ee.(Q(urro))))((rr) (4)
Note that this shows how Equation (I) in Section in the main paper is derived.
Instead of optimizing Q to maximize ELBO as is typically done in variational inference, diffusion model fixes Q predefined Markov chain:
Q(vlltr-) N(VI Beyt-1, B,T)
T
Q(1,, ... yr "0r (b) = II Q (()))))-))
where -1,-.. ,9T is sequence of predefined scalars controlling the variance schedule. Since Q is fixed, the last term (egKkQ(Q1 yo) P(yr)) in Equation (4) is constant, and we only need to optimize
T DKL Dkc.@ffu-nlee.oolll00---
Eq
log P(vo)1))
1=2
Equn ly)) log P(uolvr)
Eq DKL Pll (5)
16

---

c). Published as a conference paper at ICLR 2023
Since in practice T is usually set to large number (this paper 1000), to efficiently compute the gradients, we use stochastic gradient descent and sample t uniformly from {1.... . instead of computing the full sum Σi=2 DKL (Q(ve 1et. P 1lxe)) to estimate gradients. The key observation is that sampling from Qv (yo) is efficient since Q3)ll0)) = pp 21-1 (v)na) has an analytical form:
Quvl330) NN v1 a,!), where ār II. a and at sampling the intermediate steps y1. ... Br. This enables fast sampling of yt QyelBoo) without 18
e~ N(0,1 Vt Vay0o v1 0:c.
We refer readers Ho al. 2020
for detailed derivations.
Based on this observation,
Q(vc-113e vo)
Q(w Mr-1 i
is also Gaussian:
Q(1 (vo)
04 18t
var(1 Ōx-1)
(1 rt-1)) B.T)
QU N/
+
V/:
1 04
1 at
1
(1 a1-1)
=M(
(ye
()
3eT)
/1
30
v1 ai
1 αx
Since our goal is to minimize ΣT= DKL(Q(U1-1) be, (0 P())-1 to match the covariance
matrix of P(yc-1ly)) to that of Q yt, you, simply set
k1-1)
a
3r.
αz
To match the mean, (e(e, t) should match
(3l
Var
V1
Since y given input to the model, Ho et al. (200)) propose to parameterize [o () similar form by parameterizing another function co (Jtt. 1):
B. eo(y 1))-
polya, 1)
(yr
Vat
VI
ai
With this parameterization the objective in Equation (5)
can be written as (ignoring constants)
8 |lee cu 1)
E Equn (o) 20fa āz)
where 34- Be-yoego vT-a, final objective becomes
Ho et al. (020) further proposes to ignore the weights 20fa.(1-a.)'
so the
T
Equa mo)
11
Vāryo
co(2r. 12
(6)
V1
C ALGORITHM OUTLINE
The training algorithm of scheduled sampling is shown in Algorithm where the differences from the original DDPM training (shown in Algorthm 2) are highlighted red. We refer readers to Ho al. (2020) for the sampling algorithm, since approach doesn't change sampling.
17

---

Published as a conference paper at ICLR 2023
Algorithm Scheduled Sampling
Algorithm 2 No Scheduled Sampling
Require: m 2 0
Require:
repeat
1: repeat
2: yo data
2: data
3: Uniform (1,..., T-m))
Uniform((1......
4: c~ (O,(
4: c N).,T)
5: It+m f ve+mm30 + vI Cm 1, - 2, do 7: Ft+m' ~ P(ye+m/ (3++m(+1)
5: Y/ + Trty0 + vI āre
6: for m'
6:
7:
8: end for
8:
9: Take gradient descent step on
9:
Take gradient descent step on
Ve || - ea ( t)]l
Ve ll ( I)l
until converged
10: until converged
850
Q
750
q
650
Q
600
(
550
@
500
0 (w))
00
450
Q
7= (1)
350
Q 0 ##0
1 0 #
250
Q 0 #
2 ((.)) 00
0
- σ #
@ g #
Figure 10: The generation process of diffusion (left) versus diffusion++shedule sampling (right). The numbers on the y-axis are the number of diffusion steps (l). The ground truth LaTeX is Atilde (2) (I, (I, \tilde(Q) (I, Aneg 0
D MORE QUALITATIVE EXAMPLES OF INTERMEDIATE STEPS
More qualitative examples of the generative process comparing a model trained with scheduled sampling and a model trained without scheduled sampling can be found at Figurr 10 Figure 11 and Figure 12
E MORE PERTURBATION ANALYSIS
We expand our perturbation analysis in Figure 4 to more perturbation patterns. In addition to the deletion of symbols, we add the results for insertion and substitution. The distribution the newly introduced symbols follows the empirical distribution of those symbols in the training data. We summarize the results Table and Figure 13
F STABLE DIFFUSION EXPERIMENTAL DETAILS
We use the HuggingFace implementation of Stable Diffusion (Rombach et al. 2022). Stable Dif- fusion is originally trained on images with the size of 500 500. During inference, it generates
18

---

Published as a conference paper at ICLR 2023
850 900 = 1,
900 = 1,
750 900 1,
900 1,
650 900 1,
9t
900 1,
600
900 1,
R2
900 1,
2
550 900
96
900 1,
96t
1 + 12
2
500 900
95
900
1 + #
2
2 H
)7gy.
450
900
9y=
900
96
1 +
2
2 Z 2 Z
96g-
350 900
9
900
2
250
900
90j
900
95
2
2
0 900
90
)dgg-
900
96
$ 766.
The generation process of diffusion (left) versus diffusion+scheduule sampling (right). The numbers on the y-axis are the number of diffusion steps (f). The ground truth LaTeX
Figure 11:
=_100)=1, Aqquad g_(ij} =- (f\frac (1+\eec {2)))) '(2) (djlta_(ij).
850
(x(x), 6x
{1²(x,, ( =
750
(#(x), -
(v((x), -m)))=((==g))
650
(1(x), +(xz)) 6(*-V.
(v(x),, f/(6)) =(xx-y)
600
[v(x),, +(x)) = 8(x-v- v
{v(xx), (xx)) = 5(x vy)
550
{+x()),+<))) = 6(x y.
(1)(x), ((x)) 5(x yy
500
(v(x)),(())) = 5(x
(1)(x), +(x)) 5(x
450
(v(s),+(()) = 6(x- y.
(1(xs), (y)) = 5(x-v. y.
350
(())) ().
(*b(s), (y)) 5(x
250
(d))), = 6(x-y).
(1(x),,t<0) = 6(x-)).
0
(1(x), +()}) = 6(x-y).
{v(xs), ot (y)) 6xx-y).
Figure 12: The generation process of diffusion (left) versus diffusion++chedule sampling (right). The numbers on the y-axis are the number of diffusion steps (t). The ground truth LaTeX is Mppsi (1 (bb x)), \psi" (Adagger) ((Abf y)) {)=Adelta (bb x-y!).
19

---

Published as a conference paper at ICLR 2023
Math DTW RMSE
Simple Tables DTWL RMSE
Music DTW RMSE
Molecules
Approach
DTW RMSE,
Gold Images
-1 Symbol
12.33
27.17
10.72
40.93
39.38
25.04
19.42
30.36
-2 Symbols
16.00
35.55
16.37
60.46
48.45
30.82
23.43
36.76
-3 Symbols
18.49
37.84
18.45
67.86
57.08
35,49
24.73
37.80
-4 Symbols
20.76
40.17
20.80
73.72
66.90
39.59
23.96
36.68
-8 Symbols
24.60
43.18
23.56
81.52
75.08
43.48
21.99
32.84
-16 Symbols
27.49
43.54
25.09
84.66
71.95
42.37
21.10
31.06
Symbol
11.94
26.23
5.92
22.80
40.07
25.00
22.81
35.23
+2 Symbols
16.25
35.75
11.00
39.83
57.56
35.22
24.55
38.32
+3 Symbols
20.66
38.09
16.15
56.92
63.86
38.63
24.09
36.74
+4 Symbols
21.50
40.91
19.72
69.60
67.30
40.23
22.52
33.98
+8 Symbols
27.62
45.91
23.86
80.75
76.06
44.79
20.43
29.87
+16 Symbols
31.62
48.79
25.75
85.77
81.56
47.61
20.34
29.72
A1 Symbol
16.55
31.42
11.20
41.31
52.88
32.20
15.25
24.27
A2 Symbols
19.04
38.90
18.43
64.99
64.21
37.50
20.25
30.85
A3 Symbols
23.42
41.58
20.96
72.71
67.73
39.60
21.11
31.27
44 Symbols
25.65
43.04
23.35
79.71
66.76
39.62
20.85
30.73
A8 Symbols
28.61
45.89
25.57
86.93
75.37
43.97
20.34
29.72
A16 Symbols
31.34
47.67
25.95
85.96
74.01
44.12
20.34
29.72
Scheduled Sampling 18.81
37.19
5.64
21.11
79.81
44.70
24.80
37.92
Table 3: Perturbation results. Perturbation noise is denoted by (-) deletion, (+) insertion, and (A)
substitution.
poor-quality images if we query significantly smaller-sized images. Therefore, for our Math domain experiments, first generate images with the size of 320 1600 and then down-sample the gener- ated images to 64 x 320. Figure 14] illlusstraees some of the images generated by Stable Diffusion.
20

---

Published as a conference paper at ICLR 2023
Math
Tabbles
Molecules
25
70
23
DTW
20
60
22
50
21
20
40
Perturbed Diffusion
Symbols Removed
Math
Tables
Music
Molecules
25
25
80
61W
23
60
22
10
50
40
10
10
15
0
10
15
Perturbed Diffusion
Symbols Inserted
Moth
Toblees
Music
Molecules
24
75
20
22
DTW
25
70
15
65
20
60
55
10
10
15
15
Perturbed
Symbols Substituted
Figure 13: Perturbation results. Perturbation noise is denoted by (-) deletion, (+) insertion, and (A)
substitution.
w =n (()) - w()))
PPW.---arro arcQUo OZ Nuthh -W Wi: Wiichn Poesh
J dx f(x, f g(x,
Ta
Du 5f(w)) 8Ap
D DOO (8,At)
F(ant) (E(ans)) =
q T
O n
0
Figure 14: Qualitative results from Stable Diffusion (Rombach et al. 2022 in the Math domain.
Left column: ground truth images. Right columm: generations from Stable Diffusion.
21


Reviews:
[{"review_id": "ysScOX8WAkL", "reviewer": "Reviewer_uJv3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the task of markup-to-image generation and adapts a procedure of scheduled sampling to improve diffusion model performance. Here, the objective is to synthesize images from latex, musical notation, or HTML snippets, or any markup language that defines an output image in a deterministic fashion. In this context, scheduled sampling is applied to incorporate intermediate outputs of the diffusion model during training. Empirically, this results in some ability for the model to self-correct when it begins to generate erroneous outputs. The method is evaluated on a few different proposed datasets (HTML, music notation, molecule diagrams, latex) with an array of ablation studies. Overall the results show convincing performance, and this research direction may be significant for advances in markup compilers that are robust to typos or which incorporate generative capabilities.", "summary_of_the_review": "Overall the paper is well-written and contributes an interesting new technical idea for improving diffusion models as well as a new and potentially impactful task of markup-to-image generation. While there are areas where the clarity of the manuscript and the evaluation could be improved, I think the paper meets the bar for acceptance.\n", "strength_and_weaknesses": "The paper is mostly well-written and I found the description of the method to be clear. The paper makes a convincing case for the markup-to-image task, and I think this direction has potential for impact in the community. Moreover, the evaluation seems thorough, and while there is still room for improvment in the results, the manuscript and method are of high quality.\n\nWhile I think the paper is fairly strong, I do have some reservations about the evaluation and clarity of the writing in a few sections as I detail below.\n- The main quantitative results shown in Table 2 do not necessarily show the most convincing argument for the benefits of scheduled sampling. Some metrics are very close between the baselines, and scheduled sampling seems to worsen performance in other metrics (e.g., RASE). The paper also seems to be missing an analysis of how the performance changes with choosing m>1 (the number of scheduled sampling steps to use). Since I perceive the scheduled sampling to be the main technical contribution, I was expecting a more thorough analysis on this point. \n- I see only one qualitative result that shows the effect of including scheduled sampling (Fig. 2). I think including more such results could help make a more convincing argument for using this procedure. \n- I couldn't follow the perturbation analysis section. The paper repeatedly refers to a \"gold\" image without explaining what this is.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QWdL1iROnS", "reviewer": "Reviewer_NDpz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel scheduled sampling algorithm to guide the training of diffusion models for markup-to-image generation. Unlike the traditional diffusion models, which sample the training data from the Markov chain quantified Gaussian distribution Q(yt|y0), the paper take the m earlier predictions into consideration and thus can address the exposure bias issues in the image generative models to some extent. ", "summary_of_the_review": "In general, the paper proposed a novel sampling strategy for diffusion models to handle exposure issues. Even though the paper provided some reasonable results, lacking essential comparison experiments and poor paper organization make me hesitate to give it a high rank.", "strength_and_weaknesses": "Strength:\n+ The proposed algorithm seems to be effective. \n\nWeakness:\n- The algorithm is not well explained. I believe that readers require a lot of background in diffusion model to understand the paper content. Some notions are proposed without any explanation or reference, which makes the paper even harder to follow. To name just a few, starting from Eqn. 1 (Sec.3), the paper talks about the Markov chain Q and its sampling. However, what is Q and how Q is related to diffusion model is not discussed. How to derive Eqn.1, and Eqn. 2 are also unknown. I would like to suggest the authors either briefly introduce the pipeline and refer readers to a more detailed deduction, or carefully explain the terms to avoid ambiguity. \n- The overall network pipeline is unclear. I think including a pipeline figure and an algorithm flow tables for both training and sampling procedures are necessary.\n- Lacking essential experiments. The paper only show results from their own and don't include any comparison results with state-of-the-art (SOTA) text-to-image generative models. Considering the large amount of SOTAs, I believe that a comprehensive comparison result is needed. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ICwUW0SF0BO", "reviewer": "Reviewer_LE3F", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes markup-to-image generation as a task to measure the quality of image generation models, and examines the quality of the images generated by diffusion-based models for the task. They considered four domains, math, table, sheet music, molecules. As an improvement to the diffusion-based models for the task, they propose to use scheduled sampling. The experiments confirm that it improves the performance. The code and data are open-sourced.", "summary_of_the_review": "The introduction of the new task seems valuable and it could potentially have a huge impact by resulting in a number of following studies. However, the discussion and the content in the paper are rather shallow and the quality of the paper could be further improved. I do not see any concerns and fine to see it accepted, but it is not a very strong support.", "strength_and_weaknesses": "Strengths:\n\nThere is a value to introduce the markup-to-image generation task for image generation models. It can assess the aspects of the models that cannot be assessed by the existing tasks and there is a potential that the new task brings fundamental improvements to the models.\n\nIt is reasonable to use scheduled sampling for diffusion-based models and the effectiveness is confirmed by the experiments.\n\nWeaknesses:\n\nIt more or less just applied a diffusion-based image generation model to the markup-to-image generation task with a technique inspired by a known technique. We can see the results by the latest image generation algorithm on the task, but it is not clear what scientific value it has to show the results by the models optimized for other tasks. It could have a deeper discussion on why the new task is needed and why the existing method does not work well yet, and a scientific contribution based on the deeper insight.\n\nProbably, the experiments on assessing the absolute quality of the model are not enough. The \"Perturbed\" results in Figure 4 seem to have been generated by removing symbols for the amount in the x-axis. This can only tell how the evaluation metric is affected by random deletion errors. I do not feel it tells much to compare the simulated results with the ones by the diffusion-based models. The readers do not know how critical it is to miss a symbol to a particular renderer. How about insertion and substitution errors?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HT1kMdLDIY", "reviewer": "Reviewer_aUAh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper experiments on a deterministic generation (renderining) markup-to-image problem with diffusion model. They show that standard diffusion training process suffers exposure bias and using scheduled sampling algorithm is effective in fixing generation issues. ", "summary_of_the_review": "The  paper is clearly written. Treating markup-to-image as a generative process is a debatablly meaningful but novel point, especially when considering evaluating generative models. The authors use diffusion model as the generative model and adapt scheduled sampling to improve generation results. The methods are evaluated on four datasets with limited copmarison to other works if exist. The analysis of results is not totally convincing though, especially when the evaluation metrics are not convincing enough, more analysis on qualitative results would be helpful. ", "strength_and_weaknesses": "Strength\n1. This paper explores a unique generation problem with diffusion model where the mapping between input and generation target is deterministic, this provides a new angle for analyzing and evaluating generative models. \n2.  Authors noticed that standard diffusion training process for the markup-to-image problem often generate duplicate or misplaced symbols similar to widely studied exposure bias, so they adapt scheduled sampling by training diffusion model with its own generations and showed improved evaluation metrics on four small-scaled datasets: Math, Tables, Music and Molecules. \n\n\nWeaknesses\n1. Diffusion model has been proven effective in distribution learning, if the learning obejctive is a deterministic mapping, would regression based methods be more appropriate ? \n2. Comparisons are only made among authors' designs, are there previous works to compare with ? \n3. The authors already hinted varying performance on different domains, I would like to see more analysis on it. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster