Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive
Authors: Yumeng Li
Margret Keuper
Dan Zhang
Anna Khoreva
Venue: ICLR 2024 poster (2024)
Keywords: diffusion models
layout-to-image
domain generalization
Primary area: generative models
TL;DR: We propose adversarial supervision and multistep unrolling strategy for improved layout-to-image diffusion models, and further domonstarte its utility on the domain generalization of semantic segmentation.

Abstract: Despite the recent advances in large-scale diffusion models, little progress has been made on the layout-to-image (L2I) synthesis task. Current L2I models either suffer from poor editability via text or weak alignment between the generated image and the input layout. This limits their usability in practice. To mitigate this, we propose to integrate adversarial supervision into the conventional training pipeline of L2I diffusion models (ALDM). Specifically, we employ a segmentation-based discriminator which provides explicit feedback to the diffusion generator on the pixel-level alignment between the denoised image and the input layout. To encourage consistent adherence to the input layout over the sampling steps, we further introduce the multistep unrolling strategy. Instead of looking at a single timestep, we unroll a few steps recursively to imitate the inference process, and ask the discriminator to assess the alignment of denoised images with the layout over a certain time window. Our experiments show that ALDM enables layout faithfulness of the generated images, while allowing broad editability via text prompts. Moreover, we showcase its usefulness for practical applications: by synthesizing target distribution samples via text control, we improve domain generalization of semantic segmentation models by a large margin (~12 mIoU points).

Published as a conference paper at ICLR 2024
ADVERSARIAL SUPERVISION MAKES LAYOUT-TO-IMACE DIFFUSION MODELS THRIVE
Yumeng Li Margret K Dan Zhang Anna Khoreva'
"ossc Center for Artificial Intelligence Uniieesity of Mannheim 3 Max Planek Institute for Informatics UUnivesstty of Tübingen {yumeng. li, dan. zhang2, anna. www.eoo bosch.com
@euper@uni-mannee mamni-mmanheim... de Project page: https//yummengii00.giibbbbie wwt.://o.iiiuubbo/oDDMMM
ABSTRACT
Despite the recent advances large-scale diffusion models, little progress has been made on the layout-to-image (I.2I) synthesis task. Current 1.21 models either suffer from poor editability via text or weak alignment between the generated im- age and the input layout. This limits their usability in practice. To mitigate this, we propose to integrate adversarial supervision into the conventional training pipeline of L2I diffusion models (ALDM). Specifically, discriminator which provides explicit feedback to the diffusion generator on the pixel-level alignment between the denoised image and the input layout. To encour- age consistent adherence to the input layout over the sampling steps, we further introduce the multistep unrolling strategy. Instead looking a single timestep. we unroll few steps recursively to imitate the inference process, and ask the discriminator assess the alignment of denoised images with the layout over a certain time window. Our experiments show that ALDM enables layout faithful- ness of the generated images, while allowing broad editability via text prompts. Moreover, we showcase its usefulness for practical applications: by synthesizing target distribution samples via text control, we improve domain generalization semantic segmentation models by a large margin (~~1 mloU points).
employ
eggmentation-basee
Ground truth
FreestyleNet
ControlNet
Ours
+ "heavy fog"
+ "snowy scene with sunshine"
"snwwy scene, night time"
Layout faithfulness Text editability
In contrast to prior L2I synthesis methods (Xue al., 2023; Zhang & Agrawala, 2023), our ALDM model can synthesize faithful samples that are well aligned with the layout input, while preserving controllability via text prompt. Equipped with these both valuable properties, we can synthesize diverse samples of practical utility for downstream tasks, such as data augmentation for improving domain generalization semantic segmentation models.
Figure 1:

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Layout-to-image synthesis (L2I) is a challenging task that aims to generate images with per-pixel correspondence to the given semantic label maps. Yet, due to the tedious and costly pixel-level layout annotations of images, availability of large-scale labelled data for extensive training on this task is limited. Meanwhile, tremendous progress has been witnessed in the field of large-scale text-to-image (T2I) diffusion models (Ramesh et al., 2022; Balaji et al., 2022; Rombach et al,, 2022). By virtue of joint vision-language training on billions of image-text pairs, such as LAION dataset (Schuhmann et al., 2022), these models have demonstrated remarkable capability of synthe- sizing photorealistic images via text prompts. A natural question is: can we adapt such pretrained diffusion models for the L2I task using a limited amount of labelled layout data while preserving their Text controllability and faithful alignment the layou? Effectively addressing this question will then foster the widespread utilization of 1.21 synthetic data.
Recently, increasing attention has been devoted to answer this question (Zhang & Agrawala, 2023; Mou et al., 2023; Xue et al., 2023). Despite the efforts, prior works have suffered to find a good trade-off between faithfulness to the layout condition and editability via text, which we also empir- ically observed in our experiments (see Fig. 1). When adopting powerful pretrained T21 diffusion models, e.g., Stable Diffusion (SD) (Rombach et al., 2022), for L2I tasks, fine-tuning the whole model fully as in (Xue et al., 2023) can lead to the loss of text controllability, as the large model easily overfits to the limited amount of training samples with layout annotations. Consequently. the model can only generate samples resembling the training set, thus negatively affecting its practical use for potential downstream tasks requiring diverse data. For example, for downstream models deployed in an open-world, variety in synthetic data augmentation is crucial, since annotated data can only partially capture the real environment and synthetic samples should complement real ones.
Conversely, when freezing the T2I model weights and introducing additional parameters to accom- modate the layout information (Mou et al., 2023; Zhang & Agrawala, 2023), the L21 diffusion mod- els naturally preserve text control of the pretrained model but do not reliably comply with the layout conditioning. In such case, the condition becomes a noisy annotation of the synthetic data, under- mining its effectiveness for data augmentation. We hypothesize the poor alignment with the layout input can be attributed to the suboptimal MSE loss for the noise prediction, where the layout infor- mation only implicitly utilized during the training process. The assumption is that the denoiser has the incentive to utilize the layout information as it poses prior knowledge of the original image and thus beneficial for the denoising task. Yet, there is no direct mechanism in place to ensure the layout alignment. To address this issue, we propose to integrate adversarial supervision on the layout alignment into the conventional training pipeline of 1.21 diffusion models, which we name ALDM. Specifically, inspired by Sushko et al. (2022), we employ a semantic segmentation model based discriminator, explicitly leveraging the layout condition to provide a direct per-pixel feedback to the diffusion model generator on the adherence of the denoised images to the input layout.
Further, to encourage consistent compliance with the given layout over the sampling steps, we pro- pose a novel multistep unrolling strategy. At inference time, the diffusion model needs to consecu- tively remove noise for multiple steps to produce the desired sample in the end. Hence, the model is required maintain consistent adherence to the conditional layout over the sampling time hori- zon. Therefore, instead of applying discriminator supervision at a single timestep, we additionally unroll backward multiple steps over a certain time window to imitate the inference time sampling. This way the adversarial objective is designed over a time horizon and future steps are taken into consideration as well. Enabled by adversarial supervision over multiple sampling steps, our ALDM can effectively ensure consistent layout aligmment, while maintaining initial properties of the text controllability of the large-scale pretrained diffusion model. We experimentally show the effective- ness of adversarial supervision for different adaptation strategies (Mou et al., 2023; Qiu et al., 2023; Zhang& & Agrawala, 2023) of the SD model (Rombach et al., 2022) to the 1.21 task across different datasets, achieving the desired balance between layout faithfulness and text editability (see Table l).
Finally, we demonstrate the utility of our method on the domain generalization task, where the se- mantic segmentation network is evaluated on unseen target domains, whose samples are sufficiently different from the trained source domain. By augmenting the source domain with synthetic images generated by ALDM using text prompts aligned with the target domain, we can significantly en- hance the generalization performance of original downstream models, i.c., ~ 12 mloU points on the Cityscapes-t--ACDD generalization task (see Table 4).
2

---

Published as a conference paper at ICLR 2024
In summary, main contributions include:
We introduce adversarial supervision into the conventional diffusion model training, im- proving layout alignment without losing text controllability.
We propose a novel multistep unrolling strategy for diffusion model training, encouraging better layout coherency during the synthesis process.
We show the effectiveness of synthetic data augmentation achieved via ALDM. Benefiting from the notable layout faithfulness and text control, our ALDM improves the generaliza- tion performance of semantic segmenters by a large margin.
2 RELATED WORK
The task of pooouuttt-imageee synthesis (L2I), also known as semantic image synthesis (SIS), is to generate realistic and diverse images given the semantic label maps, which prior has been studied based on Generative Adversarial Networks (GANs) (Wang et al., 2018; Park et al., 2019; Wang et al., 2021; Tan et al., 2021; Sushko et al., 2022). The investigation can be mainly split into two groups: improving the conditional insertion in the generator (Park et al., 2019; Wang et al., 2021; Tan et al., 2021), or improving the discriminator's ability to provide more effective conditional su- pervision (Sushko et al., 2022). Notably, OASIS (Sushko et al., 2022) considerably improves the layout faithfulness by employing a segmentation- based discriminator. However, despite good lay- out alignment, the above GAN-based L21 models lack text control and the sample diversity heavily depends the availability of expensive pixel-labelled data. With the increasing prevalence of dif- fusion models, particularly the large-scale pretrained text-to-image diffusion models (Nichol et al., 2022: Ramesh et al., 2022; Balaji et al., 2022; Rombach et al., 2022). more attention has been de- voted to leveraging pretrained knowledge for the L2I task and using diffusion models. Our work falls into this field of study.
PITI (Wang et al., 2022) learns a conditional encoder to match the latent representation of GLIDE (Nichol et al., 2022) in the first stage and finetune jointly in the second stage, which un- fortunately leads to the loss of text editability. Training diffusion models in the pixel space is ex- tremely computationally expensive as well. With the emergence of latent diffusion models, i.e., Sta- ble Diffusion (SD) (Rombach et al., 2022), recent works (Xue et al., 2023; Mou et al., 2023; Zhang & Agrawala, 2023) made initial attempts to insert layout conditioning into SD. FreestyleNet (Xue et al., 2023) proposed to rectify the cross-attention maps in SD based on the label maps, while it also requires fine-tuning the whole SD, which largely compromises the text controllability, as shown in Figs. and 4. On the other hand, OFT partially updates SD, T2I-Adapter (Mou et al., 2023) and ControlNet (Zhang & Agrawala, 2023) keep SD frozen, combined with an additional adapter to ac- commodate the layout conditioning. Despite preserving the intriguing editability via text, they do not fully comply with the label map (see Fig. 1 and Table 1). We attribute this to the suboptimal diffusion model training objective, where the conditional layout information is only implicitly used without direct supervision. In light of this, we propose to incorporate the adversarial supervision to explicitly encourage alignment of images with the layout conditioning, and a multistep unrolling strategy during training to enhance conditional coherency across sampling steps.
Prior works (Xiao et al., 2022; Wang et al., 2023b) have also made links between GANs and diffusion models. Nevertheless, they primarily build upon GAN backbones, and the diffusion process is considered as an aid to smoothen the data distribution (Xiao et al., 2022), and stabilize the GAN training (Wang et al., 2023b), as GANs are known to suffer from training instability and mode collapse. By contrast, our ALDM aims at improving L2I diffusion models, where the discriminator supervision serves valuable learning signal for layout alignment.
ADVERSARIAL SUPERVISION FOR L21 DIFFUSION MODELS
1.21 diffusion model aims to generate images based on the given layout. Its current training and inference procedure is inherited from unconditional diffusion models, where the design focus has been on how the layout as the condition is fed into the UNet for noise estimation, as illustrated in Fig. 2 (A). It is yet under-explored how to enforce the faithfulness of L2I image synthesis via direct loss supervision. Here, we propose novel adversarial supervision which is realized via 1) a semantic segmenter- based discriminator (Sec. 3.1 and Fig. 2 (B)); and 2) multistep unrolling of UNet (Sec. 3.2 and Fig. (C)) to induce faithfulness already from early sampling steps and consistent adherence to the condition over consecutive steps.
3

---

Published as a conference paper at ICLR 2024
Unrolling Horizon
11-1
Diflusion Model
Diffusion Model
Diffusion Model
Truditional DM training (A)
2/1
(B)
Segmenter Discriminator
N 2eal/1 / fake
Figure 2: Method overview. To enforce faithfulness, we propose two novel training strategies to improve the traditional L2I diffusion model training (area (A)): adversarial supervision via segmenter-baase based discriminator illustrated in area (B), and multistep unrolling strategy in area (C).
3.1 DISCRIMINATOR SUPERVISION ON LAYOUT ALIGNMENT
For training the L2I diffusion model, Gaussian noise NO,, is added to the clean variable zo with a randomly sampled timestep yielding Tz.
Tt Vatza + -I- 0ge,,
(1)
where de defines the level of noise. A UNet (Ronneberger et al., 2015) denoiser co is then trained to estimate the added noise via the MSE loss:
Lnoise Ee.N(o, ||ee [|e--e((re,
y. e)] zo.ut [||e es (Vara + v1 -1- are, w)1
Besides the noisy image Iy and the time step I, the UNet additionally takes the layout input y. Since y contains the layout information of zo which simplify the noise estimation, it then influences implicitly the image synthesis via the denoising step. From 2e and the noise prediction we can generate denoised version of the clean image 400 (1)
as:
Tt V1 (ne v.
(1)
Va:
However, due to the lack of explicit supervision on the layout information y for minimizing Lnoises the output ( often lacks faithfulness y, as shown in Fig. 3. It is particularly challenging when carries detailed information about the image, as the alignment with the layout condition needs to be fulfilled on each pixel. Thus, seek direct supervision a(() to enforce the layout alignment. A straightforward option would be to simply adopt frozen pre-trained segmenter to provide guidance with respect the label map. However, we observe that the diffusion model tends learn a mean mode to meet the requirement of the segmenter, exhibiting little variation (see Table 3 and Fig. 6).
To encourage diversity in addition to alignment, we make the segmenter trainable along with the UNet training. Inspired by Sushko et al. (2022), we formulate adversarial game between the UNet and the segmenter. Specifically, the segmenter acts as a discriminator that is trained classify per-pixel class labels of real images, using the paired ground-truth label maps; while the fake images generated by UNet as in (Eq. (3)) are classified by as one extra "fake" class, as illustrated in area (B) of Fig. 2. As the task of the discriminator is essentially to solve multi-class semantic segmentation problem, its training objective is derived from the standard cross coos--nntropy loss:
HxW
HxW
log (Dis(a!") )escoNt:)) (Diis(af"))sccmn)>))) (4)
Lapis -E
Σ 7
Σ
Vi.i. log (Dis(so)ise
E
Σ $
W
4

---

Published as a conference paper at ICLR 2024
where N is the number of real semantic classes, and H x W denotes spatial size of the input. The class-dependent weighting To is computed via inverting the per per-pixel class frequency
H
X W
Yo Σ E II
(5)
=
for balancing between frequent and rare classes. To fool such a segmenter-based discriminator, #: produced by the UNet as in (Eq. (3)) shall comply with the input layout y to minimize the loss
(a(0)
N
HxW
log (Dis(a)))..a) ).s.))
Lado -
Te
(6)
Such loss poses explicit supervision the UNet for using the layout information, complementary to the original MSE loss. The total loss for training the UNet is thus
Laas = Inoise + Aado Ladoo
where Aade is the weighting factor. The whole adversarial training process is illustrated Fig. (B). As the discriminator is improved along with UNet training, we no longer observe the mean mode collapsing as with the use of a frozen semantic segmenter. The high recall reported in Table confirms the diversity of synthetic images produced by our method.
3.2 MULTISTEP UNROLLING
Admittedly, is impossible for the UNet to produce high-quality image Z0 via a single denoising step in (Bq. (3)), especially if the input H, very noisy (i.e., / is large). On the other hand, adding such adversarial supervision only at low noise inputs (i.e., f is small) is not very effective, as the alignment with the layout should be induced early enough during the sampling process. To improve the effectiveness of the adversarial supervision, we propose a multistep unrolling design for training the UNet. Extending from a single step denoising, we perform multiple denoising steps, which are recursively unrolled from the previous step:
0460 (160(1+,V.1) + V a:- c0(xx,
Tt-1 vai-1
(8)
Va:
(1-1) 40
It-1 v1 a4-10 v, v,
va--1
As illustrated in area (C) of Fig. 2, we can repeat (Eq. (8) and (Eq. (9)) times, yielding
{aft) -aft (e(t-1) .... 20 (t-K)), All these denoised images are fed into the segmenter-based discriminator the "fake" examples:
N
Eneue Telle log (Dis(al 3.)
Ladv
E
Σ
(10)
+
By doing so, the denoising model encouraged to follow the conditional label map consistently over the time horizon. It is important to note that while the number of unrolled steps K is pre-specified, the starting time step is still randomly sampled.
Such unrolling process resembles the inference time denoising with a sliding window of size K. As pointed out by Fan & Lee (2023), diffusion models can seen as control systems, where the denoising model essentially learns to mimic the ground-truth trajectory of moving from noisy image to clean image. In this regard, the proposed multistep unrolling strategy also resembles the advanced control algorithm- Model Predictive Control (MPC), where the objective function is defined in terms of both present and future system variables within a prediction horizon. Similarly, our multistep unrolling strategy takes future timesteps along with the current timestep into consideration, hence yielding a more comprehensive learning criteria.
While unrolling is simple feed-forward pass, the challenge lies in the increased computational complexity during training. Apart from the increased training time due to multistep unrolling, the memory and computation cost for training the UNet be also largely increased along with K.
5

---

Published as a conference paper at ICLR 2024
Ground truth
Label
T2I-Adapter
FreestyleNet
ControINet
ALDM
Figure 3: Qualitative comparison of faithfulness to the layout condition on ADE20K.
Since the denoising UNet model the same and reused for every step, we propose to simply ac- cumulate and scale the gradients for updating the model over the time window, instead of storing gradients at every unrolling step. This mechanism permits to harvest the benefit of multistep un- rolling with controllable increase of complexity during training.
Implementation details. We apply our method to the open-source text-to-image Stable Diffusion (SD) model (Rombach al., 2022) so that the resulting model not only synthesizes high quality im- ages based on the layout condition, but also accepts text prompts to change the content and style. As SD belongs the family of latent diffusion models (LDMss, where the diffusion model is trained in the latent space of an autoencoder, the UNet denoises the corrupted latents which are further passed through the SD decoder for the final pixel space output, i.c., &0 ((2)). We employ Uper- Net (Xiao et al., 2018) as the discriminator, nonetheless, we also ablate other types of backbones in Table 3. Since Stable Diffusion can already generate photorealistic images, randomly initialized discriminator falls behind and cannot provide useful guidance immediately from scratch. We thus warm up the discriminator firstly, then start the joint adversarial training. In the unrolling strategy, we use K 9 as the moving horizon. An ablation study on the choice of K is provided in Table 5. Considering the computing overhead, we apply unrolling every optimization steps.
4 EXPERIMENTS
Sec. 4.1 compares 1.21 diffusion models in terms of layout faithfulness and text editability. Sec, 4.2 further evaluates their use for data augmentation to improve domain generalization.
4.1 LAYOUT-TO-IMAGE SYNTHESIS
Experimental Details. We conducted experiments on two challenging datasets: ADE20K (Zhou et al., 2017) and Cityscapes (Cordts et al., 2016). ADE20K consists of 20K training and 2K vali- dation images, with 150 semantic classes. Cityscapes has 19 classes, whereas there are only 2975 training and 500 validation images, which poses special challenge for avoiding overfitting and pre- serving prior knowledge of Stable Diffusion. Following ControlNet (Zhang Agrawala, 2023), we use BLIP (Li et al., 2022b) to generate captions for both datasets.
By default, our ALDM adopts ControlNet (Zhang & Agrawala, 2023) architecture for layout con- ditioning. Nevertheless, the proposed adversarial training strategy can be combined with other L2I models as well, as shown in Table I. For all experiments, we use DDIM sampler (Song et al., 2020) with 25 sampling steps. For more training details, refer to Appendix A.1.
Evaluation Metrics. Following (Sushko et al., 2022; Xue et al., 2023), we evaluate the image- layout alignment via mean intersection-over.
(mloU) with the aid of off-thh--heelf segmentation networks. To measure the text-based editability, we use the recently proposed TIFA score (Hu al., 2023), which is defined as the accuracy of visual question answering (VQA) model, e.g.

---

Published as a conference paper at ICLR 2024
T21-Adapter
FreestyleNet
ControlNet
ALDM
Original caption: "a red van driving down street next to tall buildings."
+ "snowy scene"
"nighttime"
"burning van"
Visual comparison text control between different 1.21 diffusion models on Cityscapes. Based on the image caption, we directly modify the underlined objects (indicated ->) or append a postfix to the caption (indicated as +). In contrast to prior work, ALDM can faithfully accomplish both global scene level modification (e.g.. "snowy scene") and local editing (e.g., "burning van").
Figure 4:
Table 1:
Effect of adversarial supervision and multistep unrolling on different L2I synthesis adapta- tion methods. Best and second best are marked in bold and underline, respectively.
Cityscapes
ADE20K
Method
FID + mloUf
FID,
mloUf
OFT (Qiu et al., 2023)
57.3
48.9
29.5
24.1
+ Adversarial supervision
56.0
54.8
31.0
29.7
+ Multistep unrolling
51.3
58.8
29.7
31.8
T21-Adapter (Mou et al., 2023)
58.3
37.1
31.8
24.0
+ Adversarial supervision
55.9
46.6
32.4
26.5
+ Multistep unrolling
51.5
50.1
30.5
29.1
ControlNet (Zhang Agrawala, 2023)
57.1
55.2
29.6
30.4
+ Adversarial supervision
50.3
61.5
30.0
34.0
+ Multistep unrolling
51.2
63.9
30.2
36.0
mPLUG (Li et al., 2022a), see Appendix A.2. Fréchet Inception Distance (FID) (Heusel et al., 2017), Precision and Recall (Sajjadi et al., 2018) are for assessing sample quality and diversity. Main Results. In Table 1, apply the proposed adversarial supervision and multistep unrolling strategy to different Stable Diffusion based L2I methods: OFT (Qiu et al., 2023), T2I-Adapter (Mou al., 2023) and ControlNet (Zhang & Agrawala, 2023). Through adversarial supervision and mul- tistep unrolling, the layout faithfulness is consistently improved across different 1.21 models, e.g., improving the mloU of T2I-Adapter from 37.1 50. on Cityscapes. In many cases, the image qual- ity is also enhanced, e.g., FID improves from 57.1 to 51.2 for ControlNet on Cityscapes. Overall, we observe that the proposed adversarial training complements different SD adaptation techniques and architecture improvements. noticeably boosting their performance. By default, ALDM represents ControlNet with adversarial supervision and multistep unrolling in other tables.
In Table 2, we quantitatively compare our ALDM with the other state-of-the-art L2I diffu- sion models: PITI (Wang al., 2022), which does not support text control; and recent SD based FreestyleNet (Xue et al., 2023), T21-Adapter and ControlNet, which support text control. FreestyleNet has shown good mloU by trading off the editability, as it requires fine-tuning of the whole SD. Its poor editability, i.e., low TIFA score, is particularly notable training set small and diversity limited, FreestyleNet tends to overfit and forgets about the pretrained knowledge. This can be reflected from the low recall value in Table 2 as well. Both T21-adapter and ControlNet freeze the SD, and T21-Adapter introduces a much smaller adapter for the conditioning compared to ControlNet. Due to limited fine-tuning capacity, T21-Adapter does not
Cityscapes. As its

---

Published as a conference paper at ICLR 2024
Quantitative comparison of the state-of-the-art 1.21 diffusion models. Best and second best are marked in bold and underline, respectively, while the worst result is in red. Our ALDM demonstrates competitive conditional alignment with notable text editability.
Table 2:
Cityscapes
ADE20K
Method FID 4
mloUt P.f
R.
TIFAt
FID4
mloUt
P.
R.
TIFAt
PITI
n/a
n/a
n/a
n/a
X
27.9
29.4
n/a
n/a
X
FreestyleNet
56.8
68.8
0.73
0.44
0.300
29.2
36.1
0.83
0.79
0.740
T21-Adapter
58.3
37.1
0.55
0.59
0.902
31.8
24.0
0.79
0.81
0.892
ControlNet
57.1
55.2
0.61
0.60
0.822
29.6
30.4
0.84
0.84
0.838
ALDM (ours)
51.2
63.9
0.66
0.68
0.856
30.2
36.0
0.86
0.82
0.888
utilize the layout effectively, leading to low mloU, yet it better preserves the editability, i.e., high TIFA score. By contrast, ControlNet improves mIoU while trading off the editability. In contrast, ALDM exhibits competitive mloU while maintaining high TIFA score, which enables its usability for practical applications, e.g., data augmentation for domain generalization detailed in Sec. 4.2.
Qualitative comparison on the faithfulness to the label map is shown in Fig. 3. T2I-Adapter often ignores the layout condition (see the first row of Fig. 3), which can be reflected in low mloU as well. FreestyleNet and ControlNet may hallucinate objects in the background. For in- stance, in the second row of Fig. 3, both methods synthesize trees where the ground-truth la- bel map is sky. In the last row, ControlNet also generates more bicycles instead of the ground truth trees the background. Contrarily, ALDM better complies with the layout Visual comparison on text editability is shown in Figs. and 4. We observe that FreestyleNet only shows little variability and minor visual differences, as evidenced by the low TIFA score. T21-Adapter and ControlNet preserve better text control, nonetheless, they may not follow well the layout condition. In Fig. 1, ControlNet fails to generate the truck, especially when the prompt is modified. And in Fig. 4, the trees on the left are only sparsely synthesized. While ALDM produces samples that adhere better to both layout and text condi- tions, inline with the quantitative results.
this case.
the other hand
Table 3: Ablation on the discriminator type. Cityscapes FID m
ADE20K
Method
FID mloUt
ControlNet
57.1
55.2
29.6
30.4
UperNet Segmenter
50.3
61.5
30.0
34.0
52.9
59.2
29.8
34.1
Feature-
53.1
59.6
29.3
33.1
Frozen UperNet
50.8
40.2
Discriminator Ablation. We conduct the ablation study on different discriminator designs, shown in Table 3. Both choices for the discriminator network: CNN-based segmentation network Uper- Net (Xiao et al., 2018) and transformer- based Segmenter (Strudel et al., 2021), improve faithfulness of the baseline ControlNet model. Instead of employing the discriminator in the pixel space, we also experiment with feature-space discriminator, which also works reasonably well. It has been shown that internal representation of SD, e.g., intermediate features and cross-attention maps, can be used for the semantic segmentation task (Zhao et al., 2023). We refer to Appendix A.3 for more details. I Lastly, we employ a frozen semantic segmentation network to provide guidance directly. Note that this case is no longer adversarial training anymore, the segmentation model does not update itself with the generator. Despite achieving high mIoU, the generator tends to learn a mean mode of the class and produce unrealistic samples (see Fig. 6), thus yielding high FID. In this case, the generator can more easily find a "cheating" way to fool the discriminator as it is not updating.
4.2 IMPROVED DOMAIN GENERALIZATION FOR SEMANTIC SEGMENTATION
We further investigate the utility of synthetic data generated by different L2I models for domain gen- eralization (DG) in semantic segmentation. Namely, the downstream model trained on source domain, and its generalization performance is evaluated on unseen target domains. We experiment with both CNN-based segmentation model HRNet (Wang et al., 2020) and transformer-based Seg- Former (Xie et al., 2021). Quantitative evaluation is provided in Table 4, where all models except the oracle are trained on Cityscapes, and tested on both Cityscapes and the unseen ACDC. The oracle model is trained on both datasets. We observe that Hendrycks- Weatherr (Hendrycks & Dietterich, 2018), which simulates weather conditions in a synthetic manner, brings limited benefits. ISSA (Li et al., 2023b) resorts simple image style mixing within the source domain. For models that accept
8

---

Published as a conference paper at ICLR 2024
Image
Ground truth
Bascline
Ours
Figure 5:
Semantic segmentation results of Cityscapes - ACDC generalization using HRNet. The HRNet is trained on Cityscapes only. Augmented with diverse synthetic data generated by our ALDM, the segmentation model can make more reliable predictions under diverse conditions.
Comparison on domain generalization. i.e., from Cityscapes (train) to ACDC (unseen). mloU is reported on Cityscapes (CS), individual scenarios of ACDC (Rain, Fog, Snow) and the whole ACDC. Hendrycks- Weether (Hendrycks & Dietterich, 2018) simulates weather conditions in synthetic manner for data augmentation. Oracle model is trained on both Cityscapes and ACDC in a supervised manner, serving as an upper bound on ACDC (not Cityscapes) for the other methods. ALDM can consistently improve generalization performance of both HRNet and SegFormer.
Table 4:
HRNet (Wang et al., 2020)
SegFormer (Xie et al., 2021)
Method
CS
Rain
Fog
Snow
ACDC
CS
Rain
Fog
Snow
ACDC
Baseline (CS)
70.47
44.15
58.68
44.20
41.48
||
67.90
50.22
60.52
48.86
47.04
Hendrycks-Went Weather
69.25
50.78
60.82
38.34
43.19
67.41
54.02
64.74
49.57
49.21
ISSA
70.30
50.62
66.09
53.30
50.05
67.52
55.91
67.46
53.19
52.45
FreestyleNet
71.73
51.78
67.43
53.75
50.27
69.70
52.70
68.95
54.27
52.20
ControlNet
71.54
50.07
68.76
52.94
51.31
68.85
55.98
68.14
54.68
53.16
ALDM (ours)
72.10
53.67
69.88
57.95
53.03
68.92
56.03
69.14
57.28
53.78
Oracle (CS+ACDC)
70.29
65.67
75.22
72.34
65.90
68.24
63.67
74.10
67.97
63.56
text prompts (FreestyleNet, ControlNet and ALDM), we can synthesize novel samples given the textual description of the target domain, as shown in Fig. 4. Nevertheless, the effectiveness of such data augmentation depends on the editability via text and faithfulness to the layout. FreestyleNet only achieves on-par performance with ISSA. We hypothesize that its poor text editability only pro- vides synthetic data close to the training set with style jittering similar to ISSA's style mixing. While ControlNet allows text editability, the misalignment between the synthetic image and the input lay- out condition, unfortunately, can even hurt the performance. While mloU averaged over classes is improved over the baseline, the per-class IoU shown in Table 7 indicates the undermined perfor- mance on small object classes, such as traffic light, rider and person. On those small objects, the alignment noticeably more challenging to pursue than on classes with large area such as truck and bus. In contrast to it, ALDM, owing to its text editability and faithfulness to the layout, consis- tently improves across individual classes and ultimately achieves pronounced gains on mIoU across different target domains, e.g., 11.6% improvement for HRNet on ACDC. Qualitative visualization illustrated in Fig. 5. The segmentation model empowered by ALDM can produce more reliable predictions under diverse weather conditions, e.g., improving predictions on objects such as traffic signs and person, which are safety critical cases.
CONCLUSION
In this work, we propose incorporate adversarial supervision to improve the faithfulness to the layout condition for L21 diffusion models. We leverage a segmenter-based discriminator to explicitly utilize the layout label map and provide strong learning signal. Further, we propose a novel mul- tistep unrolling strategy to encourage conditional coherency across sampling steps. Our ALDM can well comply with the layout condition, meanwhile preserving the text controllability. Capitalizing these intriguing properties of ALDM, we synthesize novel samples via text control for data augmen- tation on the domain generalization task, resulting in a significant enhancement of the downstream model's 's generalization performance.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We would like to express our genuine appreciation to Shin-I Cheng for her dedicated support throughout the experimental testing.
ETHICS STATEMENT
We have carefully read the ICLR 2024 code of ethics and confirm that adhere to it. The method we propose in this paper allows to better steer the image generation during layout-to-image syn- thesis. Application-wise, it is conceived to improve the generalization ability of existing semantic segmentation methods. While is fundamental research and could therefore also be used for data be- yond street scenes (having in mind autonomous driving driver assistance systems), that improving the generalization ability of semantic segmentation methods on such data will benefit safety in future autonomous driving cars and driver assistance systems. Our models are trained and evaluated on publicly available, commonly used training data, so no further privacy concerns should arise.
anticipate
REPRODUCIBILITY STATEMENT
Regarding reproducibility, implementation is based publicly available models Rombach et al. (2022); Xiao et al. (2018); Zhang & Agrawala (2023); Song et al. (2020) and datasets Zhou et al. (2017); Cordts et al. (2016); Sakaridis et al. (2021) and common coruptiions Hendrycks & Diet- terich (2018). The implementation details are provided at the end of section 3, in the paragraph Implementation Details. Details on the experimental settings given the beginning of section 4. Further training details are given in the Appendix in section A. We plan to release the code upon acceptance.
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble expert denoisers. arXiv preprint arXiv:2211.01322 2022.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-Excite Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPH, 2023.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Ying Fan and Kangwook Lee. Optimizing DDPM sampling with shorteut fine-tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.), ICML, 2023.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. In NeurIPS, 2023.
Shir Gur, Sagie Benaim, and Lior Wolf. Hierarchical patch vae-gan: Generating diverse videos from single sample. NerulPS, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In ICLR, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. NeurIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPs, 33, 2020.
10

---

Published as a conference paper at ICLR 2024
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. arXiv preprint arXiv:2303.11897 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In ICCV, 2023.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metric. In ICML, 2016.
Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, He Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, and Luo Si. mPLUG: Effective and efficient vision-language learning by cross-modal skip-connections. In EMNLP, 2022a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. /CMI, 2022h.
Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing. arXiv preprini arXiv:2307.10866 2023a.
Yumeng Li, Dan Zhang, Margret Keuper, and Anna Khoreva. Intra-& extra-source exemplar-based style synthesis for improved domain generalization. IJCV, 2023b.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
Taesung Park, Ming-Yi Yu Liu, Ting-Chun Wang, and Jun- Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:230..01952, 2023.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. arXiv preprint arXiv:2306.0720 072880 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06123 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, Assessing generative models via precision and recall. NeurIPS, 2018.
Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Acde: The adverse conditions dataset with correspondences for semantic driving scene understanding. In JCCV, 2021.
Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall, Bernt Schicle, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In ICLR, 2020.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon. Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training generation image-text models. NeurIPS, 2022.

---

Published as a conference paper at ICLR 2024
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In CVPR, 2021.
Vadim Sushko, Edgar Schönfeld, Dan Zhang. Juergen Gall, Bernt Schiele, and Anna Khoreva. Oa- sis: only adversarial supervision for semantic image synthesis. IJC,, 2022.
Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Gang Hua, and Nenghai Yu. Efficient semantic image synthesis via class-adaptive normalization. TPAMI, 2021.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, al. Deep high-resolution representation learning for visual recognition. TPAMI, 2020.
Liwen Wang, Shuo Yang, Kang Yuan, Yanjun Huang, and Hong Chen. A combined reinforce- ment learning and model predictive control for car-following maneuver of autonomous vehicles. Chinese Journal of Mechanical Engineering, 2023a.
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205.1295 2022.
Ting-Chur Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
Yi Wang. Lu Qi, Ying-Cong Chen, Xiangyu Zhang, and Jiaya Jia. Image synthesis via semantic composition. In ICCV, 2021.
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion- gan: Training gans with diffusion. In ICLR, 2023b.
Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: A feature generat- ing framework for any-shot learning. In CVPR, 2019.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. ICLR, 2022.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977 2023.
Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image syn- thesis. In CVPR, 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception. In ICCV, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Zhen Zhu, Zhiliang Xu, Ansheng You, and Xiang Bai. Semantically multi-modas image synthesis. In CVPR, 2020.
12

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
This supplementary material to the main paper is structured follows:
In Appendix A, we provide more experimental details for training and evaluation.
In Appendix B, we include the ablation study the unrolling step K, and more quantitative evaluation results.
In Appendix C, we provide more visual results for both L2I task and improved domain generalization in semantic segmentation.
In Appendix D, we discuss the failure cases of our approach, and potential solution for future research.
In Appendix E, we discuss the theoretical connection with prior works and potential future research directions, which can be interesting for the community for further exploration and development grounded in our framework.
A EXPERIMENTAL DETAILS
A.1 TRAINING DETAILS
We finetune Stable Diffusion v1.5 checkpoint and adopt ControlNet for the layout conditioning, All trainings are conducted on 512 x 512 resolution. For Cityscapes, we do random cropping and for ADE20K we directly resize images. Nevertheless, we directly synthesize 512 x 1024 Cityscapes images for evaluation. We use AdamW optimizer and the learning rate of 1 10-3 for the diffusion model, 1 10 for the discriminator, and the batch size 8. The adversarial loss weighting factor Audv is set to be 0.1. The discriminator is firstly warmed up for 5K iterations on Cityscapes and 10K iterations on ADE20K. Afterward, we jointly train the diffusion model and discriminator in an adversarial manner. We conducted all training using 2 NVIDIA Tesla A100 GPUs.
A.2 TIFA EVALUATION
Evaluation the TIFA metric is based on the performance of the visual question answering (VQA) system, e.g. mPLUG (Li et al., 2022a). By definition, the TIFA score is essentially the VQA ac- curacy, given the question-answe answer pairs. To quantitatively evaluate the text editability, we design a list of prompt templates, e.g., appending "snowy scene" to the original image caption for image generation. Based on the known prompts, we design the question-answer pairs. For instance, we can ask the VQA model "What is the weather condition?". and compute TIFA score based on the accuracy of the answers.
A.3 FEATURE-BASER DISCRIMINATOR FOR ADVERSARIAL SUPERVISION
Thanks to large-scale vision-language pretraining on massive datasets, Stable Diffusion (SD) (Rom- bach al., 2022) has acquired rich representations, endowing with the capability not only gen- erate high-quality images, but also to excel in various downstream tasks. Recent work VPD (Zhao al., 2023) has unleashed the potential of SD, and leveraged representation for visual percep- tion tasks, e.g., semantic segmentation. More specifically, they extracted cross-attention maps and feature maps from SD at different resolutions and fed them to a lightweight decoder for the specific task. Despite the simplicity of the idea, it works fairly well, presumably due to the powerful knowl- edge of SD. In the ablation study, we adopt the segmentation model of VPD as the feature-bast discriminator. Nevertheless, different from the joint training of SD and the task-specific decoder in the original VPD implementation, we only train the newly added decoder, while freezing SD to preserve the text controllability ControlNet.
13

---

Published as a conference paper at ICLR 2024
Table 5: Ablation the unrolling step K. Overhead measured seconds per training iteration.
Cityscapes mloUf
ADE20K mloU1
FID
TIFAt
FID
TIFAt
Overhead
NontrolNet
57.1
55.2
0.822
29.6
30.4
0.838
0.00
==0
50.3
61.5
0.894
30.0
34.0
0.904
0.00
=
54.9
62.7
0.856
1.55
==6
51.6
64.1
0.832
30.3
34.5
0.898
3.11
K =
51.2
63.9
0.856
30.2
36.0
0.888
4.65
K 15
50.7
64.1
0.882
30.2
36.9
0.825
7.75
Table 6: Quantitative comparison of different T2I diffusion models. P., R., and R.mloU represent Precision, Recall and robust mIoU respectively.
Method FID 4
P.
R.1
mloU1
R.mloU
TIFA
FreestyleNet
56.8
0.73
0.44
68.8
69.9
0.300
T21-Adapter
58.3
0.55
0.59
37.1
44.7
0.902
ControlNet
57.1
0.61
0.60
55.2
57.3
0.822
ALDM
51.2
0.66
0.68
63,9
65.4
0.856
B MORE ABLATION AND EVALUATION RESULTS
B.1 MULTISTEP UNROLLING ABLATION
For the unrolling strategy, we compare different number of unrolling steps in Table 5. We observe that more unrolling steps is beneficial for improving the faithfulness, the model consider more future steps to ensure alignment with the layout condition. However, the additional unrolling time overhead also increases linearly. Therefore, we choose K by default in all experiments.
B.2 ABLATION ON FROZEN SEGMENTER
We ablate the usage of a frozen segmentation model, instead of joint training with the diffusion model. As quantitatively evaluated in Table 3, despite achieving good alignment with the layout condition, i.c., high mloU, we observe that the diffusion model tends lean mean mode and produces unrealistic samples with limited diversity (see Fig. 6), thus yielding high FID values.
B.3 ROBUST MIOU EVALUATION
Conventionally, the layout alignment evaluation is conducted with the aid of off-the-shelf segmen- tation networks trained on the specific dataset, thus may not be competent enough to make reliable predictions on more diverse data samples, shown in Fig. 7. Therefore, we propose to employ robust segmentation model trained with special data augmentation techniques (Li et al., 2023b), to more accurately measure the actual alignment.
report the quantitative performance in Table Notably, there a large difference between standard mloU and robust mloU, in particular for T21-Adapter. From Fig. 7, we can see that T21- Adapter occasionally generates more stylized samples, which do not comply with the Cityscapes style, and the standard segmenter has sensitivity this.
B.4 COMPARISON WITH GAN-BASED L21 METHODS.
We additionally compare method with prior GAN-based L2I methods in Table 8. It is worth- while to mention that all GAN- approaches do not have text controllability, thus they can only produce samples resembling the training dataset, which constrains their utility downstream tasks. On the other hand, our ALDM achieves the balanced performance between faithfulness to the layout condition and editability via text, rendering itself advantageous for the domain generalization tasks.
14

---

Published as a conference paper at ICLR 2024
Ground truth
Label
Samples
Figure 6: Visual results of using a frozen segmentation network, i.e., pretrained UperNet (Xiao al., 2018), to provide conditional guidance during diffusion model training. We can observe the mode collapse issue, where the diffusion model tends to learn to mean mode and exhibits little variation in the generated samples.
Ground truth
Sample
Standard Prediction
Robust Prediction
Figure 7: Comparison of standard segmenter and robust segmenter for layout alignment evalua- tion on synthesized samples of T21-Adapter. When testing on more diverse images, the standard segmenter struggles to make reliable predictions, thus may lead to inaccurate evaluation.
B.5 PER-CLASS IOU FOR SEMANTIC SEGMENTATION
In Table 7, we report per-class IoU of the object classes on Cityscapes. For ControlNet the mis- alignment between the synthetic image and the input layout condition, unfortunately, can hurt the segmentation performance on small and fine-grained object classes, such as bike, traffic light, traffic sign, rider, and person. While ALDM demonstrates better performance on those classes, which re- flects that our method can better comply with the layout condition, small and fine-gre fineggriined object classes are typically more challenging in L2I task and pose higher requirements for the faithfulness to the layout condition.
15

---

Published as a conference paper at ICLR 2024
Table 7: Per-class IoU of Cityscapes object classes. Numbers in red indicate worse IoU compared to the bascline. The best is marked in bold. Our ALDM has demonstrated better performance on small object classes, c.g., pole, traffic light, traffic sign, person, rider, which reflects our method can better comply with the layout condition, as small object classes are typically more challenging in
L.2I task and pose higher requirement for the faithfulness to the layout condition.
Method
Pole
Traf. light Traf. sign Person Rider
Car
Truck
Bus
Train Motorbike
Bike
Baseline
48.50
59.07
67.96
72.44
52.31
92.42 70.11 77.62
64.01
50.76
68.30
ControlNet
49.53
58.47
67.37
71.45
49.68
92.30
76.91
82.98 72.40
50.84
67.32
ALDM
51.21
60.50
69.56
73.82
53.01
92.57
76.61
81.37 66.49
52.79
68.61
Table Quantitative comparison results with the state-of-the- layout-to-image GANs and diffu- sion models (DMs). Our ALDM demonstrates competitive conditional alignment with notable text editability.
Cityscapes mloUt
ADE20K
Method
FID
TIFA
FID
mloUf
TIFAt
Pix2PixHD (Wang et al., 2018)
95.0
63.0
81.8
28.8
SPADE (Park et al., 2019)
71.8
61.2
33.9
38.3
OASIS (Schönfeld al., 2020)
47.7
69.3
28.3
45.7
GANs
x
x
SCGAN (Wang al., 2021)
49.5
55.9
29.3
41.5
CLADE (Tan et al., 2021)
57.2
58.6
35.4
23.9
GroupDNet (Zhu et al., 2020)
47.3
55.3
41.7
27.6
PITI (Wang al., 2022)
n/a
n/a
27.9
29.4
FreestyleNet (Xue et al., 2023)
56.8
68.8
0.300
29.2
36.1
0.740
T21- 121-Adapter (Mou et al., 2023)
58.3
37.1
0.902
31.8
24.0
0.892
DMs
ControlNet (Zhang & Agrawala, 2023)
57.1
55.2
0.822
29.6
30.4
0.838
ALDM (ours)
51.2
63.9
0.856
30.2
36.0
0.888
C MORE VISUAL EXAMPLES
C.I LAYOUT-TO-IMAGE TASKS
In Fig. we showcase more visual comparison on ADE20K across various scenes, i.c., outdoors and indoors. Our ALDM can consistently adhere to the layout condition.
Figure presents visual examples of Cityscapes, which are synthesized via various textual descrip- tions with our ALDM, which can be further utilized on downstream tasks.
In Fig. 10, we demonstrate the editability via text of our ALDM. Our method enables both global editing (e.g., style or scene-level modification) and local editing (e.g., object attribute).
In Figs. 11 until 13, we provide qualitative comparison on the text editability between different L2I diffusion models on Cityscapes and ADE20K.
In Fig. 14, we compare our ALDM with GAN-based style transfer method ISSA (Li al., 2023b). It can be observed that ALDM produces more realistic results with faithful local details, given the label map and text prompt. In contrast, style transfer methods require two images, and mix them on the global color style, while the local details, e.g., mud, and snow may not be faithfully transferred.
Fig. 15, we provide visualization the segmentation outputs from the discriminator. It can be seen that the discriminator can categorize some regions as "fake" class (in black), meanwhile it is also fooled by some areas, where the discriminator produces reasonable segmentation predictions matched with ground truth labels. Therefore, the discriminator provide useful feedback such that the generator (diffusion model) produces realistic results meanwhile complying with the given label map.
16

---

Published as a conference paper at ICLR 2024
GT
Label
T2I-Adapter
FreestyleNet
ControlNet
Ours
Figure 8:
Qualitative comparison of faithfulness to the layout condition between different L2I methods on ADE20K. Our ALDM can comply with the label map consistently, while the other may ignore the ground truth label map and hallucinate, e.g., synthesizing trees in the background (see the third row).
C.2 IMPROVED DOMAIN GENERALIZATION
More qualitative visualization improved domain generalization is shown in Fig. 16. By employ- ing synthetic data augmentation empowered by our ALDM, the segmentation model can make more reliable predictions, which is crucial for real-world deployment.
D FAILURE CASES
As shown in Fig. 17, when editing the attribute of one object, could affect the other objects as well. Such attribute leakage is presumably inherited from Stable Diffusion, which has been observed in prior work (Li al., 2023a) with SD as well. Using a larger UNet backbone e.g. SDXL (Podell et al., 2023) or combining with other techniques, e.g., inference time latent optimization (Chefer et al., 2023; Li et al., 2023a) may mitigate this issue. This is an interesting open issue, and we would leave this for future investigation. Depsite the improvement layout aligment, ALDM is not yet perfect and may not seamlessly align with given label map, especially when the text prompt heaivly edited
scenario.
17

---

Published as a conference paper at ICLR 2024
Ground truth
Label
+"rainy scene"
+ Snowy scene"
+ ""nightimm"
Ground truth
Label
+"muddy road"
+ "heavy fog"
snow scene, nighttime
Figure 9: Visual examples of Cityscapes, synthesized by ALDM via various textual descriptions, which can be further utilized on downstream tasks.
E DISCUSSION & FUTURE WORK
E.1 THEORETICAL DISCUSSION
In the proposed adversarial training, the denoising UNet of the diffusion model can be viewed as the generator, the segmentation model acts the discriminator. For the diffusion model, the dis- criminator loss is combined with the original reconstruction loss, to further explicitly incorporate the label map condition. Prior works (Gur al., 2020; Larsen al., 2016; Xian et al., 2019) have combined VAE and GAN, and hypothesized that they can learn complementary information. Since both VAE and diffusion models (DMs) are latent variable models, the combined optimization of diffusion models with an ad adversarial model follows this same intuition yet with all the advantages of DMs over VAE. The combination of the latent diffusion model with the discriminator is thus, in principle, a combination of latent variable generative model with adversarial training. In this work, we have specified the adversarial loss such that relates our model to optimizing the expectation over Co in Eq. (6). and for the diffusion model, we refer to the MSE loss defined on the estimated noise in Eq. (2), which can be related to optimizing 2g with respect to the approximate posterior by op- timizing the variational lower bound on the log-likelihood originally shown in DDPM (Ho al., 2020). Our resulting combination loss terms in Eq. (7) can thus be understood as to optimize over the weighted sum of expectations 20-
18

---

Published as a conference paper at ICLR 2024
Original caption: "a street filled with lots of parked cars next to tall buildings"
Ground truth
Label
"muddy street" "snowy street"
"burning cars
Original caption: car driving down a street next to tall buildings"
Ground truth
Label
"purple car" + "blue car"
"red car"
Original caption: "a couple of men standing next to red car" Label
Ground truth
"purple car"
"green car"
"ink car"
Original caption: "a room with chair and window" Label
Ground truth
"sketch style"
"Picasso painting"
Cyberpunk style"
Figure 10: Visual examples of text controllability with our ALDM. Based on the original image captions generated by BLIP model, we can directly modify the underlined objects (indicated as ++), append postfix to the caption (indicated as +). Our ALDM can accomplish both local attribute editing (e.g., car color) and global image style modification (c.g., sketch style).
E.2 FUTURE WORK
In this work, we empirically demonstrated the effectiveness of proposed adversarial supervision and multistep unrolling. In the future, it is interesting direction to further investigate how to better incorporate the adversarial supervision signal into diffusion models with thorough theoretical justi- fication.
For the multistep unrolling strategy, we provided a fresh perspective and a crucial link to the ad- vanced control algorithm MPC, in Sec. 3.2. Witnessing the increasing interest in Reinforcement learning from Human Feedback (RLHF) for improving T2I diffusion models (Fan et al., 2023; Xu al., 2023), is promising direction combine our unrolling strategy with RL algorithm, where MPC has been married with RL in the context of control theory (Wang et al., 2023a) to combine the best of both world. In addition, varying the supervision signal rather than adversarial supervision, e.g., from human feedback, powerful pretrained models, can be incorporated for different purposes and tailored for various downstream applications. As formulated Eq. (10), we simply average the losses at different unrolled steps, similar to simplified diffusion MSE loss (Ho et al., 2020). Future development on the dime-dependent weighting of losses at different steps might further boost the effectiveness of the proposed unrolling strategy.
19

---

Published as a conference paper at ICLR 2024
Ground truth
FreestyleNet
ControlNet
Ours
"heavy fog"
+ "snowy scene with sunshine"
"snow scene, nightitme"
Faithfulness Editability
Qualitative comparison of text editability between different L21 diffusion models Cityscapes. FreestyleNet exhibits little variability via text control. ControlNet often does not adhere the layout condition, e.g., synthesizing buildings (Ist row) trees (2nd and 3rd row) where the ground truth label map is sky. In contrast, Our ALDM can synthesize samples well complied with both layout and text prompt condition.
Figure 11:
Ground truth
T2I-Adapter
FreestyleNet
ControlNet
Ours
fence
"snowy scene"
"night scene"
Figure 12: Qualitative comparison of text editability between different L2I diffusion models on Cityscapes.
Last but not the least, with the recent rapid development of powerful pretrained segmentation models such SAM (Kirillov al., 2023), autolabelling large datasets c.g., LAION-5B (Schuhmann al., 2022) and subsequently training a foundation model jointly for the T2I and L2I tasks may become a compelling option, which can potentially elevate the performance of both tasks unprecedented levels.
20

---

Published as a conference paper at ICLR 2024
Ground trurh
T2I-Adapter
FreestyleNet
ControlNet
Ours
+ "snowy scene"
"Van Gogh style"
Figure 13: Qualitative comparison of text editability between different L21 diffusion models on ADE20K. ALDM can synthesize samples well complied with the layout and text prompt.
Content
Style
ISSA
Label
ALDM (Ours)
+ *mmuddy street"
"snowy scene"
+ *nighttime"
Comparison between our ALDM and GAN-based style- sele-transser method ISSA (Li et al., 2023b). can seen that ALDM can produce more realistic results with faithful local details, given the label map and text. In contrast, style transfer methods require two images, and mix them on the global color style, while the local details, e.g., mud, and snow may not faithfully transferred.
Figure 14:
21

---

Published as a conference paper at ICLR 2024
Ground truth
Label
Predicted a:(")
Seg. Pred.
Figure 15: Visualization of discriminator predictions on the estimated clean image a:(") at 5. Black in the ground truth label map represents unlabelled piels, while the last segmentation predicion column black indicates the fake class predicitons.
22

---

Published as a conference paper at ICLR 2024
Image
Ground truth
Baseline
Ours
Figure 16: Semantic segmentation results Cityscapes + ACDC generalization using HRNet. The HRNet is trained on Cityscapes only. Augmented with samples synthesized by ALDM, the segmentation model can make more reliable predictions under diverse unseen conditions, which is crucial for deployment in the open-world.
Ground truth
Label
"purple car"
"pink car"
Figure 17: Editing failure cases. When editing the attribute one object, it may leak to other objects well. For instance, the color of the car the right modified as well.
23


Reviews:
[{"review_id": "UJ7CP4jxwW", "reviewer": "Reviewer_4k64", "summary": "This paper proposes to embed adversarial supervision into the training of a diffusion model conditioned on layout. By introducing adversarial supervision and the multi-step rolling strategy, the framework can get strong results, better than baseline methods like control net which does not explicitly use adversarial supervision. It also shows that the generated samples can boost the domain generalization for semantic segmentation tasks.", "questions": "In the `weaknesses`", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to understand, the proposed method is simple but effective\n- Adversarial supervision is not new though, it is the first time to be used in the context of diffusion model\n- The experimental results are strong and are better than the baseline methods.", "weaknesses": "- The qualitative results seem to be much better than baseline methods, are they cherry-picked? A non-cherry picked results can better show that the proposed methods largely exceed the current baselines.\n- What is the difference between Control-Net + Adv Supervision and multi-step rolling in Table 1 compared with ALDM?\n- Can you show some outputs of the discriminator? Since it works in the latent space, we need to be more careful about what is happening inside.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OFvPdmvWvh", "reviewer": "Reviewer_d5qA", "summary": "This paper proposes to adopt a ControlNet architecture for better semantic image synthesis using an adversarial discriminator (as in OASIS) on the per-pixel label maps. Furthermore, a multistep unrolling mechanism is presented so that adversarial supervision takes into account several denoising steps to improve the signal at low noise levels.", "questions": "-", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-structured and well-written.\n- Ideas and results are presented clearly.\n- Adapting a pre-trained diffusion model for semantic image synthesis is interesting and challenging.", "weaknesses": "There are several issues with the presented work.\n\n- The technical contributions of this manuscript are limited. Adversarial supervision on the semantic maps is identical as in OASIS. Multistep unrolling is computationally very expensive (scales linearly and hence can take up x9 longer), and has a small effect on the performance.\n- The results (while improving over the diffusion baselines) are behind OASIS, a GAN from 2020.\n- The motivation to use a strong text-to-image model and adapt it for semantic image synthesis is flawed. The paper motivates it by enabling text-conditioned content and style transfer, but the semantic mask completely specifies the content. Hence, the application is reduced to text-guided style and color transfer. \n- Furthermore, the proposed model does not perform style transfer well. When changing to a snowy scene, the whole image and all objects are resampled. Local editing is also not possible. Instead, the whole image is affected when changing \"a red van\" to \"a burning van\". \n- A simple baseline combining OASIS and a state-of-the-art style transfer model should be considered.\n\nSide remarks:\n- Layout-to-image is usually referred to as a task that transforms a list of bounding boxes and class labels into an image [1,2]. This paper tackles semantic image synthesis where the input is a label mask (each pixel is labelled).\n- The paper states that Stable Diffusion based models do not comply well with layout input, but see [3,4]\n\n[1] Image Synthesis From Reconfigurable Layout and Style, https://arxiv.org/abs/1908.07500\n[2] LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation, https://arxiv.org/abs/2303.17189\n[3] SpaText: Spatio-Textual Representation for Controllable Image Generation, https://arxiv.org/abs/2211.14305\n[4] ReCo: Region-Controlled Text-to-Image Generation, https://arxiv.org/abs/2211.15518", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T1PMBBP8cO", "reviewer": "Reviewer_rjfS", "summary": "The paper \"Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive\" proposes to augment diffusion models with adversarial methods to achieve a model that is controlable from text and segmentation map layout simultaneously. In particular, the authors propose two additions to fine-tune a Stable Diffusion model: 1) a recent adversarial learning methods incorporating segmentation networks is applied to the outputs of the diffusion network, and 2) the diffusion process is unrolled, and the segmentation network is applied already to intermediate de-noised steps. It is shown in the paper that there's incremental value in both. The method is evaluated on two datasets (ADE20K and Cityscapes), and the proposed method is showns to be superior to several recent baselines (T2I-Adapter, FreestyleNet and ControlNet) qualitatively, as well as quantitatively. Finally, the method is applied on a domain generalization task, and is shown to perform best against the baselines there.", "questions": "- why don't you train a diffusion model from scratch for the task of L2I and T2I simultaneously? Recent segmentation models are very powerful, and can produce the segmentation maps needed to augment the dataset (e.g. LAION-5B). I'd expect that to work best.\n- for the case of a frozen pre-trained segmenter: did you train a diffusion model plus an additional loss for the segmentation model? That is surprising, because I wouldn't have expected the diffusion model to collapse in such case (the same training method/loss is still there).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ the paper is well written, intuitive motivations are given, and it is well understandable.\n+ the proposed method is shown to work well. It shows both qualititatively better alignment and quantitative improvements.\n+ it also shows very good results on domain generalization.", "weaknesses": "- it's unclear if careful tuning of a frozen segmentation network's impact on the total loss wouldn't be competitive to the proposed adversarial approach. I.e. in Table 3, the frozen UperNet achieves the best mIoU, but much worse FID - consistent with the hypothesis that the impact of the segmentation network is just too strong. Verying the impact of the segmentation loss relative to the diffusion loss while fine-tuning would clarify this.\n- The introduction gave a good intuition for the whole method. However, the combination of diffusion models with adversarial methods remains ad-hoc, and there's no theoretical justification for the validaty of the approach given. It should be either worked out, or at least added for future work that propoer understanding of diffusion-adversarial coupling should be investigated (to make these methods work best together).\n\nSmaller things:\n- Figure 1 lowest right image: this example shows that the method doesn't yet work perfectly. The top part of the truck is a building (see the windows and the 'roof structure'). It should be pointed out in the paper that some failed cases still persist (even if harder to see).\n- In \"Related Work\" you write \"more attention has been devoted to leveraging pretrained knowledge for the L2I task and using diffusion models\"; I think this is an important point, and should already be part of the motivation of the general method in the Introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nTMRRTgkkL", "reviewer": "Reviewer_VHY8", "summary": "This paper aims for a better layout-to-image method. It applies a segmentation-based discriminator (Sushko et al., 2022) to the diffusion generator on the pixel-level alignment between the denoised image and the input layout. In addition, it proposes multistep unrolling, predicting the clean image at multiple timesteps and apply the segmenter-based discriminator. The experiments are on the classical segmentation datasets ADE20K and Cityscapes. The model exhibits comparable pixel level alignment and image fidelity. Different type of L2I synthesis adaptation models and segmenters are tested, to demonstrate its effectiveness.", "questions": "1. When using a frozen segmenter, mIoU is higher and FID is also higher. Why does it happen, any insight?\n2. It is mentioned in the limitations that editing one object may also affect others. However, in Figure 4, it shows better local controllability than ControlNet and FreestyleNet. Any insights why? Because of better pixel alignment? If that's the case, is better mIoU always means better local controllability? For example, can the one trained on frozen segmenter exhibits better local controllability?\n3. Why larger number of unrolling steps is always better? Have you tried any K>9? Do all steps contribute the similar gradient magnitude or some of them is more important? What if we omit some of the earlier steps?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The proposed method seems straightforward and effective.\n2. The paper contains thorough ablation tests on different settings (different L2I models and segmenters) and different hyperparameters.", "weaknesses": "1. As mentioned in the Failure Cases, when editing the attribute of one object, it could affect the other objects as well. It is claimed to be inherited from Stable Diffusion.\n2. Despite thorough ablation tests, the paper does not give any insights on the experiments. The text in the experiments only describes the results instead of analyzing the phenomenon. Thus, I think it shows limited contribution to the community. I suggest to delete the plain description of the results, as we can all see from the tables and figure captions, but add more analysis and insights of why it can work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)